{"cells":[{"metadata":{},"cell_type":"markdown","source":"In this notebook I will explore setting up a [Siamese Neural Network](http://www.cs.utoronto.ca/~gkoch/files/msc-thesis.pdf) (SNN), using the fastai/pytorch framework, to try and identify whales by their flukes (tail fins). The dataset comes from the kaggle humpback whale identification [challege](https://www.kaggle.com/c/humpback-whale-identification). The inspiriation for this technique originated from Martin Piotte's [kaggle kernel](https://www.kaggle.com/martinpiotte/whale-recognition-model-with-score-0-78563) which implemented a SNN in keras.\n\nI'll be focusing on training an SNN as they are specifically tailored for one-shot learning tasks, which consists of classification under the restriction that we may only observe a single example of each possible class before making a prediction about a test instance. This is extremely useful given that in my previous [post](http://afitts.github.io/2018/11/04/humpback/) I found that the majority of whales in the dataset only have 1-4 examples in the training set. "},{"metadata":{"_uuid":"24248b49182cbb21fff652ebeccb9c171ac0c9be"},"cell_type":"markdown","source":"## Imports"},{"metadata":{"_uuid":"44b95cd546f32a8b3db5ec2323f0b5ee9bcbde54","trusted":true},"cell_type":"code","source":"%reload_ext autoreload\n%autoreload 2\n%matplotlib inline\nfrom fastai import *\nfrom fastai.vision import *\nfrom fastai.callbacks import *","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import fastai\nfastai.__version__","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3975d2c1bda59987a4d513ed9cfead50e5f58afc","trusted":true},"cell_type":"code","source":"path = Path('../input')\ntrain = pd.read_csv(path/'train.csv'); ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(25000)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"813a7d89cba83716606aacf0bbd3bf559479588e","trusted":true},"cell_type":"code","source":"## Filtering classes with at least 60 images for quick experiments\ncnter = Counter(train.Id.values)\ntrain['cnt']=train['Id'].apply(lambda x: cnter[x])\n#Not consider new_whale images and hence considering 500 as upper cutoff\ntrain = train[(train['cnt']>60) & (train['cnt']<500)] ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eb032d722df2fd2354aa372a9843c0960672e86a"},"cell_type":"markdown","source":"> Duplicating the entire dataframe so that we can generate a positive pair and a negative pair for each image; maintaing 50% positive pairs and 50% negative pairs for the saimese network. \nWhy this is done can be understood in below sections"},{"metadata":{"_uuid":"e6be39d2fb4eabae4ba4a9263f6082c476c6d421","trusted":true},"cell_type":"code","source":"train['target'] = 1\ntrain1 = train.copy()\ntrain1['target'] = 0\ntrain = train.append(train1)\ntarget_col = 3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"38b911833a865255b0e7117cfd8418240addd6fb"},"cell_type":"markdown","source":"## Custom Itembase and ItemList for Siamese Networks\n* Much of this follows the Custom Item List tutorial(https://docs.fast.ai/tutorial.itemlist.html#Example:-ImageTuple)\n* Though blindly following the steps in the tutorial might work, i was not very comfortable doing that and hence spent a few hours digging into the library  and understanding how various pieces inside it work\n* I would suggest starting with tabular data.py as tabular is relatively easier to follow and gives an overall idea of the different pieces involved in getting the databunch ready\n* At a very high level, creation of databunch for Image application follows this path. \n    *  When factory methods like from_df are called, it instantiates the corresponding ImageItemList. Important things includes creation of items, xtra df with necessary info etc. (vision/data.py - ImageItemList class)\n    * Most imp thing to note here is the get() func responsible for retreving data given the index. This func links the ImageItemList to the Image(Itembase) by returning the retrived item as the class of Itembase. I missed this as it was very subtle in the code and couldnt connect the dots initially.\n    * .split_by_.. - Much of this happens in data_block.py, ItemLists class and easier to follow\n    * Label_from_df - This is where a lot of heavy lifting seems to happen. As ItemLists doesnt have label_from_df attri, it falls back to ItemList class(using '____get__attr__') and does the job. Also, finally the class is changed to LabelLists which gives it the functionality to create databunch etc. Refer to data_block.py and LabelList and LabelLists classes\n    * Also Deduction of target variable class, and assignment of processor, loss_func etc happen here\n    * .databunch does the rest"},{"metadata":{"_uuid":"362940a2a8c19738d5ae39d7c5a49d38219ba1d0","trusted":true},"cell_type":"code","source":"mean, std = torch.tensor(imagenet_stats)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"37021f4188aec4f5d8a8ad3b33fef621844c9348","trusted":true},"cell_type":"code","source":"# The primary difference from the tutorial is with how normalization is being done here\nclass SiamImage(ItemBase):\n    def __init__(self, img1, img2): ## These should of Image type\n        self.img1, self.img2 = img1, img2\n        self.obj, self.data = (img1, img2), [(img1.data-mean[...,None,None])/std[...,None,None], (img2.data-mean[...,None,None])/std[...,None,None]]\n    def apply_tfms(self, tfms,*args, **kwargs):\n        self.img1 = self.img1.apply_tfms(tfms, *args, **kwargs)\n        self.img2 = self.img2.apply_tfms(tfms, *args, **kwargs)\n        self.data = [(self.img1.data-mean[...,None,None])/std[...,None,None], (self.img2.data-mean[...,None,None])/std[...,None,None]]\n        return self\n    def __repr__(self): return f'{self.__class__.__name__} {self.img1.shape, self.img2.shape}'\n    def to_one(self):\n        return Image(mean[...,None,None]+torch.cat(self.data,2)*std[...,None,None])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b1ca1a9b13ca75682f4bd60eb7cf52494db91c20"},"cell_type":"markdown","source":"We define get() func such that we get similar pair of images and different pair of images based on idx value. Similar for first half index values and different for next half. Hence the duplicity in dataframe initially.\nCurrently did things dumbly just to demostrate. Can do generation of pairs, dealing with underbalanced classes and many other things in a much better way "},{"metadata":{"_uuid":"ef4f015a70ddb95c3e89be41e40527f1e9da5894","trusted":true},"cell_type":"code","source":"class SiamImageItemList(ImageList):\n    def __init__(self, *args, **kwargs):\n        super().__init__(*args, **kwargs)\n#         self._label_cls=FloatList\n    \n    def __len__(self)->int: return len(self.items) or 1 \n    \n    def get(self, i):\n        match=1\n        if i>=len(self.items)//2:#\"First set of iteration will generate similar pairs, next will generate different pairs\"\n            match = 0\n        fn = self.items[i]\n        img1 = super().get(i) # Returns Image class object\n        \n        imgs = self.inner_df.Image.values\n        ids = self.inner_df.Id.values\n        wcls = ids[i]\n        simgs = imgs[ids == wcls]\n        dimgs = imgs[ids != wcls]\n        if len(simgs)==1 and match==1:\n            fn2=fn\n        else:\n            while True:\n                np.random.shuffle(simgs)\n                np.random.shuffle(dimgs)\n                if simgs[0] != fn:\n                    fn2 = [simgs[0] if match==1 else dimgs[0]][0]\n                    break\n            fn2 = self.items[np.where(imgs==fn2)[0][0]]\n        img2 = super().open(fn2) # Returns Image class object\n        return SiamImage(img1, img2)\n    \n    def reconstruct(self, t): return SiamImage(mean[...,None,None]+t[0]*std[...,None,None], mean[...,None,None]+t[1]*std[...,None,None])\n    \n    def show_xys(self, xs, ys, figsize:Tuple[int,int]=(9,10), **kwargs):\n        rows = int(math.sqrt(len(xs)))\n        fig, axs = plt.subplots(rows,rows,figsize=figsize)\n        for i, ax in enumerate(axs.flatten() if rows > 1 else [axs]):\n            xs[i].to_one().show(ax=ax, y=ys[i], **kwargs)\n        plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"32499207aca10caf9dcb2cf514ff785a2d5c1157","trusted":true},"cell_type":"code","source":"whl_tfms = get_transforms()\nbs = 8","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"560a5f9a68921844bea359012f7df7e61e6a8553","trusted":true},"cell_type":"code","source":"data = (SiamImageItemList.from_df(df=train, path=path/'train', cols=0)\n         .split_by_rand_pct(valid_pct=0.2, seed=34)\n         .label_from_df(cols=target_col, label_cls=FloatList)\n         .transform(whl_tfms, size=224)\n         .databunch(bs=bs))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"73c4778b5cf120ef83d6924936d3561e20369af5","trusted":true},"cell_type":"code","source":"data.show_batch(2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e82d6853219c6774aae860ec6c1be944d3f133a9","trusted":true},"cell_type":"code","source":"# Checking if the normalization done above is correct\nx = next(iter(data.train_dl))\nt=x[0][0][0].cpu()\nto = mean[...,None,None] + t* std[...,None,None]\nimport torchvision\nti = torchvision.transforms.ToPILImage(to)\nplt.imshow(to.numpy().transpose(1,2,0))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a589c24a4555ed7823eb7f2f35a34aa7d771e6cd","trusted":true},"cell_type":"code","source":"plt.imshow(t.numpy().transpose(1,2,0))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d3b226102bca68f0f577133e733a817b5f0df7ae"},"cell_type":"markdown","source":"## Siamese Network"},{"metadata":{"_uuid":"1b914a95f32a65fee09282380154beeeef2bd733","trusted":true},"cell_type":"code","source":"from fastai.vision import learner\n\nclass SiameseNet(nn.Module):\n    def __init__(self, arch=models.resnet18, lin_ftrs=[256, 128], emb_sz=128,ps=0.5, bn_final=False):\n        super(SiameseNet, self).__init__()\n        self.arch, self.emb_sz = arch, emb_sz\n        self.lin_ftrs, self.ps, self.bn_final = lin_ftrs, ps, bn_final\n        self.body = learner.create_body(self.arch, True, learner.cnn_config(self.arch)['cut'])\n        self.head = learner.create_head(num_features_model(self.body), self.emb_sz, self.lin_ftrs, self.ps,self.bn_final)\n        self.cnn = nn.Sequential(self.body, self.head)\n                                  \n    def forward(self, x1, x2):\n        output1 = self.cnn(x1)\n        output2 = self.cnn(x2)\n        return output1, output2\n\n    def get_embedding(self, x):\n        return self.cnn(x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Loss function**\n\nAfter comparing several loss functions and I've found that contrastive loss works the best in the current setup. Distance based logistic loss gives similar performance when model is trained with singe precision, but worse results for training with half precision."},{"metadata":{"_uuid":"da9c1df75fe452e028016bea142f6a58cd2e426c","trusted":true},"cell_type":"code","source":"class ContrastiveLoss(nn.Module):\n    \"\"\"Takes embeddings of two samples and a target label == 1 if samples are from the same class and label == 0 otherwise\n    \"\"\"\n    def __init__(self, margin=5.):\n        super(ContrastiveLoss, self).__init__()\n        self.margin = margin\n    def forward(self, ops, target, size_average=True):\n        op1, op2 = ops[0], ops[1]\n        dist = F.pairwise_distance(op1, op2)\n        pdist = dist*target\n        ndist = dist*(1-target)\n        loss = 0.5* ((pdist**2) + (F.relu(self.margin-ndist)**2))\n        return loss.mean() if size_average else losses.sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2fe3e89f59ebf0bb18077ba2a0a4403febade327","trusted":true},"cell_type":"code","source":"model = SiameseNet().cuda()\napply_init(model.head, nn.init.kaiming_normal_)\nloss_func=ContrastiveLoss().cuda()\nsiam_learner = Learner(data, model, loss_func=loss_func, model_dir=Path(os.getcwd()))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f51fc247123008493ef25d326c5778037def4a0","trusted":true},"cell_type":"code","source":"siam_learner.lr_find()\nsiam_learner.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4dc695aed7bc69c423a0e78267aa1140d6d7aef5","trusted":true},"cell_type":"code","source":"siam_learner.fit_one_cycle(5, slice(3*1e-2))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"58fcc5e3d6d1b5a37ea07d3977bea08a5f059b58","heading_collapsed":true},"cell_type":"markdown","source":"## Extract Embedding"},{"metadata":{"_uuid":"4cc28620906a9aafe865fb0d9aef4aec1b38d5b8","hidden":true,"trusted":false},"cell_type":"code","source":"def extract_embedding(dl, ds, mdl):\n    mdl.eval()\n    with torch.no_grad():\n        preds = torch.zeros((len(ds), 128))\n        start=0\n        for cnt, (x, y) in enumerate(dl, start=0):\n            size = x[0].shape[0]\n            preds[start:start+size,:] = model.get_embedding(x[0])\n            start+= size\n        return preds","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cb9806c4d44182a289d243b1ccccd4c28493fbdb","hidden":true,"trusted":false},"cell_type":"code","source":"data = (SiamImageItemList.from_df(df=trn_imgs, path=path/'train', cols=0)\n         .random_split_by_pct(valid_pct=1, seed=34)\n         .label_from_df(cols=target_col, label_cls=FloatList)\n         .transform(whl_tfms, size=224)\n         .databunch(bs=4))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"783bd34c8cdb2f912b44d619276758ebfda6a377","hidden":true,"trusted":false},"cell_type":"code","source":"dl = data.valid_dl\nds = data.valid_ds\nres = extract_embedding(dl, ds, model)\nwhlc=np.array([trn_imgs.loc[trn_imgs.Image == Path(i).name, 'Id'].values[0] for i in ds.items])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"80cf52470a1b08726021791cafc82100f4a9cee2","hidden":true},"cell_type":"markdown","source":"Embedding Visualization"},{"metadata":{"_uuid":"66288d107c4a66db574f334666ddf9895c19b5d2","hidden":true,"trusted":false},"cell_type":"code","source":"from sklearn.decomposition import PCA\npca = PCA(n_components=2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a1be1d9ad6f65f3a3cbc202aeb2c2e1c252adbe0","hidden":true,"trusted":false},"cell_type":"code","source":"pca.fit(res)\nembs2d = pca.transform(res)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7cfc7b844e524f5dc87f61ca13337ad5cca41ddb","hidden":true,"trusted":false},"cell_type":"code","source":"import colorsys\ndef random_colors(N, bright=True):\n    # This code is borrowed from https://github.com/matterport/Mask_RCNN\n    brightness = 1.0 if bright else 0.7\n    hsv = [(i / N, 1, brightness) for i in range(N)]\n    colors = list(map(lambda c: colorsys.hsv_to_rgb(*c), hsv))\n    random.shuffle(colors)\n    return colors\n\ndef plot_embeddings(embeddings=None,whlc=None, xlim=None, ylim=None):\n# This code is borrowed from https://github.com/adambielski/siamese-triplet/blob/master/Experiments_MNIST.ipynb\n    plt.figure(figsize=(10,10))\n    for cnt, i in enumerate(set(whlc)):\n#         print(cnt, i)\n        inds = np.where(whlc==i)[0]\n        plt.scatter(embeddings[inds,0], embeddings[inds,1], alpha=0.5, color=colors[cnt])\n    if xlim:\n        plt.xlim(xlim[0], xlim[1])\n    if ylim:\n        plt.ylim(ylim[0], ylim[1])\n    plt.legend(set(whlc))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dd60cf99519b026096274d21669b40ac8dc7355a","hidden":true},"cell_type":"markdown","source":"Not expecting something pretty good, given everything done here is pretty basic and crude. Lets see!!"},{"metadata":{"_uuid":"d669c704d7a7ee7af5f072ecf60fef373d30c86f","hidden":true,"trusted":true},"cell_type":"code","source":"colors = random_colors(len(set(whlc)))\nplot_embeddings(embs2d,whlc, xlim=None, ylim=None)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.0"}},"nbformat":4,"nbformat_minor":1}