{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":false},"cell_type":"markdown","source":"### **Overview**\nThe goal of this competition is identifying individual whales in images. Despite several whales are well represented in images, most of whales are unique or shown only in a few pictures. In particular, the train dataset includes 25k images and 5k unique whale ids. In addition, ~10k of images show unique whales ('new_whale' label). Checking public kernels suggests that a classical approach for classification problems based on softmax prediction for all classes is working quite well for this particular problem. However, strong class imbalance, handling labels represented by just several images, and 'new_whale' label deteriorates this approach. In addition, form the using this model for production, the above approach doesn't sound right since expansion of the model to identify new whales not represented in the train dataset would require retraining the model with increased softmax size. Meanwhile, the task of this competition could be reconsidered as checking similarities that suggests one-shot based learning algorithm to be applicable. This approach is less susceptible to data imbalance in this competition, can naturally handle 'new_whale' class, and is scalable in terms of a model for production (new classes can be added without retraining the model).\n\nThere are several public kernels targeted at using similarity based approach. First of all, it is an amazing [kernel posted by Martin Piotte](https://www.kaggle.com/martinpiotte/whale-recognition-model-with-score-0-78563), which discusses Siamese Neural Network architecture in details. A [fork of this kernel](https://www.kaggle.com/seesee/siamese-pretrained-0-822/notebook) reports 0.822 public LB score after training for 400 epochs. There is also a quite interesting [public kernel](https://www.kaggle.com/ashishpatel26/triplet-loss-network-for-humpback-whale-prediction) discussing Triplet Neural Network architecture, which is supposed to overperform Siamese architecture (check links in [this discussion](https://www.kaggle.com/c/humpback-whale-identification/discussion/76012)). Since both positive and negative examples are provided, the gradients are appeared to be more stable, and the network is not only trying to get away from negative or get close to positive example but arranges the prediction to fulfil both.\n\nIn this kernel I provide an example of a network inspired by Triplet architecture that is capable to reach **~0.81 public LB score after training within the kernel time limit** in my preliminary test. Training for more epochs is supposed to improve the prediction even further. The main trick of this kernel is **using batch all loss**. If the forward pass is completed for all images in a batch, why shouldn't I compare all of them when calculate the loss function? why should I limit myself by just several triplets? I have designed a loss function in such a way that allows performing all vs. all comparison within each batch, in other words for a batch of size 32 instead of comparing 32 triplets or 64 pairs the network performs processing of 9216 pairs of images at the same time. If training is done on multiple GPUs, the number of compared pares could be boosted even further since it it proportional to bs^2. Such a huge number of processed pairs further stabilizes gradients in comparison with triplet loss and allows more effective mapping of the input into the embedding space since not only pairs or triplets but entire picture is seen at the same time. This approach also allows effective search for hard negative examples at the later stage of training since each image is compared with all images in a batch. I tried to boost the search of the most difficult negative examples even further by selection of most similar negative examples to an anchor image when build triplets.\n\nMoreover, I added [**metric learning**](https://en.wikipedia.org/wiki/Similarity_learning) that boosts the performance of the model really a lot. In my preliminary test **for V2 setup I got 0.606->0.655 improvement** after I started calculating distance as d^2 = (v1-v2).T x A x (v1-v2) instead of Euclidian (v1-v2).T x (v1-v2), where A is a trainable matrix parameter. It can be considered as a trainable deformation of the space. However, the above form of the metric is quite slow at the inference time when distances for all image pairs are calculated. Also, it is quite difficult to impose a constrain on A during training to make it positive semi defined. Therefore, I use an alternative approximation formulation for distance calculation that is much faster at the inference time, symmetric and always positive, and have similar (or slightly better) performance with accounting for nonlinear coordinate transformations. To prevent predictions being spread too much in the embedding space, which deteriorates generalization, I added a compactificcation term to the loss that boosted the score from **0.74 to ~0.771** (V9). When I switched from ResNeXt50 to DenseNet169 backbone **I got 0.771 -> ~0.81 improvement**, and it appeared that DenseNet121 works a little bit better.\n\nI switched to using cropped rescaled square images since they work better. The idea behind rectangular images generated without distortion of images, which I used in the first versions of the kernel, is the following. Since bounding boxes have different aspect ratio, each image has different degree of distortion when rescaled to square one, which could negatively affect training. However, it looks that the setup when the tail is occupying approximately the same area in the image, no matter what is its orientation and distortion, works better. Looking at the produced images I really do not understand why. In my preliminary test for V7 I could get a **boost from ~0.70 to ~0.75 public LB** after this modification. In the current setup, the images are cropped according to bounding boxes (thanks to [this fork](https://www.kaggle.com/suicaokhoailang/generating-whale-bounding-boxes) and to Martin Piotte for posting the original kernel) and rescaled to 224x224 square images.\n\n**Milestones of the kernel score improvement and corresponding modifications are summorized in** [**this discussion**](https://www.kaggle.com/c/humpback-whale-identification/discussion/79086#466562). This kernel is written with using fast.ai 0.7 since a newer version of fast.ai doesn't work well in kaggle: using more than one core for data loading leads to [bus error](https://www.kaggle.com/product-feedback/72606) \"DataLoader worker (pid 137) is killed by signal: Bus error\". Therefore, when I tried to write similar kernel with fast.ai 1.0, it appeared to be much slower, more than 1 hour per epoch vs. 20-30 min with this kernel if ResNet34 and images of size 576x192 are used. People interested in fast.ai 1.0 could check an example of Siamese network [here](https://www.kaggle.com/raghavab1992/siamese-with-fast-ai). Also since fast.ai 0.7 is not really designed to build Siamese and Triplet networks, some parts are a little bit far away from a standard usage of the library.\n\n**Highlights: Batch all loss, metric learning, mining hard negative examples**"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"!pip install fastai==0.7.0 --no-deps\n!pip install torch==0.4.1 torchvision==0.2.1\n!pip install imgaug","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9636cfe30b9f697340fd1dc67144d2cd13ff7e6f"},"cell_type":"code","source":"from fastai.conv_learner import *\nfrom fastai.dataset import *\n\nimport pandas as pd\nimport numpy as np\nimport os\nfrom sklearn.model_selection import train_test_split\nfrom tqdm import tqdm_notebook\nimport matplotlib.pyplot as plt\nimport random\nimport math\nimport imgaug as ia\nfrom imgaug import augmenters as iaa","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f89644af8d8a80cbfa272379dc34d18c54c144c7"},"cell_type":"code","source":"PATH = './'\nTRAIN = '../input/humpback-whale-identification/train/'\nTEST = '../input/humpback-whale-identification/test/'\nLABELS = '../input/humpback-whale-identification/train.csv'\nBOXES = '../input/generating-whale-bounding-boxes/bounding_boxes.csv'\nMODLE_INIT = '../input/pytorch-pretrained-models/'\n\nn_embedding = 512\nbs = 32\nsz = 224 #increase the image size at the later stage of training\nnw = 2","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"_uuid":"6735c2010738adcbad39f6934d1031f4bd8f0c62"},"cell_type":"markdown","source":"### **Data**\nThe class Loader creates crops with sizes 224x224 based on the bounding boxes. In addition, data augmentation based on [imgaug library](https://github.com/aleju/imgaug) is applied. This library is quite interesting in the context of the competition since it supports hue and saturation augmentations as well as conversion to gray scale. Following the idea of progressive resizing on the later stage of training one can switch to higher resolution images, but at the beginning low resolution is used to speed up the convergence and improve generalization ability of the model. "},{"metadata":{"trusted":true,"_uuid":"5088875811058ee7a5c286dbbde3fc68cbe2dfc5"},"cell_type":"code","source":"def open_image(fn):\n    flags = cv2.IMREAD_UNCHANGED+cv2.IMREAD_ANYDEPTH+cv2.IMREAD_ANYCOLOR\n    if not os.path.exists(fn):\n        raise OSError('No such file or directory: {}'.format(fn))\n    elif os.path.isdir(fn):\n        raise OSError('Is a directory: {}'.format(fn))\n    else:\n        try:\n            im = cv2.imread(str(fn), flags)\n            if im is None: raise OSError(f'File not recognized by opencv: {fn}')\n            return cv2.cvtColor(im, cv2.COLOR_BGR2RGB)\n        except Exception as e:\n            raise OSError('Error handling image at: {}'.format(fn)) from e\n\nclass Loader():\n    def __init__(self,path,tfms_g=None, tfms_px=None):\n        #tfms_g - geometric augmentation (distortion, small rotation, zoom)\n        #tfms_px - pixel augmentation and flip\n        self.boxes = pd.read_csv(BOXES).set_index('Image')\n        self.path = path\n        self.tfms_g = iaa.Sequential(tfms_g,random_order=False) \\\n                        if tfms_g is not None else None\n        self.tfms_px = iaa.Sequential(tfms_px,random_order=False) \\\n                        if tfms_px is not None else None\n    def __call__(self, fname):\n        fname = os.path.basename(fname)\n        x0,y0,x1,y1 = tuple(self.boxes.loc[fname,['x0','y0','x1','y1']].tolist())\n        img = open_image(os.path.join(self.path,fname))\n        l1,l0,_ = img.shape\n        b0,b1 = x1-x0, y1-y0\n        #padding\n        x0n,x1n = max(int(x0 - b0*0.05),0), min(int(x1 + b0*0.05),l0-1)\n        y0n,y1n = max(int(y0 - b1*0.05),0), min(int(y1 + b1*0.05),l1-1)\n         \n        if self.tfms_g != None: img = self.tfms_g.augment_image(img)\n        img = cv2.resize(img[y0n:y1n,x0n:x1n,:], (sz,sz))\n        if self.tfms_px != None: img = self.tfms_px.augment_image(img)\n        return img.astype(np.float)/255","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d1d6372c8ea26c641c02df8c12631c6c805d017c"},"cell_type":"markdown","source":"The pdFilesDataset class below generates triplets of images: original image, different image with the same label, an image with different label (**including new_label images**). Image_selection class performs of selection of the most difficult negative examples used in the last part of the kernel. When this class is created, 64 most similar images with different label are selected for each image. So instead of random sampling of negative examples, sampling of these 64 most difficult images for each anchor image can be used during training."},{"metadata":{"trusted":true,"_uuid":"fd4e00567068279daca4ed915472c77111f4238f"},"cell_type":"code","source":"def get_idxs0(names, df, n=64):\n    idxs = []\n    for name in names:\n        label = df[df.Image == name].Id.values[0]\n        idxs.append(df[df.Id != label].Image.sample(n).values)\n    return idxs\n\nclass Image_selection:\n    def __init__(self,fnames,data,emb_df=None,model=None):\n        if emb_df is None or model is None:\n            df = data.copy()#.set_index('Image')\n            counts = Counter(df.Id.values)\n            df['c'] = df['Id'].apply(lambda x: counts[x])\n            df['label'] = df.Id\n            df.loc[df.c == 1,'label'] = 'new_whale'\n            df = df.sort_values(by=['c'])\n            df.label = pd.factorize(df.label)[0]\n            l1 = 1 + df.label.max()\n            l2 = len(df[df.label==0])\n            df.loc[df.label==0,'label'] = range(l1, l1+l2) #assign unique ids\n            df = df.set_index('label')\n            l = len(fnames)\n            idxs = Parallel(n_jobs=nw)(delayed(get_idxs0)\\\n                (fnames[int(i*l/nw):int((i+1)*l/nw)], df) for i in range(nw))\n            idxs = [y for x in idxs for y in x]\n        else:\n            data = data.copy().set_index('Image')\n            trn_emb = emb_df.copy()\n            trn_emb.set_index('files',inplace=True)\n            trn_emb['emb'] = [[float(i) for i in s.split()] for s in trn_emb['emb']]\n            trn_emb = data.join(trn_emb)\n            trn_emb = trn_emb.reset_index()\n            trn_emb['idx'] = np.arange(len(trn_emb))\n            trn_emb = trn_emb.set_index('Id')\n            emb = np.array(trn_emb.emb.tolist())\n            l = len(fnames)\n            idxs = []\n            model.eval()\n            with torch.no_grad():\n                #selection of the most difficult negative examples\n                m = model.module if isinstance(model,FP16) else model\n                emb = torch.from_numpy(emb).half().cuda()\n                for name in tqdm_notebook(fnames):\n                    label = trn_emb.loc[trn_emb.Image == name].index[0]\n                    v0 = np.array(trn_emb.loc[trn_emb.Image == name,'emb'].tolist()[0])\n                    v0 = torch.from_numpy(v0).half().cuda()\n                    d = m.get_d(v0,emb)\n                    ids = trn_emb.loc[trn_emb.index!=label].idx.tolist()\n                    sorted, indices = torch.sort(d[ids])\n                    idxs.append([ids[i] for i in indices[:64]]) \n            trn_emb = trn_emb.set_index('idx')\n            idxs = [trn_emb.loc[idx,'Image'] for idx in idxs]\n        self.df = pd.DataFrame({'Image':fnames,'idxs':idxs}).set_index('Image')\n        \n    def get(self,name):\n        return np.random.choice(self.df.loc[name].values[0],1)[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"78aeb73926f7707c12dd5fdef74dd1cc4e2d0d9b"},"cell_type":"code","source":"class pdFilesDataset(FilesDataset):\n    def __init__(self, data, path, transform):\n        df = data.copy()\n        counts = Counter(df.Id.values)\n        df['c'] = df['Id'].apply(lambda x: counts[x])\n        #in the production runs df.c>1 should be used\n        fnames = df[(df.c>2) & (df.Id != 'new_whale')].Image.tolist()\n        df['label'] = df.Id\n        df.loc[df.c == 1,'label'] = 'new_whale'\n        df = df.sort_values(by=['c'])\n        df.label = pd.factorize(df.label)[0]\n        l1 = 1 + df.label.max()\n        l2 = len(df[df.label==0])\n        df.loc[df.label==0,'label'] = range(l1, l1+l2) #assign unique ids\n        self.labels = df.copy().set_index('Image')\n        self.names = df.copy().set_index('label')\n        if path == TRAIN:\n            #data augmentation: 8 degree rotation, 10% stratch, shear\n            tfms_g = [iaa.Affine(rotate=(-8, 8),mode='reflect',\n                scale={\"x\": (0.9, 1.1), \"y\": (0.9, 1.1)}, shear=(-16,16))]\n            #data augmentation: horizontal flip, hue and staturation augmentation,\n            #gray scale, blur\n            tfms_px = [iaa.Fliplr(0.5), iaa.AddToHueAndSaturation((-20, 20)),\n                iaa.Grayscale(alpha=(0.0, 1.0)),iaa.GaussianBlur((0, 1.0))]\n            self.loader = Loader(path,tfms_g,tfms_px)\n        else: self.loader = Loader(path)\n        self.selection = None\n        super().__init__(fnames, transform, path)\n    \n    def get_x(self, i):\n        label = self.labels.loc[self.fnames[i],'label']\n        #random selection of a positive example\n        for j in range(10): #sometimes loc call fails\n            try:\n                names = self.names.loc[label].Image\n                break\n            except: None\n        name_p = names if isinstance(names,str) else \\\n            random.sample(set(names) - set([self.fnames[i]]),1)[0]\n        #random selection of a negative example\n        if(self.selection == None):\n            for j in range(10): #sometimes loc call fails\n                try:\n                    names = self.names.loc[self.names.index!=label].Image\n                    break\n                except: names = self.fnames[i]\n            name_n = names if isinstance(names,str) else names.sample(1).values[0]\n        else:  \n            name_n = self.selection.get(self.fnames[i])\n        imgs = [self.loader(os.path.join(self.path,self.fnames[i])),\n                self.loader(os.path.join(self.path,name_p)),\n                self.loader(os.path.join(self.path,name_n)),\n                label,label,self.labels.loc[name_n,'label']]\n        return imgs\n    \n    def get_y(self, i):\n        return 0\n    \n    def get(self, tfm, x, y):\n        if tfm is None:\n            return (*x,0)\n        else:\n            x1, y1 = tfm(x[0],x[3])\n            x2, y2 = tfm(x[1],x[4])\n            x3, y3 = tfm(x[2],x[5])\n            #combine all images into one tensor\n            x = np.stack((x1,x2,x3),0)\n            return x,(y1,y2,y3)\n        \n    def get_names(self,label):\n        names = []\n        for j in range(10):\n            try:\n                names = self.names.loc[label].Image\n                break\n            except: None\n        return names\n        \n    @property\n    def is_multi(self): return True\n    @property\n    def is_reg(self):return True\n    \n    def get_c(self): return n_embedding\n    def get_n(self): return len(self.fnames)\n    \n#class for loading an individual images when embedding is computed\nclass FilesDataset_single(FilesDataset):\n    def __init__(self, data, path, transform):\n        self.loader = Loader(path)\n        fnames = os.listdir(path)\n        super().__init__(fnames, transform, path)\n        \n    def get_x(self, i):\n        return self.loader(os.path.join(self.path,self.fnames[i]))\n                           \n    def get_y(self, i):\n        return 0\n        \n    @property\n    def is_multi(self): return True\n    @property\n    def is_reg(self):return True\n    \n    def get_c(self): return n_embedding\n    def get_n(self): return len(self.fnames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6bf08dcffcd4e4cc0b53e024618720ba5ff2fe08"},"cell_type":"code","source":"def get_data(sz,bs,fname_emb=None,model=None):\n    tfms = tfms_from_model(resnet34, sz, crop_type=CropType.NO)\n    tfms[0].tfms = [tfms[0].tfms[2],tfms[0].tfms[3]]\n    tfms[1].tfms = [tfms[1].tfms[2],tfms[1].tfms[3]]\n    df = pd.read_csv(LABELS)\n    trn_df, val_df = train_test_split(df,test_size=0.2, random_state=42)\n    ds = ImageData.get_ds(pdFilesDataset, (trn_df,TRAIN), (val_df,TRAIN), tfms)\n    md = ImageData(PATH, ds, bs, num_workers=nw, classes=None)\n    if fname_emb != None and model != None:\n        emb = pd.read_csv(fname_emb)\n        md.trn_dl.dataset.selection = Image_selection(md.trn_dl.dataset.fnames,trn_df,\n                                                      emb,model)\n        md.val_dl.dataset.selection = Image_selection(md.val_dl.dataset.fnames,val_df,\n                                                      emb,model)\n    return md","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1a52a9a47400d36d7449384d1209eff496d4413a"},"cell_type":"markdown","source":"The image below demonstrates an example of triplets of rectangular 576x192 augmented images used for training. To be honest, some of those triplets are quite hard, and I don't think that I could even reach the same performance as the model after training (~99% accuracy in identifications of 2 similar images in a triplet). "},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"1bae0892446f49e1939764fb36d6c5ae1ee42e44"},"cell_type":"code","source":"md = get_data(sz,bs)\n\nx,y = next(iter(md.trn_dl))\nprint(x.shape, y[0].shape)\n\ndef display_imgs(x,lbs=None):\n    columns = 3\n    rows = min(bs,16)\n    ig,ax = plt.subplots(rows, columns, figsize=(columns*5, rows*5))\n    for i in range(rows):\n        for j in range(columns):\n            idx = j+i*columns\n            ax[i,j].imshow((x[j][i,:,:,:]*255).astype(np.int))\n            ax[i,j].axis('off')\n            if lbs is not None:\n                ax[i,j].text(10, 25, lbs[j][i], size=20, color='red')\n    plt.subplots_adjust(wspace=0, hspace=0)\n    plt.show()\n\nto_lb = md.trn_ds.names.Id.to_dict()\nlbs = [[to_lb[idx] for idx in y_cur.tolist()] for y_cur in y]\ndisplay_imgs((md.trn_ds.denorm(x[:,0,:,:,:]),md.trn_ds.denorm(x[:,1,:,:,:]),\\\n              md.trn_ds.denorm(x[:,2,:,:,:])),lbs)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6c6a947158a6f6596e595a01046cd09ec043ab11"},"cell_type":"markdown","source":"### **Model**\nLooking to the images I would expect that larger models should perform better in this competition since there are many really difficult examples that require perception capability surpassing human ones (check examples in the last part of the kernel). The convolutional part is taken from the original DenseNet121 model pretrained on ImageNet, and adaptive pooling allows using of images of any sizes and aspect ratios. The convolutional part is followed by 2 fully connected layers added to convert the prediction into embedding space.\n\nOn the top, a learnable metric is applied that converts differences of vector components in the embedding space for all images within a batch into distances. This part replaces calculation of distances in Euclidean space by one in distorted space. The first thing that came me in mind is generalized formula for a distance in a linear space d^2 = (v1-v2).T x A x (v1-v2), where A is a trainable matrix. For Euclidian space d^2 = (v1-v2).T x (v1-v2), i.e. A is a identity matrix. Using of such distance boosted the score of the v2 version of this kernel from 0.606 to 0.655. However, the above form of the metric is quite slow at the inference time when distances for all image pairs are calculated. Also, it is quite difficult to impose a constrain on A during training to make it positive semi defined (distance is always positive). Therefore, I use an alternative approximation formulation for distance calculation that is much faster at the inference time, symmetric and always positive, and have similar (or slightly better) performance. If after calculation of differences between v1 and v2 the result goes through a linear layer, before summing the squares, the distance would include also contribution from mixed terms like dv1 * dv2 (like in an approach with matrix A), though many of these terms are correlated, and results are a little bit worse (it can be viewed as a factorization of the general approach). However, when I included also (v1-v2)^2 terms as an input, the performance of such metric surpassed one for d^2 = (v1-v2).T x A x (v1-v2). It can be explained as an approximation of some nonlinear transformation of a space including square, cubic, and quadratic terms. The concept of metric learning is similar to one in [Martin's kernel](https://www.kaggle.com/martinpiotte/whale-recognition-model-with-score-0-78563). However, in that case the metric converts the difference in the probability of two classes to have the same label, while in the current kernel it converts the difference between vectors into a distance in a distorted space."},{"metadata":{"trusted":true,"_uuid":"f6014758211d8ce5b0c57c4d1eaaff13eb3a2f27"},"cell_type":"code","source":"class Metric(nn.Module):\n    def __init__(self, emb_sz=64):\n        super().__init__()   \n        self.l = nn.Linear(emb_sz*2,emb_sz*2,False)\n        \n    def forward(self,d):\n        d2 = d.pow(2)\n        d = self.l(torch.cat((d,d2),dim=-1))\n        x = d.pow(2).sum(dim=-1)\n        return x.view(-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"51bd808549c475d5b5ac483afb7c7e60b3f71667","_kg_hide-input":true},"cell_type":"code","source":"def resnext50(pretrained=True):\n    model = resnext_50_32x4d()\n    name = 'resnext_50_32x4d.pth'\n    if pretrained:\n        path = os.path.join(MODLE_INIT,name)\n        load_model(model, path)\n    return model\n\nclass TripletResneXt50(nn.Module):\n    def __init__(self, pre=True, emb_sz=64, ps=0.5):\n        super().__init__()\n        encoder = resnext50(pretrained=pre)\n        #add DataParallel to allow support of multiple GPUs\n        self.cnn = nn.DataParallel(nn.Sequential(encoder[0],encoder[1],nn.ReLU(),\n                        encoder[3],encoder[4],encoder[5],encoder[6],encoder[7]))\n        self.head = nn.DataParallel(nn.Sequential(AdaptiveConcatPool2d(), Flatten(),\n                        nn.Dropout(ps),nn.Linear(4096, 512), nn.ReLU(), \n                        nn.BatchNorm1d(512), nn.Dropout(ps), nn.Linear(512, emb_sz)))\n        self.metric = nn.DataParallel(Metric(emb_sz))\n        \n    def forward(self,x):\n        x1,x2,x3 = x[:,0,:,:,:],x[:,1,:,:,:],x[:,2,:,:,:]\n        x1 = self.head(self.cnn(x1))\n        x2 = self.head(self.cnn(x2))\n        x3 = self.head(self.cnn(x3))\n        x = torch.cat((x1,x2,x3))\n        sz = x.shape[0]\n        x1 = x.unsqueeze(1).expand((sz,sz,-1))\n        x2 = x1.transpose(0,1)\n        #matrix of all vs all differencies\n        d = (x1 - x2).view(sz*sz,-1)\n        return self.metric(d)\n    \n    def get_embedding(self, x):\n        return self.head(self.cnn(x))\n    \n    def get_d(self, x0, x):\n        d = (x - x0)\n        return self.metric(d)\n    \nclass ResNeXt50Model():\n    def __init__(self,pre=True,name='TripletResneXt50',**kwargs):\n        self.model = to_gpu(TripletResneXt50(pre=True,**kwargs))\n        self.name = name\n\n    def get_layer_groups(self, precompute):\n        m = self.model.module if isinstance(self.model,FP16) else self.model\n        if precompute:\n            return [m.head] + [m.metric]\n        c = children(m.cnn.module)\n        return list(split_by_idxs(c,[5])) + [m.head] + [m.metric]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0169e13de36d230c5a503051d8610d346846a362","_kg_hide-input":true},"cell_type":"code","source":"def get_densenet169(pre=True):\n    with warnings.catch_warnings():\n        warnings.simplefilter(\"ignore\")\n        layers = cut_model(cut_model(densenet169(pre), -1)[0],-1)\n    return nn.Sequential(*layers)\n\nclass TripletDenseNet169(nn.Module):\n    def __init__(self, pre=True, emb_sz=64, ps=0.5):\n        super().__init__()\n        encoder = get_densenet169(pre)\n        #add DataParallel to allow support of multiple GPUs\n        self.cnn = nn.DataParallel(nn.Sequential(encoder[0],encoder[1],nn.ReLU(),\n                        encoder[3],encoder[4],encoder[5],encoder[6],encoder[7],\n                        encoder[8],encoder[9],encoder[10]))\n        self.head = nn.DataParallel(nn.Sequential(AdaptiveConcatPool2d(), Flatten(),\n                        nn.Dropout(ps),nn.Linear(3328, 512), nn.ReLU(), \n                        nn.BatchNorm1d(512), nn.Dropout(ps), nn.Linear(512, emb_sz)))\n        self.metric = nn.DataParallel(Metric(emb_sz))\n        \n    def forward(self,x):\n        x1,x2,x3 = x[:,0,:,:,:],x[:,1,:,:,:],x[:,2,:,:,:]\n        x1 = self.head(self.cnn(x1))\n        x2 = self.head(self.cnn(x2))\n        x3 = self.head(self.cnn(x3))\n        x = torch.cat((x1,x2,x3))\n        sz = x.shape[0]\n        x1 = x.unsqueeze(1).expand((sz,sz,-1))\n        x2 = x1.transpose(0,1)\n        #matrix of all vs all differencies\n        d = (x1 - x2).view(sz*sz,-1)\n        return self.metric(d)\n    \n    def get_embedding(self, x):\n        return self.head(self.cnn(x))\n    \n    def get_d(self, x0, x):\n        d = (x - x0)\n        return self.metric(d)\n    \nclass DenseNet169Model():\n    def __init__(self,pre=True,name='TripletDenseNet169',**kwargs):\n        self.model = to_gpu(TripletDenseNet169(pre=True,**kwargs))\n        self.name = name\n\n    def get_layer_groups(self, precompute):\n        m = self.model.module if isinstance(self.model,FP16) else self.model\n        if precompute:\n            return [m.head] + [m.metric]\n        c = children(m.cnn.module)\n        return list(split_by_idxs(c,[8])) + [m.head] + [m.metric]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"80faa04a67deaa8c0aa67dc0c48608ae52481f52"},"cell_type":"code","source":"def get_densenet121(pre=True):\n    with warnings.catch_warnings():\n        warnings.simplefilter(\"ignore\")\n        layers = cut_model(cut_model(densenet121(pre), -1)[0],-1)\n    return nn.Sequential(*layers)\n\nclass TripletDenseNet121(nn.Module):\n    def __init__(self, pre=True, emb_sz=64, ps=0.5):\n        super().__init__()\n        encoder = get_densenet121(pre)\n        #add DataParallel to allow support of multiple GPUs\n        self.cnn = nn.DataParallel(nn.Sequential(encoder[0],encoder[1],nn.ReLU(),\n                    encoder[3],encoder[4],encoder[5],encoder[6],encoder[7],\n                    encoder[8],encoder[9],encoder[10]))\n        self.head = nn.DataParallel(nn.Sequential(AdaptiveConcatPool2d(), Flatten(),\n                    nn.BatchNorm1d(2048),nn.Dropout(ps),nn.Linear(2048, 1024), nn.ReLU(), \n                    nn.BatchNorm1d(1024), nn.Dropout(ps), nn.Linear(1024, emb_sz)))\n        self.metric = nn.DataParallel(Metric(emb_sz))\n        \n    def forward(self,x):\n        x1,x2,x3 = x[:,0,:,:,:],x[:,1,:,:,:],x[:,2,:,:,:]\n        x1 = self.head(self.cnn(x1))\n        x2 = self.head(self.cnn(x2))\n        x3 = self.head(self.cnn(x3))\n        x = torch.cat((x1,x2,x3))\n        sz = x.shape[0]\n        x1 = x.unsqueeze(1).expand((sz,sz,-1))\n        x2 = x1.transpose(0,1)\n        #matrix of all vs all differencies\n        d = (x1 - x2).view(sz*sz,-1)\n        return self.metric(d)\n    \n    def get_embedding(self, x):\n        return self.head(self.cnn(x))\n    \n    def get_d(self, x0, x):\n        d = (x - x0)\n        return self.metric(d)\n    \nclass DenseNet121Model():\n    def __init__(self,pre=True,name='TripletDenseNet21',**kwargs):\n        self.model = to_gpu(TripletDenseNet121(pre=True,**kwargs))\n        self.name = name\n\n    def get_layer_groups(self, precompute):\n        m = self.model.module if isinstance(self.model,FP16) else self.model\n        if precompute:\n            return [m.head] + [m.metric]\n        c = children(m.cnn.module)\n        return list(split_by_idxs(c,[8])) + [m.head] + [m.metric]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"588ff9ef22ef09568e371794cdaf9387d05a0fc2"},"cell_type":"markdown","source":"### **Loss function**\nI my tests I have performed a comparison of several loss functions and found that contrastive loss works the best in the current setup. I also, in comparison with the previous version, select only nonzero terms when perform averaging. I found that at a later stage of training many pairs are already well separated and contribute zero to gradients while decrease the magnitude of useful gradients during averaging. The drawback of such an approach is that during training **values of train and validation loss must be ignored, since they are calculated each time based on different number of pairs**, and only the value of metric (like accuracy of identifying a correct pair of images with the same label in a triplet, T_acc, or accuracy of identifying a correct pair of images with the same label within a hardest triplet in batch for each anchor image, BH_acc) must be tracked. However, after performing mining hardest negative examples, even accuracy metrics are not reliable, and the only way to check performance of the model is validation based on the entire validation dataset rather than batches with using the same metric as one in the competition. To prevent predictions being spread too much in the embedding space, which deteriorates generalization, in the V9 of the kernel I added a compactificcation term to the loss that boosted the score from **0.74 to ~0.78** and stopped unstable behaviour of val loss."},{"metadata":{"trusted":true,"_uuid":"5887f278eaac38400954d53b1cfa80c5a6a88159"},"cell_type":"code","source":"class Contrastive_loss(nn.Module):\n    def __init__(self, m=10.0, wd=1e-4):\n        super().__init__()\n        self.m, self.wd = m, wd\n        \n    def forward(self, d, target):\n        d = d.float()\n        #matrix of all vs all comparisons\n        t = torch.cat(target)\n        sz = t.shape[0]\n        t1 = t.unsqueeze(1).expand((sz,sz))\n        t2 = t1.transpose(0,1)\n        y = ((t1==t2) + to_gpu(torch.eye(sz).byte())).view(-1)\n    \n        loss_p = d[y==1]\n        loss_n = F.relu(self.m - torch.sqrt(d[y==0]))**2\n        loss = torch.cat((loss_p,loss_n),0)\n        loss = loss[torch.nonzero(loss).squeeze()]\n        loss = loss.mean() if loss.shape[0] > 0 else loss.sum()\n        loss += self.wd*(d**2).mean() #compactification term\n        return loss\n\n#accuracy within a triplet\ndef T_acc(d, target):\n    sz = target[0].shape[0]\n    lp = [3*sz*i + i+sz for i in range(sz)]\n    ln = [3*sz*i + i+2*sz for i in range(sz)]\n    dp, dn = d[lp], d[ln]\n    return (dp < dn).float().mean()\n\n#accuracy within a hardest triplet in a batch for each anchor image\ndef BH_acc(d, target):\n    t = torch.cat(target)\n    sz = t.shape[0]\n    t1 = t.unsqueeze(1).expand((sz,sz))\n    t2 = t1.transpose(0,1)\n    y = (t1==t2)\n    d = d.float().view(sz,sz)\n    BH = []\n    for i in range(sz):\n        dp = d[i,y[i,:] == 1].max()\n        dn = d[i,y[i,:] == 0].min()\n        BH.append(dp < dn)\n    return torch.FloatTensor(BH).float().mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0ab74f6f28f43c03d9e612b3aeca90e5a586dc84"},"cell_type":"markdown","source":"I also tried batch hard triplet loss introduced in https://arxiv.org/pdf/1703.07737.pdf , however I didn't get better results when my current one yet. The problem, I expect, is that triplet loss doesn't try to bring all predictions with the same label to one point in an embedding space, while keeping objects of the same kind together. If would be great if there was no new_whale class. The check I perform is based on the distance to the nearest neighbors: if there are no whales in a sphere of a particular radius, I assign new whale label. It works well if the points are packed with approximately the same density, but if for some classes the spacing between images of the same class is too large, they will be misclassified as a new_whale.\n\nThe class in the hidden cell below implements such approach. The thing I have introduced here is a parameter n that indicates how many hardest triplets should be selected per batch, which allows gradually going from batch all to batch hard loss during training, when the fraction of hard triplets drops (though, sorting takes additional time)."},{"metadata":{"trusted":true,"_uuid":"391a9b50268b8947ea63569724cc7175689703d5","_kg_hide-input":true},"cell_type":"code","source":"#batch hard loss: https://arxiv.org/pdf/1703.07737.pdf \nclass BH_loss(nn.Module):\n    def __init__(self, n=1, m=0.0):\n        super().__init__()\n        self.n = n #select n hardest triplets\n        self.m = m\n    \n    def forward(self, input, target, size_average=True):\n        #matrix of all vs all comparisons\n        t = torch.cat(target)\n        sz = t.shape[0]\n        t1 = t.unsqueeze(1).expand((sz,sz))\n        t2 = t1.transpose(0,1)\n        y = ((t1==t2) + to_gpu(torch.eye(sz).byte()))\n        d = input.float().view(sz,sz)\n        D = []\n        for i in range(2*sz//3):\n            dp = d[i,y[i,:] == 1].max()\n            dn = d[i,y[i,:] == 0]\n            dist, idxs = torch.sort(dn)\n            n = min(self.n,dn.shape[0])\n            for j in range(n):\n                D.append((self.m + dp - dn[idxs[j]]).unsqueeze(0))\n        loss = torch.log1p(torch.exp(torch.cat(D)))\n        loss = loss.mean() if size_average else loss.sum()\n        return loss","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9b6f2d53d1efb2a1252843960ece5a769d55701e"},"cell_type":"markdown","source":"### **Training**"},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"9fbb63f57b1ad3f40c85cee43ef90c14c5e4fad2"},"cell_type":"code","source":"learner = ConvLearner(md,DenseNet121Model(ps=0.0,emb_sz=n_embedding))\nlearner.opt_fn = optim.Adam\nlearner.clip = 1.0 #gradient clipping\nlearner.crit = Contrastive_loss()\nlearner.metrics = [T_acc,BH_acc]\nlearner.freeze_to(-2) #unfreez metric and head block\nlearner #click \"output\" to see details of the model","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"_uuid":"a02aad7c0a311ed1c767fb46f6081f3f62496b3b"},"cell_type":"markdown","source":"First, I train only the fully connected part of the model and the metric while keeping the rest frozen. It allows to avoid corruption of the pretrained weights at the initial stage of training due to random initialization of the head layers. So the power of transfer learning is fully utilized when the training is continued."},{"metadata":{"trusted":true,"_uuid":"31cf2eea892ab56280f7ada4db72aff9ccfd46ab","scrolled":true},"cell_type":"code","source":"lr = 1e-3\nwith warnings.catch_warnings():\n    warnings.simplefilter(\"ignore\")\n    learner.fit(lr,1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"afc18d43b9da9b0315973df29d633a84d26666e8"},"cell_type":"markdown","source":"Next, I unfreeze all weights and allow training of entire model. One trick that I use is applying different learning rates in different parts of the model: the learning rate in the fully connected part is still lr, last two blocks of ResNeXt are trained with lr/3, and first layers are trained with lr/10. Since low-level detectors do not vary much from one image data set to another, the first layers do not require substantial retraining compared to the parts of the model working with high level features. Another trick is learning rate annealing. Periodic learning rate increase followed by slow decrease drives the system out of steep minima (when lr is high) towards broader ones (which are explored when lr decreases) that enhances the ability of the model to generalize and reduces overfitting. The length of the cycles gradually increases during training. Usage of half precision doubles the maximum batch size that allows to compare more pairs in each batch."},{"metadata":{"trusted":true,"_uuid":"b9c2d0bcc53b6cdf01f9d7c8e10ec573aea8f85b"},"cell_type":"code","source":"learner.unfreeze() #unfreeze entire model\nlrs=np.array([lr/10,lr/3,lr,lr])\nlearner.half() #half precision","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e4ca0787ae5e927920c4ae868328ee0840a6f5cd"},"cell_type":"markdown","source":"**Since the loss is calculated as an average of nonzero terms, as mentioned above, it's value is not relaiable and must be ignored.** Instead the values of T_acc and BH_acc metrics should be considered. "},{"metadata":{"trusted":true,"_uuid":"d50ef8f1b26bced8872933422f26abfeaa32908f","scrolled":true},"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter(\"ignore\")\n    learner.fit(lrs/2,4,cycle_len=1,use_clr=(10,20))\n    learner.fit(lrs/4,4,cycle_len=2,use_clr=(10,20))\nlearner.save('model0')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2f5648fc035df75b0a371c05fbf83aa377e4d8fc"},"cell_type":"markdown","source":"### **Embedding**\nThe following code converts images into embedding vectors, which are used later to generate predictions based on the nearest neighbor search in a space deformed by a metric (see Metric class)."},{"metadata":{"trusted":true,"_uuid":"3ca98b9da1eee10c8f61efe03ea3d4fb23b6b796"},"cell_type":"code","source":"def extract_embedding(model,path):\n    tfms = tfms_from_model(resnet34, sz, crop_type=CropType.NO)\n    tfms[0].tfms = [tfms[0].tfms[2],tfms[0].tfms[3]]\n    tfms[1].tfms = [tfms[1].tfms[2],tfms[1].tfms[3]]\n    ds = ImageData.get_ds(FilesDataset_single, (None,TRAIN), (None,TRAIN),\n         tfms, test=(None,path))\n    md = ImageData(PATH, ds, 3*bs, num_workers=nw, classes=None)\n    model.eval()\n    with torch.no_grad():\n        preds = torch.zeros((len(md.test_dl.dataset), n_embedding))\n        start=0\n        for i, (x,y) in tqdm_notebook(enumerate(md.test_dl,start=0),\n                                       total=len(md.test_dl)):\n            size = x.shape[0]\n            m = model.module if isinstance(model,FP16) else model\n            preds[start:start+size,:] = m.get_embedding(x.half())\n            start+= size\n        return preds, [os.path.basename(name) for name in md.test_dl.dataset.fnames]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ccf3cd5fd576aca25e7bd2bcda779b4e6d34bd97"},"cell_type":"code","source":"emb, names = extract_embedding(learner.model,TRAIN)\ndf = pd.DataFrame({'files':names,'emb':emb.tolist()})\ndf.emb = df.emb.map(lambda emb: ' '.join(list([str(i) for i in emb])))\ndf.to_csv('train_emb.csv', header=True, index=False)\n\nemb, names = extract_embedding(learner.model,TEST)\ndf_test = pd.DataFrame({'files':names,'emb':emb.tolist()})\ndf_test.emb = df_test.emb.map(lambda emb: ' '.join(list([str(i) for i in emb])))\ndf_test.to_csv('test_emb.csv', header=True, index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e11ae99694ba3210eec298d94988487e23bca1cd"},"cell_type":"markdown","source":"### **Validation**"},{"metadata":{"_uuid":"69f3b5e84bc4e0c8970a5a799c9c77e61d7ffa73"},"cell_type":"markdown","source":"Find 16 nearest train neighbors in embedding space for each validation image. Since there can be several neighbors with the same label, instead of 5 I use 16 here. The following code will select 5 nearest neighbors with different labels. \"new_whale\" label can be assigned as a prediction at a distance dcut. In this case, if the number of neighbors at a distance shorter than dcut is less than 5, the image is considered to be different from others, and \"new_whale\" is assigned."},{"metadata":{"trusted":true,"_uuid":"d4180d10027a4307cbcae705ee71d8d0b08272a5"},"cell_type":"code","source":"def get_nbs(model,x,y,n=16):\n    d, idxs = [], []\n    sz = x.shape[0]\n    model.eval()\n    with torch.no_grad():\n        m = model.module if isinstance(model,FP16) else model\n        m = m.module if isinstance(m,nn.DataParallel) else m\n        for i in tqdm_notebook(range(sz)):\n            preds = m.get_d(x[i],y)\n            sorted, indices = torch.sort(preds)\n            d.append(to_np(sorted[:n]))\n            idxs.append(to_np(indices[:n]))\n    return np.stack(d), np.stack(idxs)\n\ndef get_val_nbs(model,emb_df,out='val.csv',dcut=None):\n    emb_df = emb_df.copy()\n    data = pd.read_csv(LABELS).set_index('Image')\n    emb_df['emb'] = [[float(i) for i in s.split()] for s in emb_df['emb']]\n    emb_df.set_index('files',inplace=True)\n    train_df = data.join(emb_df)\n    train_df = train_df.reset_index()\n    #the split should be the same as one used for training\n    trn_df, val_df = train_test_split(train_df,test_size=0.2, random_state=42)\n    trn_preds = np.array(trn_df.emb.tolist())\n    val_preds = np.array(val_df.emb.tolist())\n    trn_df = trn_df.reset_index()\n    val_df = val_df.reset_index()\n    \n    trn_preds = torch.from_numpy(trn_preds).half().cuda()\n    val_preds = torch.from_numpy(val_preds).half().cuda()\n    trn_d,trn_idxs = get_nbs(model,val_preds,trn_preds)\n    \n    s = []\n    for l1 in trn_d.tolist():\n        s.append(' '.join([str(l2) for l2 in l1]))\n    val_df['d'] = s\n    val_df['nbs'] = [' '.join(trn_df.loc[trn_idxs[index]].Id.tolist()) \\\n                     for index, row in val_df.iterrows()]\n    val_df[['Image','Id','nbs','d']].to_csv(out, header=True, index=False)\n    \n    if dcut is not None:\n        scores = []\n        for idx in val_df.index:\n            l0 = val_df.loc[idx].Id\n            nbs = dict()\n            for i in range(16): #16 neighbors\n                nb = trn_idxs[idx,i]\n                l, s = trn_df.loc[nb].Id, trn_d[idx,i]\n                if s > dcut and 'new_whale' not in nbs: nbs['new_whale'] = dcut\n                if l not in nbs: nbs[l] = s\n                if len(nbs) >= 5: break\n            nbs_sorted = list(nbs.items())\n            score = 0.0\n            for i in range(min(len(nbs),5)):\n                if nbs_sorted[i][0] == l0:\n                    score = 1.0/(i + 1.0)\n                    break\n            scores.append(score)\n        print(np.array(scores).mean(), flush=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"33222cdd4dd6eafabff6b77dcde975ce8dbae295"},"cell_type":"code","source":"dcut = 18.0 #fit this parameter based on validation\nget_val_nbs(learner.model,df,dcut=dcut,out='val0.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6ee2d1fa46d07a49cdc665e6ca15888896637a7e"},"cell_type":"markdown","source":"### **Submission**"},{"metadata":{"trusted":true,"_uuid":"803379b14b2b8b768a9cc338ce9ec44d1e1782ba"},"cell_type":"code","source":"def get_test_nbs(model,trn_emb,test_emb,out='test.csv',\\\n                 submission='submission.csv',dcut=None):\n    trn_emb = trn_emb.copy()\n    data = pd.read_csv(LABELS).set_index('Image')\n    trn_emb['emb'] = [[float(i) for i in s.split()] for s in trn_emb['emb']]\n    trn_emb.set_index('files',inplace=True)\n    train_df = data.join(trn_emb)\n    train_df = train_df.reset_index()\n    train_preds = np.array(train_df.emb.tolist())\n    test_emb = test_emb.copy()\n    test_emb['emb'] = [[float(i) for i in s.split()] for s in test_emb['emb']]\n    test_emb['Image'] = test_emb['files']\n    test_emb.set_index('files',inplace=True)\n    test_df = test_emb.reset_index()\n    test_preds = np.array(test_df.emb.tolist())\n    train_preds = torch.from_numpy(train_preds).half().cuda()\n    test_preds = torch.from_numpy(test_preds).half().cuda()\n    test_d,test_idxs = get_nbs(model,test_preds,train_preds)\n    \n    s = []\n    for l1 in test_d.tolist():\n        s.append(' '.join([str(l2) for l2 in l1]))\n    test_df['d'] = s\n    test_df['nbs'] = [' '.join(train_df.loc[test_idxs[index]].Id.tolist()) \\\n                      for index, row in test_df.iterrows()]\n    test_df[['Image','nbs','d']].to_csv(out, header=True, index=False)\n    \n    if dcut is not None:\n        pred = []\n        for idx, row in test_df.iterrows():\n            nbs = dict()\n            for i in range(0,16):\n                nb = test_idxs[idx,i]\n                l, s = train_df.loc[nb].Id, test_d[idx,i]\n                if s > dcut and 'new_whale' not in nbs: nbs['new_whale'] = dcut\n                if l not in nbs: nbs[l] = s\n                if len(nbs) >= 5: break\n            nbs_sorted = list(nbs.items())\n            p = ' '.join([lb[0] for lb in nbs_sorted])\n            pred.append({'Image':row.files,'Id':p})\n        pd.DataFrame(pred).to_csv(submission,index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cd860e2429c025967334410e6207efa061fd6f4f"},"cell_type":"code","source":"get_test_nbs(learner.model,df,df_test,dcut=dcut,out='test0.csv',\\\n    submission='submission0.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"630eb1f72dbf5c5dbdc8041e54db224854dcea38"},"cell_type":"markdown","source":"### **Hard negative example mining **"},{"metadata":{"trusted":true,"_uuid":"a5c6e1d587de295a1b1d3a94abab3ae73f079154"},"cell_type":"markdown","source":"The code below selects the most similar negative examples to images in the dataset. **They are really tough**, and as [Haider Alwasiti](https://www.kaggle.com/hwasiti) wrote in comments, I feel really sorry for the network to do such job. The following stage of training is performed on triplets with these hard negative example. Since distribution of these examples is different from one used at the previous stage of training, not only loss, but also metrics (T_acc, BH_acc) don't give a reliable estimation of the model performance, and only a way to check the model is validation based on the entire validation dataset with using the same metric as one in the competition."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"8feb7320985ae8b4c848678e010485af151e59d3"},"cell_type":"code","source":"md = get_data(sz,bs,'train_emb.csv',learner.model)\nlearner.set_data(md)\nlearner.unfreeze()\nlearner.half()\nx,y = next(iter(md.trn_dl))\n    \nto_lb = md.trn_ds.names.Id.to_dict()\nlbs = [[to_lb[idx] for idx in y_cur.tolist()] for y_cur in y]\ndisplay_imgs((md.trn_ds.denorm(x[:,0,:,:,:]),md.trn_ds.denorm(x[:,1,:,:,:]),\\\n              md.trn_ds.denorm(x[:,2,:,:,:])),lbs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0a628c95a907f4a60fa09b684f5f0c9749e17209"},"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter(\"ignore\")\n    learner.fit(lrs/16,5,cycle_len=4,use_clr=(10,20)) \nlearner.save('model')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e7a882ba1ad4a7aa2a946ddf19a63647a192a31f"},"cell_type":"code","source":"emb, names = extract_embedding(learner.model,TRAIN)\ndf = pd.DataFrame({'files':names,'emb':emb.tolist()})\ndf.emb = df.emb.map(lambda emb: ' '.join(list([str(i) for i in emb])))\ndf.to_csv('train_emb.csv', header=True, index=False)\n\nemb, names = extract_embedding(learner.model,TEST)\ndf_test = pd.DataFrame({'files':names,'emb':emb.tolist()})\ndf_test.emb = df_test.emb.map(lambda emb: ' '.join(list([str(i) for i in emb])))\ndf_test.to_csv('test_emb.csv', header=True, index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e0253ef4503500783442e288e38a142a0ae4bc70"},"cell_type":"code","source":"dcut = 23.0 #fit this parameter based on validation\nget_val_nbs(learner.model,df,dcut=dcut,out='val.csv')\nget_test_nbs(learner.model,df,df_test,dcut=dcut,out='test.csv',\\\n    submission='submission.csv')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}