{"cells":[{"metadata":{"_uuid":"bf443bd68f2d62ae9cb8e7f45815df2704c6c400"},"cell_type":"markdown","source":"## Overview\n\nA few days ago I made a naive classification baseline which treated every class equally. The result was not that terrible compared to the other kernels of the same idea, however if you look at the leaderboard right now you can clearly see there is something I missed.\n\nSo beside that, there are two other rather obvious approaches:\n\n- One/few-shot detection, this was nicely demonstrated by **@martinpiotte** in the playground competition, so we can be sure that it works: https://www.kaggle.com/martinpiotte/whale-recognition-model-with-score-0-78563. However trying to fit this into a kernel seems to be hard work. (I'll try again in the weekend?)\n\n- The other one is more simple, we remove class 0 (**new_whale**) from the training set and train a classifier on the rest. At inference, we try to find a threshold value to decide whether and where to insert **new_whale** to the final predictions.\n\nFor example, suppose that the top 5 predictions are [class1, class2, class3, class4, class5] which have confidence values of [0.8, 0.05, 0.02, 0.02, 0.01] and the threshold found with the validation set was 0.3, then the predictions become: [class1, new_whale, class2, class3, class4]"},{"metadata":{"_uuid":"88e20188ddffce29fc4c3af7b6fc09bb48cc1fee"},"cell_type":"markdown","source":"## Training "},{"metadata":{"_uuid":"132b11ca41afe41627ed3c0df8b2be39d30f93d2"},"cell_type":"markdown","source":"Let's start by importing our libararies."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"from fastai.conv_learner import *\nfrom fastai.dataset import *\nfrom tqdm import tqdm\nimport pandas as pd\nimport numpy as np\nimport os\nfrom sklearn.model_selection import train_test_split, StratifiedShuffleSplit\nimport matplotlib.pyplot as plt\nimport math","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"MODEL_PATH = 'Resnet18_v1'\nTRAIN = '../input/train/'\nTEST = '../input/test/'\nLABELS = '../input/train.csv'\nSAMPLE_SUB = '../input/sample_submission.csv'\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e0624ab350e370dbff80cac45f33744c48e5633b"},"cell_type":"markdown","source":"The architecture is flexible, I chose Resnet18 since it can fit quite well into a kernel. You may play with this if you want to. "},{"metadata":{"trusted":true,"_uuid":"6ea9033e0200d3d9142b4ee05c45c1dd4f2d8c1d"},"cell_type":"code","source":"arch = resnet34\nnw = 4","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bf2a5c5342e855974efaeb7fe5c2b90f2cf636cf"},"cell_type":"markdown","source":"Next, we prapare out dataset to work with Fastai's pipeline."},{"metadata":{"trusted":true,"_uuid":"d9adfc15b56c7f80f291c66dc6d6f38d4d55e6a2"},"cell_type":"code","source":"df = pd.read_csv(LABELS).set_index('Image')\nnew_whale_df = df[df.Id == \"new_whale\"] # only new_whale dataset\ntrain_df = df[~(df.Id == \"new_whale\")] # no new_whale dataset, used for training\nunique_labels = np.unique(train_df.Id.values)\n\nlabels_dict = dict()\nlabels_list = []\nfor i in range(len(unique_labels)):\n    labels_dict[unique_labels[i]] = i\n    labels_list.append(unique_labels[i])\nprint(\"Number of classes: {}\".format(len(unique_labels)))\ntrain_names = train_df.index.values\ntrain_df.Id = train_df.Id.apply(lambda x: labels_dict[x])\ntrain_labels = np.asarray(train_df.Id.values)\ntest_names = [f for f in os.listdir(TEST)]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a910097d19053c50d60ea7ee9496ed2a55746e2"},"cell_type":"markdown","source":"Let's draw a simple histogram to see the sample-per-class distribution."},{"metadata":{"trusted":true,"_uuid":"ddef1744553be7723709a1e14253612a18c6f7e2"},"cell_type":"code","source":"labels_count = train_df.Id.value_counts()\n_, _,_ = plt.hist(labels_count,bins=100)\nlabels_count","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5c9a8d0b818a13929e3b18771165956b3f751d7d"},"cell_type":"code","source":"dup = []\nfor idx,row in train_df.iterrows():\n    if labels_count[row['Id']] < 5:\n        dup.extend([idx]*math.ceil((5 - labels_count[row['Id']])/labels_count[row['Id']]))\ntrain_names = np.concatenate([train_names, dup])\ntrain_names = train_names[np.random.RandomState(seed=42).permutation(train_names.shape[0])]\nlen(train_names)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0fc648f57bcc32e48bb043da3854eb46f2d91540"},"cell_type":"code","source":"sss = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42069)\nfor train_idx, val_idx in sss.split(train_names, np.zeros(train_names.shape)):\n    tr_n, val_n = train_names[train_idx], train_names[val_idx]\nprint(len(tr_n), len(val_n))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"037cdcbea35cf6a0785e5707202b26b5f707b716"},"cell_type":"markdown","source":"The image sizes seem to vary, so we'll try to see what the average width and height are:"},{"metadata":{"trusted":true,"_uuid":"7062c686741f15788a24f6f1aecc0b5d9ce57e8e"},"cell_type":"code","source":"avg_width = 0\navg_height = 0\nfor fn in os.listdir(TRAIN)[:1000]:\n    img = cv2.imread(os.path.join(TRAIN,fn))\n    avg_width += img.shape[1]\n    avg_height += img.shape[0]\navg_width //= 1000\navg_height //= 1000\nprint(avg_width, avg_height)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a855425946822696c5aa4d37401f3b5c1d0a88c5"},"cell_type":"markdown","source":"They turn out to be quite big, especially the width, so below you'll see I resize everything back to **384**. You may consider continue training on bigger size, but that probably won't fit in a kernel. "},{"metadata":{"trusted":true,"_uuid":"94af91d70819db979d39a4d77b2e30493498978b"},"cell_type":"code","source":"class HWIDataset(FilesDataset):\n    def __init__(self, fnames, path, transform):\n        self.train_df = train_df\n        super().__init__(fnames, transform, path)\n\n    def get_x(self, i):\n        img = open_image(os.path.join(self.path, self.fnames[i]))\n        # We crop the center of the original image for faster training time\n        img = cv2.resize(img, (self.sz, self.sz))\n        return img\n\n    def get_y(self, i):\n        if (self.path == TEST): return 0\n        return self.train_df.loc[self.fnames[i]]['Id']\n\n\n    def get_c(self):\n        return len(unique_labels)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"267ee406354b98daa4682d7fcb6f08106bd7bee6"},"cell_type":"markdown","source":""},{"metadata":{"trusted":true,"_uuid":"140dfae2b41cbe4f770f8d80fcaba0ebc772e983"},"cell_type":"code","source":"class RandomLighting(Transform):\n    def __init__(self, b, c, tfm_y=TfmType.NO):\n        super().__init__(tfm_y)\n        self.b, self.c = b, c\n\n    def set_state(self):\n        self.store.b_rand = rand0(self.b)\n        self.store.c_rand = rand0(self.c)\n\n    def do_transform(self, x, is_y):\n        if is_y and self.tfm_y != TfmType.PIXEL: return x  # add this line to fix the bug\n        b = self.store.b_rand\n        c = self.store.c_rand\n        c = -1 / (c - 1) if c < 0 else c + 1\n        x = lighting(x, b, c)\n        return x\n    \ndef get_data(sz, bs, test_names=test_names, test_dir=TEST):\n    aug_tfms = [RandomRotateZoom(deg=20, zoom=2, stretch=1),\n                RandomLighting(0.05, 0.05, tfm_y=TfmType.NO),\n                RandomBlur(blur_strengths=3,tfm_y=TfmType.NO),\n                RandomFlip(tfm_y=TfmType.NO)]\n    tfms = tfms_from_model(arch, sz, crop_type=CropType.NO, tfm_y=TfmType.NO,\n                           aug_tfms=aug_tfms)\n    ds = ImageData.get_ds(HWIDataset, (tr_n[:-(len(tr_n) % bs)], TRAIN),\n                          (val_n, TRAIN), tfms, test=(test_names, test_dir))\n    md = ImageData(\"./\", ds, bs, num_workers=nw, classes=None)\n    return md\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f8258255beb8fb608abb8a292b07c7161580007e"},"cell_type":"code","source":"# sz = (avg_width//2, avg_height//2)\nbatch_size = 64\nmd = get_data(384, batch_size)\nlearn = ConvLearner.pretrained(arch, md) \nlearn.opt_fn = optim.Adam","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c2a3accea73d13e6b8f81febd988b9e377fb5572"},"cell_type":"markdown","source":"Uncomment these lines to run Fastai's automatic learning rate finder. \n"},{"metadata":{"trusted":true,"_uuid":"04b0332bd91ee3752b8da857d34e566c96a638d4"},"cell_type":"code","source":"# learn.lr_find()\n# learn.sched.plot()\nlr = 5e-3","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"115459f2b3756d4f029cb223a57a80abba5f2992"},"cell_type":"markdown","source":"We start by training only the newly initialized weights, then unfreeze the model and finetune the pretrained weights with reduced learning rate."},{"metadata":{"trusted":true,"_uuid":"f3118d5e2dbe61c8d51d0e33642ea5bb0b516a54"},"cell_type":"code","source":"learn.fit(lr, 2, cycle_len=3)\nlearn.unfreeze()\nlrs = np.array([lr/10, lr/20, lr/40])\nlearn.fit(lrs, 4, cycle_len=4, use_clr=(20, 16))\nlearn.save(MODEL_PATH)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5481963802b300f4e76f757514d5395948373c62"},"cell_type":"markdown","source":"## Choosing a threshold\n\nThe validation set now contains two subset.\nFirst, we evaluate the model on the new whale-free validation set"},{"metadata":{"trusted":true,"_uuid":"fcb801182b1440e0c29b4626510a49d93ac91d6e"},"cell_type":"code","source":"preds_v,y_v = learn.TTA(is_test=False,n_aug=2)\npreds_v = np.stack(preds_v, axis=-1)\npreds_v = np.exp(preds_v)\npreds_v = preds_v.mean(axis=-1)\ny_v += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5658d46f25e80fc6517b422f116fbe373165f562"},"cell_type":"code","source":"preds_v_max = np.max(preds_v,axis=1)\n_,_,_ = plt.hist(preds_v_max)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6ee722a0d42c4bfb89eb7fec665d18701d7f3ff1"},"cell_type":"markdown","source":"And then on the all-whale set that we left out at the beginning"},{"metadata":{"trusted":true,"_uuid":"d66865dfd58fe3eac08450d646040c2550900675"},"cell_type":"code","source":"TEST=TRAIN # sorry\ntotal_new_whale = len(new_whale_df.index.values)\nmd = get_data(384, batch_size, test_names=new_whale_df.index.values[:int(total_new_whale*0.2)], test_dir=TRAIN)\nlearn.set_data(md)\npreds_w,y_w = learn.TTA(is_test=True,n_aug=2)\npreds_w = np.stack(preds_w, axis=-1)\npreds_w = np.exp(preds_w)\npreds_w = preds_w.mean(axis=-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"def65eed5d907e4d89a7c4ec4e036d329117b657"},"cell_type":"code","source":"preds_w_max = np.max(preds_w,axis=1)\n_,_,_ = plt.hist(preds_w_max)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5e4e482937eeb06cbf2991206e64d822451e450f"},"cell_type":"code","source":"y = np.concatenate([y_v,y_w])\npreds = np.concatenate([preds_v, preds_w],axis=0)\npreds = np.concatenate([np.zeros((preds.shape[0],1)), preds],axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8c31a73683fb5f9e47b12c9a40cd9c86cc538d49"},"cell_type":"markdown","source":"Trying to choose the right threshold to maximize MAP@5 score"},{"metadata":{"trusted":true,"_uuid":"25c3980c7c7004668b25bed2910d910f6f10ba1a"},"cell_type":"code","source":"def map5(X, y):\n    score = 0\n    for i in range(X.shape[0]):\n        pred = X[i].argsort()[-5:][::-1]\n        for j in range(pred.shape[0]):\n            if pred[j] == y[i]:\n                score += (5 - j)/5\n                break\n    return score/X.shape[0]\n\nbest_th = 0\nbest_score = 0\nfor th in np.arange(0.1, 0.801, 0.01):\n    preds[:,0] = th\n    score = map5(preds, y)\n    if score > best_score:\n        best_score = score\n        best_th = th\n    print(\"Threshold = {:.3f}, MAP5 = {:.3f}\".format(th,score))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"601948472f64ef81160ab6006daac3033772c1b4"},"cell_type":"markdown","source":"## Prediction"},{"metadata":{"trusted":true,"_uuid":"6cbfaedbad6bac01b06d87eaf3723dd260b7a51e"},"cell_type":"code","source":"TEST = '../input/test/'\nmd = get_data(384, batch_size, test_names=test_names, test_dir=TEST)\nlearn.set_data(md)\npreds_t,y_t = learn.TTA(is_test=True,n_aug=8)\npreds_t = np.stack(preds_t, axis=-1)\npreds_t = np.exp(preds_t)\npreds_t = preds_t.mean(axis=-1)\npreds_t = np.concatenate([np.zeros((preds_t.shape[0],1))+best_th, preds_t],axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"82b4e79d2a05267790200de4860fd25a0a669f7f"},"cell_type":"markdown","source":"Finally, our submission."},{"metadata":{"trusted":true,"_uuid":"f5fbd91e970d375debc3270ebd5b08bb41eeb66e"},"cell_type":"code","source":"sample_df = pd.read_csv(SAMPLE_SUB)\nsample_list = list(sample_df.Image)\nlabels_list = [\"new_whale\"]+labels_list\npred_list = [[labels_list[i] for i in p.argsort()[-5:][::-1]] for p in preds_t]\npred_dic = dict((key, value) for (key, value) in zip(learn.data.test_ds.fnames,pred_list))\npred_list_cor = [' '.join(pred_dic[id]) for id in sample_list]\ndf = pd.DataFrame({'Image':sample_list,'Id': pred_list_cor})\ndf.to_csv('submission.csv'.format(MODEL_PATH), header=True, index=False)\ndf.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}