{"cells":[{"metadata":{"_uuid":"5fe77716d00b69f882dbca1cf605d10070fb73de"},"cell_type":"markdown","source":"**Credits for the notebook goes to Khoi Nguyen's : https://www.kaggle.com/suicaokhoailang/wip-resnet18-baseline-with-fastai-0-375-lb .\nI just wanted to play with resnet34 and see if we can fit a bigger architecture in kaggle kernels.**  <br>\n**If the notebooks gets published then we certainly can.**"},{"metadata":{"_uuid":"88e20188ddffce29fc4c3af7b6fc09bb48cc1fee"},"cell_type":"markdown","source":"## Training "},{"metadata":{"_uuid":"132b11ca41afe41627ed3c0df8b2be39d30f93d2"},"cell_type":"markdown","source":"Let's start by importing our libararies."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"from fastai.conv_learner import *\nfrom fastai.dataset import *\nfrom tqdm import tqdm\nimport pandas as pd\nimport numpy as np\nimport os\nfrom sklearn.model_selection import train_test_split, StratifiedShuffleSplit\nimport matplotlib.pyplot as plt\nimport math","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"MODEL_PATH = 'Resnet34_v1'\nTRAIN = '../input/train/'\nTEST = '../input/test/'\nLABELS = '../input/train.csv'\nSAMPLE_SUB = '../input/sample_submission.csv'\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e0624ab350e370dbff80cac45f33744c48e5633b"},"cell_type":"markdown","source":"The architecture is flexible, I chose Resnet18 since it can fit quite well into a kernel. You may play with this if you want to. "},{"metadata":{"trusted":true,"_uuid":"6ea9033e0200d3d9142b4ee05c45c1dd4f2d8c1d"},"cell_type":"code","source":"arch = resnet18\nnw = 6","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bf2a5c5342e855974efaeb7fe5c2b90f2cf636cf"},"cell_type":"markdown","source":"Next, we prapare out dataset to work with Fastai's pipeline."},{"metadata":{"trusted":true,"_uuid":"d9adfc15b56c7f80f291c66dc6d6f38d4d55e6a2"},"cell_type":"code","source":"train_df = pd.read_csv(LABELS).set_index('Image')\nunique_labels = np.unique(train_df.Id.values)\n\nlabels_dict = dict()\nlabels_list = []\nfor i in range(len(unique_labels)):\n    labels_dict[unique_labels[i]] = i\n    labels_list.append(unique_labels[i])\nprint(\"Number of classes: {}\".format(len(unique_labels)))\ntrain_names = train_df.index.values\ntrain_df.Id = train_df.Id.apply(lambda x: labels_dict[x])\ntrain_labels = np.asarray(train_df.Id.values)\ntest_names = [f for f in os.listdir(TEST)]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a910097d19053c50d60ea7ee9496ed2a55746e2"},"cell_type":"markdown","source":"Let's draw a simple histogram to see the sample-per-class distribution."},{"metadata":{"trusted":true,"_uuid":"ddef1744553be7723709a1e14253612a18c6f7e2"},"cell_type":"code","source":"labels_count = train_df.Id.value_counts()\n_, _,_ = plt.hist(labels_count,bins=100)\nlabels_count","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0a3a2f26fd6613685c994bf0a4514a276aa5f047"},"cell_type":"markdown","source":"Ugh, okay, let's kick the elephant out of the room and try again"},{"metadata":{"trusted":true,"_uuid":"fcd012819740e8f7c2ebcc0274c8547f90434629"},"cell_type":"code","source":"print(\"Count for class new_whale: {}\".format(labels_count[0]))\n\nplt.figure(figsize=(20,8))\n\nplt.hist(labels_count[1:],bins=100,range=[0,100])\nplt.hist(labels_count[1:],bins=100,range=[0,100])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"53cd78eae54b541399f324a8531ffd1fbca90a7d"},"cell_type":"markdown","source":"So most of the classes have only one or two sample(s), making **train_test_split** directly on the data impossible. We'll try a simple fix by duplicating the minor classes so that each class have a minimum of 5 samples."},{"metadata":{"trusted":true,"_uuid":"5c9a8d0b818a13929e3b18771165956b3f751d7d"},"cell_type":"code","source":"dup = []\nfor idx,row in train_df.iterrows():\n    if labels_count[row['Id']] < 5:\n        dup.extend([idx]*math.ceil((5 - labels_count[row['Id']])/labels_count[row['Id']]))\ntrain_names = np.concatenate([train_names, dup])\ntrain_names = train_names[np.random.RandomState(seed=42).permutation(train_names.shape[0])]\nlen(train_names)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0fc648f57bcc32e48bb043da3854eb46f2d91540"},"cell_type":"code","source":"sss = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42069)\nfor train_idx, val_idx in sss.split(train_names, np.zeros(train_names.shape)):\n    tr_n, val_n = train_names[train_idx], train_names[val_idx]\nprint(len(tr_n), len(val_n))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"037cdcbea35cf6a0785e5707202b26b5f707b716"},"cell_type":"markdown","source":"The image sizes seem to vary, so we'll try to see what the average width and height are:"},{"metadata":{"trusted":true,"_uuid":"7062c686741f15788a24f6f1aecc0b5d9ce57e8e"},"cell_type":"code","source":"avg_width = 0\navg_height = 0\nfor fn in os.listdir(TRAIN)[:1000]:\n    img = cv2.imread(os.path.join(TRAIN,fn))\n    avg_width += img.shape[1]\n    avg_height += img.shape[0]\navg_width //= 1000\navg_height //= 1000\nprint(avg_width, avg_height)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a855425946822696c5aa4d37401f3b5c1d0a88c5"},"cell_type":"markdown","source":"They turn out to be quite big, especially the width, so below you'll see I resize everything back to **average_width/4**. You may consider continue training on bigger size, but that probably won't fit in a kernel. "},{"metadata":{"trusted":true,"_uuid":"94af91d70819db979d39a4d77b2e30493498978b"},"cell_type":"code","source":"class HWIDataset(FilesDataset):\n    def __init__(self, fnames, path, transform):\n        self.train_df = train_df\n        super().__init__(fnames, transform, path)\n\n    def get_x(self, i):\n        img = open_image(os.path.join(self.path, self.fnames[i]))\n        # We crop the center of the original image for faster training time\n        img = cv2.resize(img, (self.sz, self.sz))\n        return img\n\n    def get_y(self, i):\n        if (self.path == TEST): return 0\n        return self.train_df.loc[self.fnames[i]]['Id']\n\n\n    def get_c(self):\n        return len(unique_labels)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"140dfae2b41cbe4f770f8d80fcaba0ebc772e983"},"cell_type":"code","source":"class RandomLighting(Transform):\n    def __init__(self, b, c, tfm_y=TfmType.NO):\n        super().__init__(tfm_y)\n        self.b, self.c = b, c\n\n    def set_state(self):\n        self.store.b_rand = rand0(self.b)\n        self.store.c_rand = rand0(self.c)\n\n    def do_transform(self, x, is_y):\n        if is_y and self.tfm_y != TfmType.PIXEL: return x  # add this line to fix the bug\n        b = self.store.b_rand\n        c = self.store.c_rand\n        c = -1 / (c - 1) if c < 0 else c + 1\n        x = lighting(x, b, c)\n        return x\n    \ndef get_data(sz, bs):\n    aug_tfms = [RandomRotateZoom(deg=20, zoom=2, stretch=1),\n                RandomLighting(0.05, 0.05, tfm_y=TfmType.NO),\n                RandomBlur(blur_strengths=3,tfm_y=TfmType.NO),\n                RandomFlip(tfm_y=TfmType.NO)]\n    tfms = tfms_from_model(arch, sz, crop_type=CropType.NO, tfm_y=TfmType.NO,\n                           aug_tfms=aug_tfms)\n    ds = ImageData.get_ds(HWIDataset, (tr_n[:-(len(tr_n) % bs)], TRAIN),\n                          (val_n, TRAIN), tfms, test=(test_names, TEST))\n    md = ImageData(\"./\", ds, bs, num_workers=nw, classes=None)\n    return md\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f8258255beb8fb608abb8a292b07c7161580007e"},"cell_type":"code","source":"sz = (avg_width//2, avg_height//2)\nbatch_size = 128\nmd = get_data(avg_width//4, batch_size)\nlearn = ConvLearner.pretrained(arch, md) \nlearn.opt_fn = optim.Adam","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c2a3accea73d13e6b8f81febd988b9e377fb5572"},"cell_type":"markdown","source":"Uncomment these lines to run Fastai's automatic learning rate finder. \n"},{"metadata":{"trusted":true,"_uuid":"04b0332bd91ee3752b8da857d34e566c96a638d4"},"cell_type":"code","source":"# learn.lr_find()\n# learn.sched.plot()\nlr = 1e-3","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"115459f2b3756d4f029cb223a57a80abba5f2992"},"cell_type":"markdown","source":"We start by training only the newly initialized weights, then unfreeze the model and finetune the pretrained weights with reduced learning rate."},{"metadata":{"trusted":true,"_uuid":"f3118d5e2dbe61c8d51d0e33642ea5bb0b516a54"},"cell_type":"code","source":"learn.fit(lr, 1, cycle_len=1)\nlearn.unfreeze()\nlrs = np.array([lr/10, lr/20, lr/40])\nlearn.fit(lrs, 4, cycle_len=4, use_clr=(20, 16))\nlearn.fit(lrs/4, 2, cycle_len=4, use_clr=(10, 16))\nlearn.fit(lrs/16, 1, cycle_len=4, use_clr=(10, 16))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b3867b2a2b604c2dfb91b94b49a96980d5883ec"},"cell_type":"markdown","source":"May be keep training on bigger image for potential performance boost."},{"metadata":{"trusted":true,"_uuid":"fcb801182b1440e0c29b4626510a49d93ac91d6e"},"cell_type":"code","source":"# batch_size = 32\n# md = get_data(avg_width//2, batch_size)\n# learn.set_data(md)\n# learn.fit(lrs/4, 3, cycle_len=2, use_clr=(10, 8))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d66865dfd58fe3eac08450d646040c2550900675"},"cell_type":"code","source":"# batch_size = 16\n# md = get_data(avg_width, batch_size)\n# learn.set_data(md)\n# learn.fit(lrs/16, 1, cycle_len=4, use_clr=(10, 8))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eabe555ace20e3b36c6266432108d31de1e58282"},"cell_type":"markdown","source":"## Predictions"},{"metadata":{"trusted":true,"_uuid":"6cbfaedbad6bac01b06d87eaf3723dd260b7a51e"},"cell_type":"code","source":"# preds_t,y_t = learn.predict_with_targs(is_test=True) # Predicting without TTA\npreds_t,y_t = learn.TTA(is_test=True,n_aug=8)\npreds_t = np.stack(preds_t, axis=-1)\npreds_t = np.exp(preds_t)\npreds_t = preds_t.mean(axis=-1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"82b4e79d2a05267790200de4860fd25a0a669f7f"},"cell_type":"markdown","source":"Finally, our submission."},{"metadata":{"trusted":true,"_uuid":"f5fbd91e970d375debc3270ebd5b08bb41eeb66e"},"cell_type":"code","source":"sample_df = pd.read_csv(SAMPLE_SUB)\nsample_list = list(sample_df.Image)\npred_list = [[labels_list[i] for i in p.argsort()[-5:][::-1]] for p in preds_t]\npred_dic = dict((key, value) for (key, value) in zip(learn.data.test_ds.fnames,pred_list))\npred_list_cor = [' '.join(pred_dic[id]) for id in sample_list]\ndf = pd.DataFrame({'Image':sample_list,'Id': pred_list_cor})\ndf.to_csv('submission.csv'.format(MODEL_PATH), header=True, index=False)\ndf.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}