{"cells":[{"metadata":{"_uuid":"45589718c0b39cac203379fc372a38df058c1173"},"cell_type":"markdown","source":"## **Intro**\nfastai is a deep-learning framework built on top of Pytorchv1. Recently it got update to [v1](https://github.com/fastai/fastai) which is a huge update from previous version and contains preety amazing and cool apis to load data, create models and train them. You can get the overview on how to use that from their [MOOC](https://course.fast.ai/) and can get started with deep learning.\n\nThis kernel uses **fastai.vison** for image classification task. We will use transfer learning using densenet121 model from [torchvision](https://pytorch.org/docs/stable/torchvision/index.html) along with [focal loss](https://arxiv.org/pdf/1708.02002.pdf). Following techniques we will be using are already implemented in fastai.\n \n1. Learning rate finder \n2. Cyclic learning rate ([paper](https://arxiv.org/pdf/1506.01186.pdf))\n3. Discriminative learning rates\n4. Data augmentation\n5. Transfer learning with pretrained model\n6. Test time augmentation"},{"metadata":{"_uuid":"455cd1c21f566b12bfb4e06c412c2293b5fa2394"},"cell_type":"markdown","source":"## Let's get started\nLoad the vision package which also loads other required packages <br>\nLoad required metric function <br>"},{"metadata":{"_uuid":"01810353f33e07c1e20daa170f1b7f1c753290c9","colab":{},"colab_type":"code","id":"R-MWUMZ5jo2k","trusted":true},"cell_type":"code","source":"from fastai.vision import *\nfrom sklearn.metrics import roc_auc_score\n\nimport warnings\nwarnings.simplefilter('ignore', category=FutureWarning)\nwarnings.simplefilter('ignore', category=UserWarning)\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"27408fa5a97dac95dd5b83e97c6e5c079477fb87"},"cell_type":"markdown","source":"Define the path using [pathlib](https://docs.python.org/3/library/pathlib.html)."},{"metadata":{"_uuid":"9bdec8e8adb654e4ef1b5d0f0ea62a298dad91d0","colab":{},"colab_type":"code","id":"cW7W6u8ahvLU","trusted":true},"cell_type":"code","source":"PATH = Path('../input')\nPATH.ls()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3dd7641edc87f156107f4f10bccd7e7a2d5bb015"},"cell_type":"markdown","source":"## **Data loading**"},{"metadata":{"_uuid":"6f5e96605739e782a162a502e47802e266fe2e65","colab":{"base_uri":"https://localhost:8080/","height":224},"colab_type":"code","id":"PaarqomYjxGy","outputId":"4df545ab-16dc-4fc5-e547-1ce145f0332f","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(PATH/'train_labels.csv')\nprint(train_df.shape)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a76ba9f51fa0a1843358f9517a8e9941f2dca8dd","trusted":true},"cell_type":"code","source":"train_df['label'].value_counts(normalize=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d1486d44102f7fde6d0692fd86cf4439a275bc65"},"cell_type":"markdown","source":"So the ratio is around 3:2"},{"metadata":{"_uuid":"86c548732c4be7d651ef1cf52bfad37d7d9c99a8","colab":{},"colab_type":"code","id":"UqOdRS8qkCib","trusted":true},"cell_type":"code","source":"src = (ImageItemList.from_csv(PATH, folder='train', csv_name='train_labels.csv', suffix='.tif')\n      .random_split_by_pct(0.1, seed=77)\n      .label_from_df()\n      .add_test_folder())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72b9b88cbfd1103d4bad473ee0b2cc31a512e38e"},"cell_type":"markdown","source":"**get_transforms()** defines all the required data augmentation we need such as vertical flip,  horizontal flip, zoom, rotation etc. You can read them [here](https://docs.fast.ai/vision.transform.html#get_transforms)."},{"metadata":{"_uuid":"aa7a66208a76a469f9d65d65a610855de805d8b4","colab":{},"colab_type":"code","id":"Gn7_S-d4pgky","trusted":true},"cell_type":"code","source":"tfms = get_transforms(do_flip=True, flip_vert=True, max_rotate=0, max_zoom=1., max_lighting=0.05, max_warp=0)\n\ndata = (src.transform(tfms, size=96, resize_method=ResizeMethod.SQUISH)\n       .databunch(bs=64, path='.'))\n\ndata.normalize(imagenet_stats);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dcbea1e6512131294e687647b4b0b735f340f320","colab":{"base_uri":"https://localhost:8080/","height":869},"colab_type":"code","id":"E5Aa8eNOz4fQ","outputId":"680ba25c-c8cd-4136-ffeb-4aa108976589","trusted":true},"cell_type":"code","source":"data.show_batch(rows=5, figsize=(15, 15))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e7d1c698507f7c84d6557c4ba953f70c440a954f"},"cell_type":"markdown","source":"## **Model and training**\nWe will define focal loss and roc metric"},{"metadata":{"_uuid":"780f878c229cc476c5f06e339bb0d190c0b38a5d","colab":{},"colab_type":"code","id":"BTgiSh595Uld","trusted":true},"cell_type":"code","source":"class FocalLoss(nn.Module):\n    def __init__(self, alpha=1., gamma=1.):\n        super().__init__()\n        self.alpha = alpha\n        self.gamma = gamma\n\n    def forward(self, inputs, targets, **kwargs):\n        CE_loss = nn.CrossEntropyLoss(reduction='none')(inputs, targets)\n        pt = torch.exp(-CE_loss)\n        F_loss = self.alpha * ((1-pt)**self.gamma) * CE_loss\n        return F_loss.mean()\n\n    \ndef roc_score(inp, target):\n    _, indices = inp.max(1)\n    return torch.Tensor([roc_auc_score(target, indices)])[0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"86dd695a710d1710f9cf16d8934088b9beaf83ea"},"cell_type":"markdown","source":"Let's create a learner which will contain our data, model and metrics. <br>\nAlso we need to define the loss function we want to use. By default fastai uses softmax or cross-entropy loss for classification task but we want to use focal loss."},{"metadata":{"_uuid":"18229fed04b32fbb0363b4a60662a3441b4c718f","colab":{"base_uri":"https://localhost:8080/","height":106},"colab_type":"code","id":"ZZDSksaCz6ly","outputId":"1e50d6de-1930-4bb9-896f-56c73aee3ca0","trusted":true},"cell_type":"code","source":"loss_func = FocalLoss(gamma=1.)\nlearn = create_cnn(data, models.densenet121, metrics=[accuracy, roc_score], loss_func=loss_func)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b5e9c3649e8b172fddeb891c2eb4bdc1fea92d79"},"cell_type":"markdown","source":"Find learning rate and plot it"},{"metadata":{"_uuid":"23efceb67e2f69ba31d5cc0c77c0644f9ffa648f","colab":{"base_uri":"https://localhost:8080/","height":108},"colab_type":"code","id":"z0SRXnGx0i6r","outputId":"6e6a95d3-1a2e-4a96-bc19-6135b9092b65","trusted":true},"cell_type":"code","source":"learn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"03df2c8e542d00c49861fff16cb42aa4233552fb","colab":{"base_uri":"https://localhost:8080/","height":363},"colab_type":"code","id":"PyXKbxrF0sKn","outputId":"45dd976f-8c85-46d2-bd66-9ea3909dbb56","trusted":true},"cell_type":"code","source":"learn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d6cdc3923c4e139f45fb56ec26e63577258ab6e4"},"cell_type":"markdown","source":"To choose the optimal learning rate find the point where the loss is minimum and then either find a point before that where the plot has a steepest slope or just divide that point by 10, both works fine."},{"metadata":{"_uuid":"5d4000fff767da5bc7b888857fcb2226f2073a81"},"cell_type":"markdown","source":"Fit the plot for 2 cycle"},{"metadata":{"_uuid":"49fb92132a22019857007bdc43a1402179283c57","colab":{"base_uri":"https://localhost:8080/","height":138},"colab_type":"code","id":"bl1ExHg10n5y","outputId":"cee13842-b76d-46ea-c1f6-2112f3c04af7","trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(2, 1e-3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2d4efb3e55500488443ea6470d91aacfade41cbc"},"cell_type":"markdown","source":"See how the learning rate and momentum varies with the training"},{"metadata":{"_uuid":"6b50f85cb82750c80d1ad6760232b9d57016887a","trusted":true},"cell_type":"code","source":"learn.recorder.plot_lr(show_moms=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e59155a2e7a246b382b147e13fd61182fa94acdb"},"cell_type":"markdown","source":"Let's see the losses."},{"metadata":{"_uuid":"0eade559cd0e99cf681043677d626491d0c25b04","trusted":true},"cell_type":"code","source":"learn.recorder.plot_losses()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ad5c6c0f8ee379892d30aa72e05d45aeb99b9d6f"},"cell_type":"markdown","source":"Here you will see the loss gets a little bump after the initial drop. This is due to the increase in learning rate in the first half cycle and it will drive the model out of the local minima. In the second cycle the learning rate will decrease gradually which will help obtain the global minima."},{"metadata":{"_uuid":"4222e3c463631bd03bbf748d598278ef2388d022"},"cell_type":"markdown","source":"Save and load your model"},{"metadata":{"_uuid":"553e0cba7b3966fb924da4625c590e34c49bc3b5","colab":{},"colab_type":"code","id":"RRAGqSmHq9l6","trusted":true},"cell_type":"code","source":"learn.save('stage-1')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cb0c77d4f4ae4216853703edb0e7fd5929cc5134","colab":{},"colab_type":"code","id":"GXHP38GNq_wE","trusted":true},"cell_type":"code","source":"learn.load('stage-1');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f013104735f16de43d14fe6ea3dd53c8dde02b9c","trusted":true},"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)\ninterp.plot_confusion_matrix(title='Confusion matrix')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b77c7f75201256c7a2f1f3797af063e1346d5891"},"cell_type":"markdown","source":"Unfreeze the other two layers and find an optimal learning rate again"},{"metadata":{"_uuid":"d6b8e24b21b4cf35b8ce3887f1854f4827a1dc39","colab":{},"colab_type":"code","id":"emMuew8jrT_M","trusted":true},"cell_type":"code","source":"learn.unfreeze()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eaa9f60ef322c61c790eb3bce9951ae156b4631c","colab":{"base_uri":"https://localhost:8080/","height":108},"colab_type":"code","id":"w2wumEQf5Cj1","outputId":"5fed9db6-f6ff-493a-ed3d-ea5425491204","trusted":true},"cell_type":"code","source":"learn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dd9465265bd4f9b951ddcef8f4827f803cd63207","colab":{"base_uri":"https://localhost:8080/","height":361},"colab_type":"code","id":"4a9YuYriJyn-","outputId":"baa1292c-a0e3-4ca1-c49e-429c4f45aeef","trusted":true},"cell_type":"code","source":"learn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4991b7a6d295b75cfef7433e1c36c321d3bace7a"},"cell_type":"markdown","source":"Now here we define the learning rates using slice(). This function will create different learning rates for different groups to use discriminative learning. In short the main idea is to fine-tune the layers with different learning rates in which the initial layers will be optimized with lower learning rate and final layers with higher learning rate. <br>\nTrain more..."},{"metadata":{"_uuid":"81c11e513212bed148c8dc0ad9c0116597b1825b","colab":{"base_uri":"https://localhost:8080/","height":215},"colab_type":"code","id":"tPGYmD_k5W2n","outputId":"40db4681-b824-4030-c884-d1496e072d2e","trusted":true},"cell_type":"code","source":"max_lr = 1e-4\nlearn.fit_one_cycle(4, slice(1e-6, max_lr))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6072dbeed7325e85d193a1e14f722c544fcd5e5e","colab":{},"colab_type":"code","id":"LhPtpe3u85TI","trusted":true},"cell_type":"code","source":"learn.save('stage-2')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9378d684b492fbbd71a6cdb2030288c7ea63ca9a","colab":{},"colab_type":"code","id":"dW4EtWILp_pR","trusted":true},"cell_type":"code","source":"learn.load('stage-2');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"32af45596f8a5c8de65b59ebe4075e5925c32079","trusted":true},"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)\ninterp.plot_confusion_matrix(title='Confusion matrix')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"858996a9407b84d128b66fd5cf997e9ce16ea09e","trusted":true},"cell_type":"code","source":"auc_val = learn.validate()[2].item()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ca46f7c21bccfadc482c04b3f94159d16468f76f"},"cell_type":"markdown","source":"## **Testing**\nTTA is basically test time augmentation. You can read more about it in the fastai [docs](https://docs.fast.ai/basic_train.html#TTA\n). <br>\nHere we need to apply sigmoid since we have to submit the probability. TTA will give logits not the probabilities because we have used FocalLoss here which is not defined in fastai yet but it's not needed for the submission."},{"metadata":{"_uuid":"ed11807ab17245219ae78b9060f5421cf6173a10","colab":{},"colab_type":"code","id":"R9AxJ5ultoC9","trusted":true},"cell_type":"code","source":"preds, y = learn.TTA(beta=0.4, ds_type=DatasetType.Test)\npreds = torch.softmax(preds, dim=1)[:, 1].numpy()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66c4c487a89f1e2a3e53009636dea9283ca0b3f0"},"cell_type":"markdown","source":"Take care of the sequence of the ids to be submitted in **sample_submission.csv**"},{"metadata":{"_uuid":"6a6914f678580113636ceb6f4d600f3b9b48557a","colab":{},"colab_type":"code","id":"xvYmrlli5WD-","trusted":true},"cell_type":"code","source":"test_ids = [f.stem for f in learn.data.test_ds.items]\nsubm = pd.read_csv(PATH/'sample_submission.csv')\norig_ids = list(subm['id'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9acfebd279f02b5b346f5dc20b46049b0c8b3238","colab":{},"colab_type":"code","id":"-7QNMHNAxStv","trusted":true},"cell_type":"code","source":"def create_submission(orig_ids, test_ids, preds):\n    preds_dict = dict((k, v) for k, v in zip(test_ids, preds))\n    pred_cor = [preds_dict[id] for id in orig_ids]\n    df = pd.DataFrame({'id':orig_ids,'label':pred_cor})\n    df.to_csv(f'submission_{auc_val}.csv', header=True, index=False)\n    \n    return df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8da15a2ce888db06c99bb437d9973adf553d7a69","colab":{},"colab_type":"code","id":"pNQRTAmKvpvF","trusted":true},"cell_type":"code","source":"test_df = create_submission(orig_ids, test_ids, preds)\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"31c3496946aabb72bbb00686817d715f432eb133"},"cell_type":"markdown","source":"## **Useful links**\n* [How fastai data block api works](https://blog.usejournal.com/finding-data-block-nirvana-a-journey-through-the-fastai-data-block-api-c38210537fe4)\n* [One cycle scheduler](https://sgugger.github.io/the-1cycle-policy.html)\n* [Motivation](https://www.fast.ai/2019/01/02/one-year-of-deep-learning)\n* [What can you do with deep learning?](https://www.fast.ai/2019/02/21/dl-projects/)\n* [Humpback Whale Identification Competition Starter Pack](https://github.com/radekosmulski/whale)\n\n"},{"metadata":{"trusted":true,"_uuid":"85d1140d1c4e4ad1d8874f041318c81bf5a44910"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"accelerator":"GPU","colab":{"collapsed_sections":[],"name":"densenet-focal.ipynb","provenance":[],"version":"0.3.2"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}