{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Fastai starter\n\nThis is some basic starter code for using [fastai](https://docs.fast.ai/) for this dataset. The code/model is based on [this kernel](https://www.kaggle.com/xhlulu/densenet-transfer-learning-iwildcam-2019) and uses a pretrained DenseNet121, along with [Mixup](https://arxiv.org/abs/1710.09412) as implemented by the fastai library."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"%reload_ext autoreload\n%autoreload 2\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from fastai import *\nfrom fastai.vision import *\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Make sure cuda is installed:', torch.cuda.is_available())\nprint('Make sure cudnn is enabled:', torch.backends.cudnn.enabled)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = Path('../input')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load train dataframe\ntrain_df = pd.read_csv(path/'train.csv')\ntrain_df = pd.concat([train_df['id'],train_df['category_id']],axis=1,keys=['id','category_id'])\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load sample submission\ntest_df = pd.read_csv(path/'test.csv')\ntest_df = pd.DataFrame(test_df['id'])\ntest_df['predicted'] = 0\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train, test = [ImageList.from_df(df, path=path, cols='id', folder=folder, suffix='.jpg') \n               for df, folder in zip([train_df, test_df], ['train_images', 'test_images'])]\ndata = (train.split_by_rand_pct(0.2, seed=123)\n        .label_from_df(cols='category_id')\n        .add_test(test)\n        .transform(get_transforms(), size=32)\n        .databunch(path=Path('.'), bs=64).normalize())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.show_batch()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train model\n\nHere we will create a model using the [`cnn_learner`](https://docs.fast.ai/vision.learner.html#cnn_learner) function, which will automatically download and load the pretrained weights. Mixup is easily added as a callback, which is done by the `mixup()` function. If you are interested in the mixup implementation in fastai, you can read more over [here](https://docs.fast.ai/callbacks.mixup.html).\n\nWe will fine-tune the pretrained model, then unfreeze and train the whole model."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = cnn_learner(data, base_arch=models.densenet121, metrics=[FBeta(),accuracy], wd=1e-5).mixup()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The fastai library provides an implementation of a learning rate finder as described by [this paper](https://arxiv.org/abs/1506.01186). This allows us to choose the optimal learning rate for efficient training.\n\nIn a nutshell, the learning rate is adjusted over a single epoch, and the loss is plotted against the learning rate. The optimal learning rate is when the loss decreases the fastest."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will now fine-tune the final layer of our pretrained model."},{"metadata":{"trusted":true},"cell_type":"code","source":"lr = 2e-2\nlearn.fit_one_cycle(2, slice(lr))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.save('stage-1-sz32')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We now will unfreeze the model, to retrain the entire model. The optimal learning rate has to be determined again."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.unfreeze()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here, we use discriminative learning rates, where lower learning rates are used for the earlier layers in the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"lr = 1e-3\nlearn.fit_one_cycle(4, slice(lr/100, lr))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.save('stage-2-sz32')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Interpretation\n\nThe fastai library also provides some functions for interpreting the models, such as displaying images with the top losses, displaying confusion matrices, and [more](https://docs.fast.ai/vision.learner.html#ClassificationInterpretation)."},{"metadata":{"trusted":true},"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)\n\nlosses,idxs = interp.top_losses()\n\nlen(data.valid_ds)==len(losses)==len(idxs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"interp.plot_confusion_matrix(figsize=(12,12), dpi=60)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_preds = learn.TTA(DatasetType.Test)\ntest_df['predicted'] = test_preds[0].argmax(dim=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Future work:\n\nSome more tricks include:\n- label smoothing\n- Focal loss\n\nAlso, it would be helpful to implement cross-validation, and try some other pretrained models and do ensembling.\n\nIf you enjoyed this kernel, please give it an upvote! Thanks for reading!\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.4"}},"nbformat":4,"nbformat_minor":1}