{"cells":[{"metadata":{},"cell_type":"markdown","source":"# CutMix with FastAI"},{"metadata":{},"cell_type":"markdown","source":"In this notebook I  want to share how to implimente CutMix usnig fastai callback system. I used [this](https://www.kaggle.com/muellerzr/fastai-efficientnet-b3-with-ranger-83-5) notebook by Zach Mueller as a reference, which is a great fastai starter."},{"metadata":{},"cell_type":"markdown","source":"After writing this I found that this has already been developed in fastai. It has been discussed and developed in the [forums ](https://forums.fast.ai/t/implementing-cutmix-in-fastaiv2/67350) by Aman Arora, Sanyam Bhutani, Akash Palrecha, Rekil Prashanth, Asimo, and PRed in the [repo](https://github.com/fastai/fastai/pull/3037) by Zach Mueller."},{"metadata":{},"cell_type":"markdown","source":"## Installs and imports"},{"metadata":{},"cell_type":"markdown","source":"We install `timm` to have access to EfficientNet with pretrained weights. \nTo install a new library without internet access I download the wheels file form [pypi](https://pypi.org/project/timm/#files) and I add it as a dataset clicking this button on the  top right corner:\n![](https://i.ibb.co/X73N5PZ/Screenshot-from-2020-11-25-17-02-25.png)\n\nThen I install it with this command:\n```python\n!pip install ../input/pytorchimagemodels/\n```\nBut for this notebook we can turn internet access on and download it with pip"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install timm","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"from fastai.vision.all import *\nfrom fastai.callback.mixup import *\nfrom torch.distributions.beta import Beta\nimport timm\nset_seed(314)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## CutMix"},{"metadata":{},"cell_type":"markdown","source":"We define a `CutMix` callback for regularization ([paper](https://arxiv.org/pdf/1905.04899.pdf))."},{"metadata":{"trusted":true},"cell_type":"code","source":"class CutMix(MixUp):\n    def __init__(self, alpha=1.): self.distrib = Beta(tensor(alpha), tensor(alpha))\n    def before_batch(self):\n        lam = self.distrib.sample().squeeze().to(self.x.device)\n        shuffle = torch.randperm(self.y.size(0)).to(self.x.device)\n        self.yb1 = tuple(L(self.yb).itemgot(shuffle))\n        nx_dims = len(self.x.size())\n        bs, c, h, w = self.x.shape\n        rx, ry = w*self.distrib.sample(), h*self.distrib.sample()\n        rw, rh = w*(1-lam).sqrt(), h*(1-lam).sqrt()\n        x1 = (rx-rw/2).clamp(min=0).round().to(int)\n        x2 = (rx+rw/2).clamp(max=w).round().to(int)\n        y1 = (ry-rh/2).clamp(min=0).round().to(int)\n        y2 = (ry+rh/2).clamp(max=h).round().to(int)\n        self.learn.xb[0][:,:,y1:y2,x1:x2] = self.learn.xb[0][shuffle,:,y1:y2,x1:x2]\n        self.lam = 1- float(x2-x1)*(y2-y1)/(h*w)\n        \n        if not self.stack_y:\n            ny_dims = len(self.y.size())\n            self.learn.yb = tuple(L(self.yb1,self.yb).map_zip(torch.lerp,weight=unsqueeze(self.lam, n=ny_dims-1)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This callback is basically a copy of the `MixUp` callback that's already in `fastai` ([here](https://github.com/fastai/fastai/blob/master/nbs/19_callback.mixup.ipynb) is the code)."},{"metadata":{},"cell_type":"markdown","source":"We load the csv, map the labels and add a `valid` column indicating if a row belongs to the training or to validation dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv('../input/cassava-leaf-disease-classification/train.csv')\ndf['image_id'] = df['image_id'].apply(lambda x: f'train_images/{x}')\ndf.head()\nidx2label = json.load(open('../input/cassava-leaf-disease-classification/label_num_to_disease_map.json'))\ndf.label = df.label.map(str).map(idx2label)\nidxs = L.range(len(df)).shuffle()\ndf['valid'] = False\ndf.loc[idxs[:4279], 'valid'] = True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With the dataset defined as above, we can use `ImageDataLoaders.from_df` to load our data. We use `size=300` because efficientnet_b3 was trained with images of that resolution ([here](https://github.com/rwightman/pytorch-image-models/blob/9c406532bde4ffe281d356de6e597717d2e53205/timm/models/efficientnet.py#L225)).\nWe use `mult=2` and `max_zoom=2` to increase augmentation."},{"metadata":{"trusted":true},"cell_type":"code","source":"dls = ImageDataLoaders.from_df(df, path='../input/cassava-leaf-disease-classification/', bs=64,\n                               item_tfms=Resize(320),\n                               valid_col='valid',\n                               batch_tfms=aug_transforms(size=300,  mult=2, max_zoom=2.))\ndls.show_batch()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see `CutMix` in action."},{"metadata":{"trusted":true},"cell_type":"code","source":"mixup = CutMix()\nwith Learner(dls, nn.Linear(3,4), loss_func=CrossEntropyLossFlat(), cbs=mixup) as learn:\n    learn.epoch,learn.training = 0,True\n    learn.dl = dls.train\n    b = dls.one_batch()\n    learn._split(b)\n    learn('before_batch')\n\n_,axs = plt.subplots(3,3, figsize=(9,9))\ndls.show_batch(b=(mixup.xb[0],mixup.y), ctxs=axs.flatten())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## EfficientNet"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = timm.create_model('tf_efficientnet_b3_ns', pretrained=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_state_dict(torch.load('../input/timm-pretrained-efficientnet/efficientnet/tf_efficientnet_b3_ns-9d44bf68.pth'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.classifier = nn.Linear(model.classifier.in_features, len(dls.vocab))\nlearn = Learner(dls, model, loss_func=LabelSmoothingCrossEntropy(), splitter=methodcaller('parameters'), metrics=accuracy, model_dir='/kaggle/working/models')\nlearn.to_fp16()\nlearn.freeze()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fine_tune(16, base_lr=8.3e-4, cbs=[ShowGraphCallback(), CutMix()])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predict"},{"metadata":{},"cell_type":"markdown","source":"We can calculate the accuracy with tta"},{"metadata":{"trusted":true},"cell_type":"code","source":"dl = dls.valid\na1, target = learn.tta(dl=dl, n=16)\npred_1 = a1.argmax(dim=1)\n(pred_1==target).to(float).mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Submit the results"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df = pd.read_csv('../input/cassava-leaf-disease-classification/sample_submission.csv')\nsample_df.head()\nsample_copy = sample_df.copy()\nsample_copy['image_id'] = sample_copy['image_id'].apply(lambda x: f'test_images/{x}')\ntest_dl = learn.dls.test_dl(sample_copy)\ntest_dl.show_batch()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"a, _ = learn.tta(dl=test_dl, n=16)\npred = a.argmax(dim=1).numpy()\nsample_df['label'] = pred","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}