{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Recreating Abhishek's 89.3% Start Kernel with fastai\n\n\nI saw Abhishek's [notebook](https://www.kaggle.com/abhishek/tez-faster-and-easier-training-for-leaf-detection?scriptVersionId=47408263) a number of days ago and wanted to try and recreate it in a 1:1 fashion with fastai to ensure that it could perform similar.\n\nThis notebook will be the result of that process, and by the end of it we will have trained an equivalent model that can achieve 89.4% on the public LB."},{"metadata":{},"cell_type":"markdown","source":"## A few key differences between his and what other fastai-kernels are doing\n\nThere are a few pieces that differ in most other fastai kernels that don't quite breach the 89% threshold with a single model. I believe it comes down to two factors:\n\n1. Data Augmentation\n2. How they are performing TTA\n3. The scheduling being used\n\n**Data Augmentation**\n\nIn his notebook there are a few transforms that fastai does not have equivalents of, such as `HueSaturationValue`. We have `Hue` and `Saturation`, and `Value` has not been implemented yet (and I don't know quite enough yet about those transforms to work out the logic). \n\n**TTA**\n\nIn most of the fastai kernels I've seen, they follow a standard TTA regiment. Abhishek performs TTA 15 times, which I saw had a significant boost in accuracy compared to the standard 3+1 that fastai will do\n\n**Scheduling**\n\nMore PyTorch related kernels are now fitting with a `CosineAnnealingWarmRestarts` scheduler, but fastai doesn't have an equivalent, right?\n\nIt actually does. So `WarmRestarts` `T_0` value simply dictates how long we should be calling a `fit` for, and as we will see later the proper scheduler to use is `fit_flat_cos` with a `start_pct` of 0 (thanks to @tanlikesmath for pointing this out to me!)"},{"metadata":{},"cell_type":"markdown","source":"## Installing what we need\n\nWe'll be installing the `efficientnet-pytorch` library:"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install efficientnet-pytorch -qqq","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Importing from the libraries\n\nNext we'll want to bring in everything we want."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"from fastai.vision.all import *\n\n\nimport albumentations # Data Augmentation\nfrom efficientnet_pytorch import EfficientNet # The Model\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Setting up our transforms\n\nNow we need to use `albumentations` for everything, as that's what he does!\n\nSo, how can we manage this?\n\nfastai has a helpful [tutorial](https://docs.fast.ai/tutorial.albumentations.html) on how to get started with albumentations, and we'll take it a step further.\n\nFirst here is what we currently have:\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"class AlbumentationsTransform(DisplayedTransform):\n    def __init__(self, aug): self.aug = aug\n    def encodes(self, img: PILImage):\n        aug_img = self.aug(image=np.array(img))['image']\n        return PILImage.create(aug_img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"While this transform *does* work, it assumes that everything is applied to both the training and the validation set, and doesn't let us put in custom pipelines for both. Let's change that:"},{"metadata":{"trusted":true},"cell_type":"code","source":"class AlbumentationsTransform(RandTransform):\n    \"A transform handler for multiple `Albumentation` transforms\"\n    split_idx,order=None,2\n    def __init__(self, train_aug, valid_aug): store_attr()\n    \n    def before_call(self, b, split_idx):\n        self.idx = split_idx\n    \n    def encodes(self, img: PILImage):\n        if self.idx == 0:\n            aug_img = self.train_aug(image=np.array(img))['image']\n        else:\n            aug_img = self.valid_aug(image=np.array(img))['image']\n        return PILImage.create(aug_img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What we have done here is allow for a `train_aug` and `valid_aug` transform pipeline (these can be `albumentation.Compose`'d transforms, as we will see later) and we've ensured that some transforms will only occur on the training set (when split_idx is 0) and on the validation set (1). Let's now recreate his data augmentation:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_train_aug(): return albumentations.Compose([\n            albumentations.RandomResizedCrop(256,256),\n            albumentations.Transpose(p=0.5),\n            albumentations.HorizontalFlip(p=0.5),\n            albumentations.VerticalFlip(p=0.5),\n            albumentations.ShiftScaleRotate(p=0.5),\n            albumentations.HueSaturationValue(\n                hue_shift_limit=0.2, \n                sat_shift_limit=0.2, \n                val_shift_limit=0.2, \n                p=0.5\n            ),\n            albumentations.RandomBrightnessContrast(\n                brightness_limit=(-0.1,0.1), \n                contrast_limit=(-0.1, 0.1), \n                p=0.5\n            ),\n            albumentations.CoarseDropout(p=0.5),\n            albumentations.Cutout(p=0.5)\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_valid_aug(): return albumentations.Compose([\n    albumentations.CenterCrop(256,256, p=1.),\n    albumentations.Resize(256,256)\n], p=1.)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since albumentations are operating on `PILImage`'s, these need to be in the item transforms\n> Warning: do not include `Normalize` in here! Our transform expects to be able to generate a PILImage at the end, and adjusting the datatypes will cause issues!"},{"metadata":{"trusted":true},"cell_type":"code","source":"item_tfms = AlbumentationsTransform(get_train_aug(), get_valid_aug())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Building our `DataBlock`\n\nWe now have everything setup to build our data! We'll follow my similar notebook's data approach from [here](https://www.kaggle.com/muellerzr/cassava-fastai-starter)"},{"metadata":{"trusted":true},"cell_type":"code","source":"set_seed(999)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = Path(\"../input\")\ndata_path = path/'cassava-leaf-disease-classification'\ndf = pd.read_csv(data_path/'train.csv')\ndf['image_id'] = df['image_id'].apply(lambda x: f'train_images/{x}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"idx2lbl = {0:\"Cassava Bacterial Blight (CBB)\",\n          1:\"Cassava Brown Streak Disease (CBSD)\",\n          2:\"Cassava Green Mottle (CGM)\",\n          3:\"Cassava Mosaic Disease (CMD)\",\n          4:\"Healthy\"}\n\ndf['label'].replace(idx2lbl, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"blocks = (ImageBlock, CategoryBlock)\nsplitter = RandomSplitter(valid_pct=0.2, seed=999)\ndef get_x(row): return data_path/row['image_id']\ndef get_y(row): return row['label']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"block = DataBlock(blocks=blocks,\n                 get_x=get_x,\n                 get_y=get_y,\n                 splitter=splitter,\n                 item_tfms=item_tfms,\n                 batch_tfms=[Normalize.from_stats(*imagenet_stats)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And build the `DataLoaders`:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dls = block.dataloaders(df, bs=32, val_bs=64)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I can now safely say we have *equivalent* data to Abhishek's. Let's look at a batch:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dls.show_batch(figsize=(12,12))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Building the Model\n\nNext we'll want to build our model. I've recreated it in raw PyTorch below:"},{"metadata":{"trusted":true},"cell_type":"code","source":"class LeafModel(Module):\n    def __init__(self, num_classes):\n\n        self.effnet = EfficientNet.from_pretrained(\"efficientnet-b3\")\n        self.dropout = nn.Dropout(0.1)\n        self.out = nn.Linear(1536, num_classes)\n\n    def forward(self, image):\n        batch_size, _, _, _ = image.shape\n\n        x = self.effnet.extract_features(image)\n        x = F.adaptive_avg_pool2d(x, 1).reshape(batch_size, -1)\n        outputs = self.out(self.dropout(x))\n        return outputs","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And now we can build it:"},{"metadata":{"trusted":true},"cell_type":"code","source":"net = LeafModel(dls.c)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training\n\nAll that's left is to train our model.\n\nWe're not going to follow the freeze + unfreeze transfer learning methodology of fastai *only because I want this as close to a 1:1 as possible*. \n\nLet's build our `Learner`:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = Learner(dls, net, loss_func=CrossEntropyLossFlat(),\n               metrics=[accuracy]).to_native_fp16()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And now we'll train our model. To be equivalent we'll use `fit_flat_cos` with a `pct_start` of zero, along with `EarlyStopping`:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_flat_cos(10, 3e-4, pct_start=0.0,\n                  cbs=[EarlyStoppingCallback(patience=3),\n                      SaveModelCallback()])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We've achieved roughly 84-85% accuracy, so now we will export our model away and do inference with it in a seperate kernel [here](https://www.kaggle.com/muellerzr/fastai-abhishek-inference)"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.export('baseline')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}