{"cells":[{"metadata":{},"cell_type":"markdown","source":"**BEFORE YOU COPY AND EDIT NOTEBOOK, PLEASE SUPPORT AND UPVOTE**\n\nv10: Switch to EfficientNet-B5 and larger image size (456x456) with Ranger, gradient accumulation, mixup\n\nv7: Switch to EfficientNet-B4 and larger image size (380x380)\n\nv6: Back to 10 epochs (public LB: 0.884)\n\nv5: Switch to 5 epochs of EfficientNet-B3 model with Ranger (w/ flat+cosine anneal LR), 8x TTA (public LB: 0.881)\n\nv4: Train 5-->10 epochs (public LB: 0.874)\n\nv3: add TTA (public LB: 0.873)\n\nv2: add mixup and label smoothing (public LB: 0.870)\n\nv1: Initial version (public LB: 0.860)\n\n\n# Cassava Leaf Disease Classification - Simple EDA and fastai starter\n\nIn this competition, we are trying to identify common diseases of cassava crops using data science and machine learning.\nPrevious methods of disease detection require farmers to solicit the help of government-funded agricultural experts to visually inspect and diagnose the plants. This suffers from being labor-intensive, low-supply and costly. Instead, it would be preferred if an automated pipeline based on mobile-quality photos of the cassava leafs could be developed.\n\nThis competition provides a farmer-crowdsourced dataset, labeled by experts at the National Crops Resources Research Institute (NaCRRI).\n\nIn this kernel, I will present a quick 'n dirty EDA and fastai starter. \n"},{"metadata":{},"cell_type":"markdown","source":"## A look at the data"},{"metadata":{},"cell_type":"markdown","source":"Let's start out by setting up our environment by installing and importing the required modules and setting a random seed:"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install ../input/pytorch-image-models/timm-0.3.1-py3-none-any.whl","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport os\nimport pandas as pd\nfrom fastai.vision.all import *\nimport albumentations","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"set_seed(999,reproducible=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check what is available to us:"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"dataset_path = Path('../input/cassava-leaf-disease-classification')\nos.listdir(dataset_path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that we have our train csv file with the train image names and labels, the sample submission csv with the test image names, and the train and test image folders. We also have the images in tfrecords format which is useful for quick loading of images, especially for TensorFlow and TPUs. We won't use this for today though.\n\nLet's check the train csv file:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(dataset_path/'train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's do some quick processing of the image filenames to make it easier to access:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['path'] = train_df['image_id'].map(lambda x:dataset_path/'train_images'/x)\ntrain_df = train_df.drop(columns=['image_id'])\ntrain_df = train_df.sample(frac=1).reset_index(drop=True) #shuffle dataframe\ntrain_df.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Okay let's check how many images are available in the training dataset:"},{"metadata":{"trusted":true},"cell_type":"code","source":"len_df = len(train_df)\nprint(f\"There are {len_df} images\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have >21,000 images! Hopefully, we can develop a highly-predictive, robust, and generalizable model with this dataset. \n\nLet's check the distribution of the different classes:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['label'].hist(figsize = (10, 5))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this case, we have 5 labels (4 diseases and healthy):\n0. Cassava Bacterial Blight (CBB)\n1. Cassava Brown Streak Disease (CBSD)\n2. Cassava Green Mottle (CGM)\n3. Cassava Mosaic Disease (CMD)\n4. Healthy\n\nIn this case label 3, [Cassava Mosaic Disease (CMD)](https://en.wikipedia.org/wiki/Cassava_mosaic_virus) is the most common label. This imbalance may have to be addressed with a weighted loss function or oversampling. I might try this in a future iteration of this kernel or in a new kernel.\n\nLet's check an example image to see what it looks like:"},{"metadata":{"trusted":true},"cell_type":"code","source":"from PIL import Image\n\nim = Image.open(train_df['path'][1])\nwidth, height = im.size\nprint(width,height) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"im","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data loading\n\nAfter my quick 'n dirty EDA, let's load the data into fastai as `DataLoaders` objects. \n\nFirst let's define item and batch transforms. These transforms are the albumentations transforms applied in fastai with the help of some code written by @muellerzr (see [here](https://www.kaggle.com/muellerzr/recreating-abhishek-s-tez-with-fastai) for more details). The batch size is set to 32 here.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"class AlbumentationsTransform(RandTransform):\n    \"A transform handler for multiple `Albumentation` transforms\"\n    split_idx,order=None,2\n    def __init__(self, train_aug, valid_aug): store_attr()\n    \n    def before_call(self, b, split_idx):\n        self.idx = split_idx\n    \n    def encodes(self, img: PILImage):\n        if self.idx == 0:\n            aug_img = self.train_aug(image=np.array(img))['image']\n        else:\n            aug_img = self.valid_aug(image=np.array(img))['image']\n        return PILImage.create(aug_img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_train_aug(sz): return albumentations.Compose([\n            albumentations.RandomResizedCrop(sz,sz),\n            albumentations.Transpose(p=0.5),\n            albumentations.HorizontalFlip(p=0.5),\n            albumentations.VerticalFlip(p=0.5),\n            albumentations.ShiftScaleRotate(p=0.5),\n            albumentations.HueSaturationValue(\n                hue_shift_limit=0.2, \n                sat_shift_limit=0.2, \n                val_shift_limit=0.2, \n                p=0.5\n            ),\n            albumentations.RandomBrightnessContrast(\n                brightness_limit=(-0.1,0.1), \n                contrast_limit=(-0.1, 0.1), \n                p=0.5\n            ),\n            albumentations.CoarseDropout(p=0.5),\n            albumentations.Cutout(p=0.5)\n])\n\ndef get_valid_aug(sz): return albumentations.Compose([\n    albumentations.CenterCrop(sz,sz, p=1.),\n    albumentations.Resize(sz,sz)\n], p=1.)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"While fastai provides various ways of doing custom dataloading (even just taking plain PyTorch DataLoaders), traditional image classification problems work well the high-level data API. Here, we can pass all the required info to create an `DataLoaders` object."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_dls(sz,bs):\n    item_tfms = AlbumentationsTransform(get_train_aug(sz), get_valid_aug(sz))\n    batch_tfms = [Normalize.from_stats(*imagenet_stats)]\n    dls = ImageDataLoaders.from_df(train_df, #pass in train DataFrame\n                                   valid_pct=0.2, #80-20 train-validation random split\n                                   seed=999, #seed\n                                   label_col=0, #label is in the first column of the DataFrame\n                                   fn_col=1, #filename/path is in the second column of the DataFrame\n                                   bs=bs, #pass in batch size\n                                   item_tfms=item_tfms, #pass in item_tfms\n                                   batch_tfms=batch_tfms) #pass in batch_tfms\n    return dls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dls = get_dls(456,16)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To confirm successful dataloader creation, we can use the `show_batch` command, which shows a subset of the batch:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dls.show_batch()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model training:"},{"metadata":{},"cell_type":"markdown","source":"Let's train a simple EfficientNet-B5 model. We will use the wonderful [timm](https://github.com/rwightman/pytorch-image-models) package by Ross Wightman to define the model. Since this competition doesn't allow internet access, I have added the pretrained weights from timm as a dataset, and the below code cell will allow timm to find the file:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Making pretrained weights work without needing to find the default filename\nif not os.path.exists('/root/.cache/torch/hub/checkpoints/'):\n        os.makedirs('/root/.cache/torch/hub/checkpoints/')\n!cp '../input/timmefficientnet/tf_efficientnet_b5_ns-6f26d0cf.pth' '/root/.cache/torch/hub/checkpoints/tf_efficientnet_b5_ns-6f26d0cf.pth'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In fastai, the trainer class is the `Learner`, which takes in the data, model, optimizer, loss function, etc. and allows you to train models, make predictions, etc.\n\nWhen training common CNN models like ResNets, we typically can use the `cnn_learner` function which creates a `Learner` object that allows us to train a provided model with the given dataloaders. However, `cnn_learner` doesn't support the models from timm out-of-the-box. Zach Mueller (@muellerzr) [has written some simple functions](https://walkwithfastai.com/vision.external.timm) to make it very easy to create Learner objects for timm models."},{"metadata":{"trusted":true},"cell_type":"code","source":"from timm import create_model\nfrom fastai.vision.learner import _update_first_layer\n\ndef create_timm_body(arch:str, pretrained=True, cut=None, n_in=3):\n    \"Creates a body from any model in the `timm` library.\"\n    model = create_model(arch, pretrained=pretrained, num_classes=0, global_pool='')\n    _update_first_layer(model, n_in, pretrained)\n    if cut is None:\n        ll = list(enumerate(model.children()))\n        cut = next(i for i,o in reversed(ll) if has_pool_type(o))\n    if isinstance(cut, int): return nn.Sequential(*list(model.children())[:cut])\n    elif callable(cut): return cut(model)\n    else: raise NamedError(\"cut must be either integer or function\")\n        \ndef create_timm_model(arch:str, n_out, cut=None, pretrained=True, n_in=3, init=nn.init.kaiming_normal_, custom_head=None,\n                     concat_pool=True, **kwargs):\n    \"Create custom architecture using `arch`, `n_in` and `n_out` from the `timm` library\"\n    body = create_timm_body(arch, pretrained, None, n_in)\n    if custom_head is None:\n        nf = num_features_model(nn.Sequential(*body.children())) * (2 if concat_pool else 1)\n        head = create_head(nf, n_out, concat_pool=concat_pool, **kwargs)\n    else: head = custom_head\n    model = nn.Sequential(body, head)\n    if init is not None: apply_init(model[1], init)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def timm_learner(dls, arch:str, loss_func=None, pretrained=True, cut=None, splitter=None,\n                y_range=None, config=None, n_out=None, normalize=True, **kwargs):\n    \"Build a convnet style learner from `dls` and `arch` using the `timm` library\"\n    if config is None: config = {}\n    if n_out is None: n_out = get_c(dls)\n    assert n_out, \"`n_out` is not defined, and could not be inferred from data, set `dls.c` or pass `n_out`\"\n    if y_range is None and 'y_range' in config: y_range = config.pop('y_range')\n    model = create_timm_model(arch, n_out, default_split, pretrained, y_range=y_range, **config)\n    learn = Learner(dls, model, loss_func=loss_func, splitter=default_split, **kwargs)\n    if pretrained: learn.freeze()\n    return learn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's now create our `Learner` object. We will also use common state-of-the-art training techniques like label smoothing and Ranger optimizer, which are provided in fastai. We can also use mixed precision very easily:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = timm_learner(dls, \n                    'tf_efficientnet_b5_ns',\n                     opt_func=ranger,\n                     loss_func=LabelSmoothingCrossEntropy(),\n                     cbs=[GradientAccumulation(n_acc=32)],\n                     metrics = [accuracy]).to_native_fp16()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We are now provided with a Learner object which has a frozen model (only the weights of the head of the model can be updated). In order to train a model, we need to find the most optimal learning rate, which can be done with fastai's learning rate finder:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's start training the model. Often, the best way to train a model is to train the frozen pretrained model for a single epoch then train the whole pretrained model for several epochs. The Ranger optimizer performs best with a flat+cosine annealing learning rate schedule. We will now train the frozen model for one epoch.\n\nAs shown above, the optimal learning rate for training the frozen model is where the loss is decreasing most rapidly: around ~1e-1. To be safe, I will use high weight decay to help prevent overfitting. We will also use another common state-of-the-art training technique: mixup."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.freeze()\nlearn.fit_flat_cos(1,1e-1, wd=0.1, cbs=[MixUp()])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.save('stage-1')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = learn.load('stage-1')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.recorder.plot_loss()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's now unfreeze the model and find a good learning rate:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.unfreeze()\nlearn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's train for 10 epochs with the unfrozen model."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.unfreeze()\nlearn.fit_flat_cos(10, 1e-3,cbs=[MixUp(),SaveModelCallback()])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.recorder.plot_loss()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = learn.to_native_fp32()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.save('stage-2')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We plotted the loss, put the model back to fp32, and now we can export the model if we want to use later (i.e. for an inference kernel):"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.export()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference\n\nIt's very simple to perform inference with fastai. The `dls.test_dl` function allows you to create test dataloader using the same pipeline defined earlier."},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df = pd.read_csv(dataset_path/'sample_submission.csv')\nsample_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_sample_df = sample_df.copy()\n_sample_df['path'] = _sample_df['image_id'].map(lambda x:dataset_path/'test_images'/x)\n_sample_df = _sample_df.drop(columns=['image_id'])\ntest_dl = dls.test_dl(_sample_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can easily confirm that the test_dl is correct:"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dl.show_batch()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's pass the dataloader to the model and get predictions. We will use a common inference technique known as test-time augmentation (average predictions when passing in various augmented versions of the test image). This is also implemented in fastai. Let's do 8x TTA:"},{"metadata":{"trusted":true},"cell_type":"code","source":"preds, _ = learn.tta(dl=test_dl, n=15, beta=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's make a submission with these predictions!"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df['label'] = preds.argmax(dim=-1).numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, **WE ARE DONE**!\n\nIf you enjoyed this kernel, please give it an upvote. If you have any questions or suggestions, please leave a comment!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}