{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport cv2\nimport sys\nimport math\nimport time\nimport copy\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\nfrom pathlib import Path\nfrom tqdm.notebook import tqdm\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split, StratifiedShuffleSplit\nPath.ls = lambda x: list(x.iterdir())\n\nimport albumentations\nfrom albumentations.pytorch import ToTensor, ToTensorV2\n\nimport torch\nfrom torch import nn, optim\nfrom torchvision import transforms, models\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\neffnet_path = '../input/efficientnet-pytorch/'\nsys.path.append(effnet_path)\nfrom efficientnet_pytorch import EfficientNet","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loading the dataframe and filenames"},{"metadata":{"trusted":true},"cell_type":"code","source":"path = Path(\"../input/cassava-leaf-disease-classification\")\ndf_path = path/\"train.csv\"\ntrain_path = path/\"train_images\"\ntrain_fnames = train_path.ls()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"df = pd.read_csv(df_path)\nnum_classes = df['label'].nunique() # number of clases (5 in our case)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems like we have huge class imbalance here! We should take care of it."},{"metadata":{},"cell_type":"markdown","source":"## Train/Validation Split"},{"metadata":{},"cell_type":"markdown","source":"I'll split the main dataframe into train_df and val_df with stratification based on the labels. We are using 20% of the data for validation."},{"metadata":{"trusted":true},"cell_type":"code","source":"df.reset_index(inplace=True, drop=True)\nsss = StratifiedShuffleSplit(n_splits=2, test_size=0.2, random_state=42)\nfor train_idx, val_idx in sss.split(X=df, y=df['label']):\n    train_df = df.loc[train_idx]\n    val_df = df.loc[val_idx]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.shape, val_df.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Normalization"},{"metadata":{},"cell_type":"markdown","source":"One thing that is really common but doesn't sound right is that people usually use the ImageNet means and standard deviations to normalize this data. But I think it would be better to normalize this data using its own statistics to make sure that resulting mean of images will be zero and their std will be 1. This would not neccesarily be the case if ImageNet stats are used becasue this competition's images are much different from the ImageNet data (having green-ish colors much more!) so using ImageNet stats is not guaranteed to yield good classification results because the meaning of colors in different channels would differ if they are not properly centerd on zero with std of 1!"},{"metadata":{},"cell_type":"markdown","source":"Here, I'm using the stats of this very dataset and not ImageNet stats. I've iterated throgh all training images and calculated the means and stds of the channels which are the numbers below."},{"metadata":{"trusted":true},"cell_type":"code","source":"mean = [0.4589, 0.5314, 0.3236]\nstd = [0.2272, 0.2297, 0.2200]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I removed the code to calcuate this numbers from notebook in order to shorten it. But it will be something like the following. Note that the **train_dl** here must be a dataloader that does not do augmentation or normalization (it should just resize the images and convert them to tensor)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# means, stds = [], []\n# for batch, _ in train_dl:\n#     means.append(batch.mean(dim=(0, 2, 3)))\n#     stds.append(batch.std(dim=(0, 2, 3)))\n    \n# mean = torch.stack(means).mean(0)\n# std = torch.stack(stds).mean(0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dataset and DataLoaders"},{"metadata":{},"cell_type":"markdown","source":"I've borrowed some code from [this notebook](https://www.kaggle.com/abhishek/using-tez-in-leaf-disease-classification) here to define the augmentations that I'll use. But I've modefied it and removed some of the augmentaions that did not seem to be good in our case. We are using image size of 256."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_tfms = albumentations.Compose([\n            albumentations.RandomResizedCrop(256, 256),\n            albumentations.HorizontalFlip(p=0.5),\n            albumentations.ShiftScaleRotate(p=0.5),\n            albumentations.HueSaturationValue(\n                hue_shift_limit=0.2, \n                sat_shift_limit=0.2, \n                val_shift_limit=0.2, \n                p=0.5\n            ),\n            albumentations.RandomBrightnessContrast(\n                brightness_limit=(-0.1,0.1), \n                contrast_limit=(-0.1, 0.1), \n                p=0.5\n            ),\n            albumentations.Normalize(\n                mean=mean, \n                std=std, \n                max_pixel_value=255.0, \n                p=1.0\n            ),\n            albumentations.CoarseDropout(p=0.5),\n            albumentations.Cutout(p=0.5),\n            ToTensorV2()], p=1.)\n\n        \nvalid_tfms = albumentations.Compose([\n            albumentations.CenterCrop(256, 256, p=1.),\n            albumentations.Resize(256, 256),\n            albumentations.Normalize(\n                mean=mean, \n                std=std, \n                max_pixel_value=255.0, \n                p=1.0\n            ),\n            ToTensorV2()], p=1.)\n\nclass LeafData(Dataset):\n    def __init__(self, df, split=\"train\"):\n        if split == \"train\":\n            self.transforms = train_tfms\n        elif split == \"val\":\n            self.transforms = valid_tfms\n            \n        self.paths = [train_path/id_ for id_ in df['image_id'].values]\n        self.labels = df['label'].values\n    \n    def __getitem__(self, idx):\n        img = cv2.imread(str(self.paths[idx]))[..., ::-1] # ::-1 is here because cv2 loads the images in BGR rather than RGB\n        img = self.transforms(image=img)['image']\n        label = self.labels[idx]\n       \n        return img, label\n    \n    def __len__(self):\n        return len(self.paths)\n\ndef make_dataloaders(batch_size=32, num_workers=4, pin_memory=True, **kwargs):\n    dataset = LeafData(**kwargs)\n    dataloader = DataLoader(dataset, batch_size=batch_size, num_workers=num_workers,\n                            pin_memory=pin_memory, shuffle=True if kwargs['split'] == \"train\" else False)\n    return dataloader","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dl = make_dataloaders(df=train_df, split=\"train\")\nval_dl = make_dataloaders(df=val_df, split=\"val\")\nxb, yb = next(iter(train_dl))\nxb.shape, yb.shape, xb.mean(dim=(0, 2, 3)), xb.std(dim=(0, 2, 3))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"By looking at the stats of this random batch, we can make sure that the normalization is being done properly."},{"metadata":{},"cell_type":"markdown","source":"## Training helper functions"},{"metadata":{},"cell_type":"markdown","source":"Here are some utility functions which helps us easily train the model. The details are not important. All the thing they are doing is to train and validate the model properly and call the lr_schedulers at the right time during training. Also, the best weights of the model (according to the validation loss) get saved after each epoch if the loss improves."},{"metadata":{"trusted":true},"cell_type":"code","source":"def accuracy(preds, target):\n    preds = preds.argmax(dim=1)\n    return (preds == target).float().mean()\n    \ndef one_epoch(model, dl, loss_func, opt=None, lr_schedule=None):\n    running_loss = 0.\n    running_acc = 0\n    \n    for xb, yb in tqdm(dl):\n        xb, yb = xb.to(device), yb.to(device)\n        preds = model(xb)\n        loss = loss_func(preds, yb)\n        \n        if opt is not None:\n            opt.zero_grad()\n            loss.backward()\n            opt.step()\n            if lr_schedule is not None:\n                lr_schedule.step()\n    \n        running_acc += accuracy(preds, yb).item()\n        running_loss += loss.item()\n        \n    return running_loss / len(dl), running_acc / len(dl)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_lr(opt):\n    for param_group in opt.param_groups:\n        return param_group['lr']\n    \ndef train_val(model, params):\n   \n    num_epochs = params[\"num_epochs\"]\n    loss_func = params[\"loss_func\"]\n    opt = params[\"optimizer\"]\n    train_dl = params[\"train_dl\"]\n    val_dl = params[\"val_dl\"]\n    lr_scheduler = params[\"lr_scheduler\"]\n    path2weights = params[\"path2weights\"]\n    one_cycle = params[\"one_cycle\"]\n    \n    loss_history = {\n        \"train\": [],\n        \"val\": [],\n    }\n   \n    metric_history = {\n        \"train\": [],\n        \"val\": [],\n    }\n    \n    best_model_wts = copy.deepcopy(model.state_dict())\n    \n    best_loss=float('inf')\n    \n    for epoch in range(num_epochs):\n        start = time.time()\n        current_lr = get_lr(opt)\n        print(f'Epoch {epoch + 1}/{num_epochs}, current lr = {current_lr:5f}')\n      \n        model.train()\n        train_loss, train_metric = one_epoch(model, train_dl, loss_func, opt, lr_scheduler if one_cycle else None)\n\n        loss_history[\"train\"].append(train_loss)\n        metric_history[\"train\"].append(train_metric)\n  \n        model.eval()\n        with torch.no_grad():\n            val_loss, val_metric = one_epoch(model, val_dl, loss_func, opt=None)\n        \n       \n        if val_loss < best_loss:\n            best_loss = val_loss\n            best_model_wts = copy.deepcopy(model.state_dict())\n            torch.save(model.state_dict(), path2weights)\n            print(\"Copied best model weights!\")\n    \n        loss_history[\"val\"].append(val_loss)\n        metric_history[\"val\"].append(val_metric)\n        \n        if not one_cycle:\n            lr_scheduler.step(val_loss)\n            if current_lr != get_lr(opt):\n                print(\"Loading best model weights!\")\n                model.load_state_dict(best_model_wts) \n        \n        print(f\"Train Loss: {train_loss:.4f}, Val Loss: {val_loss:.4f}\\n\"\n              f\"Train Acc: {train_metric:.4f}, Val Acc: {val_metric:.4f}\\n\"\n              f\"Completed in {time.time() - start:.3f}\")\n        \n        print(\"-\"*10) \n\n    model.load_state_dict(best_model_wts)\n        \n    return model, loss_history, metric_history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## EfficientNet + Building a powerful head for it"},{"metadata":{},"cell_type":"markdown","source":"In the following code, you can see that I am using the EfficientNet model as the base (feature extractor) and then I append a powerful head to the backbone to be able to classify images into the needed classes (5 in our case). I'm using both Adaptive Average Pooling and Adaptive Max Pooling and then concatenating the two (this was suggested by Jeremy Howard in a fastai course). Then by using a Linear layer I map the features to 512 activations and then after that with another Linear layer I make the final 5 predictions for each class."},{"metadata":{"trusted":true},"cell_type":"code","source":"class EfficientNetModel(nn.Module):\n    def __init__(self, arch=\"b4\", dropout=0.2, n_out=5, \n                 pretrained=True, freeze=True):\n        super().__init__()\n        if pretrained:\n            self.model = EfficientNet.from_pretrained(f\"efficientnet-{arch}\")\n            if freeze:\n                for p in self.model.parameters():\n                    p.requires_grad = False\n        else:\n            self.model = EfficientNet.from_name(f\"efficientnet-{arch}\")\n        \n        self.lin1 = nn.Linear(1792 * 2, 512) # 1792 is the final output shape of the efficientnet backbone.\n        self.lin2 = nn.Linear(512, n_out)    # I'm multiplying by two because we are concatenating the avg pool\n        self.bn1 = nn.BatchNorm1d(1792 * 2)  # and max pool layers.\n        self.bn2 = nn.BatchNorm1d(512)\n        self.dropout = dropout\n        \n    def forward(self, x):\n        x = self.model.extract_features(x)\n        avg = F.adaptive_avg_pool2d(x, 1)\n        max_ = F.adaptive_max_pool2d(x, 1)\n        cat = torch.cat((avg.squeeze(), max_.squeeze()), dim=1)\n        x = self.bn1(cat)\n        x = F.dropout(x, self.dropout)\n        x = F.relu(self.bn2(self.lin1(x)))\n        x = self.lin2(x)\n        return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training with OneCycle Policy"},{"metadata":{},"cell_type":"markdown","source":"Here we put everything together and train the model with OneCycle Policy. You can refer to PyTorch documentation or the actual paper by Leslie Smith to learn about the OneCycle policy but the brief explanation is that it starts the training with low learning rate, increases it until 25% of iterations have passed, and then starts to reduce the learning rate until the training is finished (notice that the whole cycle will be done after that training is finished and it is not for each epoch separately)."},{"metadata":{"trusted":true},"cell_type":"code","source":"model = EfficientNetModel(pretrained=True, freeze=False, \n                          arch=\"b4\", n_out=num_classes, dropout=0.2).to(device) # I'm using pretrained weights but not freezing the backbone\n\ncriterion = nn.CrossEntropyLoss()\nopt = optim.Adam(model.parameters())\nepochs = 15\nlr_sch = optim.lr_scheduler.OneCycleLR(opt, max_lr=1e-3, epochs=epochs,\n                                       steps_per_epoch=len(train_dl), pct_start=0.25,)\n\nparams_train = {\n \"num_epochs\": epochs,\n \"optimizer\": opt,\n \"loss_func\": criterion,\n \"train_dl\": train_dl,\n \"val_dl\": val_dl,\n \"lr_scheduler\": lr_sch,\n \"path2weights\": \"/kaggle/working/effnet.pt\",\n \"one_cycle\": True\n}\n\nmodel, loss_hist, metric_hist = train_val(model, params_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(model.state_dict(), \"last_epoch.pt\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}