{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Motivation \n\nCassava is a starchy root vegetable which is staple food in Africa. Almost 80% of the farms grow these plants. But yields are generally poor due to viral diseases. It's crucial to detect these diseases early on to curb the spread.\n\n## Goal\n\nWe have to identify correct category for each image out of 5 classes. There are 4 diseased categories and 1 healthy category\n\n* Cassava Bacterial Blight (CBB),\n* Cassava Brown Streak Disease (CBSD),\n* Cassava Green Mottle (CGM),\n* Cassava Mosaic Disease (CMD),\n* Healthy"},{"metadata":{},"cell_type":"markdown","source":"In this notebook , I am creating a solution in PyTorch and PyTorch Lightning. PyTorch lightning is a way of organizing your PyTorch code. When the code is organised in simple lightning modules than we can easily use a lot of advanced features like \n\n- Mixed Precision\n- GPU/ Multi GPU/ TPU\n- Gradient Accumulation\n- Gradient Clipping\n- Moving tensors to/from GPU\n- Logging \n\nThe entire solution is structured in a way that allows us to run multiple experiments with different models, image sizes, batch sizes and other commonly used hyper parameters easily. \n\n\nIf you prefer coding in editors like visual studio, you can download entire code from my github repo over [here](https://github.com/svishnu88/Cassava)."},{"metadata":{},"cell_type":"markdown","source":"## Import the libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport sys\nfrom typing import Tuple\nimport PIL\nfrom torch.utils.data import Dataset\nfrom pathlib import Path\nfrom PIL import Image\nfrom PIL.Image import Image as PILImage\nfrom torch.utils.data.dataloader import DataLoader\nimport numpy as np\nimport pandas as pd\nfrom pytorch_lightning import LightningDataModule\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\nimport albumentations as A\nfrom albumentations.pytorch.transforms import ToTensor\n\nfrom torchvision import models\nimport torch.nn as nn\nimport torch\nimport torch.nn.functional as F\nimport pytorch_lightning as pl\nfrom torch import optim\nfrom pytorch_lightning.callbacks import LearningRateMonitor, ModelCheckpoint\n\nsys.path.append('../input/geneffnet/gen-efficientnet-pytorch-master')\n\nimport geffnet\n\npath = Path(\"/kaggle/input/cassava-leaf-disease-classification/\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Block\n\n- Create a Pytorch Dataset.\n- Create a Pytorch Lightning Data Module block which contains all the code for creating data loaders."},{"metadata":{"trusted":true},"cell_type":"code","source":"class CassavaDataset(Dataset):\n    def __init__(self, path, df, transform=None) -> None:\n        super().__init__()\n        self.df = df\n        self.path = path\n        self.transform = transform\n        self.num_workers = 2\n\n    def __getitem__(self, index) -> Tuple[PILImage, int]:\n        img_id, label = self.df.iloc[index]\n        image = Image.open(self.path / img_id)\n        image = np.array(image)\n        if self.transform is not None:\n            transformed = self.transform(image=image)\n            image = transformed[\"image\"]\n        return image, label\n\n    def __len__(self):\n        return self.df.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class CassavaDataModule(LightningDataModule):\n    def __init__(\n        self,\n        path: str = None,\n        aug_p: float = 0.5,\n        val_pct: float = 0.2,\n        img_sz: int = 224,\n        batch_size: int = 64,\n        num_workers: int = 4,\n        fold_id: int = 0,\n    ):\n        super().__init__()\n        self.path = Path(path)\n        self.aug_p = aug_p\n        self.val_pct = val_pct\n        self.img_sz = img_sz\n        self.batch_size = batch_size\n        self.num_workers = num_workers\n        self.fold_id = fold_id\n\n    def prepare_data(self):\n        # only called on 1 GPU/TPU in distributed\n        df = pd.read_csv(self.path / \"train.csv\")\n        skf = StratifiedKFold(n_splits=5)\n        t = df.label\n        train_index, valid_index = list(skf.split(np.zeros(len(t)), t))[self.fold_id]\n        train_df = df.loc[train_index]\n        valid_df = df.loc[valid_index]\n\n        train_df.to_pickle(\"train_df.pkl\")\n        valid_df.to_pickle(\"valid_df.pkl\")\n\n    def setup(self):\n        # called on every process in DDP\n        self.train_transform, self.test_transform = get_augmentations(\n            p=self.aug_p, image_size=self.img_sz\n        )\n        self.train_df = pd.read_pickle(\"train_df.pkl\")\n        self.valid_df = pd.read_pickle(\"valid_df.pkl\")\n\n    def train_dataloader(self):\n        train_dataset = CassavaDataset(\n            self.path / \"train_images\", df=self.train_df, transform=self.train_transform\n        )\n        return DataLoader(\n            train_dataset,\n            batch_size=self.batch_size,\n            num_workers=self.num_workers,\n            shuffle=True,\n            pin_memory=True,\n        )\n\n    def val_dataloader(self):\n        valid_dataset = CassavaDataset(\n            self.path / \"train_images\", df=self.valid_df, transform=self.test_transform\n        )\n        return DataLoader(\n            valid_dataset,\n            batch_size=self.batch_size,\n            num_workers=self.num_workers,\n            shuffle=False,\n            pin_memory=True,\n        )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv(path/'train.csv')\nds = CassavaDataset(path/'train_images',df=df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Apply Augmentations\n\nWe are choosing some transformations from Albumentations library to apply on the dataset, which will give our neural network more information to learn form and become better at predictions. \n\nI have not played much with the augmentation pipeline, so the parametes may not be optimized. So tweaking them can improve the model further."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_augmentations(p=0.5, image_size=224):\n    imagenet_stats = {\"mean\": [0.485, 0.456, 0.406], \"std\": [0.229, 0.224, 0.225]}\n    train_tfms = A.Compose(\n        [\n            # A.Resize(image_size, image_size),\n            A.RandomResizedCrop(image_size, image_size),\n            A.ShiftScaleRotate(shift_limit=0.15, scale_limit=0.4, rotate_limit=45, p=p),\n            A.Cutout(p=p),\n            A.RandomRotate90(p=p),\n            A.Flip(p=p),\n            A.OneOf(\n                [\n                    A.RandomBrightnessContrast(\n                        brightness_limit=0.2,\n                        contrast_limit=0.2,\n                    ),\n                    A.HueSaturationValue(\n                        hue_shift_limit=20, sat_shift_limit=50, val_shift_limit=50\n                    ),\n                ],\n                p=p,\n            ),\n            A.OneOf(\n                [\n                    A.IAAAdditiveGaussianNoise(),\n                    A.GaussNoise(),\n                ],\n                p=p,\n            ),\n            A.CoarseDropout(max_holes=10, p=p),\n            A.OneOf(\n                [\n                    A.MotionBlur(p=0.2),\n                    A.MedianBlur(blur_limit=3, p=0.1),\n                    A.Blur(blur_limit=3, p=0.1),\n                ],\n                p=p,\n            ),\n            A.OneOf(\n                [\n                    A.OpticalDistortion(p=0.3),\n                    A.GridDistortion(p=0.1),\n                    A.IAAPiecewiseAffine(p=0.3),\n                ],\n                p=p,\n            ),\n            ToTensor(normalize=imagenet_stats),\n        ]\n    )\n\n    valid_tfms = A.Compose(\n        [A.CenterCrop(image_size, image_size), ToTensor(normalize=imagenet_stats)]\n    )\n\n    return train_tfms, valid_tfms","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create a PyTorch Model\n- Any Resnet/Resnext model supported by torch hub\n- Efficientnet models from [geffnet](https://github.com/rwightman/gen-efficientnet-pytorch/tree/master/geffnet)"},{"metadata":{"trusted":true},"cell_type":"code","source":"ssl_models = [\n    \"resnet18_ssl\",\n    \"resnet50_ssl\",\n    \"resnext50_32x4d_ssl\",\n    \"resnext101_32x4d_ssl\",\n    \"resnext101_32x8d_ssl\",\n    \"resnext101_32x16d_ssl\",\n]\n\nclass Resnext(nn.Module):\n    def __init__(\n        self,\n        model_name=\"resnet18_ssl\",\n        pool_type=F.adaptive_avg_pool2d,\n        num_classes=1000,\n        kaggle=False,\n    ):\n        super().__init__()\n        self.pool_type = pool_type\n\n        if kaggle:\n            backbone = eval(model_name)()\n        else:\n            backbone = torch.hub.load(\n                \"facebookresearch/semi-supervised-ImageNet1K-models\", model_name\n            )\n        list(backbone.children())[:-2]\n        self.backbone = nn.Sequential(*list(backbone.children())[:-2])\n        in_features = getattr(backbone, \"fc\").in_features\n        self.classifier = nn.Linear(in_features, num_classes)\n\n    def forward(self, x):\n        features = self.pool_type(self.backbone(x), 1)\n        features = features.view(x.size(0), -1)\n        return self.classifier(features)\n\n\ndef get_efficientnet(model_name, pretrained=True, num_classes=5):\n    model = geffnet.create_model(model_name, pretrained=pretrained)\n    model.classifier = nn.Linear(model.classifier.in_features, num_classes)\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Pytorch Lightning Module\nCreate a Pytorch Lightning Module where we write the essential parts of our training pipeline like\n\n- training step\n- validation step\n- choose optimizer\n- choose scheduler \n- do any logging as required"},{"metadata":{"trusted":true},"cell_type":"code","source":"class CassavaModel(pl.LightningModule):\n    def __init__(\n        self,\n        model_name: str = None,\n        num_classes: int = None,\n        data_path: Path = None,\n        loss_fn=F.cross_entropy,\n        lr=1e-4,\n        wd=1e-6,\n    ):\n        super().__init__()\n\n        if model_name.find(\"res\") > -1:\n            self.model = Resnext(model_name=model_name, num_classes=num_classes)\n        elif model_name.find(\"effi\") > -1:\n            self.model = get_efficientnet(model_name)\n        self.data_path = data_path\n        self.loss_fn = loss_fn\n        self.lr = lr\n        self.accuracy = pl.metrics.Accuracy()\n        self.wd = wd\n\n    def forward(self, x):\n        return self.model(x)\n\n    def training_step(self, batch, batch_idx):\n        x, y = batch\n        y_hat = self(x)\n        loss = self.loss_fn(y_hat, y)\n        self.log(\"train_loss\", loss, prog_bar=True)\n        return loss\n\n    def validation_step(self, batch, batch_idx):\n        x, y = batch\n        y_hat = self(x)\n        loss = self.loss_fn(y_hat, y)\n        self.log(\"valid_loss\", loss, prog_bar=True)\n        self.log(\"val_acc\", self.accuracy(y_hat, y), prog_bar=True)\n\n    def configure_optimizers(self):\n        optimizer = optim.AdamW(\n            self.model.parameters(), lr=self.lr, weight_decay=self.wd\n        )\n        scheduler = optim.lr_scheduler.CosineAnnealingLR(\n            optimizer, self.trainer.max_epochs, 0\n        )\n\n        return [optimizer], [scheduler]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"fold_id = 0\naug_p = 0.5\nimg_sz= 224\nbatch_size = 64\nnum_workers = 4\nnum_classes = 5\nloss_fn = F.cross_entropy\nlr = 1e-4\nepochs = 1\ngradient_clip_val = 0.1\nprecision = 16\nmodel_name=ssl_models[2]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_module = CassavaDataModule(\n    path=path,\n    aug_p=aug_p,\n    img_sz=img_sz,\n    batch_size=batch_size,\n    num_workers=num_workers,\n    fold_id=fold_id,\n)\ndata_module.prepare_data()\ndata_module.setup()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"model = CassavaModel(\n    model_name=model_name,\n    num_classes=num_classes,\n    data_path=path,\n    lr=lr,\n    loss_fn=loss_fn,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir /kaggle/working/weights","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"weights_path = Path(f\"/kaggle/working/weights\")\n\ncheckpoint_callback = ModelCheckpoint(\n    dirpath=weights_path,\n    save_weights_only=True,\n    monitor=\"val_acc\",\n    mode=\"max\",\n    save_last=True,\n    filename=f\"{fold_id}\",\n)\ntrainer = pl.Trainer(\n    gpus=1,\n    callbacks=[checkpoint_callback],\n    max_epochs=epochs,\n    gradient_clip_val=gradient_clip_val,\n    precision=precision,\n   )\n\ntrainer.fit(model=model, datamodule=data_module)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Further improvements\n\nYou can increase the number of epochs to get further improvement. Also, right now I just ran on 1-fold, you can run multiple folds to get better score. You can play with hyperparameters and augmentations as well.\n\nWith same code I was able to reach 0.895 with a single fold on leaderboard.\nWe have completed the training pipeline here. \nI will show inference in another kernel.[Link to be added]\n\nHope you liked the notebook."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}