{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Hello! Welcome to my Cassava Leaf Disease Classification notebook, modeled using PyTorch.** \n\nIn my approach to this Kaggle problem, I (like many others) selected EfficientNet as my model to train and later infer the Cassava leaf disease for each image. More specifically, I elected to use Backbone 4 of the scaled EfficientNet model and used the pretrained TIMM model that was trained on the Noisy Student dataset, due to many forum posts sharing their success by using this configuration.\n\nI would also like to thank the authors of the following notebooks for making their code available to other Kagglers for this competition. I've used code from both notebooks in my notebook, and they were instrumental in helping me get a successful notebook up and running. I've also implemented a custom Loss function in my EfficientNet model that has shown improved performance in the presence of noisy data. Code used to create the custom Loss function was also borrowed from the below notebook. Please go and check out these notebooks if you want some additional helpful and easy-to-follow resources, I would highly recommend them!:\n* General code usage: https://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-train-amp-aug\n* Bi-Tempered Logistic Loss: https://www.kaggle.com/capiru/cassavanet-starter-easy-gpu-tpu-cv-0-9\n"},{"metadata":{},"cell_type":"markdown","source":"Now, speaking of Bi-Tempered Logistic Loss, I decided to use this more uncommon loss function as opposed to the popular CrossEntropyLoss() function in PyTorch. As many Kagglers know by now, this competition contains training and testing data that have significantly noisy labels. Bi-Tempered Logistic Loss aims to solve this problem by accounting for the presence of noise while training by using two \"temperature\" parameters to better generalize. See this Google Blog post for more information on this: https://ai.googleblog.com/2019/08/bi-tempered-logistic-loss-for-training.html"},{"metadata":{},"cell_type":"markdown","source":"Lastly, as seen from the title, I am also freezing the Batch Normalization layers in EfficientNet. It is common practice to freeze these layers when finetuning a pretrained EfficientNet model (see sources below). However, it is still required to do this manually, so in this notebook I show you how to do accomplish this.\n* https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/203594\n* https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/172882\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"package_paths = [\n    '../input/pytorch-image-models/pytorch-image-models-master' #'../input/efficientnet-pytorch-07/efficientnet_pytorch-0.7.0'\n]\nimport sys; \n\nfor pth in package_paths:\n    sys.path.append(pth)\n    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"'''\nIMPORTS\n'''\n\nimport cv2\nimport torch\nimport os\nfrom torch import nn\nfrom datetime import datetime\nimport time\nimport random\nimport torchvision\nfrom torchvision import transforms\nimport pandas as pd\nimport numpy as np\nfrom tqdm import tqdm\n\n\nimport matplotlib.pyplot as plt\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.utils.data import SequentialSampler, RandomSampler, WeightedRandomSampler\nfrom torch.cuda.amp import autocast, GradScaler\nimport torch.nn.functional as F\n\nimport timm\n\nimport sklearn\nfrom sklearn.metrics import roc_auc_score, log_loss\nfrom sklearn import metrics\nfrom sklearn.model_selection import GroupKFold, StratifiedKFold","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nLoad Data\n'''\n\ndata = pd.read_csv('../input/cassava-leaf-disease-classification/train.csv')\ndata.head()\n\n\nlen(data)    # Number of samples = 21397","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nCONFIGURATION\n'''\nconfig = {\n    'seed': 419,\n    'img_size': 512,\n    'tta':3,\n    'num_folds': 5,\n    \n    # input_size = 3, 380, 380. pool_size = 12, 12.\n    # DOC: https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/efficientnet.py\n    'model_arch':'tf_efficientnet_b4_ns',    \n    \n    'train_bs':16,\n    'valid_bs':32,\n    'num_workers': 2,\n    'epochs':10,\n    'device':'cuda:0',\n    \n    'T1':0.2,\n    'T2':1.0,\n    'label_smooth': 0.2,\n    \n    'lr':1e-4,\n    'min_lr':1e-6,\n    'T_0': 10,\n    'weight_decay':1e-6,\n    'ep_patience':4,\n    'factor':0.2,\n    'num_workers':2,\n    #'accum_iter':2,\n    'update_on_batch':True,\n    'use_wrs':False\n    \n    \n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nLOSS FUNCTIONS\n'''\n\ndef log_t(u, t):\n    \"\"\"Compute log_t for `u'.\"\"\"\n    if t==1.0:\n        return u.log()\n    else:\n        return (u.pow(1.0 - t) - 1.0) / (1.0 - t)\n\ndef tempered_softmax(activations, t, num_iters=5):\n    \"\"\"Tempered softmax function.\n    Args:\n      activations: A multi-dimensional tensor with last dimension `num_classes`.\n      t: Temperature > 1.0.\n      num_iters: Number of iterations to run the method.\n    Returns:\n      A probabilities tensor.\n    \"\"\"\n    if t == 1.0:\n        return activations.softmax(dim=-1)\n\n    normalization_constants = compute_normalization(activations, t, num_iters)\n    return exp_t(activations - normalization_constants, t)\n\ndef bi_tempered_logistic_loss(activations,\n        labels,\n        t1 = config['T1'],\n        t2 = config['T2'],\n        label_smoothing=config['label_smooth'],\n        num_iters=5,\n        reduction = 'mean'):\n\n    \"\"\"Bi-Tempered Logistic Loss.\n    Args:\n      activations: A multi-dimensional tensor with last dimension `num_classes`.\n      labels: A tensor with shape and dtype as activations (onehot), \n        or a long tensor of one dimension less than activations (pytorch standard)\n      t1: Temperature 1 (< 1.0 for boundedness).\n      t2: Temperature 2 (> 1.0 for tail heaviness, < 1.0 for finite support).\n      label_smoothing: Label smoothing parameter between [0, 1). Default 0.0.\n      num_iters: Number of iterations to run the method. Default 5.\n      reduction: ``'none'`` | ``'mean'`` | ``'sum'``. Default ``'mean'``.\n        ``'none'``: No reduction is applied, return shape is shape of\n        activations without the last dimension.\n        ``'mean'``: Loss is averaged over minibatch. Return shape (1,)\n        ``'sum'``: Loss is summed over minibatch. Return shape (1,)\n    Returns:\n      A loss tensor.\n    \"\"\"\n\n    if len(labels.shape)<len(activations.shape): #not one-hot\n        labels_onehot = torch.zeros_like(activations)\n        labels_onehot.scatter_(1, labels[..., None], 1)\n    else:\n        labels_onehot = labels\n\n    if label_smoothing > 0:\n        num_classes = labels_onehot.shape[-1]\n        labels_onehot = ( 1 - label_smoothing * num_classes / (num_classes - 1) ) \\\n                * labels_onehot + \\\n                label_smoothing / (num_classes - 1)\n\n    probabilities = tempered_softmax(activations, t2, num_iters)\n\n    loss_values = labels_onehot * log_t(labels_onehot + 1e-10, t1) \\\n            - labels_onehot * log_t(probabilities, t1) \\\n            - labels_onehot.pow(2.0 - t1) / (2.0 - t1) \\\n            + probabilities.pow(2.0 - t1) / (2.0 - t1)\n    loss_values = loss_values.sum(dim = -1) #sum over classes\n\n    if reduction == 'none':\n        return loss_values\n    if reduction == 'sum':\n        return loss_values.sum()\n    if reduction == 'mean':\n        return loss_values.mean()\n\nclass BiTemperedLogistic(nn.Module):\n    def __init__(self, T1 = config['T1'], T2 = config['T2'], LABEL_SMOOTH = config['label_smooth']):\n        super().__init__()\n        self.T1 = T1\n        self.T2 = T2\n        self.LABEL_SMOOTH = LABEL_SMOOTH\n\n    def forward(self, logits,labels):\n        return bi_tempered_logistic_loss(logits, labels,t1 = self.T1,t2 = self.T2, label_smoothing = self.LABEL_SMOOTH)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nAUGMENTATIONS\n'''\n\nfrom albumentations import (\n    ShiftScaleRotate, Normalize, Compose, CenterCrop, Resize, HorizontalFlip,\n    VerticalFlip, Transpose, RandomResizedCrop, HueSaturationValue, RandomBrightnessContrast,\n    CoarseDropout, Cutout\n)\n\nfrom albumentations.pytorch import ToTensorV2\n\ndef get_train_transforms():\n    return Compose([\n            RandomResizedCrop(config['img_size'], config['img_size']),\n            Transpose(p=0.5),\n            HorizontalFlip(p=0.5),\n            VerticalFlip(p=0.5),\n            ShiftScaleRotate(p=0.5),\n            HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n            RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            CoarseDropout(p=0.5),\n            Cutout(p=0.5),\n            ToTensorV2(p=1.0),\n        ], p=1.0)\n\ndef get_valid_transforms():\n    return Compose([\n                CenterCrop(config['img_size'], config['img_size'], p=1.0),\n                Resize(config['img_size'], config['img_size']),\n                Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n                ToTensorV2(p=1.0),\n        ], p=1.0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nHELPER FUNCTIONS\n'''\n\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True\n    \ndef get_img(path):\n    im_bgr = cv2.imread(path)\n    #print(np.shape(im_bgr))\n    #im_rgb = im_bgr[:, :, ::-1]\n    #print(im_rgb)\n    im_rgb = cv2.cvtColor(im_bgr, cv2.COLOR_BGR2RGB)\n    #print(np.shape(im_rgb))\n    return im_rgb\n\ndef prep_dataloader(df, train_idx, val_idx, use_wrs, sampler, data_dir):\n    \n    train_ref =  df.loc[train_idx, :].reset_index(drop=True)\n    valid_ref = df.loc[val_idx, :].reset_index(drop=True)\n    \n    train_ds = LeafDataset(train_ref, data_dir, transforms=get_train_transforms(), include_labels = True)\n    valid_ds = LeafDataset(valid_ref, data_dir, transforms=get_valid_transforms())\n    \n    if use_wrs:\n        train_loader = torch.utils.data.DataLoader(\n            train_ds,\n            batch_size=config['train_bs'],\n            drop_last=True,\n            shuffle=False,\n            num_workers=config['num_workers'],\n            sampler=sampler\n        )\n    else:\n        train_loader = torch.utils.data.DataLoader(\n            train_ds,\n            batch_size=config['train_bs'],\n            drop_last=True,\n            shuffle=True,\n            num_workers=config['num_workers'],\n        )\n    val_loader = torch.utils.data.DataLoader(\n        valid_ds,\n        batch_size=config['valid_bs'],\n        drop_last=True,\n        shuffle=False,\n        num_workers=config['num_workers']\n    )\n    return train_loader, val_loader","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nDATASET\n'''\n\nclass LeafDataset(Dataset):\n    def __init__(self, df, img_dir, transforms=None, include_labels=True):\n        super().__init__()\n        self.df = df     #.reset_index(drop=True).copy()\n        self.img_dir = img_dir\n        self.transforms = transforms\n        self.include_labels = include_labels\n        \n        if include_labels:\n            self.labels = self.df['label'].values\n        \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, index: int):\n        img = get_img(\"{}/{}\".format(self.img_dir, self.df.loc[index]['image_id']))\n        if self.transforms:\n            img = self.transforms(image=img)['image']\n        \n        if self.include_labels:\n            label = self.labels[index]\n            return img, label\n        else:\n            return img;\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nMODEL\n'''\n\nclass LeafDiseaseClassifier(nn.Module):\n    def __init__(self, model_arch, num_classes, pretrained=False):\n        super().__init__()\n        self.model = timm.create_model(model_arch, pretrained=pretrained)\n        n_features = self.model.classifier.in_features\n        self.model.classifier = nn.Linear(n_features, num_classes)\n\n    \n    def forward(self, x):\n        x = self.model(x)\n        return x\n        \n    def freeze_batch_norm(self):\n        layers = [mod for mod in self.model.children()]\n        for layer in layers:\n            if isinstance(layer, nn.BatchNorm2d):\n                #print(layer)\n                for param in layer.parameters():\n                    param.requires_grad = False\n                \n            elif isinstance(layer, nn.Sequential):\n                for seq_layers in layer.children():\n                    if isinstance(layer, nn.BatchNorm2d):\n                        #print(layer)\n                        param.requires_grad = False\n    \n            ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nMAIN\n'''\n\nif __name__ == '__main__':\n    seed_everything(config['seed'])\n    \n    # Calculate class stats for WeightedRandomSampler\n    class_counts = []\n    n_classes = data.label.nunique()\n    for n in range(n_classes):\n        n_count = 0\n        for class_ in data['label']:\n            if class_ == n:\n                n_count += 1\n        class_counts.append(n_count)\n  \n    num_samples = len(data)\n    targs = data['label']\n    class_weights = [num_samples/class_counts[i] for i in range(n_classes)]\n    weights = [class_weights[targs[i]] for i in range(num_samples)]\n    sampler = WeightedRandomSampler(torch.DoubleTensor(weights), n_classes)\n    \n    # Initialize lists for tracking batch and epoch losses\n    train_batch_loss = []\n    val_batch_loss = []\n    mean_val_epoch_loss = []\n    epoch_acc = []\n    \n    folds = StratifiedKFold(n_splits=config['num_folds'], shuffle=True, random_state=config['seed']).split(X=np.zeros(len(data)), y=data.label.values)\n    print('Initializing folds...')\n    \n    # K-Fold Cross Validation Loop\n    for fold, (trn_idx, val_idx) in enumerate(folds):\n        \n        # Already trained network and extracted best model, so break after first fold\n        if fold > 0:\n            break\n\n        train_loader, val_loader = prep_dataloader(data, trn_idx, val_idx, config['use_wrs'], sampler, data_dir='../input/cassava-leaf-disease-classification/train_images/')\n                                                                                                    \n        print('Initializing model, fold {} selected.'.format(fold))                                                                                                   \n        device = torch.device(config['device'] if torch.cuda.is_available() else \"cpu\")\n        \n        model = LeafDiseaseClassifier(config['model_arch'], data.label.nunique(), pretrained=True).to(device)\n        model.freeze_batch_norm()   \n        \n        scaler = GradScaler()\n        optimizer = torch.optim.Adam(model.parameters(), lr=config['lr'], weight_decay=config['weight_decay'])\n        #scheduler =  torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer=optimizer, mode='max', patience=config['patience'], verbose=True, factor=config['factor'])\n        scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=config['T_0'], T_mult=1, eta_min=config['min_lr'], last_epoch=-1)\n        \n        #loss_tr = nn.CrossEntropyLoss().to(device)\n        #loss_val = nn.CrossEntropyLoss().to(device)\n        \n        loss_tr = BiTemperedLogistic()\n        loss_val = BiTemperedLogistic()\n        best_acc = 0\n        bad_ep_count = 1\n        \n        for epoch in range(config['epochs']):\n            \n            # TRAINING LOOP\n            model.train()\n            t = time.time()\n            running_loss = None\n            \n            pbar = tqdm(enumerate(train_loader), total=len(train_loader))\n            for step, trn_batch in pbar:\n                trn_x, trn_labels = trn_batch\n                trn_x = trn_x.to(device).float()\n                trn_labels = trn_labels.to(device).long()\n                \n\n                with autocast():\n                    trn_preds = model(trn_x)\n                    loss = loss_tr(trn_preds, trn_labels)\n                    \n                scaler.scale(loss).backward()\n                scaler.step(optimizer)\n                scaler.update()\n                optimizer.zero_grad()\n                \n                if running_loss is None:\n                    running_loss = loss.item()\n                else:\n                    running_loss = running_loss * .99 + loss.item() * .01\n                if scheduler is not None and config['update_on_batch']:\n                    scheduler.step()    \n                if step+1 == len(train_loader):\n                    desc = 'Fold {0}, Epoch {1} Train Loss: {2:.4f}'.format(fold, epoch, running_loss)\n                    pbar.set_description(desc)\n                    \n            if scheduler is not None and not config['update_on_batch']:\n                scheduler.step()\n            \n            \n            # VALIDATION LOOP\n            with torch.no_grad():\n                model.eval()\n                \n                t = time.time()\n                loss_sum = 0\n                num_samples = 0\n                val_preds_list = []\n                val_labels_list = []\n                \n                pbar = tqdm(enumerate(val_loader), total=len(val_loader))\n                for step, val_batch in pbar:\n                    val_x, val_labels = val_batch\n                    val_x = val_x.to(device).float()\n                    val_labels = val_labels.to(device).long()\n                    \n                    val_preds = model(val_x)\n                    val_preds_list += [torch.argmax(val_preds, 1).detach().cpu().numpy()]\n                    val_labels_list += [val_labels.detach().cpu().numpy()]\n                    \n                    loss = loss_val(val_preds, val_labels)\n                    num_samples += val_labels.shape[0]\n                    loss_sum += loss.item() * val_labels.shape[0]\n                    val_batch_loss.append(loss_sum/num_samples)\n                    \n                    if step+1 == len(val_loader):\n                        desc = 'Fold {0}, Epoch {1} Val Loss: {2:.4f}'.format(fold, epoch, loss_sum/num_samples)\n                        pbar.set_description(desc)\n                    \n                val_preds_list = np.concatenate(val_preds_list)\n                val_labels_list = np.concatenate(val_labels_list)\n                acc = (val_preds_list==val_labels_list).mean()\n\n                if scheduler is not None:\n                    scheduler.step()\n                \n                # Checking if it's the best accuracy of this epoch\n                if acc > best_acc:\n                    best_acc = acc\n                    torch.save(model.state_dict(), '{0}_Fold{1}_Epoch{2}_Acc_{3}.pth'.format(config['model_arch'], fold, epoch, round(best_acc, 4)))\n                    bad_ep_count = 0\n\n                epoch_acc.append(acc)\n                mean_val_epoch_loss.append(sum(val_batch_loss)/len(val_batch_loss))\n                bad_ep_count += 1\n                \n                # Exit early if not improving\n                if bad_ep_count >= config['ep_patience']:\n                    print('Early stopping due to model not improving for {0} epochs'.format(config['ep_patience']))\n                    break\n\n                print('Validated on Fold {0}, epoch {1}; Validation Loss: {2}, Accuracy: {3}. Best Accuracy: {4}'.format(fold, epoch, loss_sum/num_samples, acc, best_acc))\n    \n    \n    ## Plot Results\n#    plt.figure()\n#    plt.plot(len(train_batch_loss), train_batch_loss)\n#    plt.title('Training Loss Per Batch')\n#    plt.show()\n    \n#    plt.figure()\n#    plt.plot(len(val_batch_loss), val_batch_loss)\n#    plt.title('Validation Loss Per Batch')\n#    plt.show()\n    \n#    plt.figure()\n#    plt.plot(len(mean_val_epoch_loss), mean_val_epoch_loss)\n#    plt.title('Mean Validtion Loss each Epoch')\n#    plt.show()\n    \n#    plt.figure()\n#    plt.plot(len(epoch_acc), epoch_acc)\n#    plt.title('Accuracy each Epoch')\n#    plt.show()\n    \n  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del model, optimizer, train_loader, val_loader, scaler, scheduler\ntorch.cuda.empty_cache()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**And thus concludes my Cassava Leaf Disease Classification notebook.**\n\nI hope you've enjoyed reading this and learned something. This is my first notebook and submission to a Kaggle contest, so please feel free to comment below any suggestions or advice for next time! I'm always looking to learn more and grow. Furthermore, if you have any questions about the code implementation or design decisions, I'd be happy to answer any questions in the comments below.\n\nLastly, here are the sources I've used for helping me get started writing the code for this notebook. Please give them and this notebook and upvote if you've found this notebook useful:\n* General code usage: https://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-train-amp-aug\n* Bi-Tempered Logistic Loss: https://www.kaggle.com/capiru/cassavanet-starter-easy-gpu-tpu-cv-0-9"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}