{"cells":[{"metadata":{},"cell_type":"markdown","source":"![](https://storage.googleapis.com/kaggle-competitions/kaggle/13836/logos/header.png?t=2020-10-01-17-22-54)"},{"metadata":{},"cell_type":"markdown","source":"Motivation: I wanted to test a new criterion called CutMix, which is shown to greatly improve performance.\nThe following ideas are implemented in this Notebook:\n* Optimizers : Adam, AdamW, AdamP, Ranger\n* Schedulers : ['ReduceLROnPlateau', 'CosineAnnealingLR', 'CosineAnnealingWarmRestarts', 'OneCycleLR' ,'GradualWarmupSchedulerV2']\n* Models    : ['deit_base_patch16_384','vit_large_patch16_384','tf_efficientnet_b4_ns','resnext50_32x4d'] it also works for different effnets and resnexts\n* Loss Fns   : ['CutMix' + 'TaylorSmoothed']\n* Augmentations: LightAug+Cutout(Train),LightAug (only rotations), HeavyAug, AutoAugment(imagenet)\n\nAny feedback is appreciated!\nBe sure to also checkout:\n* Inference: https://www.kaggle.com/capiru/cassavanet-inference-tta-easy-submission\n"},{"metadata":{},"cell_type":"markdown","source":"# What is Cutmix?\nCutmix is a form of transformation, which mixes two images on the same one, similar to mixup as shown below, but with only a portion of the original image. Because we are adding information about another label, the label is also adjusted to match proportionally to what is being inserted.\n\nAccording to: https://github.com/clovaai/CutMix-PyTorch\n* Cutmix Outperforms both CutOut and MixUp"},{"metadata":{},"cell_type":"markdown","source":"> ![](https://raw.githubusercontent.com/clovaai/CutMix-PyTorch/master/img1.PNG)"},{"metadata":{},"cell_type":"markdown","source":"Credits:\n* Loss Functions: https://www.kaggle.com/piantic/train-cassava-starter-using-various-loss-funcs/data\n* TaylorSmoothed: https://www.kaggle.com/yerramvarun/cassava-taylorce-loss-label-smoothing-combo\n* GradualWarmup: https://www.kaggle.com/mobassir/faster-pytorch-tpu-baseline-for-cld-cv-0-9#Train-and-eval-function\n* FineTuning: https://www.kaggle.com/piantic/how-to-finetuning-models-pytorch-xla-tpu/data#Dataset\n* CutMix: https://github.com/clovaai/CutMix-PyTorch"},{"metadata":{},"cell_type":"markdown","source":"Best CV results for each fold so far (seed 42):\nTBA\n\n\nInference: https://www.kaggle.com/capiru/cassavanet-inference-tta-easy-submission\n\nCurrent goal is improving other folds. Any suggestions greatly appreciated :)\nInference is coming soon"},{"metadata":{},"cell_type":"markdown","source":"v1 - Initial Release / Debug Run\n\nv2 - First Run - TPU\n\nv3 - Finetuning Model - GPU\n\nv4 - Upon further inspection, CutMix was not behaving as intended. Bug should be fixed, as proof the augmentation visualizations are also fixed now as well. Debug Run.\n\nv5 - Proof of CV > 0.9"},{"metadata":{},"cell_type":"markdown","source":"# Run Config"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Configurations\n# ====================================================\nimport os\nclass CFG:\n    DEBUG = False\n    \n    #Model Params\n    device = 'GPU' #['CPU','GPU','TPU']\n    N_FOLDS = 5\n    MODEL_NAME = 'tf_efficientnet_b4_ns' # Recommended : ['deit_base_patch16_384','vit_large_patch16_384','tf_efficientnet_b4_ns','resnext50_32x4d']\n    pretrained = True\n    N_CLASSES = 5\n    #TRAIN_FOLDS = [0,1,2,3,4]\n    TRAIN_FOLDS = [3] #Folds to be Trained\n    \n    scheduler_name = 'CosineAnnealingWarmRestarts'\n    # ['ReduceLROnPlateau', 'CosineAnnealingLR', 'CosineAnnealingWarmRestarts', 'OneCycleLR', 'GradualWarmupSchedulerV2','LambdaLR']\n    scheduler_update = 'batch' #['batch','epoch']\n    criterion_name = 'TaylorSmoothedLoss'\n    # ['CrossEntropyLoss', 'LabelSmoothing', 'FocalLoss','FocalCosineLoss', 'SymmetricCrossEntropyLoss', 'BiTemperedLoss', 'TaylorCrossEntropyLoss', 'TaylorSmoothedLoss']\n    optimizer_name = 'AdamP' #['Adam','AdamW','AdamP','Ranger'] -> AdamP doesn't work on TPUs\n    LR_RAMPUP_EPOCHS = 1\n    LR_SUSTAIN_EPOCHS = 0\n    \n    FREEZE = False #If you fine tune after START_FREEZE epochs\n    START_FREEZE = 0\n    \n    #Image Size\n    HEIGHT = 512 #If VIT or deit is chosen as model: need 384 x 384\n    WIDTH = 512\n    CHANNELS = 3\n    TRAIN_AUG_TYPE = 'lightaug' #['train','lightaug','heavyaug','autoaugment']\n    VALID_AUG_TYPE = 'valid' #['valid']\n    USE_CUTMIX = True\n    CM_START = 0\n    CM_alpha = 1\n    \n    #Training Params\n    BATCH_SIZE = 16 # PER REPLICA FOR TPUS\n    #RECOMMENDED : effnet = 16 ; resnext = 8 ; vit = 4 ; deit = 4\n    EPOCHS = 20# more is definitely plausible and recommended around 10\n    LR = 1.8e-4\n    LR_START =1e-5\n    LR_MIN = 5e-6\n    weight_decay = 0\n    eps = 1e-8\n    PATIENCE = 4\n    \n    #BiTemperedLoss\n    T1 = 0.2\n    T2 = 1.1\n    LABEL_SMOOTH = 0.05\n    \n    #CosineAnnealingWarmRestarts\n    T_0 = EPOCHS\n    \n    #CosineAnnealingLR\n    T_max = EPOCHS\n    \n    NUM_WORKERS = 4\n    \n    model_print = False #If the model architecture is printed\n    tqdm = True #If training bar is shown\n    \n    IMG_MEAN = [0.485, 0.456, 0.406] #Mean for normalization Transform cassava = [0.4303, 0.4967, 0.3134] imgnet = [0.485, 0.456, 0.406]\n    IMG_STD = [0.229, 0.224, 0.225] #STD for normalization Transform cassava = [0.2142, 0.2191, 0.1954] imgnet = [0.229, 0.224, 0.225]\n    \n    USE_2019 = True #Use 2019 images?\n    \n    #n_procs = number of replicas -> TPU\n    n_procs = 1 #You can set it to 1 and run a TPU as a GPU if you want\n    BGR = False #Alternate method for loading images -> set to true is a bit slower \n    SEED = 32\n    CUTMIX = False","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Required Installations\n# ====================================================\n\n\nif CFG.device == 'TPU':\n    import os\n    !curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n    !python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev\n    os.system('export XLA_USE_BF16=1')\n    os.system('export XLA_TENSOR_ALLOCATOR_MAXSIZE=100000000')\n    import torch_xla.core.xla_model as xm\n    import torch_xla.distributed.parallel_loader as pl\n    import torch_xla.distributed.xla_multiprocessing as xmp\n    import ignite.distributed as idist\n    #CFG.LR = CFG.LR * CFG.n_procs\n    #CFG.BATCH_SIZE = CFG.BATCH_SIZE * CFG.n_procs\n    \n!pip install --quiet timm\n    \nif CFG.optimizer_name == 'Ranger':\n    !pip install --quiet '../input/pytorch-ranger'\nelif CFG.optimizer_name == 'AdamP':\n    !pip install adamp\n    \nif CFG.scheduler_name == 'GradualWarmupSchedulerV2':\n    !pip install git+https://github.com/ildoonet/pytorch-gradual-warmup-lr.git\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Library"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Library\n# ====================================================\nimport random\nimport math\nimport time\n\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom matplotlib.pyplot import imread\nimport numpy as np\nimport cv2\nfrom sklearn.model_selection import GroupKFold, StratifiedKFold\n\nimport torch\nimport torch.nn as nn\nimport torchvision\nfrom torchvision import models as tvmodels\nfrom torch.cuda.amp import autocast, GradScaler\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import transforms, utils\nimport torch.nn.functional as F\nfrom tqdm import tqdm\nimport timm\n\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.model_selection import KFold,StratifiedKFold\n\nimport albumentations as A\nfrom albumentations import Compose\nfrom albumentations.pytorch import ToTensorV2\n\n\n\nfrom PIL import Image, ImageOps, ImageEnhance, ImageChops\n\n\nif CFG.optimizer_name == 'Ranger':\n    from pytorch_ranger import Ranger\nelif CFG.optimizer_name == 'AdamP':\n    from adamp import AdamP\n    \nif CFG.scheduler_name == 'GradualWarmupSchedulerV2':\n    from warmup_scheduler import GradualWarmupScheduler\n\n\n# Ignore warnings\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loss Functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"class LabelSmoothingLoss(nn.Module): \n    def __init__(self, classes=5, smoothing=0.0, dim=-1): \n        super(LabelSmoothingLoss, self).__init__() \n        self.confidence = 1.0 - smoothing \n        self.smoothing = smoothing \n        self.cls = classes \n        self.dim = dim \n    def forward(self, pred, target): \n        if CFG.criterion_name == 'LabelSmoothingLoss':\n            pred = pred.log_softmax(dim=self.dim) \n        with torch.no_grad():\n            true_dist = torch.zeros_like(pred) \n            true_dist.fill_(self.smoothing / (self.cls - 1)) \n            true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence) \n        return torch.mean(torch.sum(-true_dist * pred, dim=self.dim))\n    \nclass TaylorSoftmax(nn.Module):\n    '''\n    This is the autograd version\n    '''\n    def __init__(self, dim=1, n=2):\n        super(TaylorSoftmax, self).__init__()\n        assert n % 2 == 0\n        self.dim = dim\n        self.n = n\n\n    def forward(self, x):\n        '''\n        usage similar to nn.Softmax:\n            >>> mod = TaylorSoftmax(dim=1, n=4)\n            >>> inten = torch.randn(1, 32, 64, 64)\n            >>> out = mod(inten)\n        '''\n        fn = torch.ones_like(x)\n        denor = 1.\n        for i in range(1, self.n+1):\n            denor *= i\n            fn = fn + x.pow(i) / denor\n        out = fn / fn.sum(dim=self.dim, keepdims=True)\n        return out\n    \nclass TaylorCrossEntropyLoss(nn.Module):\n    '''\n    This is the autograd version\n    '''\n    def __init__(self, n=2, ignore_index=-1, reduction='mean'):\n        super(TaylorCrossEntropyLoss, self).__init__()\n        assert n % 2 == 0\n        self.taylor_softmax = TaylorSoftmax(dim=1, n=n)\n        self.reduction = reduction\n        self.ignore_index = ignore_index\n\n    def forward(self, logits, labels):\n        '''\n        usage similar to nn.CrossEntropyLoss:\n            >>> crit = TaylorCrossEntropyLoss(n=4)\n            >>> inten = torch.randn(1, 10, 64, 64)\n            >>> label = torch.randint(0, 10, (1, 64, 64))\n            >>> out = crit(inten, label)\n        '''\n        log_probs = self.taylor_softmax(logits).log()\n        loss = F.nll_loss(log_probs, labels, reduction=self.reduction,\n                ignore_index=self.ignore_index)\n        return loss\n    \nclass TaylorSmoothedLoss(nn.Module):\n\n    def __init__(self, n=2, ignore_index=-1, reduction='mean', smoothing=0.2):\n        super(TaylorSmoothedLoss, self).__init__()\n        assert n % 2 == 0\n        self.taylor_softmax = TaylorSoftmax(dim=1, n=n)\n        self.reduction = reduction\n        self.ignore_index = ignore_index\n        self.lab_smooth = LabelSmoothingLoss(CFG.N_CLASSES, smoothing=CFG.LABEL_SMOOTH)\n\n    def forward(self, logits, labels):\n\n        log_probs = self.taylor_softmax(logits).log()\n        #loss = F.nll_loss(log_probs, labels, reduction=self.reduction,\n        #        ignore_index=self.ignore_index)\n        loss = self.lab_smooth(log_probs, labels)\n        return loss\n    \nclass CutMixCriterion(nn.Module):\n    def __init__(self, criterion):\n        super(CutMixCriterion, self).__init__()\n        self.criterion = criterion\n\n    def forward(self, preds, targets):\n        targets1 = targets[:,0]\n        targets2 = targets[:,1]\n        lam = targets[0,2]\n        return lam * self.criterion.forward(\n            preds, targets1) + (1 - lam) * self.criterion.forward(preds, targets2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Augmentations"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Augmentations\n# ====================================================\nAug_Norm = A.Normalize(mean=CFG.IMG_MEAN, std=CFG.IMG_STD, max_pixel_value=255.0, p=1.0)\nDrop_Rand = A.CoarseDropout(max_holes=12, max_height=int(0.11*CFG.HEIGHT), max_width=int(0.11*CFG.WIDTH),\n                            min_holes=1, min_height=int(0.03*CFG.HEIGHT), min_width=int(0.03*CFG.WIDTH),\n                            always_apply=False, p=0.5)\nRand_Crop = A.RandomCrop(height= CFG.HEIGHT, width = CFG.WIDTH,always_apply=True, p=1.0)\nResize_Crop = A.RandomResizedCrop(CFG.HEIGHT, CFG.WIDTH,p=1.0)\ntrain_transforms = Compose([\n            A.Transpose(p=0.5),\n            A.HorizontalFlip(p=0.5),\n            A.VerticalFlip(p=0.5),\n            A.ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=45, p=0.5),\n            A.HueSaturationValue(\n                hue_shift_limit=0.2, \n                sat_shift_limit=0.2, \n                val_shift_limit=0.2, \n                p=0.5\n            ),\n            A.RandomBrightnessContrast(\n                    brightness_limit=(-0.1,0.1), \n                    contrast_limit=(-0.1, 0.1), \n                    p=0.5\n                ),\n            Resize_Crop,\n            Drop_Rand,           \n            Aug_Norm,   \n            ToTensorV2(p=1.0),\n        ], p=1.)\n\nlight_transforms = Compose([\n            A.Transpose(p=0.5),\n            A.HorizontalFlip(p=0.5),\n            A.VerticalFlip(p=0.5),\n            A.ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=45, p=0.5),\n            A.HueSaturationValue(\n                        hue_shift_limit=0.2, \n                        sat_shift_limit=0.2, \n                        val_shift_limit=0.2, \n                        p=0.5),\n            A.RandomBrightnessContrast(\n                            brightness_limit=(-0.1,0.1), \n                            contrast_limit=(-0.1, 0.1), \n                            p=0.5),\n            Resize_Crop,\n            Aug_Norm,   \n            ToTensorV2(p=1.0),\n        ], p=1.)\n\nheavy_transforms = Compose([\n    A.HorizontalFlip(p=0.5),\n    \n    A.Resize(CFG.HEIGHT, CFG.WIDTH),\n    \n    A.Transpose(p=0.5),\n    A.VerticalFlip(p=0.5),\n    #A.augmentations.transforms.ColorJitter(brightness=0.10, contrast=0.2, saturation=0.2, hue=0.00, always_apply=False, p=0.5),\n    A.ShiftScaleRotate(p=0.5),\n    A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n    A.RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n    A.augmentations.transforms.RGBShift (r_shift_limit=20, g_shift_limit=20, b_shift_limit=20, always_apply=False, p=0.5),\n    A.augmentations.transforms.ChannelDropout (channel_drop_range=(1, 1), fill_value=0, always_apply=False, p=0.5),\n    \n    A.augmentations.transforms.GridDistortion (num_steps=5, distort_limit=0.3, interpolation=1, border_mode=4, value=None, mask_value=None, always_apply=False, p=0.5),\n    A.CoarseDropout(p=0.5),\n    A.Cutout(p=0.5),\n    Aug_Norm,\n    ToTensorV2(p=1.0),])\n\nvalid_transforms = Compose([\n            A.CenterCrop(CFG.HEIGHT, CFG.WIDTH),\n            Aug_Norm,   \n            ToTensorV2(p=1.0),\n        ], p=1.)\n\ntest_aug = Compose([\n            A.HorizontalFlip(p=0.5),\n            A.VerticalFlip(p=0.5),\n            A.ShiftScaleRotate(p = 1.0),\n            #A.ColorJitter(brightness=0.1, contrast=0.2, saturation=0.2, hue=0.00, always_apply=False, p=1.0),\n            Rand_Crop,\n            Aug_Norm,\n            ToTensorV2(p=1.0)\n        ], p=1.)\n\nimage_net_post = Compose([\n            Resize_Crop,\n            Drop_Rand,\n            Aug_Norm,    \n            ToTensorV2(p=1.0)\n        ], p=1.)\n\nclass UnNormalize(object):\n    def __init__(self, mean, std):\n        self.mean = mean\n        self.std = std\n\n    def __call__(self, tensor):\n        \"\"\"\n        Args:\n            tensor (Tensor): Tensor image of size (C, H, W) to be normalized.\n        Returns:\n            Tensor: Normalized image.\n        \"\"\"\n        for t, m, s in zip(tensor, self.mean, self.std):\n            t.mul_(s).add_(m)\n            # The normalize code -> t.sub_(m).div_(s)\n        return tensor","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Utils"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Utils\n# ====================================================\ndef retrieve_df(df,name,idx):\n    series = df[name].iloc[idx]\n    series.reset_index(drop=True,inplace=True)\n    return series\n\ndef accuracy_metric(input, targs):\n    return accuracy_score(targs.cpu(), input.cpu())\n\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True\n    \nif CFG.scheduler_name == 'GradualWarmupSchedulerV2':\n    class GradualWarmupSchedulerV2(GradualWarmupScheduler):\n        def __init__(self, optimizer = None, multiplier = CFG.LR/CFG.LR_START, total_epoch = CFG.LR_RAMPUP_EPOCHS, after_scheduler=None):\n            super(GradualWarmupSchedulerV2, self).__init__(optimizer, multiplier, total_epoch, after_scheduler)\n            self.after_scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0 = CFG.T_0 - CFG.LR_RAMPUP_EPOCHS, T_mult=1, eta_min=CFG.LR_MIN, last_epoch=-1)\n        def get_lr(self):\n            if self.last_epoch > self.total_epoch:\n                if self.after_scheduler:\n                    if not self.finished:\n                        self.after_scheduler.base_lrs = [base_lr * self.multiplier for base_lr in self.base_lrs]\n                        self.finished = True\n                    return self.after_scheduler.get_lr()\n                return [base_lr * self.multiplier for base_lr in self.base_lrs]\n            if self.multiplier == 1.0:\n                return [base_lr * (float(self.last_epoch) / self.total_epoch) for base_lr in self.base_lrs]\n            else:\n                return [base_lr * ((self.multiplier - 1.) * self.last_epoch / self.total_epoch + 1.) for base_lr in self.base_lrs]\n\n#Choose Criterions for the Training Loop\ndef GetCriterion(criterion_name):\n    if criterion_name == 'BiTemperedLoss':\n        criterion = BiTemperedLogistic()\n    elif criterion_name == 'SymmetricCrossEntropyLoss':\n        criterion = SymmetricCrossEntropy()\n    elif criterion_name == 'CrossEntropyLoss':\n        criterion = nn.CrossEntropyLoss()\n    elif criterion_name == 'LabelSmoothingLoss':\n        criterion = LabelSmoothingLoss()\n    elif criterion_name == 'FocalLoss':\n        criterion = FocalLoss()\n    elif criterion_name == 'FocalCosineLoss':\n        criterion = FocalCosineLoss()\n    elif criterion_name == 'TaylorCrossEntropyLoss':\n        criterion = TaylorCrossEntropyLoss()\n    elif criterion_name == 'TaylorSmoothedLoss':\n        criterion = TaylorSmoothedLoss()\n    elif criterion_name == 'CutMix':\n        criterion = CutMixCriterion(GetCriterion(CFG.criterion_name))\n    return criterion\n    \n    \ndef GetScheduler(scheduler_name,optimizer,batches):\n    #['ReduceLROnPlateau', 'CosineAnnealingLR', 'CosineAnnealingWarmRestarts', 'OneCycleLR', 'GradualWarmupSchedulerV2']\n    if scheduler_name == 'OneCycleLR':\n        return torch.optim.lr_scheduler.OneCycleLR(optimizer,max_lr = 1e-2,epochs = CFG.EPOCHS,steps_per_epoch = batches+1,pct_start = 0.1)\n    elif scheduler_name == 'CosineAnnealingWarmRestarts':\n        return torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0 = CFG.T_0, T_mult=1, eta_min=CFG.LR_MIN, last_epoch=-1)\n    elif scheduler_name == 'CosineAnnealingLR':\n        return torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max = CFG.T_max, eta_min=0, last_epoch=-1)\n    elif scheduler_name == 'ReduceLROnPlateau':\n        return torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer,factor=0.1, patience=1, threshold=0.0001, cooldown=0, min_lr=CFG.LR_MIN, eps=CFG.eps)\n    elif scheduler_name == 'GradualWarmupSchedulerV2':\n        return GradualWarmupSchedulerV2(optimizer=optimizer)\n    \ndef GetOptimizer(optimizer_name,parameters):\n    #['Adam','Ranger']\n    if optimizer_name == 'Adam':\n        if CFG.scheduler_name == 'GradualWarmupSchedulerV2':\n            return torch.optim.Adam(parameters, lr=CFG.LR_START, weight_decay=CFG.weight_decay, amsgrad=False)\n        else:\n            return torch.optim.Adam(parameters, lr=CFG.LR, weight_decay=CFG.weight_decay, amsgrad=False)\n    elif optimizer_name == 'AdamW':\n        if CFG.scheduler_name == 'GradualWarmupSchedulerV2':\n            return torch.optim.AdamW(parameters, lr=CFG.LR_START, weight_decay=CFG.weight_decay, amsgrad=False)\n        else:\n            return torch.optim.Adam(parameters, lr=CFG.LR, weight_decay=CFG.weight_decay, amsgrad=False)\n    elif optimizer_name == 'AdamP':\n        if CFG.scheduler_name == 'GradualWarmupSchedulerV2':\n            return AdamP(parameters, lr=CFG.LR_START, weight_decay=CFG.weight_decay)\n        else:\n            return AdamP(parameters, lr=CFG.LR, weight_decay=CFG.weight_decay)\n    elif optimizer_name == 'Ranger':\n        return Ranger(parameters,lr = CFG.LR,alpha = 0.5, k = 6,N_sma_threshhold = 5,betas = (0.95,0.999),eps=CFG.eps,weight_decay=CFG.weight_decay)\n\ndef print_scheduler(scheduler = None,scheduler_update = CFG.scheduler_update,optimizer = None, batches = -1, epochs = -1, model = None):\n    lrs = []\n    if scheduler_update == 'epoch':\n        for epoch in range(epochs):\n            scheduler.step(epoch)\n            lrs.append(optimizer.param_groups[0][\"lr\"])\n        plt.figure(figsize=(15,4))\n        plt.plot(lrs)\n    elif scheduler_update == 'batch':\n        for epoch in range(epochs):\n            for batch in range(batches):\n                scheduler.step()\n                lrs.append(optimizer.param_groups[0][\"lr\"])\n        plt.figure(figsize=(15,4))\n        plt.plot(lrs)\n\ndef cutmix(batch):\n    batch_size = len(batch)\n    data = np.zeros((batch_size,CFG.CHANNELS,CFG.HEIGHT,CFG.WIDTH))\n    targets = np.zeros((batch_size))\n    for i in range(batch_size):\n        data[i,:,:,:] = batch[i][0]\n        targets[i] = batch[i][1]\n\n    indices = torch.randperm(batch_size)\n    shuffled_data = data[indices]\n    shuffled_targets = targets[indices]\n\n    lam = np.random.beta(CFG.CM_alpha, CFG.CM_alpha)\n\n    image_h, image_w = data.shape[2:]\n    cx = np.random.uniform(0, image_w)\n    cy = np.random.uniform(0, image_h)\n    w = image_w * np.sqrt(1 - lam)\n    h = image_h * np.sqrt(1 - lam)\n    x0 = int(np.round(max(cx - w / 2, 0)))\n    x1 = int(np.round(min(cx + w / 2, image_w)))\n    y0 = int(np.round(max(cy - h / 2, 0)))\n    y1 = int(np.round(min(cy + h / 2, image_h)))\n\n    data[:, :, y0:y1, x0:x1] = shuffled_data[:, :, y0:y1, x0:x1]\n    return_targets = torch.zeros((batch_size,3),dtype=torch.int64)\n    return_targets[:,0] = torch.from_numpy(targets)\n    return_targets[:,1] = torch.from_numpy(shuffled_targets)\n    return_targets[0,2] = lam\n\n    return torch.from_numpy(data), return_targets\n        \nclass CutMixCollator:\n    def __call__(self, batch):\n        #batch = torch.utils.data.dataloader.default_collate(batch)\n        batch = cutmix(batch)\n        return batch\n\n    \nSEED = CFG.SEED\nseed_everything(SEED)  \nDEVICE = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Datasets\n# ====================================================\nclass GetData(Dataset):\n    def __init__(self, Dir, FNames, labels,Type):\n        self.dir = Dir\n        self.fnames = FNames\n        self.lbs = labels\n        self.type = Type\n        self.auto_augment = timm.data.auto_augment.auto_augment_transform('originalr',None)\n        \n    def __len__(self):\n        return len(self.fnames)\n    \n    def get_x(self,index):\n        if CFG.BGR:\n            x = cv2.imread(os.path.join(self.dir, self.fnames[index]))\n            x = cv2.cvtColor(x, cv2.COLOR_BGR2RGB)\n        else:\n            x = imread(os.path.join(self.dir, self.fnames[index]))\n        return x\n\n    def __getitem__(self, index):\n        if \"train\" in self.type:\n            x = self.get_x(index)\n            aug_data = train_transforms(image = x)\n            return aug_data['image'], self.lbs[index]\n        elif \"lightaug\" in self.type:\n            x = self.get_x(index)\n            aug_data = light_transforms(image = x)\n            return aug_data['image'], self.lbs[index]\n        elif \"heavyaug\" in self.type:\n            x = self.get_x(index)\n            aug_data = heavy_transforms(image = x)\n            return aug_data['image'], self.lbs[index]\n        elif \"autoaugment\" in self.type:\n            x = Image.open(os.path.join(self.dir, self.fnames[index]))\n            aug_image = self.auto_augment(x)\n            aug_data = image_net_post(image = np.asarray(aug_image,dtype = np.float32))\n            return aug_data['image'], self.lbs[index]\n        elif \"valid\" in self.type:\n            x = self.get_x(index)\n            aug_data = valid_transforms(image = x)\n            return aug_data['image'], self.lbs[index]\n        elif \"tr-tst\" in self.type:\n            x = self.get_x(index)\n            return x, self.lbs[index]\n        elif \"test\" in self.type:\n            x = self.get_x(index)\n            return x, self.fnames[index]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# CV SPLIT"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# CV Split\n# ====================================================\nDATA_PATH = '../input/cassava-leaf-disease-classification/'\nTRAIN_DIR = DATA_PATH + 'train_images/'\n\nDATA_PATH_2019 = '../input/cassava-leaf-disease-merged/'\nTRAIN_DIR_2019 = DATA_PATH_2019 + 'train/'\nTEST_DIR = DATA_PATH + 'test_images/'\n\n#This guarantees that no images from 2019 contaminate the validation split\nif CFG.USE_2019:\n    train_df_merged = pd.read_csv(DATA_PATH_2019 + 'merged.csv')\n    train_df = train_df_merged.loc[train_df_merged.source == 2020]\n    if CFG.DEBUG:\n        train_df = train_df.sample(500).reset_index(drop=True)\n    train_df_2019 = train_df_merged.loc[train_df_merged.source == 2019]\n    skf = StratifiedKFold(n_splits=CFG.N_FOLDS, shuffle=True, random_state=CFG.SEED)\n    skf.get_n_splits(np.arange(train_df.shape[0]), train_df['label'])\n    folds = [(idxT,idxV) for i,(idxT,idxV) in enumerate(skf.split(np.arange(train_df.shape[0]), train_df['label']))]\n    if not CFG.DEBUG:\n        folds_2019 = [np.concatenate((idxT,idxV)) for i,(idxT,idxV) in enumerate(skf.split(np.arange(train_df_2019.shape[0]), train_df_2019['label']))]\n        for i in range(CFG.N_FOLDS):\n            (idxT,idxV) = folds[i]\n            folds[i] = (np.concatenate((idxT,train_df_2019.iloc[folds_2019[i]].index)),idxV)\n            (idxT,idxV) = folds[i]\n            print(np.bincount(train_df_merged['label'].iloc[idxT]),np.bincount(train_df['label'].iloc[idxV]))\n    DATA_FOLD = TRAIN_DIR_2019\n    del train_df_2019\nelse:\n    train_df = pd.read_csv(DATA_PATH + 'train.csv')\n    if CFG.DEBUG:\n        train_df = train_df.sample(500).reset_index(drop=True)\n    skf = StratifiedKFold(n_splits=CFG.N_FOLDS, shuffle=True, random_state=CFG.SEED)\n    skf.get_n_splits(np.arange(train_df.shape[0]), train_df['label'])\n    folds = [(idxT,idxV) for i,(idxT,idxV) in enumerate(skf.split(np.arange(train_df.shape[0]), train_df['label']))]\n    for i in range(CFG.N_FOLDS):\n        (idxT,idxV) = folds[i]\n        print(np.bincount(train_df['label'].iloc[idxT]),np.bincount(train_df['label'].iloc[idxV]))\n    \n    train_df_merged = train_df\n    DATA_FOLD = TRAIN_DIR","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Augmentation Visualization"},{"metadata":{"trusted":true},"cell_type":"code","source":"idx = [i for i in range(16)]\nx_train = retrieve_df(train_df_merged,'image_id',idx)\ny_train = retrieve_df(train_df_merged,'label',idx)\nCFG.collator = CutMixCollator()\ntrain_set = GetData(DATA_FOLD, x_train, y_train, Type = CFG.TRAIN_AUG_TYPE)\ntrain_loader = DataLoader(train_set,batch_size = 16,collate_fn = CFG.collator, shuffle=False,drop_last=True, num_workers=CFG.NUM_WORKERS,pin_memory = True)\nplt.figure(figsize=(10, 10))\ninv_normalize = UnNormalize(mean = CFG.IMG_MEAN,std = CFG.IMG_STD)\nfor i,(image,label) in enumerate(train_loader):\n    #image,a = cutmix((image,label),CFG.CM_alpha)\n    for j in range(16):\n        aug_image = inv_normalize(image[j,:,:,:]).reshape((CFG.CHANNELS,CFG.HEIGHT,CFG.WIDTH))\n        aug_image = np.transpose(aug_image.numpy(),[1,2,0])\n        plt.imshow(aug_image)\n        ax = plt.subplot(4, 4, j+1)\n        label = train_df['label'].iloc[j]\n        plt.title(j)\n        plt.axis(\"off\")\n    if i == 0:\n        break\ndel aug_image,label,image,ax,train_set,train_loader","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Model\n# ====================================================\nclass CassavaNet(nn.Module):\n    def __init__(self, model_name=CFG.MODEL_NAME, pretrained=CFG.pretrained):\n        super().__init__()\n        self.model_name = model_name\n        if model_name == 'deit_base_patch16_224' or model_name == 'deit_base_patch16_384':\n            self.model = torch.hub.load('facebookresearch/deit:main', model_name, pretrained=pretrained)\n        else:\n            self.model = timm.create_model(model_name, pretrained=pretrained)\n        if 'efficientnet' in model_name:\n            self.n_features = self.model.classifier.in_features\n            self.model.classifier = nn.Linear(self.n_features, CFG.N_CLASSES)\n        elif model_name == 'vit_large_patch16_384' or model_name == 'deit_base_patch16_224' or model_name == 'deit_base_patch16_384':\n            self.n_features = self.model.head.in_features\n            self.model.head = nn.Linear(self.n_features, CFG.N_CLASSES)\n        elif 'resnext' in model_name:\n            self.n_features = self.model.fc.in_features\n            self.model.fc = nn.Linear(self.n_features, CFG.N_CLASSES)\n        \n    def forward(self, x):\n        return self.model(x)\n    \n    def freeze(self):\n        # To freeze the residual layers\n        for param in self.model.parameters():\n            param.requires_grad = False\n            \n        if 'efficientnet' in self.model_name:\n            for param in self.model.classifier.parameters():\n                param.requires_grad = True\n        elif self.model_name == 'vit_large_patch16_384' or 'deit_base_patch16_224':\n            for param in self.model.head.parameters():\n                param.requires_grad = True\n        elif 'resnext' in self.model_name:\n            for param in self.model.fc.parameters():\n                param.requires_grad = True\n            \n    def unfreeze(self):\n        # Unfreeze all layers\n        for param in self.model.parameters():\n            param.requires_grad = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = CassavaNet()\nif CFG.model_print:\n    print(model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train Loop"},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_one_epoch(model,optimizer,scheduler,scaler,train_loader,criterion,batches,epoch,DEVICE):   \n    tr_loss = 0.0\n    scores = 0.0\n    trn_epoch_result = dict()\n    model.train()\n    if CFG.tqdm:\n        progress = tqdm(enumerate(train_loader), desc=\"Loss: \", total=len(train_loader))\n    else:\n        progress = enumerate(train_loader)\n    for i, (images,labels) in progress:\n        images = images.to(DEVICE, dtype=torch.float)\n        labels = labels.to(DEVICE)\n        \n        optimizer.zero_grad()\n        \n        if CFG.device == 'TPU':\n            logits = model(images)\n            loss = criterion(logits, labels)\n            loss.backward()\n            xm.optimizer_step(optimizer)\n        else:\n            with autocast():\n                logits = model(images)\n                loss = criterion(logits, labels)\n            scaler.scale(loss).backward()\n            scaler.step(optimizer)\n            scaler.update()\n        \n        preds = F.softmax(logits).argmax(axis = 1)\n        if not CFG.CUTMIX:\n            scores += (preds==labels).sum().cpu().numpy()\n        else:\n            if labels[0,2] >= 0.5:\n                scores += (preds==labels[:,0]).sum().cpu().numpy()\n            else:\n                scores += (preds==labels[:,1]).sum().cpu().numpy()\n\n        \n        if CFG.scheduler_update == 'batch':\n            if not CFG.scheduler_name == 'OneCycleLR':\n                scheduler.step(epoch + i/len(train_loader))\n            else:\n                scheduler.step()\n\n        tr_loss += loss.detach().item()\n        \n        if CFG.tqdm:\n            trn_epoch_result['Epoch'] = epoch\n            trn_epoch_result['train_loss'] = round(tr_loss/(i+1), 4)\n            trn_epoch_result['train_acc'] = round(scores/(i+1)/CFG.BATCH_SIZE, 4)\n            trn_epoch_result['LR'] = round(optimizer.param_groups[0][\"lr\"],7)\n\n            progress.set_description(str(trn_epoch_result))\n        else:\n            print(tr_loss/(i+1))\n    if CFG.scheduler_update == 'epoch':\n            scheduler.step(epoch+1)\n        \ndef val_one_epoch(model,DEVICE,loader,val_criterion,epoch,get_output = False):\n    val_loss = 0.0\n    scores = 0.0\n    model.eval()\n    val_progress = tqdm(enumerate(loader), desc=\"Loss: \", total=len(loader))\n    with torch.no_grad():\n        for i, (images,labels) in val_progress:\n            images = images.to(DEVICE)\n            labels = labels.to(DEVICE)\n\n            logits = model(images)\n            val_loss_value = val_criterion(logits,labels)\n            val_loss += val_loss_value.detach().item()\n\n            preds = F.softmax(logits).argmax(axis = 1)\n            scores += (preds==labels).sum().cpu().numpy()\n\n            val_epoch_result = dict()\n            val_epoch_result['Epoch'] = epoch\n            val_epoch_result['val_loss'] = round(val_loss/(i+1), 4)\n\n            val_epoch_result['val_acc'] = round(scores/(i+1)/CFG.BATCH_SIZE, 4)\n            val_progress.set_description(str(val_epoch_result))\n    if get_output:\n        return val_loss/len(loader),scores/len(loader)/CFG.BATCH_SIZE\n\ndef get_loaders(dev=None,train_set=None,val_set=None):\n    if dev == 'TPU':\n        train_sampler = torch.utils.data.distributed.DistributedSampler(\n            train_set,\n            num_replicas=xm.xrt_world_size(),\n            rank=xm.get_ordinal(),\n            shuffle=True)\n        train_loader = DataLoader(train_set, batch_size=CFG.BATCH_SIZE, sampler=train_sampler,drop_last=True,collate_fn=CFG.collator, num_workers=CFG.NUM_WORKERS)\n\n        val_sampler = torch.utils.data.distributed.DistributedSampler(\n            val_set,\n            num_replicas=xm.xrt_world_size(),\n            rank=xm.get_ordinal(),\n            shuffle=False)\n        val_loader = DataLoader(val_set, batch_size=CFG.BATCH_SIZE, sampler=val_sampler,drop_last=True, num_workers=CFG.NUM_WORKERS)\n        scaler = None\n    else:\n        train_loader = DataLoader(train_set, batch_size=CFG.BATCH_SIZE, shuffle=True,drop_last=True,collate_fn=CFG.collator, num_workers=CFG.NUM_WORKERS,pin_memory = True)\n        val_loader = DataLoader(val_set, batch_size=CFG.BATCH_SIZE, shuffle=False,drop_last=True, num_workers=CFG.NUM_WORKERS,pin_memory = True)\n        scaler = GradScaler()\n    return train_loader,val_loader,scaler\n\ndef model_train():\n    if CFG.device == 'TPU':\n        DEVICE = xm.xla_device()\n    else:\n        DEVICE = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n        \n    for fold,(idxT, idxV) in enumerate(folds):\n        if fold not in CFG.TRAIN_FOLDS:\n            continue\n        #if xm.is_master_ordinal():\n        #    xm.master_print(fold)\n        #______INSTANTIATE TRAINING DATASETS_____\n        x_train = retrieve_df(train_df_merged,'image_id',idxT)\n        y_train = retrieve_df(train_df_merged,'label',idxT)\n        x_val = retrieve_df(train_df_merged,'image_id',idxV)\n        y_val = retrieve_df(train_df_merged,'label',idxV)\n        train_set = GetData(DATA_FOLD, x_train, y_train, Type = CFG.TRAIN_AUG_TYPE)\n        val_set = GetData(DATA_FOLD, x_val, y_val, Type = CFG.VALID_AUG_TYPE)\n        CFG.collator = torch.utils.data.dataloader.default_collate\n        train_loader,val_loader,scaler = get_loaders(dev=CFG.device,train_set=train_set,val_set=val_set)\n            \n        batches = len(train_loader)\n        val_batches = len(val_loader)\n\n        #INSTANTIATE FOLD MODEL\n        if CFG.model is None:\n            if CFG.device == 'TPU':\n                if xm.is_master_ordinal(local=True):\n                    CFG.model = CassavaNet(model_name=CFG.MODEL_NAME, pretrained=True)\n                xm.rendezvous('ModelDone')\n            else:\n                CFG.model = CassavaNet(model_name=CFG.MODEL_NAME, pretrained=True)\n        model = CFG.model.to(DEVICE, dtype=torch.float)\n\n        criterion = GetCriterion(CFG.criterion_name)\n        val_criterion = GetCriterion(CFG.criterion_name)\n\n        optimizer = GetOptimizer(CFG.optimizer_name, model.parameters())\n        scheduler = GetScheduler(CFG.scheduler_name, optimizer,batches)\n        \n        saved_model = None\n        best_val_acc = 0.0\n        best_val_loss = 1e3\n        fold_patience = 0.0\n        for epoch in range(CFG.EPOCHS):\n            if epoch >= CFG.START_FREEZE and CFG.FREEZE:\n                print('Model Frozen -> Train Classifier Only')\n                if saved_model is not None:\n                    info = torch.load(saved_model,map_location = torch.device(DEVICE))\n                    model.load_state_dict(info)\n                model.freeze()\n                \n                CFG.FREEZE = False\n            if epoch>= CFG.CM_START and CFG.USE_CUTMIX:\n                print('Using CutMix')\n                CFG.CUTMIX = True\n                CFG.collator = CutMixCollator()\n                criterion = GetCriterion('CutMix')\n                val_criterion = GetCriterion(CFG.criterion_name)\n                train_loader,val_loader,scaler = get_loaders(dev=CFG.device,train_set=train_set,val_set=val_set)\n                \n                CFG.USE_CUTMIX = False\n            #______TRAINING______\n            if CFG.device == 'TPU':\n                para_train_loader = pl.ParallelLoader(train_loader, [DEVICE])\n                train_one_epoch(model,optimizer,scheduler,scaler,para_train_loader.per_device_loader(DEVICE),criterion,batches,epoch,DEVICE)\n                del para_train_loader\n                gc.collect()\n            else:\n                train_one_epoch(model,optimizer,scheduler,scaler,train_loader,criterion,batches,epoch,DEVICE)\n            \n            #______VALIDATION_______\n            if CFG.device == 'TPU':\n                para_val_loader = pl.ParallelLoader(val_loader, [DEVICE])\n                val_loss, val_acc = val_one_epoch(model,DEVICE,para_val_loader.per_device_loader(DEVICE),val_criterion,epoch,get_output = True)\n                del para_val_loader\n                gc.collect()\n                val_loss = np.sum(idist.all_gather(torch.tensor(val_loss)).to('cpu').numpy())/CFG.n_procs\n                val_acc = np.sum(idist.all_gather(torch.tensor(val_acc)).to('cpu').numpy())/CFG.n_procs\n                xm.master_print(f'Fold Ended at {round(val_acc, 4)} val accuracy')\n            else:\n                val_loss, val_acc = val_one_epoch(model,DEVICE,val_loader,val_criterion,epoch,get_output = True)\n            \n            if val_acc > best_val_acc:\n                fold_patience = 0\n                best_val_loss = val_loss/val_batches\n                best_val_acc = val_acc\n                if CFG.device == 'TPU':\n                    xm.save(CFG.model.state_dict(),\n                                f'{CFG.MODEL_NAME}_f{fold}_TPU_b{round(best_val_acc, 4)}.pth')\n                    if saved_model is not None:\n                        try:\n                            os.remove(\"./\"+saved_model)\n                        except:\n                            a = 1\n                    xm.master_print(f'Model Saved at {round(best_val_acc, 5)} accuracy')\n                    saved_model = f'{CFG.MODEL_NAME}_f{fold}_TPU_b{round(best_val_acc, 4)}.pth'\n                else:\n                    torch.save(model.state_dict(),\n                            f'{CFG.MODEL_NAME}_f{fold}_b{round(best_val_acc, 4)}.pth')\n                    if saved_model is not None:\n                        try:\n                            os.remove(\"./\"+saved_model)\n                        except:\n                            a = 1\n                    saved_model = f'{CFG.MODEL_NAME}_f{fold}_b{round(best_val_acc, 4)}.pth'\n                \n                    print(f'Model Saved at {round(best_val_acc, 5)} accuracy')\n            else:\n                fold_patience += 1\n                if fold_patience >= CFG.PATIENCE:\n                    print(f'Early stopping due to model not improving for {CFG.PATIENCE} epochs')\n                    CFG.model = None\n                    break\n            if torch.cuda.is_available():\n                torch.cuda.empty_cache()\n        CFG.model = None\n                \ndef _map_fn(index,flags):\n    torch.set_default_tensor_type('torch.FloatTensor')\n    a = model_train()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CFG.model = CassavaNet(model_name=CFG.MODEL_NAME, pretrained=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if CFG.device == 'TPU':\n    FLAGS = {}\n    xmp.spawn(_map_fn, args=(FLAGS,), nprocs=CFG.n_procs, start_method='fork')\nelse:\n    torch.set_default_tensor_type('torch.FloatTensor')\n    a = model_train()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}