{"cells":[{"metadata":{},"cell_type":"markdown","source":"![](https://pbs.twimg.com/media/EmpjIGjXEAAtqXK.jpg:large)"},{"metadata":{},"cell_type":"markdown","source":"# Motivation\n\nHi kagglers,\n\ni hope you are doing well in this pandemic time. This is Going to be my 2nd Kaggle Notebook of this year and again i am going to use pytorch tpu for cassava leaf disease classification problem.in my last tpu notebook of this competition [ViT - Pytorch xla (TPU) for leaf disease](https://www.kaggle.com/mobassir/vit-pytorch-xla-tpu-for-leaf-disease?scriptVersionId=51452003)  i tried to convert awesome gpu baseline of [Kun Hao Yeh](https://www.kaggle.com/khyeh0719) into tpu and tried to describe each code cell thoroughly so that anyone can understand but this notebook work is going to be my personal favorite one,why?\n\nBecause In this Notebook i will try to convert @haqishen san's  magnificent [Baseline - Modified From Previous Competition](https://www.kaggle.com/haqishen/baseline-modified-from-previous-competition) which is a very good boilerplate to use in pretty much all future projects, so thank you [Qishen Ha](https://www.kaggle.com/haqishen) san and thank you for all the wisdom you shared, the main reason why i think this notebook work is going to be my personal favorite because we all know that this competition is very similar to last pandas challenge where we had label noise and in that competition my team used this notebook of qishen ha san [Train EfficientNet-B0 w/ 36 tiles_256 [LB0.87]](https://www.kaggle.com/haqishen/train-efficientnet-b0-w-36-tiles-256-lb0-87) and missed silver by only 1 place in private lb :( we tried a lot of different things and lot of different models using that baseline of qishen ha san which helped me get into 51st private lb position. we trained all models on rtx 2080ti and didn't use TPU back then in pandas competition so here in this kernel i am planning to convert that baseline into TPU for faster experiments. i hope it will help those who don't have personal GPU,love pytorch like me,want to do more experiments by investing less time(compared to GPU training) and also for beginners like me who want to learn how to make TPU baseline for a given data problem."},{"metadata":{},"cell_type":"markdown","source":"![](https://cdn-ak.f.st-hatena.com/images/fotolife/t/tereka/20200330/20200330235102.png)"},{"metadata":{},"cell_type":"markdown","source":"# ChangeLog\n"},{"metadata":{},"cell_type":"markdown","source":"* v1 : trying tf_efficientnet_lite4 for first fold only and for 5 epochs\n* v2 : forgot to select TPU hardware in version 1,fixing that issue\n* v3 : training fold 1 for 15 epochs without using any scheduler(failed)\n* v4 : saving weight after each and every epoch,going for 10 epoch single fold for quick check if it fails again or not(failed)\n* v5 : saving weight at the end of fold and using tpu 1.7 version,using scheduler\n* v6 : AdamW and gradual warmup scheduler\n* v7 : trying vit_large_patch16_384 (instead of efficientnet lite models) for image_size = 384 and epochs = 10(failed)\n* v8 : 5 epoch(1 epoch takes only 10 minutes). don't know why kernel failing to finish commit for more epochs when i use xm.save(),if anybody knows the solution then please let me know\n* v9 : running for 10 epoch and trying to save best weight file again inside epoch loop\n* v10 : **bi tempered logistic loss** (failed)\n* v11 : reducing batch size\n* v12 : Fancy augmentation in train phase\n* v13 : [TaylorCE loss + Label Smoothing Combo](https://www.kaggle.com/yerramvarun/cassava-taylorce-loss-label-smoothing-combo) related   [DISCUSSION](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/209782) \n* v14 : smoothing 0.2\n* v15 : tried normalization using datasets mean and std and it made cv worst somehow,using bi tempered logistic loss and more augmentation\n* v16 : error fix\n* v17 : got cv 0.896 on first fold, so now training  other 4 folds with same setting\n* v18 : v17 failed because of SIGKILL (DON'T know the reason),now training fold 1 only\n* v19 : fold 2\n* v20 : fold 3\n* v21 : fold 4 (failed)\n* v22 : fold 4\n* v23 : **fold 3** had lowest validation accuracy,trying to improve that using  Finetuning original model by freezing few layers as shown [here](https://www.kaggle.com/piantic/how-to-finetuning-models-pytorch-xla-tpu) and also applying scheduler after 4th epoch\n* v24 :  scheduler for every epoch (commit stuck)\n* v25 : retrying(stopped)\n* v26 :  SymmetricCrossEntropy loss\n* v27 :  fixing bug in train_model() function,added more augmentations,switching back to bi_tempered_logistic_loss\n* v28 :  it seems in version 27 i got OOM,posted about it [here](https://www.kaggle.com/discussion/214063) , now reducing batch size and retrying (oom again)\n* v29 : reducing augmentation and epoch  \n* v30 : fixing syntaxError\n* v31 : vit_base_patch16_384 long training"},{"metadata":{},"cell_type":"markdown","source":"# install  torch xla nightly"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"\n!python3.7 -m pip install --upgrade pip\n!pip install torch==1.7.0\n!pip install torchvision \n!pip install cloud-tpu-client==0.10 https://storage.googleapis.com/tpu-pytorch/wheels/torch_xla-1.7-cp37-cp37m-linux_x86_64.whl\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true,"_kg_hide-input":false},"cell_type":"code","source":"#!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n#!python pytorch-xla-env-setup.py --version \"nightly\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Installing Gradual Warmup Scheduler"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install git+https://github.com/ildoonet/pytorch-gradual-warmup-lr.git","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Imports"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import os\n\nimport torch_xla\nimport torch_xla.debug.metrics as met\nimport torch_xla.distributed.data_parallel as dp\nimport torch_xla.distributed.parallel_loader as pl\nimport torch_xla.utils.utils as xu\nimport torch_xla.core.xla_model as xm\nimport torch_xla.utils.serialization as xser\nimport torch_xla.distributed.xla_multiprocessing as xmp\nimport torch_xla.test.test_utils as test_utils\n\nimport sys; \n\npackage_paths = [\n    '../input/pytorch-image-models/pytorch-image-models-master', #'../input/efficientnet-pytorch-07/efficientnet_pytorch-0.7.0'\n    '../input/image-fmix/FMix-master'\n]\nfor pth in package_paths:\n    sys.path.append(pth)\n    \n\nimport torch\n\nimport warnings\n\nimport pandas as pd\nimport numpy as np\nimport torch.nn as nn\n\nfrom sklearn.model_selection import train_test_split\n\nfrom sklearn import metrics\nfrom transformers import AdamW\nfrom transformers import get_linear_schedule_with_warmup\n\nimport time\nimport torchvision\nimport torch.nn as nn\nfrom tqdm import tqdm_notebook as tqdm\n\nfrom PIL import Image, ImageFile\n\nimport torch.optim as optim\nfrom torchvision import transforms\nfrom torch.optim import lr_scheduler\n\n\n\nimport gc\n\nimport random\n\nimport skimage.io\n\nfrom PIL import Image\nimport scipy as sp\n\nimport sklearn.metrics\nfrom sklearn.metrics import accuracy_score\n\nfrom functools import partial\n\nfrom torch.utils.data import DataLoader, Dataset\nimport torchvision.models as models\n\nfrom albumentations import Compose, Normalize, HorizontalFlip, VerticalFlip\n\nfrom albumentations import (\n    HorizontalFlip, VerticalFlip, IAAPerspective, ShiftScaleRotate, CLAHE, RandomRotate90,\n    Transpose, ShiftScaleRotate, Blur, OpticalDistortion, GridDistortion, HueSaturationValue,\n    IAAAdditiveGaussianNoise, GaussNoise, MotionBlur, MedianBlur, IAAPiecewiseAffine, RandomResizedCrop,\n    IAASharpen, IAAEmboss, RandomBrightnessContrast, Flip, OneOf, Compose, Normalize, Cutout, CoarseDropout, ShiftScaleRotate, CenterCrop, Resize\n)\n\nfrom albumentations.pytorch import ToTensorV2\n\n\nfrom contextlib import contextmanager\nfrom pathlib import Path\nfrom collections import defaultdict, Counter\n\n\n\nfrom fmix import sample_mask, make_low_freq_image, binarise_mask\n\nfrom glob import glob\nfrom sklearn.model_selection import GroupKFold, StratifiedKFold\nimport cv2\nfrom skimage import io\n\nfrom torch import nn\n\nfrom datetime import datetime\nimport time\nimport random\n\n\nfrom torchvision import transforms\n\nfrom tqdm import tqdm\n\nimport matplotlib.pyplot as plt\n\nfrom torch.utils.data.sampler import SequentialSampler, RandomSampler\n\nfrom torch.nn.modules.loss import _WeightedLoss\nimport torch.nn.functional as F\n\nimport timm\n\nimport sklearn\nimport warnings\nimport joblib\nfrom sklearn.metrics import roc_auc_score, log_loss\nfrom sklearn import metrics\n\n\nimport pydicom\n\nfrom scipy.ndimage.interpolation import zoom\n\n\nimport PIL.Image\n\nimport torch.nn.functional as F\n\nfrom torch.utils.data.sampler import SubsetRandomSampler, RandomSampler, SequentialSampler\nfrom warmup_scheduler import GradualWarmupScheduler\n\nimport albumentations\n\nfrom sklearn.metrics import cohen_kappa_score\nfrom tqdm import tqdm_notebook as tqdm\nfrom pylab import rcParams\n\n#from efficientnet_pytorch import EfficientNet\n\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# version of torch"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(torch.__version__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# For parallelization in TPUs"},{"metadata":{"trusted":true},"cell_type":"code","source":"os.environ[\"XLA_USE_BF16\"] = \"1\"\nos.environ[\"XLA_TENSOR_ALLOCATOR_MAXSIZE\"] = \"100000000\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Config"},{"metadata":{"trusted":true},"cell_type":"code","source":"DEBUG = 0\n\ndata_dir = '../input/cassava-leaf-disease-classification'\ndf_train = pd.read_csv(os.path.join(data_dir, 'train.csv'))\nimage_folder = os.path.join(data_dir, 'train_images')\n\nkernel_type = 'vit_base_patch16_384' \n\nnet_type = 'vit_base_patch16_384'\nfolds = 5\n#fold = 0\nimage_size = 384\nbatch_size = 6\nnum_workers = 4\nout_dim = 5\ninit_lr = 1e-4\nwarmup_factor = 7\nwarmup_epo = 1\n\nsmoothing = 0.3\nt1 = 0.8\nt2 = 1.4\nfreeze=True\n\n\nn_epochs = 1 if DEBUG else 40\ndf_train = df_train.sample(500).reset_index(drop=True) if DEBUG else df_train\n\n#device = torch.device('cuda')\n\nprint(image_folder)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create Folds"},{"metadata":{},"cell_type":"markdown","source":"we will use StratifiedKFold"},{"metadata":{"trusted":true},"cell_type":"code","source":"skf = StratifiedKFold(folds, shuffle=True, random_state=42)\ndf_train['fold'] = -1\nfor i, (train_idx, valid_idx) in enumerate(skf.split(df_train, df_train['label'])):\n    df_train.loc[valid_idx, 'fold'] = i\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataset"},{"metadata":{},"cell_type":"markdown","source":"Custom dataset class that will process our dataset,it returns images and corresponding labels as torch tensor."},{"metadata":{"trusted":true},"cell_type":"code","source":"\nclass LEAFDataset(Dataset):\n    def __init__(self, df, transforms=None):\n\n        self.df = df.reset_index(drop=True)\n        self.transforms = transforms\n\n    def __len__(self):\n        return self.df.shape[0]\n\n    def __getitem__(self, index):\n        row = self.df.iloc[index]\n        img_id = row.image_id\n        \n        image_file = os.path.join(image_folder, img_id)\n        image = cv2.imread(image_file)\n        image = image[:, :, ::-1]\n\n        if self.transforms is not None:\n            image = self.transforms(image=image)['image']\n        image = image.astype(np.float32)\n        image /= 255\n        image = image.transpose(2, 0, 1)\n\n        return torch.tensor(image), torch.tensor(row.label)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Augmentations"},{"metadata":{},"cell_type":"markdown","source":"horizontal flip and resize for training and only resize for validation"},{"metadata":{"trusted":true},"cell_type":"code","source":"transforms_train = albumentations.Compose([\n    #torchvision.transforms.Resize((image_size1, image_size)),\n    albumentations.Resize(image_size, image_size),\n    \n    #albumentations.HorizontalFlip(p=0.5),\n    albumentations.Flip(always_apply=True), # Either Horizontal, Vertical or both flips\n   \n    albumentations.Transpose(p=0.5),\n    #albumentations.VerticalFlip(p=0.5),\n    #albumentations.augmentations.transforms.ColorJitter(brightness=0.10, contrast=0.2, saturation=0.2, hue=0.00, always_apply=False, p=0.5),\n    #albumentations.ShiftScaleRotate(p=0.5),\n    #albumentations.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n    #albumentations.RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n    #albumentations.augmentations.transforms.GridDistortion(num_steps=5, distort_limit=0.3, p=0.5),\n    albumentations.CoarseDropout(max_holes=12, max_height=4, max_width=4, always_apply=False, p=0.5),\n    #CustomCutout(p=1),\n    #albumentations.Cutout(num_holes=5, max_h_size=24, max_w_size=24, fill_value=0, always_apply=False, p=0.5),\n    #albumentations.Normalize(mean=[0.4303, 0.4967, 0.3134], std=[0.2142, 0.2191, 0.1954], max_pixel_value=255.0, p=1.0),\n    \n])\ntransforms_val = albumentations.Compose([\n    #torchvision.transforms.Resize((image_size1, image_size)),\n    albumentations.Resize(image_size, image_size),\n\n    #albumentations.Normalize(mean=[0.4303, 0.4967, 0.3134], std=[0.2142, 0.2191, 0.1954], max_pixel_value=255.0, p=1.0),\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualizing dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_show = LEAFDataset(df_train, transforms=transforms_train)\nrcParams['figure.figsize'] = 20,10\nfor i in range(2):\n    f, axarr = plt.subplots(1,5)\n    for p in range(5):\n        idx = np.random.randint(0, len(dataset_show))\n        img, label = dataset_show[idx]\n        axarr[p].imshow(img.transpose(0, 1).transpose(1,2).squeeze())\n        axarr[p].set_title(str(label))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"class cassavamodel(nn.Module):\n    def __init__(self, model_arch, n_class, pretrained=True):\n        super().__init__()\n        self.model = timm.create_model(model_arch, pretrained=pretrained)\n        #print(self.model)\n        n_features = self.model.head.in_features\n        self.model.head = nn.Linear(n_features, n_class)\n\n    def forward(self, x):\n        x = self.model(x)\n        return x\n    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"''' \n!pip install efficientnet_pytorch\nfrom efficientnet_pytorch import EfficientNet\n\nmodel = EfficientNet.from_name('efficientnet-b2')\n\n#model.avg_pool = nn.AdaptiveAvgPool2d(1)\nnum_ftrs = model._fc.in_features\nmodel._fc = nn.Linear(num_ftrs, out_dim)\n'''","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loss Function"},{"metadata":{},"cell_type":"markdown","source":"we'll use [**bi tempered logistic loss**](https://arxiv.org/pdf/1906.03361.pdf)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Code taken from https://github.com/fhopfmueller/bi-tempered-loss-pytorch/blob/master/bi_tempered_loss_pytorch.py\n\ndef log_t(u, t):\n    \"\"\"Compute log_t for `u'.\"\"\"\n    if t==1.0:\n        return u.log()\n    else:\n        return (u.pow(1.0 - t) - 1.0) / (1.0 - t)\n\ndef exp_t(u, t):\n    \"\"\"Compute exp_t for `u'.\"\"\"\n    if t==1:\n        return u.exp()\n    else:\n        return (1.0 + (1.0-t)*u).relu().pow(1.0 / (1.0 - t))\n\ndef compute_normalization_fixed_point(activations, t, num_iters):\n\n    \"\"\"Returns the normalization value for each example (t > 1.0).\n    Args:\n      activations: A multi-dimensional tensor with last dimension `num_classes`.\n      t: Temperature 2 (> 1.0 for tail heaviness).\n      num_iters: Number of iterations to run the method.\n    Return: A tensor of same shape as activation with the last dimension being 1.\n    \"\"\"\n    mu, _ = torch.max(activations, -1, keepdim=True)\n    normalized_activations_step_0 = activations - mu\n\n    normalized_activations = normalized_activations_step_0\n\n    for _ in range(num_iters):\n        logt_partition = torch.sum(\n                exp_t(normalized_activations, t), -1, keepdim=True)\n        normalized_activations = normalized_activations_step_0 * \\\n                logt_partition.pow(1.0-t)\n\n    logt_partition = torch.sum(\n            exp_t(normalized_activations, t), -1, keepdim=True)\n    normalization_constants = - log_t(1.0 / logt_partition, t) + mu\n\n    return normalization_constants\n\ndef compute_normalization_binary_search(activations, t, num_iters):\n\n    \"\"\"Returns the normalization value for each example (t < 1.0).\n    Args:\n      activations: A multi-dimensional tensor with last dimension `num_classes`.\n      t: Temperature 2 (< 1.0 for finite support).\n      num_iters: Number of iterations to run the method.\n    Return: A tensor of same rank as activation with the last dimension being 1.\n    \"\"\"\n\n    mu, _ = torch.max(activations, -1, keepdim=True)\n    normalized_activations = activations - mu\n\n    effective_dim = \\\n        torch.sum(\n                (normalized_activations > -1.0 / (1.0-t)).to(torch.int32),\n            dim=-1, keepdim=True).to(activations.dtype)\n\n    shape_partition = activations.shape[:-1] + (1,)\n    lower = torch.zeros(shape_partition, dtype=activations.dtype, device=activations.device)\n    upper = -log_t(1.0/effective_dim, t) * torch.ones_like(lower)\n\n    for _ in range(num_iters):\n        logt_partition = (upper + lower)/2.0\n        sum_probs = torch.sum(\n                exp_t(normalized_activations - logt_partition, t),\n                dim=-1, keepdim=True)\n        update = (sum_probs < 1.0).to(activations.dtype)\n        lower = torch.reshape(\n                lower * update + (1.0-update) * logt_partition,\n                shape_partition)\n        upper = torch.reshape(\n                upper * (1.0 - update) + update * logt_partition,\n                shape_partition)\n\n    logt_partition = (upper + lower)/2.0\n    return logt_partition + mu\n\nclass ComputeNormalization(torch.autograd.Function):\n    \"\"\"\n    Class implementing custom backward pass for compute_normalization. See compute_normalization.\n    \"\"\"\n    @staticmethod\n    def forward(ctx, activations, t, num_iters):\n        if t < 1.0:\n            normalization_constants = compute_normalization_binary_search(activations, t, num_iters)\n        else:\n            normalization_constants = compute_normalization_fixed_point(activations, t, num_iters)\n\n        ctx.save_for_backward(activations, normalization_constants)\n        ctx.t=t\n        return normalization_constants\n\n    @staticmethod\n    def backward(ctx, grad_output):\n        activations, normalization_constants = ctx.saved_tensors\n        t = ctx.t\n        normalized_activations = activations - normalization_constants \n        probabilities = exp_t(normalized_activations, t)\n        escorts = probabilities.pow(t)\n        escorts = escorts / escorts.sum(dim=-1, keepdim=True)\n        grad_input = escorts * grad_output\n        \n        return grad_input, None, None\n\ndef compute_normalization(activations, t, num_iters=5):\n    \"\"\"Returns the normalization value for each example. \n    Backward pass is implemented.\n    Args:\n      activations: A multi-dimensional tensor with last dimension `num_classes`.\n      t: Temperature 2 (> 1.0 for tail heaviness, < 1.0 for finite support).\n      num_iters: Number of iterations to run the method.\n    Return: A tensor of same rank as activation with the last dimension being 1.\n    \"\"\"\n    return ComputeNormalization.apply(activations, t, num_iters)\n\ndef tempered_sigmoid(activations, t, num_iters = 5):\n    \"\"\"Tempered sigmoid function.\n    Args:\n      activations: Activations for the positive class for binary classification.\n      t: Temperature tensor > 0.0.\n      num_iters: Number of iterations to run the method.\n    Returns:\n      A probabilities tensor.\n    \"\"\"\n    internal_activations = torch.stack([activations,\n        torch.zeros_like(activations)],\n        dim=-1)\n    internal_probabilities = tempered_softmax(internal_activations, t, num_iters)\n    return internal_probabilities[..., 0]\n\n\ndef tempered_softmax(activations, t, num_iters=5):\n    \"\"\"Tempered softmax function.\n    Args:\n      activations: A multi-dimensional tensor with last dimension `num_classes`.\n      t: Temperature > 1.0.\n      num_iters: Number of iterations to run the method.\n    Returns:\n      A probabilities tensor.\n    \"\"\"\n    if t == 1.0:\n        return activations.softmax(dim=-1)\n\n    normalization_constants = compute_normalization(activations, t, num_iters)\n    return exp_t(activations - normalization_constants, t)\n\ndef bi_tempered_binary_logistic_loss(activations,\n        labels,\n        t1,\n        t2,\n        label_smoothing = 0.0,\n        num_iters=5,\n        reduction='mean'):\n\n    \"\"\"Bi-Tempered binary logistic loss.\n    Args:\n      activations: A tensor containing activations for class 1.\n      labels: A tensor with shape as activations, containing probabilities for class 1\n      t1: Temperature 1 (< 1.0 for boundedness).\n      t2: Temperature 2 (> 1.0 for tail heaviness, < 1.0 for finite support).\n      label_smoothing: Label smoothing\n      num_iters: Number of iterations to run the method.\n    Returns:\n      A loss tensor.\n    \"\"\"\n    internal_activations = torch.stack([activations,\n        torch.zeros_like(activations)],\n        dim=-1)\n    internal_labels = torch.stack([labels.to(activations.dtype),\n        1.0 - labels.to(activations.dtype)],\n        dim=-1)\n    return bi_tempered_logistic_loss(internal_activations, \n            internal_labels,\n            t1,\n            t2,\n            label_smoothing = label_smoothing,\n            num_iters = num_iters,\n            reduction = reduction)\n\ndef bi_tempered_logistic_loss(activations,\n        labels,\n        t1,\n        t2,\n        label_smoothing=0.0,\n        num_iters=5,\n        reduction = 'mean'):\n\n    \"\"\"Bi-Tempered Logistic Loss.\n    Args:\n      activations: A multi-dimensional tensor with last dimension `num_classes`.\n      labels: A tensor with shape and dtype as activations (onehot), \n        or a long tensor of one dimension less than activations (pytorch standard)\n      t1: Temperature 1 (< 1.0 for boundedness).\n      t2: Temperature 2 (> 1.0 for tail heaviness, < 1.0 for finite support).\n      label_smoothing: Label smoothing parameter between [0, 1). Default 0.0.\n      num_iters: Number of iterations to run the method. Default 5.\n      reduction: ``'none'`` | ``'mean'`` | ``'sum'``. Default ``'mean'``.\n        ``'none'``: No reduction is applied, return shape is shape of\n        activations without the last dimension.\n        ``'mean'``: Loss is averaged over minibatch. Return shape (1,)\n        ``'sum'``: Loss is summed over minibatch. Return shape (1,)\n    Returns:\n      A loss tensor.\n    \"\"\"\n\n    if len(labels.shape)<len(activations.shape): #not one-hot\n        labels_onehot = torch.zeros_like(activations)\n        labels_onehot.scatter_(1, labels[..., None], 1)\n    else:\n        labels_onehot = labels\n\n    if label_smoothing > 0:\n        num_classes = labels_onehot.shape[-1]\n        labels_onehot = ( 1 - label_smoothing * num_classes / (num_classes - 1) ) \\\n                * labels_onehot + \\\n                label_smoothing / (num_classes - 1)\n\n    probabilities = tempered_softmax(activations, t2, num_iters)\n\n    loss_values = labels_onehot * log_t(labels_onehot + 1e-10, t1) \\\n            - labels_onehot * log_t(probabilities, t1) \\\n            - labels_onehot.pow(2.0 - t1) / (2.0 - t1) \\\n            + probabilities.pow(2.0 - t1) / (2.0 - t1)\n    loss_values = loss_values.sum(dim = -1) #sum over classes\n\n    if reduction == 'none':\n        return loss_values\n    if reduction == 'sum':\n        return loss_values.sum()\n    if reduction == 'mean':\n        return loss_values.mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n[**Taylor Cross Entropy Loss Implementation**](https://www.kaggle.com/yerramvarun/cassava-taylorce-loss-label-smoothing-combo)"},{"metadata":{},"cell_type":"markdown","source":"Discussion : https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/209782"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\n# implementations reference - https://github.com/CoinCheung/pytorch-loss/blob/master/pytorch_loss/taylor_softmax.py\n# paper - https://www.ijcai.org/Proceedings/2020/0305.pdf\n\nclass TaylorSoftmax(nn.Module):\n\n    def __init__(self, dim=1, n=2):\n        super(TaylorSoftmax, self).__init__()\n        assert n % 2 == 0\n        self.dim = dim\n        self.n = n\n\n    def forward(self, x):\n        \n        fn = torch.ones_like(x)\n        denor = 1.\n        for i in range(1, self.n+1):\n            denor *= i\n            fn = fn + x.pow(i) / denor\n        out = fn / fn.sum(dim=self.dim, keepdims=True)\n        return out\n\nclass LabelSmoothingLoss(nn.Module):\n\n    def __init__(self, classes, smoothing=0.0, dim=-1): \n        super(LabelSmoothingLoss, self).__init__() \n        self.confidence = 1.0 - smoothing \n        self.smoothing = smoothing \n        self.cls = classes \n        self.dim = dim \n    def forward(self, pred, target): \n        \"\"\"Taylor Softmax and log are already applied on the logits\"\"\"\n        #pred = pred.log_softmax(dim=self.dim) \n        with torch.no_grad(): \n            true_dist = torch.zeros_like(pred) \n            true_dist.fill_(self.smoothing / (self.cls - 1)) \n            true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence) \n        return torch.mean(torch.sum(-true_dist * pred, dim=self.dim))\n    \n\nclass TaylorCrossEntropyLoss(nn.Module):\n\n    def __init__(self, n=2, ignore_index=-1, reduction='mean', smoothing=0.2):\n        super(TaylorCrossEntropyLoss, self).__init__()\n        assert n % 2 == 0\n        self.taylor_softmax = TaylorSoftmax(dim=1, n=n)\n        self.reduction = reduction\n        self.ignore_index = ignore_index\n        self.lab_smooth = LabelSmoothingLoss(out_dim, smoothing=smoothing)\n\n    def forward(self, logits, labels):\n\n        log_probs = self.taylor_softmax(logits).log()\n        loss = F.nll_loss(log_probs, labels, reduction=self.reduction,\n                ignore_index=self.ignore_index)\n        #loss = self.lab_smooth(log_probs, labels)\n        return loss\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"criterion = TaylorCrossEntropyLoss(n=2, smoothing = smoothing )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# SymmetricCrossEntropy Loss"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nimport torch.nn.functional as F\n#https://github.com/HanxunH/SCELoss-Reproduce/blob/master/loss.py\n\nclass SymmetricCrossEntropy(torch.nn.Module):\n    def __init__(self, alpha = 0.1, beta=1.0, num_classes=5):\n        super(SymmetricCrossEntropy, self).__init__()\n        self.device = 'cuda' if torch.cuda.is_available() else 'cpu'\n        self.alpha = alpha\n        self.beta = beta\n        self.num_classes = num_classes\n        self.cross_entropy = torch.nn.CrossEntropyLoss()\n\n    def forward(self, pred, labels):\n        # CCE\n        ce = self.cross_entropy(pred, labels)\n\n        # RCE\n        pred = F.softmax(pred, dim=1)\n        pred = torch.clamp(pred, min=1e-7, max=1.0)\n        label_one_hot = torch.nn.functional.one_hot(labels, self.num_classes).float().to(self.device)\n        label_one_hot = torch.clamp(label_one_hot, min=1e-4, max=1.0)\n        rce = (-1*torch.sum(pred * torch.log(label_one_hot), dim=1))\n\n        # Loss\n        loss = self.alpha * ce + self.beta * rce.mean()\n        return loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train and eval function"},{"metadata":{},"cell_type":"markdown","source":"\n* when we will call **train_epoch()** function, we will be training our model for 1 epoch every time using the train_loader that we are passing to this function for 8 core tpu training,from train_loader we are using batch images and labels and later feeding them to our model,calculating loss,doing backpropagation,taking optimizer step and returning training loss for each epoch\n\n* **val_epoch()** function uses our validation dataloader that we are passing to this function for 8 core training on tpu.it calculates validation loss and validation accuracy using our validation dataset and returns both validation accuracy and validation loss, while calculatin validation accuracy we should do this : accuracy = xm.mesh_reduce('test_accuracy', acc, np.mean),if you don't do this then you won't be able to save best weight file for each fold and xm.save() will hang for forever(i made this silly mistake few days ago while working on this kernel,so keep this in mind)"},{"metadata":{},"cell_type":"markdown","source":"NOTE : Latest Torch nightly and version 1.7 has some issue with  softmax i guess,so i couldn't make qishen ha san's val_epoch() working on tpu. i changed val_epoch() function code after this [discussion](https://github.com/pytorch/xla/issues/2712)  and it is working fine now"},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_epoch(model,device,loader, optimizer):\n \n    model.train()\n    train_loss = []\n    bar = tqdm(loader)\n    for (data, target) in bar:\n        data, target = data.to(device), target.to(device)\n        optimizer.zero_grad()\n        logits = model(data)\n        #lossfn = SymmetricCrossEntropy()\n        #loss = lossfn(logits, target)\n        loss = bi_tempered_logistic_loss(logits, target, t1=t1, t2=t2, label_smoothing=smoothing)\n            \n        loss.backward()\n        xm.optimizer_step(optimizer)\n        loss_np = loss.detach().cpu().numpy()\n        train_loss.append(loss_np)\n        #bar.set_description('loss: %.5f' % (loss_np))\n    return train_loss\n\ndef val_epoch(model,device,loader, get_output=False):\n    model.eval()\n\n    t = time.time()\n\n    val_loss = []\n    sample_num = 0\n    image_preds_all = []\n    image_targets_all = []\n\n    pbar = tqdm(enumerate(loader), total=len(loader))\n    for step, (imgs, image_labels) in pbar:\n        imgs = imgs.to(device).float()\n        image_labels = image_labels.to(device).long()\n        \n        image_preds = model(imgs)   #output = model(input)\n        image_preds_all += [torch.argmax(image_preds, 1).detach().cpu().numpy()]\n        image_targets_all += [image_labels.detach().cpu().numpy()]\n        #lossfn = SymmetricCrossEntropy()\n        #loss = lossfn(image_preds, image_labels)\n        loss = bi_tempered_logistic_loss(image_preds, image_labels, t1=t1, t2=t2, label_smoothing = smoothing)\n           \n        val_loss.append(loss.detach().cpu().numpy())\n        \n        sample_num += image_labels.shape[0]  \n\n       \n    val_loss = np.mean(val_loss)\n    image_preds_all = np.concatenate(image_preds_all)\n    image_targets_all = np.concatenate(image_targets_all)\n    acc = (image_preds_all==image_targets_all).mean()\n\n    accuracy = xm.mesh_reduce('test_accuracy', acc, np.mean)\n    xm.master_print(\"Validation Accuracy = \",accuracy)\n\n    return val_loss,accuracy\n    \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Cassava Model"},{"metadata":{},"cell_type":"markdown","source":"in previous versions of this kernel i tried efficientnet lite models and they performer poorly,couldn't cross 75% validation accuracy,in my previous notebook [ViT - Pytorch xla (TPU) for leaf disease](https://www.kaggle.com/mobassir/vit-pytorch-xla-tpu-for-leaf-disease?scriptVersionId=51559451) i tried ViT base model which performed well so here in this notebook will try **vit_large_patch16_384** model now."},{"metadata":{},"cell_type":"markdown","source":"![](https://www.programmersought.com/images/83/cc5dacb45f0ea361e032b940ca8ed5c3.png)"},{"metadata":{},"cell_type":"markdown","source":"# How it works?\n\n[Visual Transformers (ViT)](https://pypi.org/project/pretrained-vit-pytorch/) are a straightforward application of the transformer architecture to image classification. Even in computer vision, it seems, attention is all you need.\n\nThe ViT architecture works as follows: (1) it considers an image as a 1-dimensional sequence of patches, (2) it prepends a classification token to the sequence, (3) it passes these patches through a transformer encoder (like BERT), (4) it passes the first token of the output of the transformer through a small MLP to obtain the classification logits. ViT is trained on a large-scale dataset (ImageNet-21k) with a huge amount of compute.\n\n![](https://warehouse-camo.ingress.cmh1.psfhosted.org/49345a5e2e5aa356f28f4bbb54b89fccc5505603/68747470733a2f2f7261772e67697468756275736572636f6e74656e742e636f6d2f676f6f676c652d72657365617263682f766973696f6e5f7472616e73666f726d65722f6d61737465722f666967757265312e706e67)"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = cassavamodel(net_type, n_class=out_dim)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Scheduler"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nclass GradualWarmupSchedulerV2(GradualWarmupScheduler):\n    def __init__(self, optimizer, multiplier, total_epoch, after_scheduler=None):\n        super(GradualWarmupSchedulerV2, self).__init__(optimizer, multiplier, total_epoch, after_scheduler)\n    def get_lr(self):\n        if self.last_epoch > self.total_epoch:\n            if self.after_scheduler:\n                if not self.finished:\n                    self.after_scheduler.base_lrs = [base_lr * self.multiplier for base_lr in self.base_lrs]\n                    self.finished = True\n                return self.after_scheduler.get_lr()\n            return [base_lr * self.multiplier for base_lr in self.base_lrs]\n        if self.multiplier == 1.0:\n            return [base_lr * (float(self.last_epoch) / self.total_epoch) for base_lr in self.base_lrs]\n        else:\n            return [base_lr * ((self.multiplier - 1.) * self.last_epoch / self.total_epoch + 1.) for base_lr in self.base_lrs]\n\n\ninit_lr = init_lr * xm.xrt_world_size()\n\noptimizer = optim.Adam(model.parameters(), lr=init_lr)\nscheduler_cosine = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, n_epochs-warmup_epo)\nscheduler_warmup = GradualWarmupSchedulerV2(optimizer, multiplier=10, total_epoch=warmup_epo, after_scheduler=scheduler_cosine)\n\n\n\nlrs = []\nfor epoch in range(1, n_epochs+1):\n    scheduler_warmup.step(epoch-1)\n    lrs.append(optimizer.param_groups[0][\"lr\"])\nrcParams['figure.figsize'] = 20,3\nplt.plot(lrs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# prepare Dataloader and Run Training"},{"metadata":{},"cell_type":"markdown","source":"\n* unfortunately torch xla can't use our **dataset_train and dataset_valid** (that we are preparing below using the **LEAFDataset()** class that we have defined above) directly for 8 core training and that's why we use **DistributedSampler()** for preparing train_sampler and valid_sampler that will appropriately distribute the dataset across the 8 cores.\n\n* using train_sampler and valid_sampler we finally create our train_loader and valid_loader that we will use for 8 core training on pytorch tpu by passing them in train_epoch() and val_epoch() function respectively"},{"metadata":{},"cell_type":"markdown","source":"# Layer freezing in transfer learning"},{"metadata":{},"cell_type":"markdown","source":"we will not freeze all layers like @piantic did [here](https://www.kaggle.com/piantic/how-to-finetuning-models-pytorch-xla-tpu), instead we will [TRAIN SOME LAYERS AND LEAVE OTHER FROZEN](https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088/2)"},{"metadata":{},"cell_type":"markdown","source":"The basic idea is that all models have a function model.children() which returns it’s layers. Within each layer, there are parameters (or weights), which can be obtained using .param() on any children (i.e. layer). Now, every parameter has an attribute called requires_grad which is by default True. True means it will be backpropagrated and hence to freeze a layer you need to set requires_grad to False for all parameters of a layer.inside epoch loop we are freezing  first 3 layers in the total N layers of ViT large model. Hope this helps!\n\n![](https://qph.fs.quoracdn.net/main-qimg-96376d794775a37a272dac2a7a38f29e)"},{"metadata":{"trusted":true},"cell_type":"code","source":"#model = cassavamodel(net_type, n_class=out_dim)\n\ndef train_model():\n    device = xm.xla_device()\n    #global model\n    #model = xm.send_cpu_data_to_device(model, device)\n    model.to(device)\n    for fold in range(folds):\n        if(fold != 3):\n            continue\n\n        optimizer = optim.Adam(model.parameters(), lr=init_lr/warmup_factor)\n        scheduler_cosine = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, n_epochs-warmup_epo)\n        scheduler = GradualWarmupScheduler(optimizer, multiplier=warmup_factor, total_epoch=warmup_epo, after_scheduler=scheduler_cosine)\n\n        train_idx = np.where((df_train['fold'] != fold))[0]\n        valid_idx = np.where((df_train['fold'] == fold))[0]\n   \n        df_this  = df_train.loc[train_idx]\n        df_valid = df_train.loc[valid_idx]\n\n        dataset_train = LEAFDataset(df_this , transforms=transforms_train)\n        dataset_valid = LEAFDataset(df_valid, transforms=transforms_val)\n\n        train_sampler = torch.utils.data.distributed.DistributedSampler(\n            dataset_train,\n            num_replicas=xm.xrt_world_size(),\n            rank=xm.get_ordinal(),\n            shuffle=True)\n\n\n        valid_sampler = torch.utils.data.distributed.DistributedSampler(\n            dataset_valid,\n            num_replicas=xm.xrt_world_size(),\n            rank=xm.get_ordinal(),\n            shuffle=False,\n            )\n\n\n        train_loader = torch.utils.data.DataLoader(\n            dataset=dataset_train,\n            batch_size=batch_size,\n            sampler=train_sampler,\n            drop_last=True,\n            num_workers=num_workers,\n        )\n\n        valid_loader = torch.utils.data.DataLoader(\n            dataset=dataset_valid, \n            batch_size=batch_size,\n            sampler=valid_sampler,\n            drop_last=True,\n            num_workers=num_workers,\n        )\n\n        acc_max = 0.\n        count = 0\n        for epoch in range(1, n_epochs+1):\n            scheduler.step(epoch-1)\n            \n            #https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088/32\n            \n            if epoch < 4:\n                # freeze backbone layers\n                for param in model.parameters():\n                    count +=1\n                    if count < 4: #freezing first 3 layers\n                        param.requires_grad = False\n\n            else:\n                for param in model.parameters():\n                    param.requires_grad = True\n\n            para_loader = pl.ParallelLoader(train_loader, [device])\n\n            train_loss = train_epoch(model,device,para_loader.per_device_loader(device), optimizer)\n            del para_loader\n            gc.collect()\n        \n            para_loader = pl.ParallelLoader(valid_loader, [device])\n\n            val_loss, acc = val_epoch(model,device,para_loader.per_device_loader(device))\n            \n            del para_loader\n            gc.collect()\n            \n            #xm.master_print(device)\n         \n\n            content = time.ctime() + ' ' + f'FOLD -> {fold} --> Epoch {epoch}, lr: {optimizer.param_groups[0][\"lr\"]:.7f}, train loss: {np.mean(train_loss):.5f}, val loss: {np.mean(val_loss):.5f}, acc: {(acc):.5f}'\n\n            with open(f'log_{kernel_type}.txt', 'a') as appender:\n                appender.write(content + '\\n')\n            \n\n            #xm.save(model.state_dict(), os.path.join(f'{kernel_type}_fold{fold}_epoch_{epoch}_val_acc_{acc}.pth'))\n        \n            best_file = f'{kernel_type}_best_fold.pth'\n\n            if acc > acc_max:\n                xm.save(model.state_dict(), os.path.join(best_file))\n                #xser.save(model.state_dict(), os.path.join(best_file))\n                acc_max = acc\n            \n        #xm.save(model.state_dict(), os.path.join(f'{kernel_type}_fold{fold}.pth'))    \n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Start training processes"},{"metadata":{},"cell_type":"markdown","source":"**In order to train the model, we need to spawn the training processes on each of the TPU cores.we will call train_model() function during spawn call**"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"%%time\n\ndef _mp_fn(rank, flags):\n    global acc_list\n    torch.set_default_tensor_type('torch.FloatTensor')\n    res = train_model()\n\nFLAGS={}\nxmp.spawn(_mp_fn, args=(FLAGS,), nprocs=8, start_method='fork')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"cv 0.9 in previous versions of this kernel"},{"metadata":{},"cell_type":"markdown","source":"# Full Training Log"},{"metadata":{"trusted":true},"cell_type":"code","source":"f = open(f'./log_{kernel_type}.txt', \"r\")\n\nprint(f.read())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Thank you for your time!\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}