{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Note on current version:\nAfter trying to solve this problem with both classifiers and regressors, I've worked out that the model can learn much faster when exposed to both losses at the same time.\n\nThis model uses 6 outputs: 5 for the target classes and 1 for the direct kappa regressor. It minimises the sum of both FocalLoss and a continuous version of the kappa score."},{"metadata":{},"cell_type":"markdown","source":"# Finetuning of ImageNet pretrained EfficientNet-B4 for Blindness Detection with PyTorch Ignite\n\nRecently new ConvNets architectures have been proposed in [\"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks\"](https://arxiv.org/pdf/1905.11946.pdf) paper. According to the paper, model's compound scaling starting from a 'good' baseline provides an network that achieves state-of-the-art on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet.\n\n\n\n[Official implementation](https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet) of EfficientNet uses Tensorflow, \nfor our case we will borrow the code from [katsura-jp/efficientnet-pytorch](https://github.com/katsura-jp/efficientnet-pytorch), \n[rwightman/pytorch-image-models](https://github.com/rwightman/pytorch-image-models) and [lukemelas/EfficientNet-PyTorch](https://github.com/lukemelas/EfficientNet-PyTorch/) repositories (kudos to authors!). We will download pretrained weights from [lukemelas/EfficientNet-PyTorch](https://github.com/lukemelas/EfficientNet-PyTorch/) repository.\n\n(sourced from [Ignite](https://github.com/pytorch/ignite/tree/master/examples))\n\n## Network architecture review\nThe architecture of EfficientNet-B0 is the following:\n```\n1 - Stem - Conv3x3|BN|Swish\n\n2 - Blocks - MBConv1, k3x3 \n - MBConv6, k3x3 repeated 2 times\n - MBConv6, k5x5 repeated 2 times\n - MBConv6, k3x3 repeated 3 times\n - MBConv6, k5x5 repeated 3 times\n - MBConv6, k5x5 repeated 4 times\n - MBConv6, k3x3\n totally 16 blocks\n\n3 - Head - Conv1x1|BN|Swish \n - Pooling\n - Dropout\n - FC\n```\n\nwhere \n```\nSwish(x) = x * sigmoid(x)\n```\nand `MBConvX` stands for mobile inverted bottleneck convolution, X - denotes expansion ratio:\n``` \nMBConv1 : \n -> DepthwiseConv|BN|Swish -> SqueezeExcitation -> Conv|BN\n\nMBConv6 : \n -> Conv|BN|Swish -> DepthwiseConv|BN|Swish -> SqueezeExcitation -> Conv|BN\n\nMBConv6+IdentitySkip : \n -.-> Conv|BN|Swish -> DepthwiseConv|BN|Swish -> SqueezeExcitation -> Conv|BN-(+)->\n \\___________________________________________________________________________/\n```"},{"metadata":{"trusted":true},"cell_type":"code","source":"def is_interactive():\n return 'runtime' in get_ipython().config.IPKernelApp.connection_file\n\nprint('Interactive?', is_interactive())\n\n# debug = is_interactive()\ndebug = False\n\n# use mixed float precision for training\nuse_amp = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os, sys, gc\nimport numpy as np\nimport random\nimport torch\nimport ignite\n\ndef seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n\nseed_everything(21)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.__version__, ignite.__version__","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"%%time\nif use_amp:\n try:\n from apex import amp\n except ImportError:\n# !git clone https://github.com/NVIDIA/apex\n# !pip install --no-cache-dir --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" apex/\n !pip install -v --no-cache-dir --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" ../input/nvidia-apex/repository/*\n from apex import amp","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model\n\n\nLet's define some helpful modules:\n- Flatten \n- Swish \n\nThe reason why Swish is not implemented in `torch.nn` can be found [here](https://github.com/pytorch/pytorch/pull/3182).\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nimport torch.nn as nn\n\nclass Swish(nn.Module):\n def forward(self, x):\n return x * torch.sigmoid(x)\n\nclass Flatten(nn.Module):\n def forward(self, x):\n return x.reshape(x.shape[0], -1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's visualize Swish transform vs ReLU:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pylab as plt\n%matplotlib inline\n\nd = torch.linspace(-10.0, 10.0)\ns = Swish()\nres = s(d)\nres2 = torch.relu(d)\n\nplt.title(\"Swish transformation\")\nplt.plot(d.numpy(), res.numpy(), label='Swish')\nplt.plot(d.numpy(), res2.numpy(), label='ReLU')\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's define `SqueezeExcitation` module"},{"metadata":{"trusted":true},"cell_type":"code","source":"class SqueezeExcitation(nn.Module):\n \n def __init__(self, inplanes, se_planes):\n super(SqueezeExcitation, self).__init__()\n self.reduce_expand = nn.Sequential(\n nn.Conv2d(inplanes, se_planes, \n kernel_size=1, stride=1, padding=0, bias=True),\n Swish(),\n nn.Conv2d(se_planes, inplanes, \n kernel_size=1, stride=1, padding=0, bias=True),\n nn.Sigmoid()\n )\n\n def forward(self, x):\n x_se = torch.mean(x, dim=(-2, -1), keepdim=True)\n x_se = self.reduce_expand(x_se)\n return x_se * x\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we can define `MBConv`.\n\n**Note on implementation**: in Tensorflow (and PyTorch ports) convolutions use `SAME` padding option which in PyTorch requires\na specific padding computation and additional operation to apply. We will use built-in padding argument of the convolution."},{"metadata":{"trusted":true},"cell_type":"code","source":"from torch.nn import functional as F\n\nclass MBConv(nn.Module):\n def __init__(self, inplanes, planes, kernel_size, stride, \n expand_rate=1.0, se_rate=0.25, \n drop_connect_rate=0.2):\n super(MBConv, self).__init__()\n\n expand_planes = int(inplanes * expand_rate)\n se_planes = max(1, int(inplanes * se_rate))\n\n self.expansion_conv = None \n if expand_rate > 1.0:\n self.expansion_conv = nn.Sequential(\n nn.Conv2d(inplanes, expand_planes, \n kernel_size=1, stride=1, padding=0, bias=False),\n nn.BatchNorm2d(expand_planes, momentum=0.01, eps=1e-3),\n Swish()\n )\n inplanes = expand_planes\n\n self.depthwise_conv = nn.Sequential(\n nn.Conv2d(inplanes, expand_planes,\n kernel_size=kernel_size, stride=stride, \n padding=kernel_size // 2, groups=expand_planes,\n bias=False),\n nn.BatchNorm2d(expand_planes, momentum=0.01, eps=1e-3),\n Swish()\n )\n\n self.squeeze_excitation = SqueezeExcitation(expand_planes, se_planes)\n \n self.project_conv = nn.Sequential(\n nn.Conv2d(expand_planes, planes, \n kernel_size=1, stride=1, padding=0, bias=False),\n nn.BatchNorm2d(planes, momentum=0.01, eps=1e-3),\n )\n\n self.with_skip = stride == 1\n self.drop_connect_rate = torch.tensor(drop_connect_rate, requires_grad=False)\n \n def _drop_connect(self, x): \n keep_prob = 1.0 - self.drop_connect_rate\n drop_mask = torch.rand(x.shape[0], 1, 1, 1) + keep_prob\n drop_mask = drop_mask.type_as(x)\n drop_mask.floor_()\n return drop_mask * x / keep_prob\n \n def forward(self, x):\n z = x\n if self.expansion_conv is not None:\n x = self.expansion_conv(x)\n\n x = self.depthwise_conv(x)\n x = self.squeeze_excitation(x)\n x = self.project_conv(x)\n \n # Add identity skip\n if x.shape == z.shape and self.with_skip: \n if self.training and self.drop_connect_rate is not None:\n self._drop_connect(x)\n x += z\n return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And finally, we can implement generic `EfficientNet':"},{"metadata":{"trusted":true},"cell_type":"code","source":"from collections import OrderedDict\nimport math\n\n\ndef init_weights(module): \n if isinstance(module, nn.Conv2d): \n nn.init.kaiming_normal_(module.weight, a=0, mode='fan_out')\n elif isinstance(module, nn.Linear):\n init_range = 1.0 / math.sqrt(module.weight.shape[1])\n nn.init.uniform_(module.weight, a=-init_range, b=init_range)\n \n \nclass EfficientNet(nn.Module):\n \n def _setup_repeats(self, num_repeats):\n return int(math.ceil(self.depth_coefficient * num_repeats))\n \n def _setup_channels(self, num_channels):\n num_channels *= self.width_coefficient\n new_num_channels = math.floor(num_channels / self.divisor + 0.5) * self.divisor\n new_num_channels = max(self.divisor, new_num_channels)\n if new_num_channels < 0.9 * num_channels:\n new_num_channels += self.divisor\n return new_num_channels\n\n def __init__(self, num_classes, \n width_coefficient=1.0,\n depth_coefficient=1.0,\n se_rate=0.25,\n dropout_rate=0.2,\n drop_connect_rate=0.2):\n super(EfficientNet, self).__init__()\n \n self.width_coefficient = width_coefficient\n self.depth_coefficient = depth_coefficient\n self.divisor = 8\n \n list_channels = [32, 16, 24, 40, 80, 112, 192, 320, 1280]\n list_channels = [self._setup_channels(c) for c in list_channels]\n \n list_num_repeats = [1, 2, 2, 3, 3, 4, 1]\n list_num_repeats = [self._setup_repeats(r) for r in list_num_repeats] \n \n expand_rates = [1, 6, 6, 6, 6, 6, 6]\n strides = [1, 2, 2, 2, 1, 2, 1]\n kernel_sizes = [3, 3, 5, 3, 5, 5, 3]\n\n # Define stem:\n self.stem = nn.Sequential(\n nn.Conv2d(3, list_channels[0], kernel_size=3, stride=2, padding=1, bias=False),\n nn.BatchNorm2d(list_channels[0], momentum=0.01, eps=1e-3),\n Swish()\n )\n \n # Define MBConv blocks\n blocks = []\n counter = 0\n num_blocks = sum(list_num_repeats)\n for idx in range(7):\n \n num_channels = list_channels[idx]\n next_num_channels = list_channels[idx + 1]\n num_repeats = list_num_repeats[idx]\n expand_rate = expand_rates[idx]\n kernel_size = kernel_sizes[idx]\n stride = strides[idx]\n drop_rate = drop_connect_rate * counter / num_blocks\n \n name = \"MBConv{}_{}\".format(expand_rate, counter)\n blocks.append((\n name,\n MBConv(num_channels, next_num_channels, \n kernel_size=kernel_size, stride=stride, expand_rate=expand_rate, \n se_rate=se_rate, drop_connect_rate=drop_rate)\n ))\n counter += 1\n for i in range(1, num_repeats): \n name = \"MBConv{}_{}\".format(expand_rate, counter)\n drop_rate = drop_connect_rate * counter / num_blocks \n blocks.append((\n name,\n MBConv(next_num_channels, next_num_channels, \n kernel_size=kernel_size, stride=1, expand_rate=expand_rate, \n se_rate=se_rate, drop_connect_rate=drop_rate) \n ))\n counter += 1\n \n self.blocks = nn.Sequential(OrderedDict(blocks))\n \n # Define head\n self.head = nn.Sequential(\n nn.Conv2d(list_channels[-2], list_channels[-1], \n kernel_size=1, bias=False),\n nn.BatchNorm2d(list_channels[-1], momentum=0.01, eps=1e-3),\n Swish(),\n nn.AdaptiveAvgPool2d(1),\n Flatten(),\n nn.Dropout(p=dropout_rate),\n nn.Linear(list_channels[-1], num_classes)\n )\n\n self.apply(init_weights)\n \n def forward(self, x):\n f = self.stem(x)\n f = self.blocks(f)\n y = self.head(f)\n return y","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**All EfficientNet models can be defined using the following parametrization:\n```\n# (width_coefficient, depth_coefficient, resolution, dropout_rate)\n'efficientnet-b0': (1.0, 1.0, 224, 0.2),\n'efficientnet-b1': (1.0, 1.1, 240, 0.2),\n'efficientnet-b2': (1.1, 1.2, 260, 0.3),\n'efficientnet-b3': (1.2, 1.4, 300, 0.3),\n'efficientnet-b4': (1.4, 1.8, 380, 0.4),\n'efficientnet-b5': (1.6, 2.2, 456, 0.4),\n'efficientnet-b6': (1.8, 2.6, 528, 0.5),\n'efficientnet-b7': (2.0, 3.1, 600, 0.5),\n``` \nLet's define and train the third last one: `EfficientNet-B4"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = EfficientNet(num_classes=1000, \n width_coefficient=1.4, depth_coefficient=1.8,\n dropout_rate=0.4)\nresolution = 380\nimg_stats = [[0.485, 0.456, 0.406], [0.229, 0.224, 0.225]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Number of parameters:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def print_num_params(model, display_all_modules=False):\n total_num_params = 0\n for n, p in model.named_parameters():\n num_params = 1\n for s in p.shape:\n num_params *= s\n if display_all_modules: print(\"{}: {}\".format(n, num_params))\n total_num_params += num_params\n print(\"-\" * 50)\n print(\"Total number of parameters: {:.2e}\".format(total_num_params))\n \n\nprint_num_params(model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's compare the number of parameters with some of ResNets:"},{"metadata":{"trusted":true},"cell_type":"code","source":"from torchvision.models.resnet import resnet18, resnet34, resnet50","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print_num_params(resnet18(pretrained=False, num_classes=1000))\nprint_num_params(resnet34(pretrained=False, num_classes=1000))\nprint_num_params(resnet50(pretrained=False, num_classes=1000))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model's graph with Tensorboard\n\nWe can optionally inspect model's graph with the code below. For that we need to install\n`tensorboardX` package.\nOtherwise go directly to the next section."},{"metadata":{"trusted":true},"cell_type":"code","source":"# from tensorboardX.pytorch_graph import graph\n\n# import random\n# from IPython.display import clear_output, Image, display, HTML\n\n\n# def show_graph(graph_def):\n# \"\"\"Visualize TensorFlow graph.\"\"\"\n# if hasattr(graph_def, 'as_graph_def'):\n# graph_def = graph_def.as_graph_def()\n# strip_def = graph_def\n# code = \"\"\"\n# \n# \n# \n# \n# \n#
\n# \"\"\".format(data=repr(str(strip_def)), id='graph'+str(random.randint(0, 1000)))\n\n# iframe = \"\"\"\n# \n# \"\"\".format(code.replace('\"', '"'))\n# display(HTML(iframe))","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"# x = torch.rand(4, 3, 224, 224)\n# graph_def = graph(model, x, operator_export_type='RAW')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Display in Firefox may not work properly. Use Chrome.\n# show_graph(graph_def[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load pretrained weights\n\nLet's load pretrained weights and check the model on a single image."},{"metadata":{"trusted":true},"cell_type":"code","source":"from collections import OrderedDict\n\nmodel_state = torch.load(\"/kaggle/input/efficientnet-pytorch/efficientnet-b4-e116e8b3.pth\")\n\n# A basic remapping is required\nmapping = {\n k: v for k, v in zip(model_state.keys(), model.state_dict().keys())\n}\nmapped_model_state = OrderedDict([\n (mapping[k], v) for k, v in model_state.items()\n])\n\nmodel.load_state_dict(mapped_model_state, strict=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import json\n\nwith open(\"/kaggle/input/efficientnet-pytorch/efficientnet-pytorch/EfficientNet-PyTorch-master/examples/simple/labels_map.txt\", \"r\") as h:\n labels = json.load(h)\n\nfrom PIL import Image\nimport torchvision.transforms as transforms\n\n\nimg = Image.open(\"/kaggle/input/efficientnet-pytorch/efficientnet-pytorch/EfficientNet-PyTorch-master/examples/simple/img.jpg\")\n# Preprocess image\ntfms = transforms.Compose([transforms.Resize([resolution]*2),\n transforms.ToTensor(),\n transforms.Normalize(*img_stats),])\nx = tfms(img).unsqueeze(0)\nprint(x.shape)\n_ = plt.imshow(img, shape=[resolution]*2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Classify\nmodel.eval()\nwith torch.no_grad():\n y_pred = model(x)\n\n# Print predictions\nprint('-----')\nfor idx in torch.topk(y_pred, k=5)[1].squeeze(0).tolist():\n prob = torch.softmax(y_pred, dim=1)[0, idx].item()\n print('{label:<75} ({p:.2f}%)'.format(label=labels[str(idx)], p=prob*100))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dataflow\n\nLet's setup the dataflow:\n- load train and test datasets\n- setup train/test image transforms\n- setup train/test data loaders\n\nAccording to the paper authors borrowed training settings from other publications and the dataflow for CIFAR100 is the following:\n\n- input images to the network during training are resized to the model resolution\n- horizontally flipped randomly and augmented using cutout.\n- each mini-batch contained 256 examples\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"from torchvision.transforms import *\n\nfrom torch.utils.data import Subset\nimport torchvision.utils as vutils\n\nimport pandas as pd\nfrom sklearn.utils import shuffle","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\ndf_test = pd.read_csv('../input/aptos2019-blindness-detection/test.csv')\n\nx = df_train['id_code']\ny = df_train['diagnosis']\n\nx, y = shuffle(x, y)\n_ = y.hist()\n\n# get class stats\nn_classes = int(y.max()+1)\nclass_weights = len(y) / df_train.groupby('diagnosis').count().values.ravel() # we can use this to balance our loss function\nclass_weights *= n_classes / class_weights.sum()\nprint('class_weights:', class_weights.tolist())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ntrain_x, valid_x, train_y, valid_y = train_test_split(x.values, y.values, test_size=0.10, stratify=y, random_state=42)\ntest_x = df_test.id_code.values\n\nif debug:\n train_x, train_y = train_x[:128], train_y[:128]\n valid_x, valid_y = valid_x[:64], valid_y[:64]\n\nprint(train_x.shape)\nprint(train_y.shape)\nprint(valid_x.shape)\nprint(valid_y.shape)\nprint(test_x.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from PIL import Image\n\nclass ImageDataset(torch.utils.data.Dataset):\n\n def __init__(self, root, path_list, targets=None, transform=None, extension='.png'):\n super().__init__()\n self.root = root\n self.path_list = path_list\n self.targets = targets\n self.transform = transform\n self.extension = extension\n if targets is not None:\n assert len(self.path_list) == len(self.targets)\n self.targets = torch.LongTensor(targets)\n\n def __getitem__(self, index):\n \"\"\"\n Args:\n index (int): Index\n Returns:\n tuple: (sample, target) where target is class_index of the target class.\n \"\"\"\n path = self.path_list[index]\n sample = Image.open(os.path.join(self.root, path+self.extension))\n if self.transform is not None:\n sample = self.transform(sample)\n\n if self.targets is not None:\n return sample, self.targets[index]\n else:\n return sample, torch.LongTensor([])\n\n def __len__(self):\n return len(self.path_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from PIL.Image import BICUBIC\n\ntrain_transform = Compose([\n Resize([resolution]*2, BICUBIC),\n ColorJitter(brightness=0.05, contrast=0.05, saturation=0.01, hue=0),\n RandomAffine(degrees=15, translate=(0.01, 0.01), scale=(1.0, 1.25), fillcolor=(0,0,0), resample=BICUBIC),\n RandomHorizontalFlip(),\n# RandomVerticalFlip(),\n ToTensor(),\n Normalize(*img_stats)\n])\n\ntest_transform = Compose([\n Resize([resolution]*2, BICUBIC),\n ToTensor(),\n Normalize(*img_stats)\n])\n\ntrain_dataset = ImageDataset(root='../input/aptos2019-blindness-detection/train_images',\n path_list=train_x, targets=train_y, transform=train_transform)\ntrain_eval_dataset = ImageDataset(root='../input/aptos2019-blindness-detection/train_images',\n path_list=valid_x, targets=valid_y, transform=test_transform)\ntest_dataset = ImageDataset(root='../input/aptos2019-blindness-detection/test_images',\n path_list=test_x, transform=test_transform)\n\nprint(len(train_dataset), len(test_dataset), len(train_eval_dataset))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from torch.utils.data import DataLoader\n\ntrain_batch_size = 32\neval_batch_size = 16 ## optimized for loading speed\nnum_workers = os.cpu_count()\nprint('num_workers:', num_workers)\n\ntrain_loader = DataLoader(train_dataset, batch_size=train_batch_size, num_workers=num_workers, \n shuffle=True, drop_last=True, pin_memory=True)\n\ntest_loader = DataLoader(test_dataset, batch_size=eval_batch_size, num_workers=num_workers, \n shuffle=False, drop_last=False, pin_memory=True)\n\neval_train_loader = DataLoader(train_eval_dataset, batch_size=eval_batch_size, num_workers=num_workers, \n shuffle=False, drop_last=False, pin_memory=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Plot some training images\nbatch = next(iter(train_loader))\n\nplt.figure(figsize=(16, 8))\nplt.axis(\"off\")\nplt.title(\"Training Images\")\n_ = plt.imshow( \n vutils.make_grid(batch[0][:16], padding=2, normalize=True).cpu().numpy().transpose((1, 2, 0))\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"{'mean':batch[0].mean(dim=(0,2,3)), 'std':batch[0].std(dim=(0,2,3))}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Classify prior to fine tunning\nmodel.eval()\nwith torch.no_grad():\n y_pred = model.cuda()(batch[0][:1].cuda())\n\n# Print predictions\nprint('-----')\nfor idx in torch.topk(y_pred, k=5)[1].squeeze(0).tolist():\n prob = torch.softmax(y_pred, dim=1)[0, idx].item()\n print('{label:<75} ({p:.2f}%)'.format(label=labels[str(idx)], p=prob*100))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del batch\ntorch.cuda.empty_cache()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Finetunning model"},{"metadata":{},"cell_type":"markdown","source":"As we are interested to finetune the model to APTOS19, we will replace the classification fully-connected layer (originally for ImageNet-1000)."},{"metadata":{"trusted":true},"cell_type":"code","source":"in_features, out_features = model.head[6].in_features, model.head[6].out_features\nin_features, out_features","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.head[6] = nn.Linear(in_features, n_classes+1) # classification + kappa regressor\nclasses = ('No DR', 'Mild', 'Moderate', 'Severe', 'Proliferative DR')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.head[6].in_features, model.head[6].out_features","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will finetune the model on GPU with AMP fp32/fp16 using nvidia/apex package."},{"metadata":{"trusted":true},"cell_type":"code","source":"assert torch.cuda.is_available()\nassert torch.backends.cudnn.enabled, \"NVIDIA/Apex:Amp requires cudnn backend to be enabled.\"\ntorch.backends.cudnn.benchmark = True\n\ndevice = \"cuda\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = model.to(device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training\n\nLet's setup Focal loss as criterion and SGD as optimizer.\n\nWe will split model parameters into 2 groups: \n\n 1) feature extractor (pretrained weights)\n 2) classifier (random weights)\n\nand define different learning rates for these groups (via learning rate scheduler)."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class FocalLoss(nn.Module):\n def __init__(self, alpha=None, gamma=2., reduction='mean'):\n super().__init__()\n self.alpha = torch.tensor(alpha)\n self.gamma = gamma\n self.reduction = reduction\n\n def forward(self, inputs, targets):\n self.alpha = self.alpha.type(inputs.type(), non_blocking=True) # fix type and device\n\n CE_loss = F.cross_entropy(inputs, targets, reduction='none')\n pt = torch.exp(-CE_loss)\n F_loss = self.alpha[targets] * (1-pt)**self.gamma * CE_loss\n\n if self.reduction == 'sum':\n return F_loss.sum()\n elif self.reduction == 'mean':\n return F_loss.mean()\n return F_loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# activation = lambda y: (n_classes-1) * torch.sigmoid(y)\n# activation = lambda y: (n_classes-1) * (0.5 + 0.5 * y / (1 + y.abs())) # linear sigmoid\nactivation = lambda y: y # no-op\n\ndef cont_kappa(input, targets, activation=None):\n ''' continuos version of quadratic weighted kappa '''\n n = len(targets)\n y = targets.float().unsqueeze(0)\n pred = input.float().squeeze(-1).unsqueeze(0)\n if activation is not None:\n pred = activation(pred)\n wo = (pred - y)**2\n we = (pred - y.t())**2\n return 1 - (n * wo.sum() / we.sum())\n# adapted from keras version: https://www.kaggle.com/ryomiyazaki/keras-simple-implementation-of-qwk-for-regressor","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"kappa_loss = lambda pred, y: 1 - cont_kappa(pred, y) # from 0 to 2 instead of 1 to -1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## test loss\ny = torch.from_numpy(np.random.randint(5, size=6))\npreds = torch.ones_like(y.float()) * 2\nprint(y.tolist())\nprint(preds.tolist())\nprint(activation(preds).tolist())\ncont_kappa(preds, y), kappa_loss(preds, y), F.mse_loss(preds, y.float())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# balance between metric optimisation and classification accuracy\nclass MultiTaskLoss(FocalLoss):\n def __init__(self, alpha=None, gamma=2.0, second_loss=F.mse_loss, second_mult=0.1):\n super().__init__(alpha, gamma)\n self.second_loss = second_loss\n self.second_mult = second_mult\n\n def forward(self, inputs, targets):\n loss = super().forward(inputs[...,:-1], targets) # focal loss\n loss += self.second_mult * self.second_loss(inputs[...,-1], targets.float())\n return loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from itertools import chain\n\nimport torch.optim as optim\nimport torch.nn.functional as F\n\ncriterion = MultiTaskLoss(gamma=2., alpha=class_weights, second_loss=kappa_loss, second_mult=0.5)\nlr = 1e-2 # placeholder only! check the LR schedulers below\n\noptimizer = optim.SGD([\n {\n \"params\": chain(model.stem.parameters(), model.blocks.parameters()),\n \"lr\": lr * 0.1,\n },\n {\n \"params\": model.head[:6].parameters(),\n \"lr\": lr * 0.2,\n }, \n {\n \"params\": model.head[6].parameters(), \n \"lr\": lr\n }], \n momentum=0.99, weight_decay=1e-4, nesterov=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"if use_amp:\n # Initialize Amp\n model, optimizer = amp.initialize(model, optimizer, opt_level=\"O2\", num_losses=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, let's define a single iteration function `update_fn`. This function is then used by `ignite.engine.Engine` to update model while running over the input data."},{"metadata":{"trusted":true},"cell_type":"code","source":"from ignite.utils import convert_tensor\n\n\ndef update_fn(engine, batch):\n x = convert_tensor(batch[0], device=device, non_blocking=True)\n y = convert_tensor(batch[1], device=device, non_blocking=True)\n\n model.train()\n y_pred = model(x)\n\n # Compute loss \n loss = criterion(y_pred, y)\n\n optimizer.zero_grad()\n if use_amp:\n with amp.scale_loss(loss, optimizer, loss_id=0) as scaled_loss:\n scaled_loss.backward()\n else:\n loss.backward()\n optimizer.step()\n\n return {\n \"batchloss\": loss.item(),\n }","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check `update_fn`"},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.cuda.empty_cache()\ngc.collect()\n\ntry:\n batch = next(iter(train_loader))\n res = update_fn(engine=None, batch=batch)\n print(res)\nfinally:\n print('max_memory_allocated:', torch.cuda.max_memory_allocated())\n del batch\n torch.cuda.empty_cache()\n _ = gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's define a trainer and add some practical handlers:\n- log to tensorboard: losses, metrics, lr\n- progress bar\n- models/optimizers checkpointing"},{"metadata":{"trusted":true},"cell_type":"code","source":"from ignite.engine import Engine, Events, create_supervised_evaluator\nfrom ignite.metrics import RunningAverage, Accuracy, Precision, Recall, Loss, TopKCategoricalAccuracy\n\nfrom ignite.contrib.handlers import TensorboardLogger\nfrom ignite.contrib.handlers.tensorboard_logger import OutputHandler, OptimizerParamsHandler","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import cohen_kappa_score, accuracy_score\n\ndef qw_kappa(pred, y): ## quadratic weights\n return cohen_kappa_score(torch.argmax(pred[...,:-1], dim=1).cpu().numpy(),\n y.cpu().numpy(),\n weights='quadratic')\n\ndef cl_accuracy(pred, y):\n return accuracy_score(torch.argmax(pred[...,:-1], dim=1).cpu().numpy(),\n y.cpu().numpy())\n\ntrainer = Engine(update_fn)\n\nmetrics = {\n 'Loss': Loss(criterion),\n 'Accuracy': Loss(cl_accuracy),\n# 'Precision': Precision(average=True),\n# 'Recall': Recall(average=True),\n 'ClKappa': Loss(qw_kappa),\n 'RgKappa': Loss(lambda pred, y: cont_kappa(pred[...,-1], y)),\n}\n\nevaluator = create_supervised_evaluator(model, metrics=metrics, device=device, non_blocking=True)\n\nclass Metrics(RunningAverage):\n def __init__(self, evaluator, output_transform=None, interactive=None):\n super().__init__(alpha=0.9, output_transform=output_transform)\n self._evaluator = evaluator\n self.interactive = interactive\n self.validation_history = {}\n self.loss_history = []\n\n def attach(self, engine, name):\n super().attach(engine, name)\n engine.add_event_handler(Events.EPOCH_COMPLETED, self.run_evaluation)\n \n def compute(self):\n loss = super().compute()\n self.loss_history.append(loss)\n return loss\n\n def run_evaluation(self, engine=None):\n self._evaluator.run(eval_train_loader)\n if self.interactive:\n print(evaluator.state.metrics)\n # save validation_history\n for k,v in self._evaluator.state.metrics.items():\n if k not in self.validation_history.keys():\n self.validation_history[k] = [v]\n else:\n self.validation_history[k].append(v)\n\nhistory = Metrics(evaluator, output_transform=lambda out: out['batchloss'], interactive=is_interactive())\nhistory.attach(trainer, \"batchloss\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from datetime import datetime\n\nlog_path = \"./log\"\ntb_logger = TensorboardLogger(log_dir=log_path)\n\ntb_logger.attach(trainer, \n log_handler=OutputHandler('training', ['batchloss', ]), \n event_name=Events.ITERATION_COMPLETED)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Learning rate scheduling with warm restarts"},{"metadata":{"trusted":true},"cell_type":"code","source":"from ignite.contrib.handlers import CosineAnnealingScheduler, LinearCyclicalScheduler, ParamGroupScheduler\n\nweight_decay = 1e-4\ncycle_mult = 2\nsim_epochs = 31\nepoch_size = len(train_loader)\nlr_sched_params = {'param_name':'lr', 'cycle_size':epoch_size, 'cycle_mult':cycle_mult,\n 'start_value_mult':0.8, 'end_value_mult':0.25}\nmom_sched_params = {'param_name':'momentum', 'cycle_size':epoch_size, 'cycle_mult':cycle_mult,\n 'start_value':0.85, 'end_value':0.96}\nwd_sched_params = {'param_name':'weight_decay', 'cycle_size':epoch_size*sim_epochs, 'cycle_mult':cycle_mult,\n 'start_value':weight_decay/10, 'end_value':weight_decay*10}\n\nlr = 0.85\nbody_sched_params = {**lr_sched_params, 'start_value':lr/10, 'end_value':lr/100}\nbody_sched = CosineAnnealingScheduler(optimizer.param_groups[0], **body_sched_params)\nfeat_sched_params = {**lr_sched_params, 'start_value':lr/5, 'end_value':lr/50}\nfeat_sched = CosineAnnealingScheduler(optimizer.param_groups[1], **feat_sched_params)\nhead_sched_params = {**lr_sched_params, 'start_value':lr, 'end_value':lr/20}\nhead_sched = CosineAnnealingScheduler(optimizer.param_groups[2], **feat_sched_params)\n\nmom_sched = CosineAnnealingScheduler(optimizer, **mom_sched_params)\nwd_sched = CosineAnnealingScheduler(optimizer, **wd_sched_params)\nschedulers = [body_sched, feat_sched, head_sched, mom_sched, wd_sched]\nnames = [\"lr (body)\", \"lr (feat)\", \"lr (head)\", \"momentum\", \"wd\"]\n\nlr_values0 = np.array(body_sched.simulate_values(num_events=epoch_size*sim_epochs, **body_sched_params))\nlr_values1 = np.array(feat_sched.simulate_values(num_events=epoch_size*sim_epochs, **feat_sched_params))\nlr_values2 = np.array(head_sched.simulate_values(num_events=epoch_size*sim_epochs, **head_sched_params))\nwd_values = np.array( wd_sched.simulate_values(num_events=epoch_size*sim_epochs, **wd_sched_params))\nmom_values = np.array(mom_sched.simulate_values(num_events=epoch_size*sim_epochs, **mom_sched_params))\n\nfig = plt.figure(figsize=(16, 4))\nax = plt.subplot()\nplt.title(f\"Cosine annealing with start_value_mult={lr_sched_params['start_value_mult']}\")\nax.plot(lr_values0[:, 0], lr_values0[:, 1], label=names[0])\nax.plot(lr_values1[:, 0], lr_values1[:, 1], label=names[1])\nax.plot(lr_values2[:, 0], lr_values2[:, 1], label=names[2])\nax.plot( wd_values[:, 0], wd_values[:, 1], label=names[4])\nax.set_yscale('log')\nax.set_xlabel('Batches processed')\nax.set_ylabel(\"learning rate\")\nax.legend(frameon=False, loc='upper right')\nax2 = ax.twinx()\nax2.plot(mom_values[:, 0], mom_values[:, 1], label=names[3])\nax2.set_ylabel(\"momentum\")\nax2.legend(frameon=False, loc='lower right')\n# fig.tight_layout()\n_ = ax.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scheduler = ParamGroupScheduler(schedulers=schedulers, names=names)\n\n# Attach single scheduler to the trainer\ntrainer.add_event_handler(Events.ITERATION_STARTED, scheduler)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Log optimizer parameters\ntb_logger.attach(trainer,\n log_handler=OptimizerParamsHandler(optimizer, \"lr\"), \n event_name=Events.EPOCH_STARTED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"from ignite.contrib.handlers import ProgressBar\n\n# Iteration-wise progress bar\npbar = ProgressBar(bar_format=\"\")\npbar.attach(trainer, metric_names=['batchloss',])\n\n# Epoch-wise progress bar with display of training losses\nProgressBar(persist=True, bar_format=\"\").attach(trainer,\n event_name=Events.EPOCH_STARTED,\n closing_event_name=Events.COMPLETED)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's create two evaluators to compute metrics on train/test images and log them to Tensorboard:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Log validation metrics:\ntb_logger.attach(evaluator,\n log_handler=OutputHandler(tag=\"test\",\n metric_names=list(metrics.keys()),\n another_engine=trainer),\n event_name=Events.EPOCH_COMPLETED)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's setup logging and the best model checkpointing:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import logging\n\n# Setup engine & logger\ndef setup_logger(logger):\n handler = logging.StreamHandler()\n formatter = logging.Formatter(\"%(asctime)s %(name)-12s %(levelname)-8s %(message)s\")\n handler.setFormatter(formatter)\n logger.addHandler(handler)\n logger.setLevel(logging.INFO)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from ignite.handlers import ModelCheckpoint, EarlyStopping, TerminateOnNan\n\ntrainer.add_event_handler(Events.ITERATION_COMPLETED, TerminateOnNan())\n\n# Store the best model\ndef default_score_fn(engine):\n score = engine.state.metrics['ClKappa']\n return score\n\nbest_model_handler = ModelCheckpoint(dirname=log_path,\n filename_prefix=\"best\",\n n_saved=10,\n score_name=\"ClKappa\",\n score_function=default_score_fn,\n require_empty=False)\nevaluator.add_event_handler(Events.COMPLETED, best_model_handler, {'model': model, })\n\n# # Add early stopping\n# es_patience = 15\n# es_handler = EarlyStopping(patience=es_patience, score_function=default_score_fn, trainer=trainer)\n# evaluator.add_event_handler(Events.EPOCH_COMPLETED, es_handler)\n# setup_logger(es_handler._logger)\n\n# Clear cuda cache between training/testing\n@trainer.on(Events.EPOCH_COMPLETED)\n@evaluator.on(Events.COMPLETED)\ndef empty_cuda_cache(engine):\n torch.cuda.empty_cache()\n import gc\n gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's freeze the first layers for the first epochs "},{"metadata":{"trusted":true},"cell_type":"code","source":"@trainer.on(Events.EPOCH_STARTED)\ndef turn_on_layers(engine):\n epoch = engine.state.epoch\n if epoch == 1:\n for name, child in model.named_children():\n if name == 'head':\n pbar.log_message(f'training \"{name}\"')\n for param in child.parameters():\n param.requires_grad = True\n else:\n# pbar.log_message(f'\"{name}\" is frozen')\n for param in child.parameters():\n param.requires_grad = False\n if epoch == 2:\n pbar.log_message(f\"Epoch {epoch}: training all layers\")\n for name, child in model.named_children():\n for param in child.parameters():\n param.requires_grad = True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"BatchNorm momentum scheduler"},{"metadata":{"trusted":true},"cell_type":"code","source":"@trainer.on(Events.EPOCH_STARTED)\ndef tweak_bn_momenta(engine):\n epoch = engine.state.epoch\n if epoch <= 6:\n momentum = 0.06 / epoch\n pbar.log_message(f\"setting bn momentum to {momentum}\")\n for module in model.modules():\n if isinstance(module, nn.modules.batchnorm._BatchNorm):\n module.momentum = momentum","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_epochs = 7 if debug else sim_epochs\n\nstate = trainer.run(train_loader, max_epochs=num_epochs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Results on the validation set:"},{"metadata":{"trusted":true},"cell_type":"code","source":"evaluator.state.metrics","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(16, 4))\nax = plt.subplot()\nax.set_yscale('log')\nax.set_xlabel('Batches processed')\nax.set_ylabel(\"loss\")\nax.plot(history.loss_history, label='Train Loss')\nax2 = ax.twinx()\nax2.set_ylabel(\"score\")\nfor k,v in history.validation_history.items():\n iters = np.arange(1, len(v)+1) * len(train_loader)\n if k == 'Loss':\n ax.plot(iters, v, label='Valid Loss')\n else: ax2.plot(iters, v, label=k, ls='--')\nax.legend(frameon=False, loc='upper left')\nax2.legend(frameon=False, loc='lower left')\n_ = ax.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_grad_flow(named_parameters):\n '''Plots the gradients flowing through different layers in the net during training.\n Can be used for checking for possible gradient vanishing / exploding problems.\n Usage: Plug this function in Trainer class after loss.backwards() as \n \"plot_grad_flow(self.model.named_parameters())\" to visualize the gradient flow\n https://discuss.pytorch.org/t/check-gradient-flow-in-network/15063/8#post_10'''\n from matplotlib.lines import Line2D\n ave_grads = []\n max_grads= []\n layers = []\n for n, p in named_parameters:\n if (p.requires_grad) and (\"bias\" not in n):\n layers.append(n)\n ave_grads.append(p.grad.abs().mean())\n max_grads.append(p.grad.abs().max())\n plt.figure(figsize=(len(layers)*2, 8))\n plt.bar(np.arange(len(layers)), max_grads, alpha=0.1, lw=1, color=\"c\")\n plt.bar(np.arange(len(layers)), ave_grads, alpha=0.1, lw=1, color=\"b\")\n plt.hlines(0, 0, len(layers)+1, lw=2, color=\"k\" )\n plt.xticks(range(0, len(layers), 1), layers, rotation=\"vertical\")\n plt.xlim(left=-1, right=len(layers))\n# plt.ylim(bottom = -0.001, top=0.02) # zoom in on the lower gradient regions\n plt.xlabel(\"Layers\")\n plt.ylabel(\"average gradient\")\n plt.yscale('log')\n plt.title(\"Gradient flow\")\n plt.grid(True)\n plt.legend([Line2D([0], [0], color=\"c\", lw=4),\n Line2D([0], [0], color=\"b\", lw=4),\n Line2D([0], [0], color=\"k\", lw=4)], ['max-gradient', 'mean-gradient', 'zero-gradient'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check gradients\nplot_grad_flow([(n,p) for n,p in model.named_parameters() if 'blocks' not in n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":false},"cell_type":"code","source":"# # Load the TensorBoard notebook extension\n# %load_ext tensorboard.notebook\n# %tensorboard --logdir {log_path}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference\n\nLet's load the best model and recompute evaluation metrics on test dataset with a very basic Test-Time-Augmentation to boost the performances.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Find the last checkpoint\n!ls {log_path}\ncheckpoints = next(os.walk(log_path))[2]\ncheckpoints = sorted(filter(lambda f: f.endswith(\".pth\"), checkpoints))\nscores = [c.split('=')[-1][:-4] for c in checkpoints] # look for '*Metric=score.pth'\nbest_epoch = np.argmax(scores)\nprint(best_epoch, scores)\nif not checkpoints:\n print('No weight files in {}'.format(log_path))\nelse:\n model_path = f'efficientNet_{scores[best_epoch]}.pth'\n !cp {os.path.join(log_path, checkpoints[best_epoch])} {model_path}\n\nprint(model_path)\n!rm {log_path}/*.pth","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"best_model = model\nbest_model.load_state_dict(torch.load(model_path))\nbest_model = best_model.cuda().eval()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot some test images\nbatch = next(iter(test_loader))\n\nplt.figure(figsize=(16, 8))\nplt.axis(\"off\")\nplt.title(\"Test Images\")\n_ = plt.imshow(\n vutils.make_grid(batch[0][:16], padding=2, normalize=True).cpu().numpy().transpose((1, 2, 0))\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Classify\nwith torch.no_grad():\n y_pred = best_model(batch[0][:16].cuda())\n\nprint('Regressor activations:')\nprint(activation(y_pred[:16,-1]).reshape([2,8]).cpu(), '\\n')\n\nprint('Predictions for first item:')\nfor idx in torch.topk(y_pred[0,:-1], k=n_classes)[1].squeeze(0).tolist():\n prob = torch.softmax(y_pred, dim=-1)[0, idx].item()\n print('{label:<75} ({p:.2f}%)'.format(label=classes[idx], p=prob*100))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del batch","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Create submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"use_regressor = False\n\ndef inference_update_with_tta(engine, batch, use_regressor=use_regressor):\n global preds, targets\n best_model.eval()\n with torch.no_grad():\n x, y = batch\n x = x.cuda()\n # Let's compute final prediction as a mean of predictions on x and flipped x\n if use_regressor:\n y_pred1 = best_model(x)[...,-1]\n y_pred2 = best_model(x.flip(dims=(-1, )))[...,-1]\n # calc softmax for submission\n curr_pred = (activation(y_pred1) + activation(y_pred2)) * 0.5\n preds += curr_pred.cpu().squeeze().tolist()\n else:\n y_pred1 = best_model(x)[...,:-1]\n y_pred2 = best_model(x.flip(dims=(-1, )))[...,:-1]\n # calc softmax for submission\n curr_pred = F.softmax(y_pred1, dim=-1) + F.softmax(y_pred2, dim=-1)\n preds += curr_pred.argmax(dim=-1).cpu().squeeze().tolist()\n targets += y.cpu().squeeze().tolist()\n return y_pred1, y\n\ninferencer = Engine(inference_update_with_tta)\nProgressBar(desc=\"Inference\").attach(inferencer)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds, targets = [], []\nresult_state = inferencer.run(eval_train_loader, max_epochs=1)\nprint('valid accuracy:', (np.array(preds) == np.array(targets)).mean())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Optimise on metric:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import scipy as sp\n\nclass KappaOptimizer(nn.Module):\n def __init__(self):\n super().__init__()\n self.coef = [0.5, 1.5, 2.5, 3.5]\n # define score function:\n self.func = self.quad_kappa\n \n def predict(self, preds):\n return self._predict(self.coef, preds)\n\n @classmethod\n def _predict(cls, coef, preds):\n if type(preds).__name__ == 'Tensor':\n y_hat = preds.clone().view(-1)\n else:\n y_hat = torch.FloatTensor(preds).view(-1)\n\n for i,pred in enumerate(y_hat):\n if pred < coef[0]: y_hat[i] = 0\n elif pred < coef[1]: y_hat[i] = 1\n elif pred < coef[2]: y_hat[i] = 2\n elif pred < coef[3]: y_hat[i] = 3\n else: y_hat[i] = 4\n return y_hat.int()\n \n def quad_kappa(self, preds, y):\n return self._quad_kappa(self.coef, preds, y)\n\n @classmethod\n def _quad_kappa(cls, coef, preds, y):\n y_hat = cls._predict(coef, preds)\n return cohen_kappa_score(y, y_hat, weights='quadratic')\n\n def fit(self, preds, y):\n ''' maximize quad_kappa '''\n print('Early score:', self.quad_kappa(preds, y))\n neg_kappa = lambda coef: -self._quad_kappa(coef, preds, y)\n opt_res = sp.optimize.minimize(neg_kappa, x0=self.coef, method='nelder-mead',\n options={'maxiter':100, 'fatol':1e-20, 'xatol':1e-20})\n print(opt_res)\n self.coef = opt_res.x\n print('New score:', self.quad_kappa(preds, y))\n\n def forward(self, preds, y):\n ''' the pytorch loss function '''\n return torch.tensor(self.quad_kappa(preds, y))\n\nif use_regressor:\n kappa_opt = KappaOptimizer()\n # fit on validation set\n kappa_opt.fit(preds, targets)\n opt_preds = kappa_opt.predict(preds).tolist()\n\n _ = pd.DataFrame(preds).hist()\n _ = pd.DataFrame(opt_preds).hist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Inference on test set:"},{"metadata":{"trusted":true},"cell_type":"code","source":"preds, targets = [], []\nresult_state = inferencer.run(test_loader, max_epochs=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if use_regressor:\n preds = kappa_opt.predict(preds).tolist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, the submission csv:"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame({'id_code': pd.read_csv('../input/aptos2019-blindness-detection/sample_submission.csv').id_code.values,\n 'diagnosis': np.squeeze(preds).astype(np.int32)})\n\nsubmission.hist()\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"# clean up folders\n!rm -rf apex /tmp/*\n!ls *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Submission in https://www.kaggle.com/hmendonca/efficientnet-pytorch-ignite-aptos19-submission"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"}},"nbformat":4,"nbformat_minor":1}