{"cells":[{"metadata":{},"cell_type":"markdown","source":"Note: The io with original images is awful. Therefore I created a dataset https://www.kaggle.com/arroqc/siic-isic-224x224-images of images preprocessed to 224x224 size and saved in png (lossless).\nIf you train on original jpeg large imagesm I have put the right lines of code in comments with the added comments : # Use this when training with original images","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Pytorch Lightning Starter - SSIM Melanoma competition\n\nI use pytorch lightning both for this competition and in my day to day work. I hope it can serve as a useful tutorial for fellow kagglers.\nWhy use Pytorch-Lightning ?\n\nPytorch lighntning is designed to help you easily follow a pytorch based training loop and ease modifications that you may want. Want to use a new scheduler ? Then simply modify the configure_optimizer method ! The beauty of it is that it automates all the boring stuff that clogs a pure pytorch code. All these loops, .zero_grad(), .eval(), torch.save etc. are gone and handled by the framework. You just have to focus on the ML part of it. The best things for researchers is that it comes with automated logs through tensorboard to compare your many experiments and easy switches between GPU, DataParallel, TPU mixed precision etc. Obviously kaggle is not very friendly with logs so I suggest reproducing the code of this kernel in a local environment and use tensorboard there.\n\nYou may ask why not simply use fastai. This is now a matter of preference. Fastai automates a lot of stuff with best practices like .fit_one_cycle. But on the other hand unless you have a lot of experience with it I find it rather opaque in what is happening behind the scenes. It's a framework designed to go with doing the fastai course so that you understand the options. If like me you learnt deep learning in a more academic environment in pure pytorch or pure tensorflow then you may find fastai hard to understand without listening to J. Howard courses. Similarly as soon as you want to do something a bit different it can become hard to understand how to change anything. On a personal note, I'll wait for the fastai v2 course before delving into it.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's install it as it not in kaggle by default.\n!pip install pytorch-lightning ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## Loading data\n\nFirst let's open the csv. One thing we need to make sure when splitting data in a medical context is to split by patient ID rather than image ID. Otherwise you run the risk of having some data leakage.\n","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import os\nimport random\nimport argparse\nfrom pathlib import Path\nimport PIL.Image as Image\nimport pandas as pd\nimport numpy as np\nimport cv2\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import train_test_split\n\nimport pytorch_lightning as pl\nimport torch.utils.data as tdata\nimport torch\nimport torch.nn as nn\nimport torchvision.models as models\nfrom torchvision import transforms","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reproductibility\nSEED = 33\nrandom.seed(SEED)\nos.environ['PYTHONHASHSEED'] = str(SEED)\nnp.random.seed(SEED)\ntorch.manual_seed(SEED)\ntorch.cuda.manual_seed(SEED)\ntorch.backends.cudnn.deterministic = True\ntorch.backends.cudnn.benchmark = False","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def dict_to_args(d):\n\n    args = argparse.Namespace()\n\n    def dict_to_args_recursive(args, d, prefix=''):\n        for k, v in d.items():\n            if type(v) == dict:\n                dict_to_args_recursive(args, v, prefix=k)\n            elif type(v) in [tuple, list]:\n                continue\n            else:\n                if prefix:\n                    args.__setattr__(prefix + '_' + k, v)\n                else:\n                    args.__setattr__(k, v)\n\n    dict_to_args_recursive(args, d)\n    return args","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CSV_DIR = Path('/kaggle/input/siim-isic-melanoma-classification/')\ntrain_df = pd.read_csv(CSV_DIR/'train.csv')\ntest_df = pd.read_csv(CSV_DIR/'test.csv')\n#IMAGE_DIR = Path('/kaggle/input/siim-isic-melanoma-classification/jpeg')  # Use this when training with original images\nIMAGE_DIR = Path('/kaggle/input/siic-isic-224x224-images/')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.groupby(by=['patient_id'])['image_name'].count()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.groupby(by=['patient_id'])['target'].mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So you have patients that have multiple images. Also apparently the data is imbalanced. Let's verify:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.groupby(['target']).count()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"so we have approx 60 times more negatives than positives. We need to make sure we split good/bad patients equally.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_means = train_df.groupby(['patient_id'])['target'].mean()\npatient_ids = train_df['patient_id'].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now let's make our split\ntrain_idx, val_idx = train_test_split(np.arange(len(patient_ids)), stratify=(patient_means > 0), test_size=0.2)  # KFold + averaging should be much better considering how small the dataset is for malignant cases\npid_train = patient_ids[train_idx]\npid_val = patient_ids[val_idx]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's verify the split was correct","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[train_df['patient_id'].isin(pid_train)].groupby(['target']).count()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[train_df['patient_id'].isin(pid_val)].groupby(['target']).count()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Pytorch Dataset\n A dataset should simply return all the information necessary for a sample by defining the getitem and len magic methods.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class SIIMDataset(tdata.Dataset):\n    \n    def __init__(self, df, transform, test=False):\n        self.df = df\n        self.transform = transform\n        self.test = test\n        \n    def __len__(self):\n        return len(self.df)\n        \n    def __getitem__(self, idx):\n        meta = self.df.iloc[idx]\n        #image_fn = meta['image_name'] + '.jpg'  # Use this when training with original images\n        image_fn = meta['image_name'] + '.png'\n        if self.test:\n            img = Image.open(str(IMAGE_DIR / ('test/' + image_fn)))\n        else:\n            img = Image.open(str(IMAGE_DIR / ('train/' + image_fn)))\n        \n        if self.transform is not None:\n            img = self.transform(img)\n        \n        if self.test:\n            return {'image': img}\n        else:\n            return {'image': img, 'target': meta['target']}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class AdaptiveConcatPool2d(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.avg = nn.AdaptiveAvgPool2d(output_size=(1, 1))\n        self.max = nn.AdaptiveMaxPool2d(output_size=(1, 1))\n\n    def forward(self, x):\n        avg_x = self.avg(x)\n        max_x = self.max(x)\n        return torch.cat([avg_x, max_x], dim=1)\n\n\nclass Flatten(nn.Module):\n    def forward(self, x):\n        return x.view(x.shape[0], -1)\n    \n\nclass Model(nn.Module):\n    \n    def __init__(self, c_out=1, arch='resnet34'):\n        super().__init__()\n        if arch == 'resnet34':\n            remove_range = 2\n            m = models.resnet34(pretrained=True)\n        elif arch == 'seresnext50':\n            m = torch.hub.load('facebookresearch/semi-supervised-ImageNet1K-models', 'resnext50_32x4d_ssl')\n            remove_range = 2\n            \n        c_feature = list(m.children())[-1].in_features\n        self.base = nn.Sequential(*list(m.children())[:-remove_range])\n        self.head = nn.Sequential(AdaptiveConcatPool2d(),\n                                  Flatten(),\n                                  nn.Linear(c_feature * 2, c_out))\n\n        \n    def forward(self, x):\n        h = self.base(x)\n        logits = self.head(h).squeeze(1)\n        return logits","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## Pytorch Lightning module definition\n\nIn a normal pytorch code you probably would instantiate the model, dataloaders and make a nested for loop for epochs and batches. Pytorch lightning automates the engineering parts like the loops so that you focus on the ML part. To do that you create a pytorch lightning model and then define every ML step inside of it. To help you understand I have added comments under every method you need to implement.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn.functional as F\nclass FocalLoss(nn.Module):\n    def __init__(self, alpha=1, gamma=2, logits=False, reduce=True):\n        super(FocalLoss, self).__init__()\n        self.alpha = alpha\n        self.gamma = gamma\n        self.logits = logits\n        self.reduce = reduce\n\n    def forward(self, inputs, targets):\n        if self.logits:\n            BCE_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction='none')\n        else:\n            BCE_loss = F.binary_cross_entropy(inputs, targets, reduction='none')\n        pt = torch.exp(-BCE_loss)\n        F_loss = self.alpha * (1-pt)**self.gamma * BCE_loss\n\n        if self.reduce:\n            return torch.mean(F_loss)\n        else:\n            return F_loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class LightModel(pl.LightningModule):\n\n    def __init__(self, df_train, df_test, pid_train, pid_val, hparams):\n        # This is where paths and options should be stored. I also store the\n        # train_idx, val_idx for cross validation since the dataset are defined \n        # in the module !\n        super().__init__()\n        self.pid_train = pid_train\n        self.pid_val = pid_val\n        self.df_train = df_train\n\n        self.model = Model(arch=hparams.arch)  # You will obviously want to make the model better :)\n\n        self.hparams = hparams\n        \n        # Defining datasets here instead of in prepare_data usually solves a lot of problems for me...\n        self.transform_train = transforms.Compose([#transforms.Resize((224, 224)),   # Use this when training with original images\n                                              transforms.RandomHorizontalFlip(0.5),\n                                              transforms.RandomVerticalFlip(0.5),\n                                              transforms.ToTensor(),\n                                              transforms.Normalize(mean=[0.485, 0.456, 0.406],\n                                                                   std=[0.229, 0.224, 0.225])])\n        self.transform_test = transforms.Compose([#transforms.Resize((224, 224)),   # Use this when training with original images\n                                             transforms.ToTensor(),\n                                             transforms.Normalize(mean=[0.485, 0.456, 0.406],\n                                                                  std=[0.229, 0.224, 0.225])])\n        self.trainset = SIIMDataset(self.df_train[self.df_train['patient_id'].isin(pid_train)], self.transform_train)\n        self.valset = SIIMDataset(self.df_train[self.df_train['patient_id'].isin(pid_val)], self.transform_test)\n        self.testset = SIIMDataset(df_test, self.transform_test, test=True)\n\n    def forward(self, batch):\n        # What to do with a batch in a forward. Usually simple if everything is already defined in the model.\n        return self.model(batch['image'])\n\n    def prepare_data(self):\n        # This is called at the start of training\n        pass\n\n    def train_dataloader(self):\n        # Simply define a pytorch dataloader here that will take care of batching. Note it works well with dictionnaries !\n        train_dl = tdata.DataLoader(self.trainset, batch_size=self.hparams.batch_size, shuffle=True,\n                                    num_workers=os.cpu_count())\n        return train_dl\n\n    def val_dataloader(self):\n        # Same but for validation. Pytorch lightning allows multiple validation dataloaders hence why I return a list.\n        val_dl = tdata.DataLoader(self.valset, batch_size=self.hparams.batch_size, shuffle=False,\n                                  num_workers=os.cpu_count()) \n        return [val_dl]\n    \n    def test_dataloader(self):\n        test_dl = tdata.DataLoader(self.testset, batch_size=self.hparams.batch_size, shuffle=False,\n                                  num_workers=os.cpu_count()) \n        return [test_dl]\n    \n\n    def loss_function(self, logits, gt):\n        # How to calculate the loss. Note this method is actually not a part of pytorch lightning ! It's only good practice\n        #loss_fn = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([32542/584]).to(logits.device))  # Let's rebalance the weights for each class here.\n        loss_fn = FocalLoss(logits=True)\n        gt = gt.float()\n        loss = loss_fn(logits, gt)\n        return loss\n\n    def configure_optimizers(self):\n        # Optimizers and schedulers. Note that each are in lists of equal length to allow multiple optimizers (for GAN for example)\n        optimizer = torch.optim.Adam(self.model.parameters(), lr=self.hparams.lr, weight_decay=3e-6)\n        scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=10 * self.hparams.lr, \n                                                        epochs=self.hparams.epochs, steps_per_epoch=len(self.train_dataloader()))\n        return [optimizer], [scheduler]\n\n    def training_step(self, batch, batch_idx):\n        # This is where you must define what happens during a training step (per batch)\n        logits = self(batch)\n        loss = self.loss_function(logits, batch['target']).unsqueeze(0)  # You need to unsqueeze in case you do multi-gpu training\n        # Pytorch lightning will call .backward on what is called 'loss' in output\n        # 'log' is reserved for tensorboard and will log everything define in the dictionary\n        return {'loss': loss, 'log': {'train_loss': loss}}\n\n    def validation_step(self, batch, batch_idx):\n        # This is where you must define what happens during a validation step (per batch)\n        logits = self(batch)\n        loss = self.loss_function(logits, batch['target']).unsqueeze(0)\n        probs = torch.sigmoid(logits)\n        return {'val_loss': loss, 'probs': probs, 'gt': batch['target']}\n    \n    def test_step(self, batch, batch_idx):\n        logits = self(batch)\n        probs = torch.sigmoid(logits)\n        return {'probs': probs}\n\n    def validation_epoch_end(self, outputs):\n        # This is what happens at the end of validation epoch. Usually gathering all predictions\n        # outputs is a list of dictionary from each step.\n        avg_loss = torch.cat([out['val_loss'] for out in outputs], dim=0).mean()\n        probs = torch.cat([out['probs'] for out in outputs], dim=0)\n        gt = torch.cat([out['gt'] for out in outputs], dim=0)\n        probs = probs.detach().cpu().numpy()\n        gt = gt.detach().cpu().numpy()\n\n        auc_roc = torch.tensor(roc_auc_score(gt, probs))\n        tensorboard_logs = {'val_loss': avg_loss, 'auc': auc_roc}\n        print(f'Epoch {self.current_epoch}: {avg_loss:.2f}, auc: {auc_roc:.4f}')\n\n        return {'avg_val_loss': avg_loss, 'log': tensorboard_logs}\n    \n    def test_epoch_end(self, outputs):\n        probs = torch.cat([out['probs'] for out in outputs], dim=0)\n        probs = probs.detach().cpu().numpy()\n        self.test_predicts = probs  # Save prediction internally for easy access\n        # We need to return something \n        return {'dummy_item': 0}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## Training\n\nLet's start by specifying parameters, the seed and output folder.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# dict_to_args is a simple helper to make hparams act like args from argparse. This makes it trivial to then use argparse\nOUTPUT_DIR = './lightning_logs'\nhparams = dict_to_args({'batch_size': 64,\n                        'lr': 1e-4, # common when using pretrained\n                        'epochs': 10,\n                        'arch': 'seresnext50'\n                       })","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For training we just need to instantiate the pytorch lightning module and a trainer with a few options. Most importantly this is where you specify how many GPU to use (or TPU) and if you want to do mixed precision training (with apex). For the purpose of this kernel I just do FP32 1GPU training but please read the pytorch lightning doc if you want to try TPU and/or mixed precision.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Initiate model\nmodel = LightModel(train_df, test_df, pid_train, pid_val, hparams)\ntb_logger = pl.loggers.TensorBoardLogger(save_dir='./',\n                                         name=f'baseline', # This will create different subfolders for your models\n                                         version=f'0')  # If you use KFold you can specify here the fold number like f'fold_{fold+1}'\ncheckpoint_callback = pl.callbacks.ModelCheckpoint(filepath=tb_logger.log_dir + \"/{epoch:02d}-{auc:.4f}\",\n                                                   monitor='auc', mode='max')\n# Define trainer\n# Here you can \ntrainer = pl.Trainer(max_nb_epochs=hparams.epochs, auto_lr_find=False,  # Usually the auto is pretty bad. You should instead plot and pick manually.\n                     gradient_clip_val=1,\n                     nb_sanity_val_steps=0,  # Comment that out to reactivate sanity but the ROC will fail if the sample has only class 0\n                     checkpoint_callback=checkpoint_callback,\n                     gpus=1,\n                     early_stop_callback=False,\n                     progress_bar_refresh_rate=0\n                     )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trainer.fit(model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test time\nThe easy part :)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Grab best checkpoint file\nout = Path(tb_logger.log_dir)\naucs = [ckpt.stem[-4:] for ckpt in out.iterdir()]\nbest_auc_idx = aucs.index(max(aucs))\nbest_ckpt = list(out.iterdir())[best_auc_idx]\nprint('Using ', best_ckpt)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trainer = pl.Trainer(resume_from_checkpoint=str(best_ckpt), gpus=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trainer.test(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = model.test_predicts\ntest_df['target'] = preds\nsubmission = test_df[['image_name', 'target']]\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.head(20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Suggested improvements\n\n* KFold validation + Average predictions on test\n* Explore architectures (efficient\n* Heavy data augmentation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}