{"cells":[{"metadata":{},"cell_type":"markdown","source":"Note: This Kernel is part of our ongoing `Learn Deep learning by competing in Live Kaggle competition` [course](https://www.youtube.com/watch?v=rnJG12vIuFc&t=2s).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# SIIM-ISIC Melanoma Classification\n\nIn this competition we need to identify whether a skin lesion has Melanoma or Not. It is a very difficult problem primarily for 2 reasons -\n\n1. For an untrained human eye it is very difficult to distinguish whether a given image has Melanoma or not.\n2. The dataset is highly imbalanced.\n\n\nThis is a simple starter notebook for Kaggle's Melanoma Comp, showing how to use \n\n- PyTorch\n- Albumentations\n- Chris Deotte [triple stratified data](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164092)\n- Chris Deotte resized [data](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164092) of 256,256\n- Different EfficientNet models\n- Apply N number of TTA for predicitons.\n- Use a 80/20 split for comparing experiments across competitions. ","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install efficientnet-pytorch","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from pathlib import Path\nimport pandas as pd\nfrom torch.utils.data import Dataset,DataLoader\nfrom PIL import Image\nfrom torchvision import transforms as T\nimport torch.nn as nn\nimport torch\nimport torch.nn.functional as F\nfrom sklearn.model_selection import GroupKFold\nimport numpy as np\nfrom fastprogress.fastprogress import master_bar, progress_bar\nfrom sklearn.metrics import accuracy_score, roc_auc_score\nfrom efficientnet_pytorch import EfficientNet\nfrom torchvision import models\nimport pdb\nimport albumentations as A\nfrom albumentations.pytorch.transforms import ToTensor\nimport matplotlib.pyplot as plt\n\nimport pickle ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def list_files(path:Path):\n    return [o for o in path.iterdir()]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Exploration","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"path = Path('../input/jpeg-melanoma-256x256/')\ndf_path = Path('../input/melanoma-256x256/')\nim_sz = 256\nbs = 16","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_fnames = list_files(path/'train')\ndf = pd.read_csv(df_path/'train.csv')\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The dataset contains around 35K images out of which only 584 images are malignant. That makes only 1.8% of the total dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df.target.value_counts(),df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(\"Samples with Melanoma\")\nimgs = df[df.target==1]['image_name'].values\n_, axs = plt.subplots(2, 5, figsize=(20, 8))\naxs = axs.flatten()\nfor f_name,ax in zip(imgs[:10],axs):\n    img = Image.open(path/f'train/{f_name}.jpg')\n    ax.imshow(img)\n    ax.axis('off')\nplt.show()\n\nprint(\"Samples without Melanoma\")\nimgs = df[df.target==0]['image_name'].values\n_, axs = plt.subplots(2, 5, figsize=(20, 8))\naxs = axs.flatten()\nfor f_name,ax in zip(imgs[:10],axs):\n    img = Image.open(path/f'train/{f_name}.jpg')\n    ax.imshow(img)\n    ax.axis('off')    \nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Augmentation\n\nWe use albumentations to perform augmentations. Since the dataset is small and we are not using any external dataset in this Kernel, an increased augmentation can be helpful. You can play with the augmentation argument to either increase or decrease the amount of data augmentation applied.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_augmentations(p=0.5):\n    imagenet_stats = {'mean':[0.485, 0.456, 0.406], 'std':[0.229, 0.224, 0.225]}\n    train_tfms = A.Compose([\n        A.Cutout(p=p),\n        A.RandomRotate90(p=p),\n        A.Flip(p=p),\n        A.OneOf([\n            A.RandomBrightnessContrast(brightness_limit=0.2,\n                                       contrast_limit=0.2,\n                                       ),\n            A.HueSaturationValue(\n                hue_shift_limit=20,\n                sat_shift_limit=50,\n                val_shift_limit=50)\n        ], p=p),\n        A.OneOf([\n            A.IAAAdditiveGaussianNoise(),\n            A.GaussNoise(),\n        ], p=p),\n        A.OneOf([\n            A.MotionBlur(p=0.2),\n            A.MedianBlur(blur_limit=3, p=0.1),\n            A.Blur(blur_limit=3, p=0.1),\n        ], p=p),\n        A.ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=45, p=p),\n        A.OneOf([\n            A.OpticalDistortion(p=0.3),\n            A.GridDistortion(p=0.1),\n            A.IAAPiecewiseAffine(p=0.3),\n        ], p=p), \n        ToTensor(normalize=imagenet_stats)\n        ])\n    \n    test_tfms = A.Compose([\n        ToTensor(normalize=imagenet_stats)\n        ])\n    return train_tfms, test_tfms","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train/Validation split\n- We use a simple 80/20 split based on the triple stratified K-Fold split. \n- We remove all the duplicate images.\n- TF record - id with values 12,13,14 are put into validation split and the rest into train split.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_train_val_split(df):\n    #Remove Duplicates\n    df = df[df.tfrecord != -1].reset_index(drop=True)\n    #We are splitting data based on triple stratified kernel provided here https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/165526\n    train_tf_records = list(range(len(df.tfrecord.unique())))[:12]\n    split_cond = df.tfrecord.apply(lambda x: x in train_tf_records)\n    train_df = df[split_cond].reset_index()\n    valid_df = df[~split_cond].reset_index()\n    return train_df,valid_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class MelanomaDataset(Dataset):\n    def __init__(self,df,im_path,transforms=None,is_test=False):\n        self.df = df\n        self.im_path = im_path\n        self.transforms = transforms\n        self.is_test = is_test\n        \n    def __getitem__(self,idx):\n        img_path = f\"{self.im_path}/{self.df.iloc[idx]['image_name']}.jpg\"\n        img = Image.open(img_path)\n        if self.transforms:\n            img = self.transforms(**{\"image\": np.array(img)})[\"image\"]\n            \n        if self.is_test:\n            return img\n        target = self.df.iloc[idx]['target']\n        return img,torch.tensor([target],dtype=torch.float32)\n    \n    def __len__(self):\n        return self.df.shape[0]\n        ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model \n\nEfficientnets have proved themselves in the last year as a key to winning competitions. You can try different Efficientnet model by passing the right model name. Changing the number in the model name `efficientnet-b0` gives you different models. The larger the number the more complex/bigger and better the model is. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class MelanomaEfficientNet(nn.Module):\n    def __init__(self,model_name='efficientnet-b0',pool_type=F.adaptive_avg_pool2d):\n        super().__init__()\n        self.pool_type = pool_type\n        self.backbone = EfficientNet.from_pretrained(model_name)\n        in_features = getattr(self.backbone,'_fc').in_features\n        self.classifier = nn.Linear(in_features,1)\n\n    def forward(self,x):\n        features = self.pool_type(self.backbone.extract_features(x),1)\n        features = features.view(x.size(0),-1)\n        return self.classifier(features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Helper functions \n- Split data to train and validation split\n- Get model, choose different optimizer, freeze backbone, different learning rates/weight decay.\n- The training method by default uses cosine annealing for scheduling learning rate, you can experiment with.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_device():\n    return torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\n\ndef get_model(model_name='efficientnet-b0',lr=1e-5,wd=0.01,freeze_backbone=False,opt_fn=torch.optim.AdamW,device=None):\n    device = device if device else get_device()\n    model = MelanomaEfficientNet(model_name=model_name)\n    if freeze_backbone:\n        for parameter in model.backbone.parameters():\n            parameter.requires_grad = False\n    opt = opt_fn(model.parameters(),lr=lr,weight_decay=wd)\n    model = model.to(device)\n    return model, opt\n\ndef training_step(xb,yb,model,loss_fn,opt,device,scheduler):\n    xb,yb = xb.to(device), yb.to(device)\n    out = model(xb)\n    opt.zero_grad()\n    loss = loss_fn(out,yb)\n    loss.backward()\n    opt.step()\n    scheduler.step()\n    return loss.item()\n    \ndef validation_step(xb,yb,model,loss_fn,device):\n    xb,yb = xb.to(device), yb.to(device)\n    out = model(xb)\n    loss = loss_fn(out,yb)\n    out = torch.sigmoid(out)\n    return loss.item(),out\n\ndef get_data(train_df,valid_df,train_tfms,test_tfms,bs):\n    train_ds = MelanomaDataset(df=train_df,im_path=path/'train',transforms=train_tfms)\n    valid_ds = MelanomaDataset(df=valid_df,im_path=path/'train',transforms=test_tfms)\n    train_dl = DataLoader(dataset=train_ds,batch_size=bs,shuffle=True,num_workers=4)\n    valid_dl = DataLoader(dataset=valid_ds,batch_size=bs*2,shuffle=False,num_workers=4)\n    return train_dl,valid_dl","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fit(epochs,model,train_dl,valid_dl,opt,device=None,loss_fn=F.binary_cross_entropy_with_logits):\n    \n    device = device if device else get_device()\n    scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(opt, len(train_dl)*epochs)\n    val_rocs = [] \n    \n    #Creating progress bar\n    mb = master_bar(range(epochs))\n    mb.write(['epoch','train_loss','valid_loss','val_roc'],table=True)\n\n    for epoch in mb:    \n        trn_loss,val_loss = 0.0,0.0\n        val_preds = np.zeros((len(valid_dl.dataset),1))\n        val_targs = np.zeros((len(valid_dl.dataset),1))\n        \n        #Training\n        model.train()\n        \n        #For every batch \n        for xb,yb in progress_bar(train_dl,parent=mb):\n            trn_loss += training_step(xb,yb,model,loss_fn,opt,device,scheduler) \n        trn_loss /= mb.child.total\n\n        #Validation\n        model.eval()\n        with torch.no_grad():\n            for i,(xb,yb) in enumerate(progress_bar(valid_dl,parent=mb)):\n                loss,out = validation_step(xb,yb,model,loss_fn,device)\n                val_loss += loss\n                bs = xb.shape[0]\n                val_preds[i*bs:i*bs+bs] = out.cpu().numpy()\n                val_targs[i*bs:i*bs+bs] = yb.cpu().numpy()\n\n        val_loss /= mb.child.total\n        val_roc = roc_auc_score(val_targs.reshape(-1),val_preds.reshape(-1))\n        val_rocs.append(val_roc)\n\n        mb.write([epoch,f'{trn_loss:.6f}',f'{val_loss:.6f}',f'{val_roc:.6f}'],table=True)\n    return model,val_rocs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"df = pd.read_csv(df_path/'train.csv')\ntrain_df,valid_df = get_train_val_split(df)\ntrain_tfms,test_tfms = get_augmentations(p=0.5)\ntrain_dl,valid_dl = get_data(train_df,valid_df,train_tfms,test_tfms,bs)\nmodel, opt = get_model(model_name='efficientnet-b5',lr=1e-4,wd=1e-4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I trained the model on my local machine. You can uncomment below code to start training here.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# model,val_rocs = fit(10,model,train_dl,valid_dl,opt)\n# torch.save(model.state_dict(),f'effb5.pth')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generate test predictions\n\n- By default we use the same data augmentation techniques that we applied during training. \n- Tweak the TTA parameter in `get_preds()` to increase the number of times TTA is applied.\n- If you do not want TTA, change `transforms` to `test_transforms`.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_tfms = A.Compose([\n    A.RandomRotate90(p=p),\n        A.Flip(p=p),\n        A.OneOf([\n            A.RandomBrightnessContrast(brightness_limit=0.2,\n                                       contrast_limit=0.2,\n                                       ),\n            A.HueSaturationValue(\n                hue_shift_limit=20,\n                sat_shift_limit=50,\n                val_shift_limit=50)\n        ], p=p),\n        A.OneOf([\n            A.IAAAdditiveGaussianNoise(),\n            A.GaussNoise(),\n        ], p=p),\n    ToTensor(normalize=imagenet_stats)\n    ])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"test_df = pd.read_csv(path/'test.csv')\n\nmodel, opt = get_model(model_name='efficientnet-b5',lr=1e-4,wd=1e-4)\nmodel.load_state_dict(torch.load(f'../input/melanomaefficientnetb5/effb5.pth',map_location=device))\n\n#Testing with lighter augmentation\ntest_ds = MelanomaDataset(df=test_df,im_path=path/'test',transforms=test_tfms,is_test=True)\ntest_dl = DataLoader(dataset=test_ds,batch_size=bs*2,shuffle=False,num_workers=4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"def get_preds(model,device=None,tta=3):\n    if device is None:\n        device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\n    preds = np.zeros(len(test_ds))\n    for tta_id in range(tta):\n        test_preds = []\n        with torch.no_grad():\n            for xb in test_dl:\n                xb = xb.to(device)\n                out = model(xb)\n                out = torch.sigmoid(out)\n                test_preds.extend(out.cpu().numpy())\n            preds += np.array(test_preds).reshape(-1)\n        print(f'TTA {tta_id}')\n    preds /= tta\n    return preds\n\n#Changing tta to 25 from 10\npreds = get_preds(model,tta=25)  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"subm = pd.read_csv(path/'sample_submission.csv')\nsubm.target = preds\nsubm.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"markdown","source":"# Things to try next\n\nSome of the tricks that can help in improving score are you can.\n\n- Try training on higher image sizes.\n- Try larger/different models.\n- Try different loss functions like Focal Loss.\n- Try Label smoothing.\n- Try Focal Loss + Label smoothing\n- Different Pooling layers","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Thank you for reading. Hope you find this kernel useful. \nI will add more ideas in upcoming versions. \nPlease consider upvoting and sharing the kernel.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}