{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Still Under Minor Work!\n### If you liked my work, please give an upvote! I am a beginner and even a single upvote gives me loads of motivation to work on!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 1. Installation and Imports\nI am installing a bunch of modules that would be needed for training on TPUs\nAlso, installing EfficientNet and ResNet for PyTorch","execution_count":null},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# Trying out TPUs with PyTorch, still very prone to error tho\n# ! curl -s https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n# ! python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"! pip install -q efficientnet_pytorch torchtoolbox resnet_pytorch","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\nimport random\nfrom random import shuffle\nimport cv2\nimport warnings\nfrom tqdm.notebook import tqdm\n\nimport torch\nimport torchvision\nimport torch.nn.functional as F\nimport torch.nn as nn\nimport torchtoolbox.transform as transforms\nfrom torch.utils.data import Dataset, DataLoader, Subset\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\nfrom efficientnet_pytorch import EfficientNet\nfrom resnet_pytorch import ResNet\n\nfrom sklearn.metrics import accuracy_score, roc_auc_score\nfrom sklearn.model_selection import GroupKFold\n\n# import torch_xla.core.xla_model as xm\n# import torch_xla.distributed.parallel_loader as pl\n# import torch_xla.distributed.xla_multiprocessing as xmp\n\nwarnings.simplefilter('ignore')\nplt.style.use(\"fivethirtyeight\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For EDA and other analysis, I will be using the already provided SIIM-ISIC Dataset and their CSV files.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data = pd.read_csv(\"../input/siim-isic-melanoma-classification/train.csv\")\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Append full image path to image_name\nIMG_DIR = \"../input/siim-isic-melanoma-classification/jpeg/train\"\ndata['image_name'] = data['image_name'].apply(lambda x: os.path.join(IMG_DIR, x+\".jpg\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Since there are null values in age column, I will replace them with mean of all ages\n# Also called 'Mean Imputation'\ndef impute_age(x):\n    if np.isnan(x):\n        return int(data['age_approx'].mean())\n    else:\n        return x\n\ndata['age_approx'] = data['age_approx'].apply(impute_age)\n# Drop NaN Rows\ndata = data.dropna()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. EDA\nThis EDA is by no means extensive, but only a starter one. I plan to improve it as I try out different things.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 2.1 Target Values Pie Chart","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Target\nplt.figure(figsize=(10, 8))\ntarget = [len(data[data['target'] == 0]), len(data[data['target'] == 1])]\nlabels = [\"Benign\", \"Malignant\"]\nplt.pie(x=target, labels=labels, explode=[0.0, 0.5], autopct='%1.1f%%')\nplt.title(\"Target Values\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.2 Age Values Distribution\nWe can note that a good majority of ages in the dataset are between 40-60. One contributing factor to it is that we have replace NaN values in Age column with the mean with is ~48","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Age\nplt.figure(figsize=(10, 5))\nchart = sns.kdeplot(data['age_approx'], shade=True)\nplt.title(\"Age Distribution\")\nplt.xlabel(\"Ages\")\nplt.ylabel(\"Count\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.3 Sex Values Pie - All Cases\nThis Pie is for all the cases","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Sex - all\nplt.figure(figsize=(10, 8))\ntarget = [len(data[data['sex'] == 'male']), len(data[data['sex'] == 'female'])]\nlabels = [\"Male\", \"Female\"]\nplt.pie(x=target, labels=labels, autopct='%1.1f%%')\nplt.title(\"Gender Values for All Cases\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.4 Sex Values - Malignant Cases\nThis chart is only for patients that have Malignant Case of Melanoma.\nAnyone can be mistaken seeing the pie chart for all cases that gender values are near equally distributed, but that is not the case for malignant","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Sex - malignant\nplt.figure(figsize=(10, 8))\ntarget = [len(data.query(\"sex == 'male' and target == '1'\")), len(data.query(\"sex == 'female' and target == '1'\"))]\nlabels = [\"Male\", \"Female\"]\nplt.pie(x=target, labels=labels, autopct='%1.1f%%')\nplt.title(\"Gender Values for Malignant Cases\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.5 Site of Focus - All Cases","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Site of Focus\nplt.figure(figsize=(10, 8))\ntargets = data['anatom_site_general_challenge'].value_counts().tolist()\nlabels = dict(data['anatom_site_general_challenge'].value_counts()).keys()\nplt.pie(x=targets, labels=labels, autopct='%1.1f%%')\nplt.title(\"Site of Focus - All Cases\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.6 Site of Focus - Malignant Cases Only","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Site of Focus - Malignant Only\nplt.figure(figsize=(10, 8))\ntargets = data[data['target']==1]['anatom_site_general_challenge'].value_counts().tolist()\nlabels = dict(data[data['target']==1]['anatom_site_general_challenge'].value_counts()).keys()\nplt.pie(x=targets, labels=labels, autopct='%1.1f%%')\nplt.title(\"Site of Focus - Malignant Cases\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.7 Diasnosis with Unknowns\nThis pie-chart shows the diagnosis including the unknown diagnosis","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Diagnosis w/t Unknowns\nplt.figure(figsize=(10, 8))\ntargets = data['diagnosis'].value_counts().tolist()\nlabels = dict(data['diagnosis'].value_counts()).keys()\npatches, texts = plt.pie(targets, shadow=True, startangle=90)\nplt.legend(patches, labels, loc=\"best\", frameon=False)\nplt.axis('equal')\nplt.title(\"Diagnosis with Unknowns\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.8 Diagnosis without Unknowns\nThe Pie-chart of all the diagnosis is dominated by the huge number of unknowns, let's see it without the unknowns","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Diagnosis w/tout Unknonws\nplt.figure(figsize=(10, 8))\ntargets = data['diagnosis'].value_counts().tolist()[1:]\nlabels = list(dict(data['diagnosis'].value_counts()).keys())[1:]\npatches, texts = plt.pie(targets, shadow=True, startangle=90)\nplt.legend(patches, labels, loc=\"best\", frameon=False)\nplt.axis('equal')\nplt.title(\"Diagnosis without Unknowns\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.9 Crosstab Heatmap for Diagnosis v/s target","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Crosstab Heatmap\nplt.figure(figsize=(10, 8))\ndata_h = data.drop(['image_name','patient_id','sex','age_approx','anatom_site_general_challenge','target'], axis=1)\ncross = pd.crosstab(data['diagnosis'].values, data['benign_malignant'])\nsns.heatmap(cross, cmap=\"YlGnBu\", annot=True, cbar=False)\nplt.title(\"Diagnosis v/s Target\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.10 Peek at the Images\nLet's take a look at different images in their normal and canny-edge detected forms","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_images(num, canny=False):\n    \"\"\"\n    I modified this function to display canny edges too\n    Originally taken from the excellent EDA notebook by Tarun Paparaju\n    Here: https://www.kaggle.com/tarunpaparaju/siim-isic-melanoma-eda-pytorch-baseline\n    \"\"\"\n    IMG_PATHS = \"../input/siim-isic-melanoma-classification/jpeg/train\"\n    sq_num = np.sqrt(num)\n    assert sq_num == int(sq_num)\n\n    sq_num = int(sq_num)\n    image_ids = os.listdir(IMG_PATHS)\n    shuffle(image_ids)\n    fig, ax = plt.subplots(nrows=sq_num, ncols=sq_num, figsize=(10, 10))\n\n    for i in range(sq_num):\n        for j in range(sq_num):\n            idx = i*sq_num + j\n            ax[i, j].axis('off')\n            img = cv2.imread(IMG_PATHS + '/' + image_ids[idx])\n            if canny:\n                img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n                img = cv2.Canny(img, 50, 50)\n                ax[i, j].imshow(img); ax[i, j].set_title('Canny Image {}'.format(idx), fontsize=12)\n            else:\n                ax[i, j].imshow(img); ax[i, j].set_title('Normal Image {}'.format(idx), fontsize=12)\n\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display_images(4, canny=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display_images(4, canny=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Modelling\n\nI am currently in the process of trying out different approaches for training. As of now (14/7/20), I am only using Images as features to train the model and not other meta features such as age, sex, etc.\n\nI am beginner and I learned how to do this from [Roman's Kernel](https://www.kaggle.com/nroman/melanoma-pytorch-starter-efficientnet) (Thank you!!)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split the data into 5 folds for training\nkf = GroupKFold(n_splits=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n# device = xm.xla_device()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# I'll be using a different resized dataset for training\ndata = pd.read_csv(\"../input/melanoma-external-malignant-256/train_concat.csv\")\nsubmission_df = pd.read_csv(\"../input/siim-isic-melanoma-classification/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Remove Meta Features (might use them in a later release)\nsubmission_df = submission_df.drop(['sex', 'age_approx', 'anatom_site_general_challenge'], axis=1)\nsubmission_df['patient_id'] = submission_df['patient_id'].fillna(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = data.drop(['sex', 'age_approx', 'anatom_site_general_challenge'], axis=1)\ndata['patient_id'] = data['patient_id'].fillna(0)\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Add the complete path to the image_names for ease in loading them\nTRAIN_IMG_PATH = \"../input/melanoma-external-malignant-256/train/train\"\nTEST_IMG_PATH = \"../input/melanoma-external-malignant-256/test/test\"\ndata['image_name'] = data['image_name'].apply(lambda x: os.path.join(TRAIN_IMG_PATH, x + \".jpg\"))\nsubmission_df['image_name'] = submission_df['image_name'].apply(lambda x: os.path.join(TEST_IMG_PATH, x + \".jpg\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class MelanomaDataset(Dataset):\n    \"\"\"\n    Custom Dataset class for loading Melanoma Dataset\n    \"\"\"\n    def __init__(self, df, is_train=True, transform=None):\n        self.df = df\n        self.is_train = is_train\n        self.transform = transform\n    \n    def __getitem__(self, index):\n        \"\"\"\n        Get the image path, read it and return it with corresponding label (if is_train=True)\n        else return just the image\n        \"\"\"\n        self.img_path = self.df.iloc[index]['image_name']\n        img = cv2.imread(self.img_path)\n        if self.transform:\n            img = self.transform(img)\n        if self.is_train:\n            labl = self.df.iloc[index]['target']\n            return img, labl\n        else:\n            return img\n    def __len__(self):\n        return len(self.df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Network(nn.Module):\n    \"\"\"\n    Currently I've only implmented 2 models (EfficientNet and ResNet15)\n    \n    Friendly warning: DONOT USE RESNET150, IT WILL PROBABLY RESULT IN AN OOM ERROR (OUT OF MEMORY)\n    \"\"\"\n    def __init__(self, model_name='eff'):\n        super(Network, self).__init__()\n        self.model_name = model_name\n        if self.model_name == 'eff':\n            self.model = EfficientNet.from_pretrained('efficientnet-b1')\n            self.model._fc = nn.Linear(in_features=1280, out_features=500, bias=True)\n            self.output = nn.Linear(500, 1)\n        elif self.model_name == 'res':\n            self.model = ResNet.from_pretrained('resnet18')\n            self.model.fc = nn.Linear(in_features=512, out_features=500, bias=True)\n            self.output = nn.Linear(500, 1)\n        \n    def forward(self, img):\n        img_feat = self.model(img)\n        out = self.output(img_feat)\n        return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Microscope:\n    \"\"\"\n    Cutting out the edges around the center circle of the image\n    Imitating a picture, taken through the microscope\n\n    Args:\n        p (float): probability of applying an augmentation\n    \"\"\"\n\n    def __init__(self, p: float = 0.5):\n        self.p = p\n\n    def __call__(self, img):\n        \"\"\"\n        Args:\n            img (PIL Image): Image to apply transformation to.\n\n        Returns:\n            PIL Image: Image with transformation.\n        \"\"\"\n        if random.random() < self.p:\n            circle = cv2.circle((np.ones(img.shape) * 255).astype(np.uint8), # image placeholder\n                        (img.shape[0]//2, img.shape[1]//2), # center point of circle\n                        random.randint(img.shape[0]//2 - 3, img.shape[0]//2 + 15), # radius\n                        (0, 0, 0), # color\n                        -1)\n\n            mask = circle - 255\n            img = np.multiply(img, mask)\n        \n        return img\n\n    def __repr__(self):\n        return f'{self.__class__.__name__}(p={self.p})'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_transform = transforms.Compose([\n    transforms.RandomResizedCrop(size=256, scale=(0.7, 1.0)),\n    transforms.RandomHorizontalFlip(),\n    transforms.RandomVerticalFlip(),\n    transforms.ColorJitter(brightness=32. / 255.,saturation=0.5),\n    Microscope(p=0.6),\n    transforms.ToTensor(),\n    transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])\n])\ntest_transform = transforms.Compose([\n    transforms.ToTensor(),\n    transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test = MelanomaDataset(\n    df=submission_df,\n    is_train=False,\n    transform=test_transform\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Main training loop\nepochs = 3\nsave_model_path = \"best_model.pth\"\n\nnet = Network(model_name=\"eff\")\nnet = net.to(device)\noptim = torch.optim.Adam(net.parameters(), lr=1e-2)\nloss_fn = nn.BCEWithLogitsLoss()\n\nfor fold_, (train_idx, val_idx) in enumerate(kf.split(X=np.zeros(len(data)), y=data['target'], groups=data['patient_id'].tolist()), 1):\n    print(\"=\"*20, f\"Fold: {fold_}\", \"=\"*20)    \n    \n    # Make the datasets and initialize their respective loaders\n    train = MelanomaDataset(\n        df=data.iloc[train_idx].reset_index(drop=True),\n        is_train=True,\n        transform=train_transform\n    )\n    \n    valid = MelanomaDataset(\n        df=data.iloc[val_idx].reset_index(drop=True),\n        is_train=True,\n        transform=train_transform\n    )\n    \n    train_loader = DataLoader(train, batch_size=64, shuffle=True, num_workers=2)\n    valid_loader = DataLoader(valid, batch_size=64, shuffle=True, num_workers=2)\n    scheduler = ReduceLROnPlateau(optimizer=optim, mode='max', patience=1, verbose=True, factor=0.2)\n    \n    # For every fold, complete the required number of epochs\n    for epoch in range(epochs):\n        epoch_loss = 0\n        correct = 0\n        net.train()\n        \n        for x, y in train_loader:\n            # Convert the x (img) and y (labels) into torch tensors\n            x = torch.tensor(x, device=device, dtype=torch.float32)\n            y = torch.tensor(y, device=device, dtype=torch.float32)\n            \n            # Training Procedure\n            optim.zero_grad()\n            z = net(x)\n            loss = loss_fn(z, y.unsqueeze(1))\n            loss.backward()\n            optim.step()\n            \n            # Calculate the predictions\n            pred = torch.round(torch.sigmoid(z))\n            correct += (pred.cpu() == y.cpu().unsqueeze(1)).sum().item()\n            epoch_loss += loss.item()\n        \n        train_acc = correct / len(train_idx)\n        \n        # Validation Set Evaluation!!!\n        net.eval()\n        with torch.no_grad():\n            val_preds = torch.zeros((len(val_idx), 1), dtype=torch.float32, device=device)\n            for j, (x_val, y_val) in enumerate(valid_loader):\n                x_val = torch.tensor(x_val, device=device, dtype=torch.float32)\n                y_val = torch.tensor(y_val, device=device, dtype=torch.float32)\n                val_pred = torch.sigmoid(net(x_val))\n                val_preds[j * x_val.shape[0] : j * x_val.shape[0] + x_val.shape[0]] = val_pred\n                val_acc = accuracy_score(data.iloc[val_idx]['target'].values, torch.round(val_preds.cpu()))\n                val_roc_auc = roc_auc_score(data.iloc[val_idx]['target'].values, val_preds.cpu())\n                scheduler.step(val_roc_auc)\n                \n            print(\"-\"*10, \"Valid Values\", \"-\"*10)\n            print(f\"Epoch: {epoch+1}/{epochs} | Training Accuracy: {train_acc} | Validation Accuracy: {val_acc} | Validation ROC: {val_roc_auc}\")\n            torch.save(net, save_model_path)\n            print(f\"Model with val_acc: {val_acc} saved.\")\n    \n    # Delete these guys to free up memory\n    del val_pred, x_val, y_val, x, y, pred, train_loader, valid_loader","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Prediction time!\ntest_time_augments = 3\ntest_loader = DataLoader(dataset=test, batch_size=16, shuffle=False, num_workers=2)\npreds = torch.zeros((len(test), 1), dtype=torch.float32, device=device)\nwith torch.no_grad():\n    for _ in range(test_time_augments):\n        for i, x_test in enumerate(test_loader):\n            x_test = torch.tensor(x_test, dtype=torch.float32, device=device)\n            z_test = net(x_test)\n            z_test = torch.sigmoid(z_test)\n            preds[i*x_test.shape[0] : i*x_test.shape[0] + x_test.shape[0]] += z_test\n    preds /= test_time_augments\npreds /= kf.n_splits","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert the predictions dataframe to a csv file for submission\nsub = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/sample_submission.csv')\nsub['target'] = preds.cpu().numpy().reshape(-1,)\nsub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Thank you!\n\nPlease correct me if I’ve made any mistakes in EDA, modelling or maybe explaining some concept since I am a beginner and prone to making mistakes.\nReach me here:\n- [LinkedIn](https://linkedin.com/in/tanaymehta28/) \n- [Github](https://github.com/heytanay)\n- [Blog](https://heytanay.github.io)","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}