{"cells":[{"metadata":{},"cell_type":"markdown","source":"
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nIn this project, I will show how one can finetune EfficientNet-B3 to detect Melanoma (a variety of skin cancer) from images. This problem is important because fast and accurate automated diagnosis can help reduce burden on doctors and let them focus on curing patients. I will use the **Efficientnet Pytorch** package to get the pretrained EfficientNet-B3 model and **PyTorch XLA** to train the model on TPU. At the end, we will run inference on the test set and test the model's predictions on some sample images.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Preparing the ground\n\nIn this section, we will prepare the ground to train and test the model by installing packages, setting hyperparameters, and loading the data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Set up PyTorch XLA\n\n* We need to install a package called **PyTorch XLA** to take advantage of TPUs with PyTorch.\n* Training on TPUs can be significantly faster than on CPUs or GPUs. So we will use TPUs for this project.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Install additional packages\n\n* We will now install two other packages: **colored** and **efficientnet_pytorch**.\n* **colored** helps add foreground colors to text output in Python to make training logs more readable.\n* **efficientnet_pytorch** will help us load the EfficientNet-B3 architecture with the pretrained weights.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"!export XLA_USE_BF16=1\n!pip install -q colored\n!pip install -q efficientnet_pytorch","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Import necessary libraries\n\n* Now, we import all the libraries we need.\n* **colored, matplotlib, tqdm, and plotly** for visualization.\n* **numpy, pandas, torch, torchvision, albumentations, and efficientnet_pytorch** for modelling.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport cv2\nimport time\nimport numpy as np\nimport pandas as pd\n\nfrom colored import fg, attr\nfrom tqdm.notebook import tqdm\nimport matplotlib.pyplot as plt\nfrom sklearn.utils import shuffle\n\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport plotly.figure_factory as ff\nfrom plotly.subplots import make_subplots\n\nimport torch\nimport torch.nn as nn\nfrom torch.optim import Adam\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch import FloatTensor, LongTensor, DoubleTensor\nfrom torch.utils.data.sampler import WeightedRandomSampler\n\nimport torch_xla.core.xla_model as xm\nimport torch_xla.distributed.parallel_loader as pl\nimport torch_xla.distributed.xla_multiprocessing as xmp\n\nfrom efficientnet_pytorch import EfficientNet\nfrom albumentations import Normalize, VerticalFlip, HorizontalFlip, Compose","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define key hyperparameters and paths\n\n* Here we define the key hyperparameters: image height and width, train/val split, batch size, epochs, and LR.\n* These hyperparameters can be tuned for performance, but runtime will change if you change **EPOCHS, H, or W.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Unzip ZIP files","execution_count":null},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!unzip ../input/siim-isic-melanoma-resized-images/test.zip\n!unzip ../input/siim-isic-melanoma-resized-images/train_1.zip\n!unzip ../input/siim-isic-melanoma-resized-images/train_2.zip","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define hyperparameters and paths","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"W = 512\nH = 512\nB = 0.5\nSPLIT = 0.8\nSAMPLE = True\nMU = [0.485, 0.456, 0.406]\nSIGMA = [0.229, 0.224, 0.225]\n\nEPOCHS = 5\nLR = 1e-3, 1e-3\nBATCH_SIZE = 32\nVAL_BATCH_SIZE = 32\nMODEL = 'efficientnet-b3'\nIMG_PATHS = ['../working/test',\n '../working/train_1',\n '../working/train_2']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Get image path dictionary","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH_DICT = {}\nfor folder_path in tqdm(IMG_PATHS):\n for img_path in os.listdir(folder_path):\n PATH_DICT[img_path] = folder_path + '/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Set random seeds\n\n* The next step is to set the random seed for NumPy and PyTorch.\n* Setting the random seed helps us keep training determinstic and ensure reproducible results.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(42)\ntorch.manual_seed(42)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load the training and testing metadata\n\n* Now we need to load the training and testing metadata.\n* These dataframes contain the image file names and targets which we need for training.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(os.listdir('../input/siim-isic-melanoma-classification'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')\ntrain_df = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EDA\n\nNow, I will do some basic visualization to gain some insights from the data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Values in categorical columns\n\n* First I will look at the set of values present in each categorical column.\n* The categorical columns include **target, sex, anatom_site_general_challenge, diagnosis, and benign_malignant**.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sfn = lambda x: \"🔎 \" + str(x)\nfs = (fg('#7efc79'), attr('reset'))\ngs = (fg('#fac3c3'), attr('reset'))\n\nfor column in train_df.columns[[7, 2, 6, 4, 5]]:\n column_string = (\"%s\" + column + \"%s 🎯\\n\") % gs\n column_values = set(train_df[column].apply(sfn))\n column_values = \"\\n\".join(sorted(column_values, key=len))\n hyphens = (\"%s\" + ''.join(['-']*len(column)) + \"\\n%s\") % gs\n print(hyphens + column_string + hyphens + \"\\n\" + (\"%s\" + column_values + \"\\n%s\") % fs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Target vs. Proportion (per sex)\n\nLet us understand how **sex** affects the **target**.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"x = ['healthy', 'melanoma']\nx_1 = len(train_df.query('sex == \"male\" and target == 0'))\nx_2 = len(train_df.query('sex == \"male\" and target == 1'))\nx_3 = len(train_df.query('sex == \"female\" and target == 0'))\nx_4 = len(train_df.query('sex == \"female\" and target == 1'))\n\nx_1, x_3 = x_1/sum([x_1, x_3]), x_3/sum([x_1, x_3])\nx_2, x_4 = x_2/sum([x_2, x_4]), x_4/sum([x_2, x_4])\nfig = go.Figure(data=[go.Bar(name='male', y=x, x=[x_1, x_2], orientation=\"h\",\n marker=dict(color='#02cc45', line=dict(width=0.5, color='black'))),\n go.Bar(name='female', y=x, x=[x_3, x_4], orientation=\"h\",\n marker=dict(color='#f7766f', line=dict(width=0.5, color='black')))])\n\nfig.update_layout(xaxis_title='Target', yaxis_title='Proportion', template=\"plotly_white\",\n title='Target vs. Proportion (per sex)', barmode='stack', paper_bgcolor=\"#edebeb\"); fig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the graph above, we can see that the proportion of males (in green) under \"melanoma\" is much higher than under \"healthy\". This might suggest that most patients suffering from melanoma in this dataset are males.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Target vs. Proportion (per benign_malignant)\n\nNow let us understand how **benign_malignant** affects the **target**.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"x = ['healthy', 'melanoma']\nx_1 = len(train_df.query('benign_malignant == \"benign\" and target == 0'))\nx_2 = len(train_df.query('benign_malignant == \"benign\" and target == 1'))\nx_3 = len(train_df.query('benign_malignant == \"malignant\" and target == 0'))\nx_4 = len(train_df.query('benign_malignant == \"malignant\" and target == 1'))\n\nx_1, x_3 = x_1/sum([x_1, x_3]), x_3/sum([x_1, x_3])\nx_2, x_4 = x_2/sum([x_2, x_4]), x_4/sum([x_2, x_4])\nfig = go.Figure(data=[go.Bar(name='benign', y=x, x=[x_1, x_2], orientation=\"h\",\n marker=dict(color='#02cc45', line=dict(width=0.5, color='black'))),\n go.Bar(name='malignant', y=x, x=[x_3, x_4], orientation=\"h\",\n marker=dict(color='#f7766f', line=dict(width=0.5, color='black')))])\n\nfig.update_layout(xaxis_title='Target', yaxis_title='Proportion', template=\"plotly_white\",\n title='Target vs. Proportion (per benign_malignant)', barmode='stack', paper_bgcolor=\"#edebeb\"); fig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can clearly see from the bar chart above that the **benign_malignant** feature perfectly mirrors the target.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Target vs. Proportion (per anatom_site_general_challenge)\n\nNext, let us look at how **anatom_site_general_challenge** affects the **target**.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"x = ['healthy', 'melanoma']\n\nx_2 = len(train_df.query('anatom_site_general_challenge == \"torso\" and target == 0'))\nx_3 = len(train_df.query('anatom_site_general_challenge == \"head/neck\" and target == 0'))\nx_4 = len(train_df.query('anatom_site_general_challenge == \"palms/soles\" and target == 0'))\nx_5 = len(train_df.query('anatom_site_general_challenge == \"oral/genital\" and target == 0'))\nx_6 = len(train_df.query('anatom_site_general_challenge == \"upper extremity\" and target == 0'))\nx_7 = len(train_df.query('anatom_site_general_challenge == \"lower extremity\" and target == 0'))\nx_1 = len(train_df.query('anatom_site_general_challenge != anatom_site_general_challenge and target == 0'))\n\nx_9 = len(train_df.query('anatom_site_general_challenge == \"torso\" and target == 1'))\nx_10 = len(train_df.query('anatom_site_general_challenge == \"head/neck\" and target == 1'))\nx_11 = len(train_df.query('anatom_site_general_challenge == \"palms/soles\" and target == 1'))\nx_12 = len(train_df.query('anatom_site_general_challenge == \"oral/genital\" and target == 1'))\nx_13 = len(train_df.query('anatom_site_general_challenge == \"upper extremity\" and target == 1'))\nx_14 = len(train_df.query('anatom_site_general_challenge == \"lower extremity\" and target == 1'))\nx_8 = len(train_df.query('anatom_site_general_challenge != anatom_site_general_challenge and target == 1'))\n\ntotal = sum([x_1, x_2, x_3, x_4, x_5, x_6, x_7])\nx_1, x_2, x_3, x_4, x_5, x_6, x_7 = [y/total for y in [x_1, x_2, x_3, x_4, x_5, x_6, x_7]]\n\ntotal = sum([x_8, x_9, x_10, x_11, x_12, x_13, x_14])\nx_8, x_9, x_10, x_11, x_12, x_13, x_14 = [y/total for y in [x_8, x_9, x_10, x_11, x_12, x_13, x_14]]\n\n\nfig = go.Figure(data=[go.Bar(name='nan', y=x, x=[x_1, x_8], orientation=\"h\",\n marker=dict(color='red', line=dict(width=0.5, color='black'))),\n go.Bar(name='torso', y=x, x=[x_2, x_9], orientation=\"h\",\n marker=dict(color='orange', line=dict(width=0.5, color='black'))),\n go.Bar(name='head/neck', y=x, x=[x_3, x_10], orientation=\"h\",\n marker=dict(color='gold', line=dict(width=0.5, color='black'))),\n go.Bar(name='palms/soles', y=x, x=[x_4, x_11], orientation=\"h\",\n marker=dict(color='green', line=dict(width=0.5, color='black'))),\n go.Bar(name='oral/genital', y=x, x=[x_5, x_12], orientation=\"h\",\n marker=dict(color='blue', line=dict(width=0.5, color='black'))),\n go.Bar(name='upper extremity', y=x, x=[x_6, x_13], orientation=\"h\",\n marker=dict(color='violet', line=dict(width=0.5, color='black'))),\n go.Bar(name='lower extremity', y=x, x=[x_7, x_14], orientation=\"h\",\n marker=dict(color='indigo', line=dict(width=0.5, color='black')))])\n\nfig.update_layout(xaxis_title='Target', yaxis_title='Proportion', template=\"plotly_white\",\n title='Target vs. Proportion (per anatom_site_general_challenge)', barmode='stack', paper_bgcolor=\"#edebeb\"); fig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the bar charts above, we can see two major differences between melanoma images and non-melanoma images. The melanoma images tend to be more from the \"head/neck\" and \"upper extremity\" regions (yellow and pink) as compared to non-melanoma images. On the other hand, non-melanoma images tend to be more from the \"torso\" and \"lower extremity\" regions (orange and purple) as compared to melanoma images. This might suggest that **melanoma images are more likely to be from lower regions of the body than upper regions.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Target vs. Proportion (per diagnosis)\n\nNow let us look at how the **diagnosis** affects the **target**.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"x = ['healthy', 'melanoma']\n\nx_1 = len(train_df.query('diagnosis == \"nevus\" and target == 0'))\nx_2 = len(train_df.query('diagnosis == \"unknown\" and target == 0'))\nx_3 = len(train_df.query('diagnosis == \"melanoma\" and target == 0'))\nx_4 = len(train_df.query('diagnosis == \"lentigo NOS\" and target == 0'))\nx_5 = len(train_df.query('diagnosis == \"solar lentigo\" and target == 0'))\nx_6 = len(train_df.query('diagnosis == \"lichenoid keratosis\" and target == 0'))\nx_7 = len(train_df.query('diagnosis == \"cafe-au-lait macule\" and target == 0'))\nx_8 = len(train_df.query('diagnosis == \"seborrheic keratosis\" and target == 0'))\nx_9 = len(train_df.query('diagnosis == \"atypical melanocytic proliferation\" and target == 0'))\n\nx_10 = len(train_df.query('diagnosis == \"nevus\" and target == 1'))\nx_11 = len(train_df.query('diagnosis == \"unknown\" and target == 1'))\nx_12 = len(train_df.query('diagnosis == \"melanoma\" and target == 1'))\nx_13 = len(train_df.query('diagnosis == \"lentigo NOS\" and target == 1'))\nx_14 = len(train_df.query('diagnosis == \"solar lentigo\" and target == 1'))\nx_15 = len(train_df.query('diagnosis == \"lichenoid keratosis\" and target == 1'))\nx_16 = len(train_df.query('diagnosis == \"cafe-au-lait macule\" and target == 1'))\nx_17 = len(train_df.query('diagnosis == \"seborrheic keratosis\" and target == 1'))\nx_18 = len(train_df.query('diagnosis == \"atypical melanocytic proliferation\" and target == 1'))\n\ntotal = sum([x_1, x_2, x_3, x_4, x_5, x_6, x_7, x_8, x_9])\nx_1, x_2, x_3, x_4, x_5, x_6, x_7, x_8, x_9 = [y/total for y in [x_1, x_2, x_3, x_4, x_5, x_6, x_7, x_8, x_9]]\n\ntotal = sum([x_10, x_11, x_12, x_13, x_14, x_15, x_16, x_17, x_18])\nx_10, x_11, x_12, x_13, x_14, x_15, x_16, x_17, x_18 = [y/total for y in [x_10, x_11, x_12, x_13, x_14, x_15, x_16, x_17, x_18]]\n\n\nfig = go.Figure(data=[go.Bar(name='nevus', y=x, x=[x_1, x_10], orientation=\"h\",\n marker=dict(color='red', line=dict(width=0.5, color='black'))),\n go.Bar(name='unknown', y=x, x=[x_2, x_11], orientation=\"h\",\n marker=dict(color='orange', line=dict(width=0.5, color='black'))),\n go.Bar(name='melanoma', y=x, x=[x_3, x_12], orientation=\"h\",\n marker=dict(color='gold', line=dict(width=0.5, color='black'))),\n go.Bar(name='lentigo NOS', y=x, x=[x_4, x_13], orientation=\"h\",\n marker=dict(color='green', line=dict(width=0.5, color='black'))),\n go.Bar(name='solar lentigo', y=x, x=[x_5, x_14], orientation=\"h\",\n marker=dict(color='blue', line=dict(width=0.5, color='black'))),\n go.Bar(name='lichenoid keratosis', y=x, x=[x_6, x_15], orientation=\"h\",\n marker=dict(color='violet', line=dict(width=0.5, color='black'))),\n go.Bar(name='cafe-au-lait macule', y=x, x=[x_7, x_16], orientation=\"h\",\n marker=dict(color='indigo', line=dict(width=0.5, color='black'))),\n go.Bar(name='seborrheic keratosis', y=x, x=[x_8, x_17], orientation=\"h\",\n marker=dict(color='goldenrod', line=dict(width=0.5, color='black'))),\n go.Bar(name='atypical melanocytic proliferation', y=x, x=[x_9, x_18], orientation=\"h\",\n marker=dict(color='silver', line=dict(width=0.5, color='black')))])\n\nfig.update_layout(xaxis_title='Target', yaxis_title='Proportion', template=\"plotly_white\",\n title='Target vs. Proportion (per anatom_site_general_challenge)', barmode='stack', paper_bgcolor=\"#edebeb\"); fig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above graph clearly shows that all melanoma images are diagnosed as melanoma (as expected), and **most non-melanoma images are diagnosed as either \"unknown\" or \"nevus\".**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Approximate age vs. Target\n\nFinally let us see how the **approximate age** affects the **target**.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"nums_1 = train_df.query(\"target == 1\")[\"age_approx\"]\nnums_2 = train_df.query(\"target == 0\")[\"age_approx\"]\n\nnums_1 = nums_1.fillna(nums_1.mean())\nnums_2 = nums_2.fillna(nums_2.mean())\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n group_labels=[\"0\", \"1\"],\n colors=[\"darkorange\", \"dodgerblue\"], show_hist=False)\n\nfig.update_layout(title_text=\"Approximate age vs. Target\", xaxis_title=\"Approximate age\",\n template=\"plotly_white\", paper_bgcolor=\"#edebeb\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see from the distribution plot above that images with melanoma tend to come from younger patients with an average age of around 50, while non-melanoma images tend to come from older patients with an average age of around 65.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Display sample images\n\n* Now, I will show few sample images from the test directory to get an idea of what these skin images look like.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def display_images(num):\n sq_num = np.sqrt(num)\n assert sq_num == int(sq_num)\n\n sq_num = int(sq_num)\n image_ids = os.listdir(IMG_PATHS[0])\n fig, ax = plt.subplots(nrows=sq_num, ncols=sq_num, figsize=(20, 20))\n\n for i in range(sq_num):\n for j in range(sq_num):\n idx = i*sq_num + j\n ax[i, j].axis('off')\n img = cv2.imread(IMG_PATHS[0] + '/' + image_ids[idx])\n ax[i, j].imshow(img); ax[i, j].set_title('Test Image {}'.format(idx), fontsize=12)\n\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"display_images(36)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Modeling\n\n* Now since we understand the dataset better, it is time to build a machine learning pipeline to solve this problem.\n* I will use a classic image classification pipeline with image augmentation and a pretrained EfficientNet model.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Define PyTorch dataset\n\n* The first step is to build a PyTorch dataset to generate data for our model.\n* A PyTorch dataset has three fundamental functions: init, len, and getitem.\n* The init function initializes all the components required for data loading (image path, transforms, etc).\n* The len function simply returns the length of the dataset. This indicates the number of retrievable samples.\n* The getitem function returns a data point at a given index idx. The actual logic is written in this function.\n* The getitem function does 3 things: gets the target, loads the image, resizes it to (512, 512), and randomly flips.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def to_tensor(data):\n return [FloatTensor(point) for point in data]\n\ndef set_image_transformations(dataset, aug):\n norm = Normalize(mean=MU, std=SIGMA, p=1)\n vflip, hflip = VerticalFlip(p=0.5), HorizontalFlip(p=0.5)\n dataset.transformation = Compose([norm, vflip, hflip]) if aug else norm\n\nclass SIIMDataset(Dataset):\n def __init__(self, df, aug, targ, ids):\n set_image_transformations(self, aug)\n self.df, self.targ, self.aug, self.image_ids = df, targ, aug, ids\n\n def __len__(self):\n return len(self.image_ids)\n\n def __getitem__(self, i):\n image_id = self.image_ids[i]\n target = [self.df.target[i]] if self.targ else 0\n image = cv2.imread(PATH_DICT[image_id] + image_id)\n return to_tensor([self.transformation(image=image)['image'], target])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define EfficientNet-B3 model\n\n* Now we define the actual model which we are going to train.\n* The CancerNet model uses an **EfficientNet-B3** backbone with a **(Pooling + Dense)** head for classification.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def GlobalAveragePooling(x):\n return x.mean(axis=-1).mean(axis=-1)\n\nclass CancerNet(nn.Module):\n def __init__(self, features):\n super(CancerNet, self).__init__()\n self.avgpool = GlobalAveragePooling\n self.dense_output = nn.Linear(features, 1)\n self.efn = EfficientNet.from_pretrained(MODEL)\n \n def forward(self, x):\n x = x.view(-1, 3, H, W)\n x = self.efn.extract_features(x)\n return self.dense_output(self.avgpool(x))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define binary cross entropy and accuracy\n\n* Next, we define our loss function (**binary cross entropy**) and evaluation metric (**accuracy**).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def bce(y_true, y_pred):\n return nn.BCEWithLogitsLoss()(y_pred, y_true)\n\ndef acc(y_true, y_pred):\n y_true = y_true.squeeze()\n y_pred = nn.Sigmoid()(y_pred).squeeze()\n return (y_true == torch.round(y_pred)).float().sum()/len(y_true)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define helper function for training logs\n\n* Now we define a simple helper function to produce training logs (with foreground colors).","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def print_metric(data, batch, epoch, start, end, metric, typ):\n t = typ, metric, \"%s\", data, \"%s\"\n if typ == \"Train\": pre = \"BATCH %s\" + str(batch-1) + \"%s \"\n if typ == \"Val\": pre = \"\\nEPOCH %s\" + str(epoch+1) + \"%s \"\n time = np.round(end - start, 1); time = \"Time: %s{}%s s\".format(time)\n fonts = [(fg(211), attr('reset')), (fg(212), attr('reset')), (fg(213), attr('reset'))]\n print(pre % fonts[0] + \"{} {}: {}{}{}\".format(*t) % fonts[1] + \" \" + time % fonts[2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Split train/val (80/20)\n\n* We now split the data into training and validation sets (train: 80%, val: 20%).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"split = int(SPLIT*len(train_df))\ntrain_df, val_df = train_df.loc[:split], train_df.loc[split:]\ntrain_df, val_df = train_df.reset_index(drop=True), val_df.reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define PyTorch dataloaders, model, and optimizer\n\n* Next we define the PyTorch dataloader (for loading the train and val data).\n* We also define our model (CancerNet) and optimizer (ADAM) for training.\n* We use WeightedRandomSampler to fix data imbalance (with weight B for 1s)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Define sampling weights","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"C = np.array([B, (1 - B)])*2\nones = len(train_df.query('target == 1'))\nzeros = len(train_df.query('target == 0'))\n\nweightage_fn = {0: C[1]/zeros, 1: C[0]/ones}\nweights = [weightage_fn[target] for target in train_df.target]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define PyTorch datasets","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"length = len(train_df)\nval_ids = val_df.image_name.apply(lambda x: x + '.jpg')\ntrain_ids = train_df.image_name.apply(lambda x: x + '.jpg')\n\nval_set = SIIMDataset(val_df, False, True, ids=val_ids)\ntrain_set = SIIMDataset(train_df, True, True, ids=train_ids)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define sampling procedure and DataLoader","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_sampler = WeightedRandomSampler(weights, length)\nif_sample, if_shuffle = (train_sampler, False), (None, True)\nsample_fn = lambda is_sample, sampler: if_sample if is_sample else if_shuffle\n\nsampler, shuffler = sample_fn(SAMPLE, train_sampler)\nval_loader = DataLoader(val_set, VAL_BATCH_SIZE, shuffle=False)\ntrain_loader = DataLoader(train_set, BATCH_SIZE, sampler=sampler, shuffle=shuffler)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define model and optimizer","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"device = xm.xla_device()\nnetwork = CancerNet(features=1536).to(device)\noptimizer = Adam([{'params': network.efn.parameters(), 'lr': LR[0]},\n {'params': network.dense_output.parameters(), 'lr': LR[1]}])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train the model on one TPU core\n\n* Now, we train the model on one TPU core.\n* First, we do a **training loop**, where we train the model with back-prop to adjust the parameters.\n* Second, we do a **validation loop**, to check the model's performance on unseen data after each epoch.\n* The training loop uses **forward-prop and back-prop**, while the validation loop uses **only forward-prop.**\n* We use the torch.no_grad() flag for the validation loop as no gradients are needed for forward-prop.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"STARTING TRAINING ...\\n\")\n\nstart = time.time()\ntrain_batches = len(train_loader) - 1\n\nfor epoch in range(EPOCHS):\n fonts = (fg(48), attr('reset'))\n print((\"EPOCH %s\" + str(epoch+1) + \"%s\") % fonts)\n \n batch = 1\n network.train()\n for train_batch in train_loader:\n train_img, train_targ = train_batch\n train_targ = train_targ.view(-1, 1)\n train_img, train_targ = train_img.to(device), train_targ.to(device)\n \n if batch >= train_batches: break\n train_preds = network.forward(train_img)\n train_acc = acc(train_targ, train_preds)\n train_loss = bce(train_targ, train_preds)\n \n optimizer.zero_grad()\n train_loss.backward()\n xm.optimizer_step(optimizer, barrier=True)\n \n end = time.time()\n batch = batch + 1\n log = batch % 10 == 1\n accuracy = np.round(train_acc.item(), 3)\n if log: print_metric(accuracy, batch, 0, start, end, \"acc\", \"Train\")\n \n network.eval()\n val_loss, val_acc, val_points = 0, 0, 0\n \n with torch.no_grad():\n for val_batch in tqdm(val_loader):\n val_img, val_targ = val_batch\n val_targ = val_targ.view(-1, 1)\n val_img, val_targ = val_img.to(device), val_targ.to(device)\n\n val_points += len(val_targ)\n val_preds = network.forward(val_img)\n val_acc += acc(val_targ, val_preds).item()*len(val_preds)\n val_loss += bce(val_targ, val_preds).item()*len(val_preds)\n \n end = time.time()\n val_acc /= val_points\n val_loss /= val_points\n accuracy = np.round(val_acc, 3)\n print_metric(accuracy, 0, epoch, start, end, \"acc\", \"Val\")\n \n print(\"\")\n\nprint(\"ENDING TRAINING ...\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Run inference on the test data\n\n* Next I will run inference on the test data and store the test predictions in a list.\n* These predictions are logits and will be converted to probabilities later using sigmoid.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ids = test_df.image_name.apply(lambda x: x + '.jpg')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def sigmoid(x):\n return 1/(1 + np.exp(-x))\n\ntest_set = SIIMDataset(test_df, False, False, test_ids)\ntest_loader = tqdm(DataLoader(test_set, VAL_BATCH_SIZE, shuffle=False))\n\nnetwork.eval()\ntest_preds = []\nwith torch.no_grad():\n for test_batch in test_loader:\n test_img, _ = test_batch\n test_img = test_img.to(device)\n test_preds.extend(network.forward(test_img).squeeze().detach().cpu().numpy())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualize sample test predictions\n\n* Now since the model is trained, we will visualize predictions made on unseen test images.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def display_preds(num):\n sq_num = np.sqrt(num)\n assert sq_num == int(sq_num)\n\n sq_num = int(sq_num)\n image_ids = os.listdir(IMG_PATHS[0])\n few_preds = sigmoid(np.array(test_preds[:num]))\n pred_dict = {0: '\"No Melanoma\"', 1: '\"Melanoma\"'}\n fig, ax = plt.subplots(nrows=sq_num, ncols=sq_num, figsize=(20, 20))\n norm = Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], p=1)\n\n for i in range(sq_num):\n for j in range(sq_num):\n idx = i*sq_num + j\n ax[i, j].axis('off')\n pred = few_preds[idx]\n img = cv2.imread(IMG_PATHS[0] + '/' + image_ids[idx])\n ax[i, j].imshow(img); ax[i, j].set_title('Prediction: {}'.format(pred_dict[round(pred.item())]), fontsize=12)\n\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"display_preds(16)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Sigmoid the logits and generate submission file\n\n* Now we apply the sigmoid function to all the logits and prepare the final submission file.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '../input/siim-isic-melanoma-classification/'\nsample_submission = pd.read_csv(path + 'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission.target = sigmoid(np.array(test_preds))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Takeaways\n\n* Training is very fast on TPUs and should be used when possible.\n* Pretrained models like EfficientNet-B3 can generalize effectively to new tasks.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}