{"cells":[{"metadata":{},"cell_type":"markdown","source":"<h1>Melanoma Detection using CNN : Part-2</h1>\n<h2>Modelling and Inference</h2>\n\n---\n\nA typical deep learning pipeline involves the following steps-\n\n1. Preparing the data so that it can be used for training/inference. This involves creating custom datasets/dataloaders so that batches of data can be fed to the model.\n\n\\***NOTE**- *A general best practice is to split the training data set into training and validation sets, then training using only the training set at first, while using the validaton set for inference to test the model's performance. This gives a rough idea on how efficiently the model will perform on the test set.*\n\n2. Defining a model that will be used for training/inference. This step involves either using pretrained models, or creating a custom neural network from scratch.\n\n3. Once the model is defined, we need to define helper functions for model training. A typical training epoch involves the following operations-\n    * Forward propagation, where we use the model to predict the target values for a given training data. Then we use a criterion (loss function) to compare the predicted values against the actual target values and calculate the total information loss.\n    * Back propagation, where we calculate the gradients of the model parameters w.r.t. the total loss, then use an optimizer function to update the weights and biases accordingly.\n    * There is also an optional validation step where we evaluate the model performance on the validation data using the updated model parameters.****\n---","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Preparing the Dataset\n---\n\nIn this section, we will define the dataset and dataloaders. Let us begin with importing all the necessary project dependencies.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport torch\nimport numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.metrics import roc_auc_score\n\nfrom PIL import Image\n\nfrom torch.utils.data import DataLoader, Dataset, random_split\nimport torchvision.models as models\nimport torchvision.transforms as transforms\nimport torch.nn.functional as F\nimport torch.nn as nn\nfrom torchvision.utils import make_grid\n\n!pip install torchsummary --quiet","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's now load the training and testing dataframes.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that we have loaded the testing and training dataframes, let's create the custom PyTorch dataset objects that will allow us to use the image data with our PyTorch model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# creating custom pytorch dataset\nclass ImageDataset(Dataset):\n    \"\"\"Converts the images into a dataset that will allow us to use the image data with our model\n    \n    Args-\n        dataframe- pandas.DataFrame object containing image IDs and labels\n        data_dirs- The directory list in which the images are located\n        transforms- torchvision.transforms methods to apply transformations on the images\n        test- Boolean value, denotes whether the data is training or testing data\n    \"\"\"\n    def __init__(self, dataframe, data_dir, transform = None, test = False):\n        self.df = dataframe\n        self.data_dir = data_dir\n        self.transform = transform\n        self.test = test\n        \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, idx):\n        \n        if self.test:\n            img_name = self.df.loc[idx]['image_name']\n        else:\n            img_name, label = self.df.loc[idx][['image_name','target']]\n            label = torch.tensor(label)\n            label = label.unsqueeze(-1)\n            \n        try:\n            img_path = self.data_dir[0] + '/' + img_name + '.jpg'\n            img = Image.open(img_path)\n        except:\n            img_path = self.data_dir[1] + '/' + img_name + '.jpg'\n            img = Image.open(img_path)\n        \n        if self.transform:\n            img = self.transform(img)\n        if self.test:\n            return img \n        else:\n            return img, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"transform_1 = transforms.Compose([\n    transforms.ToTensor(),\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let us create our datasets.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# path of training data image directory\ntrain_dirs = ['../input/siimisic-melanoma-512x512-resized/train_1/train_1', '../input/siimisic-melanoma-512x512-resized/train_2/train_2']\n\n# path of test data image directory\ntest_dir = ['../input/siimisic-melanoma-512x512-resized/test/test']\n\n#creating training dataset object\ntrain_dataset = ImageDataset(train_df, train_dirs, transform = transform_1, test = False)\n\n#creating training dataset object\ntest_dataset = ImageDataset(test_df, test_dir, transform = transform_1, test = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us test our ImageDataset objects by printing the lengths of both the test set and the training set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"len(train_dataset), len(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, let us view some random images from our training and test datasets.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset[0][1].shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def image_viewer(dataset, index):\n    \"\"\"Shows image in the dataset at the index that the user provides.\n    \n    Args-\n        dataset- the custom dataset object\n        index- index of the image\n    Returns-\n        None\n    \"\"\"\n    label_dict = {0: 'Benign', 1: 'Malignant'}\n    try:\n        image, label = dataset[index] \n        print('Lesion type:', label_dict[label.squeeze().item()])\n    except:\n        image = dataset[index]\n    \n    plt.imshow(image.permute(1,2,0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_viewer(train_dataset, 69)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_viewer(test_dataset, 420)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that we know that our datasets are working fine, let us move to the next step. We need to split the training dataset into training and validation datasets. We will keep the validation set around 10% of the complete training dataset.\n\nAlso, since the dataset is very large, we will break down the trainig set into 4 parts to ease the training process.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"val_size = int(0.1 * len(train_dataset))\ntrain_size = len(train_dataset) - val_size\n\ntrain_ds, val_split_ds = random_split(train_dataset, lengths = [train_size, val_size])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us have a look at the number of instances in our training and validation splits.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"len(train_ds), len(val_split_ds) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, we need to define a method so that during the training/inference, the data is moved to the CUDA device (Nvidia GPU) in batches instead of all at once. This is very important since our dataset is huge and if we try to move the entire data to the GPU at once, it will run out of memory and we'll run into an error.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 64\n               \n# list of training dataloaders\ntrain_dl = DataLoader(\n    dataset = train_ds, \n    batch_size = batch_size, \n    shuffle = True, \n    num_workers = 4, \n    pin_memory = True)  \n\nval_dl = DataLoader(\n    dataset = val_split_ds, \n    batch_size = batch_size, \n    shuffle = False, \n    num_workers = 4, \n    pin_memory = True)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have defined the dataloaders. Now, let's view a batch of data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def batch_viewer(dataloader):\n    \"\"\"Shows the images in a batch of data\n    \n    Args-\n        dataloader- PyTorch dataloader object\n    \n    Returns-\n        None\n    \"\"\"\n    for images, labels in dataloader:\n        fig, ax = plt.subplots(figsize = (16,16))\n        ax.imshow(make_grid(images, nrow = 8).permute(1, 2, 0))\n        break  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_viewer(train_dl)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that we have prepared the data for training, we will define our model along with the helper functions required for training and inference.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Defining the Model\n\n---\n\nIn this section, we will define our **model class** along with the **criterion function**, **scoring function** and the various **helper functions** that will come handy while training and evaluating our model.\n\nFirst, let us create a base class where we will define all the helper functions as well as our loss and accuracy functions. Then we will simply inherit the base class to another class where we will define our actual model.\n\nThis will allow us to keep our code clean and modular. Instead of redefining different model functions again and again from scratch for different architectures, we can simply define a new model while our helper functions will remain the same.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def accuracy(output, labels):\n    \"\"\"Calculates the accuracy for the predicted output and the actual label values.\n    \n    Agrs-\n        output- Output tensor generated by the model\n        labels- Actual labels for the given batch of data\n    \n    Returns-\n        accuracy- accuracy for the predictions\n    \"\"\"\n    labels = labels.squeeze()\n    output = output.squeeze()\n    return (labels == torch.round(output)).float().sum()/len(labels)\n    \n\nclass MelanomaClassificationBase(nn.Module):\n    def training_step(self, batch):\n        \"\"\"Calculates the cross entropy loss for a given batch of data.\n        \n        Args-\n            batch- One batch of data as generated by the data loader\n        \n        Returns-\n            batch_loss- Total cross entropy loss for the batch\n        \"\"\"\n        images, labels = batch \n        output = self(images)\n        batch_loss = F.binary_cross_entropy(output.double(), labels.double()) # calculates loss for the predictions and actual labels\n        return batch_loss\n    \n    def validation_step(self, batch):\n        \"\"\"Calculates total validation loss and validation accuracy for a given batch data during a validation step.\n        \n        Args-\n            batch- One batch of data as generated by the data loader\n            \n        Returns-\n            A dictionary object containing validation loss and validation accuracy for the given batch\n        \"\"\"\n        images, labels = batch \n        output = self(images)\n        batch_loss = F.binary_cross_entropy(output.double(), labels.double())   # calculates batch loss\n        batch_acc = accuracy(output, labels)           # calculate batch accuracy\n        return {'val_loss': batch_loss.detach(), 'val_acc': batch_acc}\n        \n    def validation_epoch_end(self, outputs):\n        \"\"\"Calculates mean validation loss and mean validation accuracy for a one validation epoch.\n        \n        Args-\n            outputs- A list of dictionary objects containing validation accuracy and validation loss for each batch of data in one epoch\n            \n        Returns-\n            A dictionary object containing validation loss and validation accuracy for the given batch\n        \"\"\"\n        batch_losses = [batch_val_dict['val_loss'] for batch_val_dict in outputs] # creates a list of batch losses for all the batches in one validation epoch\n        epoch_loss = torch.stack(batch_losses).mean()   # calculates mean validation loss for the epoch\n        batch_accs = [batch_val_dict['val_acc'] for batch_val_dict in outputs]   # creates a list of batch accuracies for all the batches in one validation epoch \n        epoch_acc = torch.stack(batch_accs).mean()      # calculates mean validation accuracy for the epoch\n        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}\n    \n    def epoch_end(self, epoch, result):\n        print(\"Epoch [{}], last_lr: {:.5f}, train_loss: {:.4f}, val_loss: {:.4f}, val_acc: {:.4f}\".format(\n            epoch, \n            result['lrs'][-1], \n            result['train_loss'], \n            result['val_loss'], \n            result['val_acc']))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that we have defined the helper functions, let us finally define the model. We are going to use a ResNet34 model that we are going to train from scratch. \n\n**NOTE**- We are not using a pretrained model since the pretrained model was actually trained on the ImageNet dataset, which is quite a lot different from our data. So the pretrained model parameters will not add much to the model performance.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class MelanomaClassifier(MelanomaClassificationBase):\n    def __init__(self):\n        super().__init__()\n        self.network = models.resnet34(pretrained=False, progress=False)\n        # modifying the last fully-connected layer\n        num_ftrs = self.network.fc.in_features \n        self.network.fc = nn.Linear(num_ftrs, 1)\n    \n    def forward(self, xb):\n        return torch.sigmoid(self.network(xb))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = MelanomaClassifier()\nmodel","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating CUDA Wrappers\n---\nLet us now define a method that allows us to push our model as well as the dataset onto the GPU during the training/testing process. Training on the GPU supports parallalization, processing batches of data at the same time thus increasing the training and inference process. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_default_device():\n    \"\"\"Picks the trainig device-- GPU if available, else CPU.\n    \"\"\"\n    if torch.cuda.is_available():   # checks if a cuda device is available\n        return torch.device('cuda') # sets the default device as the available CUDA device\n    else:\n        return torch.device('cpu')  # if no CUDA device found, sets CPU as the default device\n    \ndef to_device(data, device):\n    \"\"\"Move tensor(s) to chosen device\n    \"\"\"\n    if isinstance(data, (list,tuple)): # asserts if the data is a list/tuple \n        return [to_device(x, device) for x in data]\n    return data.to(device, non_blocking=True)\n\nclass DeviceDataLoader():\n    \"\"\"Wrap a dataloader to move data to the default device.\n    \"\"\"\n    def __init__(self, dataloader, device):\n        self.dl = dataloader\n        self.device = device\n        \n    def __iter__(self):\n        \"\"\"Yield a batch of data after moving it to device.\n        \"\"\"\n        for batch in self.dl: \n            yield to_device(batch, self.device)\n\n    def __len__(self):\n        \"\"\"Prints the total number of batches.\n        \"\"\"\n        return len(self.dl)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With this done, we have successfully defined the method that allows batches of data generated by the dataloader to the CUDA device (our Nvidia GPU).\n\nNow, let us check the default device we have available for the runtime.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"device = get_default_device()\ndevice","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, the default device is CUDA. Now, let us move our device and dataloaders onto the default device.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dl = DeviceDataLoader(train_dl, device)\nval_dl = DeviceDataLoader(val_dl, device)\n\nmodel.to(device, non_blocking=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us now have a look at the model summary.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from torchsummary import summary\n\nsummary(model, input_size=(3, 512, 512))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With the model and the dataloader moved to the GPU, we will now test the model on a batch of data to make sure everything is working fine before we actually move on to training/inference part. This is like a precautionary step to make sure that we don't run into any major problem while the actual training/inference phase.\n\nWe will feed our model with a batch of data from the training dataloader.\n\n* The dimension of the input will be- batch_size x 3 x img_width x img_height\n* The output dimension we expect is- batch_size x num_classes(2)\n\nIf everything works as expected, we will move on to the final step of our project-- training and evaluating our model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndef try_batch(dataloader):\n    for images, labels in dataloader:\n        print('images.shape:', images.shape)\n        out = model(images)\n        print('out.shape:', out.shape)\n        print('out[0]:', out[0])\n        break\n\ntry_batch(train_dl)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model Training and Validation\n---\n\nAs we can see here, everything worked just fine. We will now define the training method.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from tqdm.notebook import tqdm\n\n@torch.no_grad()\ndef evaluate(model, val_loader):\n    model.eval()    # sets the model to evaluation mode\n    outputs = [model.validation_step(batch) for batch in val_loader] # performs validation for each batch and stores it in a list\n    return model.validation_epoch_end(outputs) # returns mean validation accuracy and validation loss for one complete epoch\n\ndef get_lr(optimizer):\n    \"\"\"Gets the learning rate of the optimizer.\n    \"\"\"\n    for param_group in optimizer.param_groups:\n        return param_group['lr']\n\ndef fit_one_cycle(epochs, max_lr, model, train_loader, val_loader, \n                  weight_decay = 0, grad_clip = None, opt_func = torch.optim.SGD):\n    \n    torch.cuda.empty_cache()    # clears cache in CUDA device\n    history = []    # declares an empty list to store result for each epoch\n    \n    # sets up custom optimizer with weight decay\n    optimizer = opt_func(model.parameters(), max_lr, weight_decay = weight_decay)\n    \n    # sets up one-cycle learning rate scheduler\n    scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr, epochs = epochs, steps_per_epoch = len(train_loader))\n    \n    for epoch in range(epochs): \n        model.train()    # initiate training phase\n        train_losses = []\n        lrs = []\n        for batch in tqdm(train_loader):    # cycles through each batch of the training data\n            loss = model.training_step(batch)    \n            train_losses.append(loss)\n            loss.backward()\n            \n            # perfomrs gradient clipping\n            if grad_clip: \n                nn.utils.clip_grad_value_(model.parameters(), grad_clip)\n            \n            optimizer.step() # updates parameters based on gradients obtained via optimizer.backwards()\n            optimizer.zero_grad() # resets gradient values\n            \n            # records & updates learning rate\n            lrs.append(get_lr(optimizer))\n            scheduler.step()\n        \n        # initaites validation phase\n        result = evaluate(model, val_loader)\n        result['train_loss'] = torch.stack(train_losses).mean().item()\n        result['lrs'] = lrs\n        model.epoch_end(epoch, result)\n        history.append(result)\n    return history","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Let us now train the model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = 20\nmax_lr = 0.01\ngrad_clip = 0.1\nweight_decay = 1e-3\nopt_func = torch.optim.Adam","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nhistory = fit_one_cycle(epochs, max_lr, model, train_dl, val_dl, \n                             grad_clip=grad_clip, \n                             weight_decay=weight_decay, \n                             opt_func=opt_func)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"training_complete = True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Saving the Model for Later Use\n\n---\nNow that we have completed the training, we will save the model state so we can use it later for more training.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(model.state_dict(), 'melanomaClassification_ResNet34.pth')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Making predictions & submission\n\n---\nNow that the training is complete. let us use the model to make predictions on our test set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dl = DataLoader(\n    dataset = test_dataset, \n    batch_size = batch_size, \n    shuffle = False, \n    num_workers = 4, \n    pin_memory = True)\n\ntest_dl = DeviceDataLoader(test_dl, device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, we will define the predictor function.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"@torch.no_grad()\ndef predict_dl(test_dl, model):\n    torch.cuda.empty_cache()\n    batch_probs = []\n    for images in tqdm(test_dl):\n        probs = model(images)\n        batch_probs.append(probs.cpu().detach())\n    batch_probs = torch.cat(batch_probs)\n    return batch_probs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_preds = predict_dl(test_dl, model)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df = pd.read_csv('../input/siim-isic-melanoma-classification/sample_submission.csv')\nsubmission_df.target = test_preds.numpy()\nsubmission_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Saving the dataframe to submission CSV.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_fname = 'resnet34_submission.csv'\nsubmission_df.to_csv(submission_fname, index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With this, we come to an end of our project. Feel free to use this starter code. Good luck and keep learning!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}