{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Efficient net + K-fold Model\n\nThis notebook presents work on the classification task of identifying malignant melanoma based on patient images. \n\nThe general approach is to adjust a pre-trained efficientnet to better reflect the presented classification task. This process, called transfer-learning, is presented in other public notebooks in this competeition and has proven to be an effective approach for classifying images. \n\nUsing PyTorch, ths work is a relatively simple approach (minimal data augmentation, minimal adjustments to efficientnet model) that leverages a k-fold learning scheme where final test predictions are the ensemble average of predictions from each fold. \n\nOrganization of the notebook:\n1. Load the efficientnet model and required libraries \n\n2. Discuss k-fold partitioning of data \n\n3. Construct data-loaders (apply noise to training images) \n\n4. Define and train the model \n\n5. Show OOF (out of fold) prediction performance  \n\n6. Create submission file ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 1. Load efficientnet and required libraries ","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!pip install efficientnet_pytorch torchtoolbox","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt \n\nimport torch \nimport torch.nn as nn\nimport torchvision\nimport torchvision.transforms as transforms\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\n\nfrom efficientnet_pytorch import EfficientNet\n\nfrom skimage.transform import resize\nfrom sklearn.metrics import confusion_matrix, roc_curve, auc\nfrom scipy.stats import ks_2samp","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Device configuration (GPU can be enabled in settings)\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n#device = 'cpu'\nprint(device)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# upload train dataframe\npath = '/kaggle/input/jpeg-melanoma-256x256/'\ntrain_df_2020 = pd.read_csv(path + 'train.csv')\ntrain_df_2019 = pd.read_csv(\"/kaggle/input/jpeg-isic2019-256x256/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv(path + 'test.csv')\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. TFRecord describes stratified k-fold to keep counts of patient replicates evenly distributed within each fold\n\nSome patients have as many as 115 images - to keep validation more representative of a real testing scenario, training and validation should have a similar distribution of the number of replicates from each patient. The tfrecord column (data organized by ****Chris Deotte) in the data table provides a k-fold partitioning scheme that keeps replicates even distributed in each fold.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"unique_patient_ids, patient_counts = np.unique(train_df_2020['patient_id'].values, return_counts=True)\npatient_to_count = {p_id:count for p_id, count in zip(unique_patient_ids, patient_counts)}\n\ntfrecords = train_df_2020['tfrecord'].values \nfolds = np.arange(15)\n\nplt.style.use('seaborn-colorblind')\nplt.rcParams.update({'font.size': 16,\n                     'legend.framealpha':.5,\n                     'legend.edgecolor':'k',\n                     'axes.edgecolor':'k'})\nplt.figure(figsize=(5*5, 6*3))\nk = 1 \nfor f in folds:\n    fold_df = train_df_2020.iloc[tfrecords == folds[f], :]\n\n    fold_patient_ids = fold_df['patient_id'].values \n    fold_patient_counts = [patient_to_count[p] for p in fold_patient_ids]\n    fold_target = fold_df['target'].values\n    fold_p_target = 100 * sum(fold_target) / len(fold_target)\n\n    plt.subplot(5, 3, k)\n    k += 1\n    plt.hist(fold_patient_counts)\n    plt.xlabel(\"Patient replicate counts\")\n    plt.ylabel(\"In-fold instances\")\n    plt.title(\"Percent of pos. target = {:.2f}\".format(fold_p_target))\n    \nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 2020 data has 15 folds, while 2019 data has 30. Need to convert to system with 5 folds for each \nold_folds = [-1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]\nnew_folds = [-1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1,  2,  2,  2,  2,  2]\nconverter = {o:n for o, n in zip(old_folds, new_folds)}\ntf_2020 = train_df_2020.tfrecord.values\ntf_2020_fixed = [converter[o] for o in tf_2020]\ntrain_df_2020['tfrecord'] = tf_2020_fixed\n\nold_folds = [-1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]\nnew_folds = [-1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2]\nconverter = {o:n for o, n in zip(old_folds, new_folds)}\ntf_2019 = train_df_2019.tfrecord.values\ntf_2019_fixed = [converter[o] for o in tf_2019]\ntrain_df_2019['tfrecord'] = tf_2019_fixed\n\ntrain_df_allsamples = pd.concat((train_df_2020, train_df_2019))\ntfrecords = train_df_allsamples.tfrecord.values\ntrain_df_allsamples.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create a dictionary that links image names to path \nimg_names_2019 = train_df_2019.image_name.values \npath_dict = {img_name:\"/kaggle/input/jpeg-isic2019-256x256/train/\" for img_name in img_names_2019}\n\nimg_names_2020 = train_df_2020.image_name.values \nfor img_name in img_names_2020:\n    path_dict[img_name] = \"/kaggle/input/jpeg-melanoma-256x256/train/\"\n    \ntest_names = test_df.image_name.values\nfor img_name in test_names:\n    path_dict[img_name] = \"/kaggle/input/jpeg-melanoma-256x256/test/\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Define data loaders using PyTorch Dataset class","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"meta_features = ['sex', 'age_approx', 'anatom_site_general_challenge'] \n\nencoder = {}\nfor feature in meta_features: \n    # determine unique features  \n    categories = np.unique(np.array(train_df_allsamples[feature].values, str))\n    for i, category in enumerate(categories): \n        if category != 'nan':\n            encoder[category] = np.float(i)\nencoder['nan'] = np.nan\n\n# define a unique transform each time a positive is resampled: \n\n# basic transform for all images \ntransform_basic = transforms.Compose([\n    transforms.ToPILImage(),\n    transforms.RandomRotation(degrees=8),\n    transforms.ColorJitter(brightness=32. / 255.,saturation=0.5),\n    transforms.RandomResizedCrop(size=256, scale=(0.9, 1.1), ratio=(0.9, 1.1)),\n    transforms.ToTensor(),\n    transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])])\n\n# additional transform to augment positive samples \ntransform_augment_h = transforms.Compose([\n    transforms.ToPILImage(), \n    transforms.RandomResizedCrop(size=256, scale=(0.9, 1.1), ratio=(0.9, 1.1)),\n    transforms.RandomHorizontalFlip(1),\n    transforms.ToTensor(),\n    transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])])\n\n# no flip or rotation for test/validation data \ntransform_valid = transforms.Compose([\n    transforms.ToTensor(),\n    transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])])\n\nclass TrainDataset(torch.utils.data.Dataset):\n    def __init__(self, df, path_dict):\n        # 1. Initialize file paths or a list of file names.\n        self.path_dict = path_dict\n        self.df = df\n\n    def __getitem__(self, index):\n        # 1. Read one data from file (e.g. using numpy.fromfile, PIL.Image.open).\n        \n        # load X \n        img_name = self.df['image_name'].values[index]\n        img_path = self.path_dict[img_name] + img_name + \".jpg\"\n        img = plt.imread(img_path) \n        \n        # determine meta data \n        meta = self.df[meta_features].values[index]\n        meta_data = np.array([encoder[str(m)] for m in meta])\n        \n        # load y \n        label = self.df[\"target\"].values[index]\n        label_encode = np.zeros(2)\n        label_encode[label] = 1\n        target = torch.tensor(label, dtype=torch.float32)\n        \n        # 2. Preprocess the data (e.g. torchvision.Transform) \n        img_processed = transform_basic(img)\n        # 3. Return a data pair (e.g. image and label).\n        return img_processed, meta_data, target\n        \n    def __len__(self):\n        # total size of your dataset.\n        return self.df.shape[0]\n\nclass ValidDataset(torch.utils.data.Dataset):\n    def __init__(self, df, path_dict):\n        # 1. Initialize file paths or a list of file names.\n        self.path_dict = path_dict\n        self.df = df\n\n    def __getitem__(self, index):\n        # 1. Read one data from file (e.g. using numpy.fromfile, PIL.Image.open).\n        \n        # load X \n        img_name = self.df['image_name'].values[index]\n        img_path = self.path_dict[img_name] + img_name + \".jpg\"\n        img = plt.imread(img_path) \n        \n        # determine meta data \n        meta = self.df[meta_features].values[index]\n        meta_data = np.array([encoder[str(m)] for m in meta])\n        \n        # load y \n        label = self.df[\"target\"].values[index]\n        label_encode = np.zeros(2)\n        label_encode[label] = 1\n        target = torch.tensor(label, dtype=torch.float32)\n        \n        # 2. Preprocess the data (e.g. torchvision.Transform).\n        img_processed = transform_valid(img)\n        # 3. Return a data pair (e.g. image and label).\n        return img_processed, meta_data, target\n        \n    def __len__(self):\n        # total size of your dataset.\n        return self.df.shape[0]\n    \nclass TestDataset(torch.utils.data.Dataset):\n    def __init__(self, df, path_dict):\n        # 1. Initialize file paths or a list of file names.\n        self.path_dict = path_dict\n        self.df = df\n\n    def __getitem__(self, index):\n        # 1. Read one data from file (e.g. using numpy.fromfile, PIL.Image.open).\n\n        # load X\n        img_name = self.df['image_name'].values[index]\n        img_path = self.path_dict[img_name] + img_name + \".jpg\"\n        img = plt.imread(img_path)\n\n        # determine meta data\n        meta = self.df[meta_features].values[index]\n        meta_data = np.array([encoder[str(m)] for m in meta])\n\n        # 2. Preprocess the data (e.g. torchvision.Transform)\n        img_processed = transform_valid(img)\n        # 3. Return a data pair (e.g. image and label).\n        return img_processed, meta_data\n\n    def __len__(self):\n        # total size of your dataset.\n        return self.df.shape[0]\n\ndef AugmentBatch(images, labels):\n    \n    # find positives in data set \n    for img, label in zip(images, labels):\n        if label == 1:\n            # transform image with horizontal flip\n            img_aug = transform_augment_h(img).unsqueeze_(0)\n            # append set of augmented images and labels \n            images = torch.cat((images, img_aug)) \n            labels = torch.cat((labels, torch.Tensor([label])))\n\n    # shuffle \n    shuffle_inds = torch.randperm(images.size()[0])\n    images = images[shuffle_inds, :, :, :]\n    labels = labels[shuffle_inds]\n    return images, labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Define and train neural network model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convolutional neural network\nclass MyENet(nn.Module):\n    def __init__(self, ENet):\n        super(MyENet, self).__init__()\n        # modify output layer of the pre-trained ENet \n        self.ENet = ENet\n        num_ftrs = self.ENet._fc.in_features\n        self.ENet._fc = nn.Linear(in_features=num_ftrs, out_features=1024)\n        # map Enet output to melanoma decision \n        self.output = nn.Sequential(nn.LeakyReLU(),\n                                    nn.Dropout(p=0.4),\n                                    nn.Linear(1024, 1),\n                                    nn.Sigmoid())\n        \n    def embedding(self, x):\n        out = self.ENet(x)\n        return out \n        \n    def forward(self, x):\n        out = self.ENet(x)\n        out = self.output(out)\n        return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train the model\n# Use the prebuilt data loader.\ntrain_path = path + \"train/\"\ntest_path  = path + \"test/\"\n\n# define batch size and accumulation steps \nnum_epochs = 10\nbatchsize  = 106\naccumulation_steps = 1 # ~ sort of like making an effective batchsize = acc.steps * batchsize ? \nevaluation_steps = 50\nset_patience = 3\nverbose = True\n\n# record out-of-fold predictions and test predictions \nnn_oof = np.zeros(len(train_df_allsamples))\nnn_predictions = np.zeros(len(test_df))\n\n# create a test loader \ntest_dataset = TestDataset(test_df, path_dict)                                              \ntest_loader = torch.utils.data.DataLoader(dataset=test_dataset, batch_size=batchsize) \n        \n# set k-folds\nfold_sets = [0, 1, 2]\n\n# loop through every fold \nfor f, fold_set in enumerate(fold_sets):\n    # initialize new model for each fold \n    ENet = EfficientNet.from_pretrained('efficientnet-b0').to(device)\n    model = MyENet(ENet).to(device)\n    path_to_model = 'model_{}.ckpt'.format(f+1)\n    \n    # reset best val and patience \n    best_val = 0\n    ks_stat = 0\n    patience = set_patience\n    \n    # define lr and optimizer \n    optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)\n    criterion = torch.nn.BCELoss()\n    \n    # scheduler reduces learning rate by factor of 10 when val auc does not improve\n    scheduler = ReduceLROnPlateau(optimizer=optimizer, min_lr=1e-6, mode='max', patience=0, verbose=True)\n    \n    # pull the training and validation data for each fold\n    inds = np.in1d(tfrecords, np.array(fold_set))\n    train_df = train_df_allsamples.iloc[~inds, :]\n    val_df = train_df_allsamples.iloc[inds, :]\n\n    # evaluate performance on validation data \n    train_dataset = TrainDataset(train_df, path_dict)                                              \n    train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batchsize, shuffle=True) \n    \n    # evaluate performance on validation data \n    valid_dataset = ValidDataset(val_df, path_dict)                                              \n    valid_loader = torch.utils.data.DataLoader(dataset=valid_dataset, batch_size=batchsize) \n    \n    # save validation statistics \n    val_roc = []\n    val_ks_stat = []\n    \n    # Reset gradients\n    for epoch in range(num_epochs):\n        \n        # set up model for training     \n        model.train()\n        for i, (images, meta_data, labels) in enumerate(train_loader):\n\n            # augment the batch with more positive samples \n            images, labels = AugmentBatch(images, labels)\n            \n            # send images and labels to gpu or cpu RAM \n            images = images.to(device)\n            labels = torch.reshape(labels, [len(labels), 1])\n            labels = labels.to(device)\n\n            # Forward pass\n            outputs = model(images)   \n            \n            # Compute loss \n            loss = criterion(outputs, labels)\n            loss = loss / accumulation_steps                # Normalize loss (if averaged)\n            loss.backward()                                 # Backward pass\n            if (i+1) % accumulation_steps == 0:             # Wait for several backward steps\n                optimizer.step()                            # Now we can do an optimizer step\n                model.zero_grad()                           # Reset gradients tensors\n            \n            # calculate performance \n            train_targets = np.array(labels.cpu().numpy(), np.int)\n            train_predictions = outputs.detach().cpu().numpy().ravel()\n\n            # if positive samples existed in batch \n            if sum(train_targets)[0] > 0:\n                fpr, tpr, _ = roc_curve(train_targets, train_predictions)\n                roc = auc(fpr, tpr)\n                \n                ks_stat, p = ks_2samp(np.array(train_predictions).ravel()[np.array(train_targets).ravel()==0], \n                                   np.array(train_predictions).ravel()[np.array(train_targets).ravel()==1])\n\n                if verbose and (i+1) % evaluation_steps == 0:\n                    # Print performance on minibatch train data \n                    print ('\\nFold [{}/{}], Epoch [{}/{}], Step [{}/{}], ROC AUC: {:.3f}, Loss: {:.3f}, K-S stat: {:.3f}\\n'\n                       .format(f+1, len(fold_sets), epoch+1, num_epochs, i+1, len(train_loader), roc, loss.detach().cpu().numpy(), ks_stat))\n                \n        \n        # Evaluate validation data at end of each epoch\n        valid_predictions = []\n        valid_targets = []\n        model.eval() \n        with torch.no_grad():\n            for j, (images, meta_data, labels) in enumerate(valid_loader):\n                images = images.to(device)\n\n                labels = torch.reshape(labels, [len(labels), 1])\n                labels = labels.to(device)\n\n                # Forward pass\n                outputs = model(images)\n\n                # Store predictions and true values \n                valid_predictions += list(outputs.detach().cpu().numpy())\n                valid_targets += list(labels.cpu().numpy())\n\n        # Calculate performance statistics on validation data \n        fpr, tpr, _ = roc_curve(np.array(valid_targets, np.int), np.array(valid_predictions).ravel())\n        val_roc_epoch = auc(fpr, tpr)\n        val_roc.append(val_roc_epoch)\n        val_ks_stat_epoch, p = ks_2samp(np.array(valid_predictions).ravel()[np.array(valid_targets).ravel()==0], \n                                        np.array(valid_predictions).ravel()[np.array(valid_targets).ravel()==1])\n        val_ks_stat.append(val_ks_stat_epoch)\n\n        print('\\nFold [{}/{}], Epoch [{}/{}], Val ROC AUC: {:.3f}, Val K-S stat: {:.3f}\\n'\n               .format(f+1, len(fold_sets), epoch+1, num_epochs, val_roc_epoch, val_ks_stat_epoch))\n\n        # learning rate is reduced if val roc doesn't improve \n        scheduler.step(val_roc_epoch)\n        \n        # Save model if validation performance improved \n        if val_roc_epoch > best_val:\n            best_val = val_roc_epoch\n            # patience = set_patience (takes too long to reset patience every time the score improves)     \n            print(\"Saving model...\")\n            torch.save(model.state_dict(), path_to_model)  \n        else:\n            # re-do epoch with the best model and a reduced learning rate\n            model.load_state_dict(torch.load(path_to_model))\n            patience -= 1\n            if patience == 0:\n                print('Early stopping. Best validation roc_auc: {:.3f}'.format(best_val))\n                break\n\n    # Load best model from fold (in case last epoch was not best)\n    model.load_state_dict(torch.load(path_to_model))\n    \n    # at the end of fold, use model to make predictions on testing data  \n    model.eval()\n    \n    print(\"Making predictions on test data...\")\n    test_predictions = []\n    with torch.no_grad():\n        for i, (images, meta_data) in enumerate(test_loader):\n            images = images.to(device)\n\n            # Forward pass\n            outputs = model(images)\n            \n            # append predictions\n            test_predictions += list(outputs.detach().cpu().numpy())\n    test_predictions = np.array(test_predictions).ravel()\n    \n    # save out of fold predictions\n    nn_oof[inds] += np.array(valid_predictions).ravel()\n    \n    # save test predictions \n    nn_predictions += test_predictions / len(fold_sets)\n\n    # plot validation performance over epochs \n    plt.style.use('seaborn-colorblind')\n    plt.rcParams.update({'font.size': 16, \n                         'legend.framealpha':1, \n                         'legend.edgecolor':'inherit'}) \n    plt.figure(figsize=(9, 6))\n\n    plt.plot(val_roc)\n    plt.title(\"Validation ROC AUC Fold {}\".format(f+1))\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Show performance on out of fold validation samples","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.style.use('seaborn-colorblind')\nplt.rcParams.update({'font.size': 16, \n                     'legend.framealpha':1, \n                     'legend.edgecolor':'inherit'}) \nplt.figure(figsize=(9, 6))\n\nvalid_pred = nn_oof\ny_valid = train_df_allsamples['target'].values\n\ncounts, bins = np.histogram(np.array(valid_pred), bins=50)\nplt.hist(valid_pred[y_valid==0], \n         density=True, bins=bins, label='p(malignant | benign img) ', alpha=.75)\nplt.hist(valid_pred[y_valid==1],\n         density=True, bins=bins, label='p(malignant | malignant img)', alpha=.75)\n\nks, p = ks_2samp(valid_pred[y_valid==0], \n                 valid_pred[y_valid==1])\n\nfpr, tpr, _ = roc_curve(y_valid, valid_pred)\nroc_auc = auc(fpr, tpr)\n\nplt.xlim([0, 1])\nplt.legend()\nplt.title(\"NN fit to validation data: K-S = {:.3f}, AUC = {:.3f}\".format(ks, roc_auc))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6. Create submission file","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"image_names = test_df['image_name'].values\n\nsubmission = pd.DataFrame()\nsubmission[\"image_name\"] = image_names\nsubmission[\"target\"] = nn_predictions\nsubmission.to_csv(\"submission.csv\", index=False)\nsubmission.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}