{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Versions\n\n* **2.0 -** Added effnet b2, changed transform size to 256, added more transforms: HorizontalFlip(), VerticalFlip(), ColorJitter(brightness=32. / 255.,saturation=0.5,hue=0.01)\n\n* **3.0 -** Added Hairs for augmentation and Microscope, Changed final prediction plot","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{"id":"FBQocLLMEOkA"}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sb\nimport cv2\nfrom PIL import Image\nimport gc\n\nimport torch\nfrom torch import nn\nfrom torch import optim\nimport torch.nn.functional as F\nfrom torchvision import datasets, transforms, models\nfrom torch.utils.data import Dataset\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\n\n!pip install torchtoolbox\nimport torchtoolbox.transform as transforms\nfrom torch.utils.data import Dataset, DataLoader, Subset\n\n\nimport time\nimport datetime\nimport random\n\nfrom sklearn.model_selection import StratifiedKFold, GroupKFold, train_test_split\nfrom sklearn.metrics import accuracy_score, roc_auc_score, confusion_matrix\n\n!pip install efficientnet_pytorch\nfrom efficientnet_pytorch import EfficientNet\n\nimport os \n\nimport warnings\nwarnings.simplefilter('ignore')","metadata":{"scrolled":true,"id":"vYU2bLwBBOA_","outputId":"1c817d42-0ae5-46e3-d1fa-c9134b145420","execution":{"iopub.status.busy":"2022-11-11T15:26:07.450176Z","iopub.execute_input":"2022-11-11T15:26:07.450625Z","iopub.status.idle":"2022-11-11T15:26:18.415963Z","shell.execute_reply.started":"2022-11-11T15:26:07.45058Z","shell.execute_reply":"2022-11-11T15:26:18.412853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Seeds","metadata":{"id":"X-i6D7jah5KX"}},{"cell_type":"code","source":"# Creating seeds to make results reproducible\ndef seed_everything(seed_value):\n    np.random.seed(seed_value)\n    torch.manual_seed(seed_value)\n    os.environ['PYTHONHASHSEED'] = str(seed_value)\n    \n    if torch.cuda.is_available(): \n        torch.cuda.manual_seed(seed_value)\n        torch.cuda.manual_seed_all(seed_value)\n        torch.backends.cudnn.deterministic = True\n        torch.backends.cudnn.benchmark = True\n\nseed = 1234\nseed_everything(seed)","metadata":{"id":"AJZ6Z9puh6By","execution":{"iopub.status.busy":"2022-11-11T15:26:18.4173Z","iopub.status.idle":"2022-11-11T15:26:18.417688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Setting up the Device ","metadata":{"id":"EJIdZdAf9rXD"}},{"cell_type":"code","source":"# Setting up GPU for processing or CPU if GPU isn't available\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint (device)","metadata":{"id":"QwoV7i5g9zrI","outputId":"8473f840-069c-45ff-9d1f-483ddb15ee24","execution":{"iopub.status.busy":"2022-11-11T15:26:18.418764Z","iopub.status.idle":"2022-11-11T15:26:18.419486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading the data\n\nHere I'll use `torchvision` to load the data ([documentation](http://pytorch.org/docs/0.3.0/torchvision/index.html)). The data should be included alongside this notebook. The dataset is split into three parts, training, validation, and testing. For the training, I'll apply transformations such as  random scaling, cropping, and flipping. This will help the network generalize leading to better performance. The input data must be resized to 256x256 pixels as required by the pre-trained networks (in this case the images are already 256x256).\n\nThe validation and testing sets are used to measure the model's performance on data it hasn't seen yet. For this no scaling or rotation transformations will be applied, but the images still need to be resized and then croped to the appropriate size.\n\nThe pre-trained networks I'll use were trained on the ImageNet dataset where each color channel was normalized separately. For all three sets I'll need to normalize the means and standard deviations of the images to what the network expects. For the means, it's `[0.485, 0.456, 0.406]` and for the standard deviations `[0.229, 0.224, 0.225]`, calculated from the ImageNet images.  These values will shift each color channel to be centered at 0 and range from -1 to 1.\n ","metadata":{"id":"20tgWUAbBOBK"}},{"cell_type":"markdown","source":"## Creating Custom Dataset Class","metadata":{"id":"ofellIJSiPc0"}},{"cell_type":"code","source":"class CustomDataset(Dataset):\n  def __init__(self, df: pd.DataFrame, img_dir, train: bool = True, transforms= None):\n    self.df = df\n    self.img_dir = img_dir\n    self.transforms = transforms\n    self.train = train\n\n  def __getitem__(self, index):\n    img_path = os.path.join(self.img_dir, self.df.iloc[index]['image_name'] + '.jpg')\n    #images = Image.open(img_path)\n    images = cv2.imread(img_path)\n\n    if self.transforms:\n        images = self.transforms(images)\n\n    if self.train:\n        labels = self.df.iloc[index]['target']\n        #return images, labels\n        return torch.tensor(images, dtype=torch.float32), torch.tensor(labels, dtype=torch.float32)\n    \n    else:\n        #return (images)\n        return torch.tensor(images, dtype=torch.float32)\n    \n  def __len__(self):\n        return len(self.df)","metadata":{"id":"LUvAW33pfTnx","execution":{"iopub.status.busy":"2022-11-11T15:26:18.421036Z","iopub.status.idle":"2022-11-11T15:26:18.421809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Creating Dataframes and image directories","metadata":{"id":"dWiQtdq3O8wu"}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/melanoma-external-malignant-256/train_concat.csv')\ntest_df = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/test.csv')\ntest_img_dir = '/kaggle/input/melanoma-external-malignant-256/test/test/'\ntrain_img_dir = '/kaggle/input/melanoma-external-malignant-256/train/train/'","metadata":{"scrolled":true,"id":"dchPK1V0BOBN","execution":{"iopub.status.busy":"2022-11-11T15:26:18.423072Z","iopub.status.idle":"2022-11-11T15:26:18.423849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Creating Validation Data from the Training Data \nBecause there is no validation dataset we have to split the training set into training and validation set","metadata":{"id":"d2CaSSfyPImV"}},{"cell_type":"code","source":"vld_size=0.20\n\ntrain, valid = train_test_split (df, stratify=df.target, test_size = vld_size, random_state=42) \n\ntrain_df=pd.DataFrame(train)\nvalidation_df=pd.DataFrame(valid)\n\nprint(len(validation_df))\nprint(len(train_df))","metadata":{"execution":{"iopub.status.busy":"2022-11-11T15:26:18.425186Z","iopub.status.idle":"2022-11-11T15:26:18.425837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#skf = StratifiedKFold(n_splits=5)\n#for fold, (train_ix, val_ix) in enumerate(skf.split(df['image_name'].to_numpy(), df['target'].to_numpy())): \n\n #print(len(train_ix), len(val_ix))                                      \n","metadata":{"id":"ViaVD3Xtlkch","outputId":"a38a71ea-7a88-418e-b67f-35e09f972149","execution":{"iopub.status.busy":"2022-11-11T15:26:18.427194Z","iopub.status.idle":"2022-11-11T15:26:18.427899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#train_df = df.iloc[train_ix].reset_index(drop=True)\n#validation_df = df.iloc[val_ix].reset_index(drop=True)","metadata":{"id":"_b9cg1Egojpi","execution":{"iopub.status.busy":"2022-11-11T15:26:18.429035Z","iopub.status.idle":"2022-11-11T15:26:18.429787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Distribution of Targets in Training and Validation sets\nTargets (in this case melanoma images) should be equally distributed","metadata":{}},{"cell_type":"code","source":"fig2 = plt.figure(figsize=(20, 5))\nax3 = fig2.add_subplot(1,2,1)\nax4 = fig2.add_subplot(1,2,2)\n\ncounts1 = train_df['target'].value_counts()\ndx = ['Benign', 'Malignant']\nax3.bar(dx, counts1)  \nax3.set_title(\"Training Set\")\nax3.legend()\n\nfor i, v in enumerate(counts1):\n    ax3.text(i-.1, \n              v/counts1[i]+200, \n              counts1[i], \n              fontsize=15,\n              )\n\n\ncounts2 = validation_df['target'].value_counts()\nax4.bar(dx, counts2)  \nax4.set_title(\"Validation Set\")\nax4.legend()\n\nfor i, v in enumerate(counts2):\n    ax4.text(i-.1, \n              v/counts2[i]+100, \n              counts2[i], \n              fontsize=15)\n \n\nplt.show()  ","metadata":{"execution":{"iopub.status.busy":"2022-11-11T15:26:18.431073Z","iopub.status.idle":"2022-11-11T15:26:18.431764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Defining Transformations","metadata":{"id":"RTeS2qQdiWbk"}},{"cell_type":"markdown","source":"### Defining random hairs simulation and microscope view for augmentation","metadata":{}},{"cell_type":"code","source":"class AdvancedHairAugmentation:\n    \"\"\"\n    Impose an image of a hair to the target image\n\n    Args:\n        hairs (int): maximum number of hairs to impose\n        hairs_folder (str): path to the folder with hairs images\n    \"\"\"\n\n    def __init__(self, hairs: int = 5, hairs_folder: str = \"../input/melanoma-hairs\"):\n        self.hairs = hairs\n        self.hairs_folder = hairs_folder\n\n    def __call__(self, img):\n        \"\"\"\n        Args:\n            img (PIL Image): Image to draw hairs on.\n\n        Returns:\n            PIL Image: Image with drawn hairs.\n        \"\"\"\n        n_hairs = random.randint(0, self.hairs)\n        \n        if not n_hairs:\n            return img\n        \n        height, width, _ = img.shape  # target image width and height\n        hair_images = [im for im in os.listdir(self.hairs_folder) if 'png' in im]\n        \n        for _ in range(n_hairs):\n            hair = cv2.imread(os.path.join(self.hairs_folder, random.choice(hair_images)))\n            hair = cv2.flip(hair, random.choice([-1, 0, 1]))\n            hair = cv2.rotate(hair, random.choice([0, 1, 2]))\n\n            h_height, h_width, _ = hair.shape  # hair image width and height\n            roi_ho = random.randint(0, img.shape[0] - hair.shape[0])\n            roi_wo = random.randint(0, img.shape[1] - hair.shape[1])\n            roi = img[roi_ho:roi_ho + h_height, roi_wo:roi_wo + h_width]\n\n            # Creating a mask and inverse mask\n            img2gray = cv2.cvtColor(hair, cv2.COLOR_BGR2GRAY)\n            ret, mask = cv2.threshold(img2gray, 10, 255, cv2.THRESH_BINARY)\n            mask_inv = cv2.bitwise_not(mask)\n\n            # Now black-out the area of hair in ROI\n            img_bg = cv2.bitwise_and(roi, roi, mask=mask_inv)\n\n            # Take only region of hair from hair image.\n            hair_fg = cv2.bitwise_and(hair, hair, mask=mask)\n\n            # Put hair in ROI and modify the target image\n            dst = cv2.add(img_bg, hair_fg)\n\n            img[roi_ho:roi_ho + h_height, roi_wo:roi_wo + h_width] = dst\n                \n        return img\n\n    def __repr__(self):\n        return f'{self.__class__.__name__}(hairs={self.hairs}, hairs_folder=\"{self.hairs_folder}\")'\n\nclass DrawHair:\n    \"\"\"\n    Draw a random number of pseudo hairs\n\n    Args:\n        hairs (int): maximum number of hairs to draw\n        width (tuple): possible width of the hair in pixels\n    \"\"\"\n\n    def __init__(self, hairs:int = 4, width:tuple = (1, 2)):\n        self.hairs = hairs\n        self.width = width\n\n    def __call__(self, img):\n        \"\"\"\n        Args:\n            img (PIL Image): Image to draw hairs on.\n\n        Returns:\n            PIL Image: Image with drawn hairs.\n        \"\"\"\n        if not self.hairs:\n            return img\n        \n        width, height, _ = img.shape\n        \n        for _ in range(random.randint(0, self.hairs)):\n            # The origin point of the line will always be at the top half of the image\n            origin = (random.randint(0, width), random.randint(0, height // 2))\n            # The end of the line \n            end = (random.randint(0, width), random.randint(0, height))\n            color = (0, 0, 0)  # color of the hair. Black.\n            cv2.line(img, origin, end, color, random.randint(self.width[0], self.width[1]))\n        \n        return img\n\n    def __repr__(self):\n        return f'{self.__class__.__name__}(hairs={self.hairs}, width={self.width})'\n","metadata":{"execution":{"iopub.status.busy":"2022-11-11T15:26:18.434953Z","iopub.status.idle":"2022-11-11T15:26:18.43575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Microscope:\n    \"\"\"\n    Cutting out the edges around the center circle of the image\n    Imitating a picture, taken through the microscope\n\n    Args:\n        p (float): probability of applying an augmentation\n    \"\"\"\n\n    def __init__(self, p: float = 0.5):\n        self.p = p\n\n    def __call__(self, img):\n        \"\"\"\n        Args:\n            img (PIL Image): Image to apply transformation to.\n\n        Returns:\n            PIL Image: Image with transformation.\n        \"\"\"\n        if random.random() < self.p:\n            circle = cv2.circle((np.ones(img.shape) * 255).astype(np.uint8), # image placeholder\n                        (img.shape[0]//2, img.shape[1]//2), # center point of circle\n                        random.randint(img.shape[0]//2 - 3, img.shape[0]//2 + 15), # radius\n                        (0, 0, 0), # color\n                        -1)\n\n            mask = circle - 255\n            img = np.multiply(img, mask)\n        \n        return img\n\n    def __repr__(self):\n        return f'{self.__class__.__name__}(p={self.p})'\n\n","metadata":{"execution":{"iopub.status.busy":"2022-11-11T15:26:18.437042Z","iopub.status.idle":"2022-11-11T15:26:18.43785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Setting up transformations","metadata":{}},{"cell_type":"code","source":"# Defining transforms for the training, validation, and testing sets\ntraining_transforms = transforms.Compose([#Microscope(),\n                                          AdvancedHairAugmentation(),\n                                          transforms.RandomRotation(30),\n                                          transforms.RandomResizedCrop(256, scale=(0.8, 1.0)),\n                                          transforms.RandomHorizontalFlip(),\n                                          transforms.RandomVerticalFlip(),\n                                          transforms.ColorJitter(brightness=32. / 255.,saturation=0.5,hue=0.01),\n                                          transforms.ToTensor(),\n                                          transforms.Normalize([0.485, 0.456, 0.406], \n                                                               [0.229, 0.224, 0.225])])\n\nvalidation_transforms = transforms.Compose([transforms.Resize(256),\n                                            transforms.CenterCrop(256),\n                                            transforms.ToTensor(),\n                                            transforms.Normalize([0.485, 0.456, 0.406], \n                                                                 [0.229, 0.224, 0.225])])\n\ntesting_transforms = transforms.Compose([transforms.Resize(256),\n                                         transforms.CenterCrop(256),\n                                         transforms.ToTensor(),\n                                         transforms.Normalize([0.485, 0.456, 0.406], \n                                                              [0.229, 0.224, 0.225])])","metadata":{"scrolled":true,"id":"UzAdvS3EBOBX","execution":{"iopub.status.busy":"2022-11-11T15:26:18.439155Z","iopub.status.idle":"2022-11-11T15:26:18.43992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loading Datasets","metadata":{"id":"UJR_kmLtiegA"}},{"cell_type":"code","source":"# Loading the datasets with the transforms previously defined\ntraining_dataset = CustomDataset(df = train_df,\n                                 img_dir = train_img_dir, \n                                 train = True,\n                                 transforms = training_transforms )\n\nvalidation_dataset = CustomDataset(df = validation_df,\n                                   img_dir = train_img_dir, \n                                   train = True,\n                                   transforms = training_transforms )\n\ntesting_dataset = CustomDataset(df = test_df,\n                                img_dir = test_img_dir,\n                                train= False, \n                                transforms = testing_transforms )","metadata":{"id":"p9llHQMpmjOv","execution":{"iopub.status.busy":"2022-11-11T15:26:18.441178Z","iopub.status.idle":"2022-11-11T15:26:18.442027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Defining Dataloaders","metadata":{"id":"sgOZpxN0idt6"}},{"cell_type":"code","source":"# Using the image datasets with the transforms, defining the dataloaders\ntrain_loader = torch.utils.data.DataLoader(training_dataset, batch_size=32, num_workers=4, shuffle=True)\nvalidate_loader = torch.utils.data.DataLoader(validation_dataset, batch_size=16, shuffle = False)\ntest_loader = torch.utils.data.DataLoader(testing_dataset, batch_size=16, shuffle = False)","metadata":{"id":"EaNxLBcTi3bw","execution":{"iopub.status.busy":"2022-11-11T15:26:18.443257Z","iopub.status.idle":"2022-11-11T15:26:18.444122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(train_loader))\nprint(len(validate_loader))\nprint(len(test_loader))","metadata":{"scrolled":true,"id":"e360v3HeBOBf","outputId":"84d22231-3021-460f-c8b9-bebe46bf6345","execution":{"iopub.status.busy":"2022-11-11T15:26:18.445342Z","iopub.status.idle":"2022-11-11T15:26:18.446158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Figuring out how much time the Transformations take","metadata":{}},{"cell_type":"code","source":"transform_start = time.time()\nfor i, data in enumerate(train_loader):\n    images = data\nend = time.time()\ntime_spent = (end-transform_start)/60\nprint(f\"{time_spent:.3} minutes\")","metadata":{"execution":{"iopub.status.busy":"2022-11-11T15:26:18.447435Z","iopub.status.idle":"2022-11-11T15:26:18.448236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Building the Model\n\nNow that the data is ready, it's time to build the Model. I will be using a pretrained model 'Efficient Net' to get the image features and then modify it to fit the Dataset","metadata":{"id":"wtE-5PEhBOBv"}},{"cell_type":"code","source":"class Net(nn.Module):\n    def __init__(self, arch):\n        super(Net, self).__init__()\n        self.arch = arch\n        if 'fgdf' in str(arch.__class__):\n            self.arch.fc = nn.Linear(in_features=1280, out_features=500, bias=True)\n        if 'EfficientNet' in str(arch.__class__):   \n            self.arch._fc = nn.Linear(in_features=1408, out_features=500, bias=True)\n            #self.dropout1 = nn.Dropout(0.2)\n            \n        self.ouput = nn.Linear(500, 1)\n        \n    def forward(self, images):\n        \"\"\"\n        No sigmoid in forward because we are going to use BCEWithLogitsLoss\n        Which applies sigmoid for us when calculating a loss\n        \"\"\"\n        x = images\n        features = self.arch(x)\n        output = self.ouput(features)\n        \n        return output","metadata":{"execution":{"iopub.status.busy":"2022-11-11T15:26:18.449461Z","iopub.status.idle":"2022-11-11T15:26:18.450273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"arch = EfficientNet.from_pretrained('efficientnet-b2')\nmodel = Net(arch=arch)  \nmodel = model.to(device)\n","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-11-11T15:26:18.45148Z","iopub.status.idle":"2022-11-11T15:26:18.452281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# If we need to freeze the pretrained model parameters to avoid backpropogating through them, turn to \"False\"\nfor parameter in model.parameters():\n    parameter.requires_grad = True","metadata":{"execution":{"iopub.status.busy":"2022-11-11T15:26:18.453494Z","iopub.status.idle":"2022-11-11T15:26:18.454317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Total Parameters (If the model is unfrozen the trainning params will be the same as the Total params)\ntotal_params = sum(p.numel() for p in model.parameters())\nprint(f'{total_params:,} total parameters.')\ntotal_trainable_params = sum(\n    p.numel() for p in model.parameters() if p.requires_grad)\nprint(f'{total_trainable_params:,} training parameters.')","metadata":{"id":"XjWG7Yh0BOB7","outputId":"a6a45e57-28e3-4757-fb81-a1822395208e","execution":{"iopub.status.busy":"2022-11-11T15:26:18.455533Z","iopub.status.idle":"2022-11-11T15:26:18.456335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Hyperparameters\n\nEasier for later tunning","metadata":{"id":"nxXejLkPTJdz"}},{"cell_type":"code","source":"# Empty variable to be stored with best validation accuracy\nbest_val = 0\n\n# Path and filename to save model to\nmodel_path = f'melanoma_model_{best_val}.pth'  \n\n# Number of Epochs\nepochs = 10\n\n# Early stopping if no change in accurancy\nes_patience = 3\n\n# Loss Function:\ncriterion = nn.BCEWithLogitsLoss()\n\n# Optimizer (gradient descent):\noptimizer = optim.Adam(model.parameters(), lr=0.0005) \n\n# Scheduler\nscheduler = ReduceLROnPlateau(optimizer=optimizer, mode='max', patience=1, verbose=True, factor=0.2)\n\n\n\n\n\n","metadata":{"scrolled":true,"id":"FbgDZOZjBOCG","execution":{"iopub.status.busy":"2022-11-11T15:26:18.457561Z","iopub.status.idle":"2022-11-11T15:26:18.458399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Trainning the Model","metadata":{}},{"cell_type":"code","source":"#from workspace_utils import active_session -> this can be used so that the session remains on and not disconnect\n\n  \nloss_history=[]  \ntrain_acc_history=[]  \nval_loss_history=[]  \nval_acc_history=[] \nval_auc_history=[]\n\n    \npatience = es_patience\nTotal_start_time = time.time()  \nmodel.to(device)\n\nfor e in range(epochs):\n    \n    start_time = time.time()\n    correct = 0\n    running_loss = 0\n    model.train()\n    \n    for images, labels in train_loader:\n        \n        \n        images, labels = images.to(device), labels.to(device)\n            \n        \n        optimizer.zero_grad()\n        \n        output = model(images) \n        loss = criterion(output, labels.view(-1,1))  \n        loss.backward()\n        optimizer.step()\n        \n        # Training loss\n        running_loss += loss.item()\n\n        # Number of correct training predictions and training accuracy\n        train_preds = torch.round(torch.sigmoid(output))\n            \n        correct += (train_preds.cpu() == labels.cpu().unsqueeze(1)).sum().item()\n                        \n    train_acc = correct / len(train_df)\n        \n        \n    #switching to validation:        \n    model.eval()\n    preds=[]            \n    # Turning off gradients for validation, saves memory and computations\n    with torch.no_grad():\n        \n        val_loss = 0\n        val_correct = 0\n    \n        for val_images, val_labels in validate_loader:\n         \n        \n            val_images, val_labels = val_images.to(device), val_labels.to(device)\n\n        \n            val_output = model(val_images)\n            val_loss += (criterion(val_output, val_labels.view(-1,1))).item() \n            val_pred = torch.sigmoid(val_output)\n            \n            preds.append(val_pred.cpu())\n        pred=np.vstack(preds).ravel()\n           \n        #val_accuracy = accuracy_score(train_df['target'].values, torch.round(pred2))\n        val_auc_score = roc_auc_score(validation_df['target'].values, pred)\n            \n        training_time = str(datetime.timedelta(seconds=time.time() - start_time))[:7]\n            \n        print(\"Epoch: {}/{}.. \".format(e+1, epochs),\n              \"Training Loss: {:.3f}.. \".format(running_loss/len(train_loader)),\n              \"Training Accuracy: {:.3f}..\".format(train_acc),\n              \"Validation Loss: {:.3f}.. \".format(val_loss/len(validate_loader)),\n              #\"Validation Accuracy: {:.3f}\".format(val_accuracy),\n              \"Validation AUC Score: {:.3f}\".format(val_auc_score),\n              \"Training Time: {}\".format( training_time))\n            \n          \n        scheduler.step(val_auc_score)\n                \n        if val_auc_score >= best_val:\n            best_val = val_auc_score\n            patience = es_patience  # Resetting patience since we have new best validation accuracy\n            torch.save(model, model_path)  # Saving current best model\n        else:\n            patience -= 1\n            if patience == 0:\n                print('Early stopping. Best Val roc_auc: {:.3f}'.format(best_val))\n                break\n        \n    loss_history.append(running_loss)  \n    train_acc_history.append(train_acc)    \n    val_loss_history.append(val_loss)  \n    #val_acc_history.append(val_accuracy)\n    val_auc_history.append(val_auc_score)\n    \n\ntotal_training_time = str(datetime.timedelta(seconds=time.time() - Total_start_time  ))[:7]                  \nprint(\"Total Training Time: {}\".format(total_training_time))\n                  \n              ","metadata":{"scrolled":true,"id":"34YYRItSBOCU","outputId":"331ab0b5-6baa-4471-9259-be514e7e8e9e","execution":{"iopub.status.busy":"2022-11-11T15:26:18.45966Z","iopub.status.idle":"2022-11-11T15:26:18.460477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Ploting losses and accuracies","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(20, 5))\nax1 = fig.add_subplot(1,2,1)\nax2 = fig.add_subplot(1,2,2)\n\nax1.plot(loss_history, label= 'Training Loss')  \nax1.plot(val_loss_history,label='Validation Loss')\nax1.set_title(\"Losses\")\nax1.set_xlabel('Epochs')\nax1.set_ylabel('Loss')\nax1.legend()\n\nax2.plot(train_acc_history,label='Training accuracy')  \n#ax2.plot(val_acc_history,label='Validation accuracy')\nax2.plot(val_auc_history,label='Validation AUC Score')\nax2.set_title(\"Accuracies\")\nax2.set_xlabel('Epochs')\nax2.set_ylabel('Accuracy')\nax2.legend()\n\nplt.show()  ","metadata":{"execution":{"iopub.status.busy":"2022-11-11T15:26:18.461686Z","iopub.status.idle":"2022-11-11T15:26:18.462508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Deleting unnecessary variables and cleaning up Cache","metadata":{}},{"cell_type":"code","source":"del training_dataset, validation_dataset, train_loader, validate_loader, images, val_images, val_labels\ngc.collect()\n","metadata":{"execution":{"iopub.status.busy":"2022-11-11T15:26:18.463813Z","iopub.status.idle":"2022-11-11T15:26:18.464629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Testing the network\n\nIt's good practice to test the trained network on test data, images the network has never seen either in training or validation. This will give a good estimate for the model's performance on completely new images. I will run the test images through the network and measure the accuracy, the same way I did validation. ","metadata":{"id":"tnqLx-40BOCg"}},{"cell_type":"code","source":"test_df['target']= np.zeros((len(test_df), 1))\n","metadata":{"execution":{"iopub.status.busy":"2022-11-11T15:26:18.465907Z","iopub.status.idle":"2022-11-11T15:26:18.466729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_labels = torch.tensor(test_df['target'], dtype=torch.float32)","metadata":{"execution":{"iopub.status.busy":"2022-11-11T15:26:18.467961Z","iopub.status.idle":"2022-11-11T15:26:18.468737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nmodel = torch.load(model_path)\nmodel.eval()\nmodel.to(device)\ntest_preds=[]\nwith torch.no_grad():\n    \n    \n    \n    for f, (test_images) in enumerate(test_loader):\n        \n        \n        test_images, test_labels = test_images.to(device), test_labels.to(device)\n        \n        \n        test_output = model(test_images)\n        test_pred = torch.sigmoid(test_output)\n            \n        test_preds.append(test_pred.cpu())\n        \n    test_pred=np.vstack(test_preds).ravel()\n    test_pred2 = torch.tensor(test_pred)\n    test_accuracy = accuracy_score(test_labels.cpu(), torch.round(test_pred2))\n      \n        \n    \nprint(\"Test Accuracy: {}\".format(test_accuracy))    \n        ","metadata":{"scrolled":true,"id":"l8ox_qGkBOCh","execution":{"iopub.status.busy":"2022-11-11T15:26:18.469922Z","iopub.status.idle":"2022-11-11T15:26:18.470739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/sample_submission.csv')\nsub.loc[:, \"target\"] = test_pred\nsub.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-11-11T15:26:18.471969Z","iopub.status.idle":"2022-11-11T15:26:18.472759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"###### Inference for classification\n\nWriting a function to use a trained network for inference. That is, I'll pass an image into the network and predict the class of the Image. \n\nWriting a function called `predict` that takes an image and a model, then returns the top $K$ most likely classes along with the probabilities. \n\nFirst the input image needs to be processed in the same manner it was for training, so that it can be used in the network. \n\n\n## Image Preprocessing\n\nUsing `PIL` to load the image ([documentation](https://pillow.readthedocs.io/en/latest/reference/Image.html)). It's best to write a function that preprocesses the image so it can be used as an input for the model. This function should process the images in the same manner used for training. \n\nIf the Image is bigger than 256x256:..\n\nFirst, resize the images where the shortest side is 256 pixels, keeping the aspect ratio. This can be done with the [`thumbnail`](http://pillow.readthedocs.io/en/3.1.x/reference/Image.html#PIL.Image.Image.thumbnail) or [`resize`](http://pillow.readthedocs.io/en/3.1.x/reference/Image.html#PIL.Image.Image.thumbnail) methods. Then crop out the center 256x256 portion of the image.\n\nColor channels of images are typically encoded as integers 0-255, but the model expected floats 0-1. I'll need to convert the values. It's easiest with a Numpy array, which can be obtained from a PIL image like so `np_image = np.array(pil_image)`.\n\nAs before, the network expects the images to be normalized in a specific way. For the means, it's `[0.485, 0.456, 0.406]` and for the standard deviations `[0.229, 0.224, 0.225]`. I'll subtract the means from each color channel, then divide by the standard deviation. \n\nAnd finally, PyTorch expects the color channel to be the first dimension but it's the third dimension in the PIL image and Numpy array. Dimensions must be reordered using [`ndarray.transpose`](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.ndarray.transpose.html). The color channel needs to be first and retain the order of the other two dimensions.","metadata":{"id":"GtXTgJRLBOC3"}},{"cell_type":"code","source":"#from PIL import Image\n\ndef process_image(image_path):\n    ''' Scales, crops, and normalizes a PIL image for a PyTorch model,\n        returns an Numpy array\n    '''\n    \n    # Process a PIL image for use in a PyTorch model\n    \n    pil_image = Image.open(image_path)\n    \n    # Resize\n    if pil_image.size[0] > pil_image.size[1]:\n        pil_image.thumbnail((5000, 256))\n    else:\n        pil_image.thumbnail((256, 5000))\n        \n    # Crop \n    left_margin = (pil_image.width-256)/2\n    bottom_margin = (pil_image.height-256)/2\n    right_margin = left_margin + 256\n    top_margin = bottom_margin + 256\n    \n    pil_image = pil_image.crop((left_margin, bottom_margin, right_margin, top_margin))\n    \n    \n    # Normalize\n    np_image = np.array(pil_image)/255\n    mean = np.array([0.485, 0.456, 0.406])\n    std = np.array([0.229, 0.224, 0.225])\n    np_image = (np_image - mean) / std\n  \n    # PyTorch expects the color channel to be the first dimension but it's the third dimension in the PIL image and Numpy array\n    # Color channel needs to be first; retain the order of the other two dimensions.\n    np_image = np_image.transpose((2, 0, 1))\n    \n    return np_image\n\n","metadata":{"scrolled":true,"id":"7VlLkC4RBOC4","execution":{"iopub.status.busy":"2022-11-11T15:26:18.473993Z","iopub.status.idle":"2022-11-11T15:26:18.474801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Checking work done untill now, the function below converts a PyTorch tensor and displays it in the notebook. If `process_image` function works, running the output through this function should return the original image (except for the cropped out portions).","metadata":{"id":"nBhKsGjlBOC9"}},{"cell_type":"code","source":"def imshow(image, ax=None, title=None):\n    if ax is None:\n        fig, ax = plt.subplots()\n    \n    # PyTorch tensors assume the color channel is the first dimension\n    # but matplotlib assumes is the third dimension\n    image = image.transpose((1, 2, 0))\n    \n    # Undo preprocessing\n    mean = np.array([0.485, 0.456, 0.406])\n    std = np.array([0.229, 0.224, 0.225])\n    image = std * image + mean\n    \n    if title is not None:\n        ax.set_title(title)\n    \n    # Image needs to be clipped between 0 and 1 or it looks like noise when displayed\n    image = np.clip(image, 0, 1)\n    \n    ax.imshow(image)\n    \n    return ax\n\nimage = process_image('../input/siim-isic-melanoma-classification/jpeg/test/ISIC_0052060.jpg')\nimshow(image)","metadata":{"scrolled":true,"id":"vfKE6_-PBOC-","execution":{"iopub.status.busy":"2022-11-11T15:26:18.476035Z","iopub.status.idle":"2022-11-11T15:26:18.476822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Class Prediction\n\nOnce the images are in the correct format, it's time to write a function to make predictions with the model. A common practice is to predict the top 5 or so (usually called top-$K$) most probable classes. I'll calculate the class probabilities then find the $K$ largest values.\n\nTo get the top $K$ largest values in a tensor use [`x.topk(k)`](http://pytorch.org/docs/master/torch.html#torch.topk). This method returns both the highest `k` probabilities and the indices of those probabilities corresponding to the classes. The indices need to be converted to the actual class labels using `class_to_idx` which was added to the model, it can also be done from an `ImageFolder` used to load the data ([see here](#Save-the-checkpoint)). \n\nAgain, this method should take a path to an image and a model checkpoint, then return the probabilities and classes.\n","metadata":{"id":"CTiPF0nmBODD"}},{"cell_type":"code","source":"#model = torch.load('../input/melanoma-model/melanoma_model_0.pth' )\n#model = model.to(device)\n#model.eval()","metadata":{"execution":{"iopub.status.busy":"2022-11-11T15:26:18.47803Z","iopub.status.idle":"2022-11-11T15:26:18.478838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Implement the code to predict the class from an image file\n\ndef predict(image_path, model, topk=1): #just 2 classes from 1 single output\n    ''' Predict the class (or classes) of an image using a trained deep learning model.\n    '''\n    \n    image = process_image(image_path)\n    \n    # Convert image to PyTorch tensor first\n    image = torch.from_numpy(image).type(torch.cuda.FloatTensor)\n    #print(image.shape)\n    #print(type(image))\n    \n    # Returns a new tensor with a dimension of size one inserted at the specified position.\n    image = image.unsqueeze(0)\n    \n    output = model(image)\n    \n    probabilities = torch.sigmoid(output)\n    \n    # Probabilities and the indices of those probabilities corresponding to the classes\n    top_probabilities, top_indices = probabilities.topk(topk)\n    \n    # Convert to lists\n    top_probabilities = top_probabilities.detach().type(torch.FloatTensor).numpy().tolist()[0] \n    top_indices = top_indices.detach().type(torch.FloatTensor).numpy().tolist()[0] \n    \n    top_classes = []\n    \n    if probabilities > 0.5 :\n        top_classes.append(\"Melanoma\")\n    else:\n        top_classes.append(\"Benign\")\n\n    \n    return top_probabilities, top_classes\n\npredict_image_path='../input/siim-isic-melanoma-classification/jpeg/train/ISIC_0502582.jpg'\n#predict_image_path='../input/siim-isic-melanoma-classification/jpeg/test/ISIC_0074618.jpg'\n\nprobs, classes = predict(predict_image_path, model)   \nprint(probs)\nprint(classes)","metadata":{"scrolled":true,"id":"gnR_JjzSBODE","execution":{"iopub.status.busy":"2022-11-11T15:26:18.480069Z","iopub.status.idle":"2022-11-11T15:26:18.480861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Confusion Matrix","metadata":{"id":"5dS38R6wBODJ"}},{"cell_type":"code","source":"test = test_df['target']= np.zeros((len(test_df), 1))\npred = np.round(test_pred)\ncm = confusion_matrix(test, pred)\n\ncm_df = pd.DataFrame(cm,\n                     index = ['Benign','Malignant'], \n                     columns = ['Benign','Malignant'])\n\nplt.figure(figsize=(5.5,4))\nsb.heatmap(cm_df, annot=True)\nplt.title('Confusion Matrix \\nAccuracy:{0:.3f}'.format(test_accuracy))\nplt.ylabel('True label')\nplt.xlabel('Predicted label')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-11T15:26:18.4821Z","iopub.status.idle":"2022-11-11T15:26:18.482885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Sanity Check\n\nNow that we can use a trained model for predictions, let's check to make sure it makes sense. Even if the testing accuracy is high, it's always good to check that there aren't obvious bugs. Using `matplotlib` to plot the diagnosis along with the input image. \n\nIntroduce the image path to be predicted in \"predict_image_path\" object above\n","metadata":{"id":"Ok_y-IikBODJ"}},{"cell_type":"code","source":"# Display an image along with the diagnosis of melanoma or benign\n\n# Plot Skin image input image\nplt.figure(figsize = (6,10))\nplot_1 = plt.subplot(2,1,1)\n\nimage = process_image(predict_image_path)\n\nimshow(image, plot_1)\nfont = {\"color\": 'g'} if 'Benign' in classes else {\"color\": 'r'}\nplot_1.set_title(\"Diagnosis: {}\".format(classes), fontdict=font);","metadata":{"scrolled":true,"id":"KO7XaIeiBODL","execution":{"iopub.status.busy":"2022-11-11T15:26:18.484252Z","iopub.status.idle":"2022-11-11T15:26:18.485061Z"},"trusted":true},"execution_count":null,"outputs":[]}]}