{"nbformat": 4, "nbformat_minor": 0, "metadata": {"language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.6-final"}, "orig_nbformat": 2, "kernelspec": {"name": "python3", "display_name": "Python 3", "language": "python"}, "colab": {"name": "main.ipynb", "provenance": []}, "accelerator": "GPU"}, "cells": [{"cell_type": "markdown", "metadata": {"id": "XvBDTao8djYJ"}, "source": ["# Melanoma Classification\n", "\n", "Kaggle Competition Page: www.kaggle.com/c/siim-isic-melanoma-classification/overview\n", "\n", "\n", "## What is Melanoma?\n", "Melanoma, the most severe type of skin cancer, develops in the cells (melanocytes) that produce melanin \u2014 the pigment that gives your skin its color. Melanoma can also form in your eyes and, rarely, inside your body, such as in your nose or throat.\n", "\n", "The exact cause of all melanomas isn't clear, but exposure to ultraviolet (UV) radiation from sunlight or tanning lamps and beds increases your risk of developing melanoma.\n", "\n", "The risk of melanoma seems to be increasing in people under 40, especially women. Knowing the warning signs of skin cancer can help ensure that cancerous changes are detected and treated before the cancer has spread. We can treat melanoma successfully if it is detected early."]}, {"cell_type": "markdown", "metadata": {"id": "PQIr6OLjdjYs"}, "source": ["![\"Drawing\"](\"https://github.com/SaschaMet/melanoma-classification/blob/master/images/melanoma.jpg?raw=1\")
"]}, {"cell_type": "markdown", "metadata": {"id": "wR5V4LGwdjYs"}, "source": ["## Symptoms & Diagnosis\n", "Melanomas can develop anywhere on your body. They most often develop in areas with exposure to the sun, such as your back, legs, arms, and face.\n", "Melanomas can also occur in areas that don't receive much sun exposure, such as the soles of your feet, palms of your hands, and fingernail beds. These hidden melanomas are more common in people with darker skin.\n", "\n", "To help you identify characteristics of melanomas or other skin cancers, think of the letters ABCDE:\n", "- A is for asymmetrical shape. Look for moles with irregular shapes, such as two very different-looking halves.\n", "- B is for irregular border. Look for moles with rough, notched, or scalloped edges \u2014 characteristics of melanomas.\n", "- C is for color changes. Look for growths that have many colors or an uneven distribution of color.\n", "- D is for diameter. Look for new growth in a mole larger than 1/4 inch (about 6 millimeters).\n", "- E is for evolving. Look for changes over time, such as a mole that grows in size or changes color or shape.\n"]}, {"cell_type": "markdown", "metadata": {"id": "3_Vl79qwdjYw"}, "source": ["\n", "\n", "Source: https://www.health.harvard.edu/cancer/melanoma-overview"]}, {"cell_type": "markdown", "metadata": {"id": "Bm8mD1YSdjYx"}, "source": ["The facts about Melanoma:\n", "- Melanoma is the most severe form of skin cancer\n", "- It makes up 2% of skin cancers but is responsible for 75% of skin cancer deaths\n", "- Australia and New Zealand have the highest melanoma rates in the world\n", "- 1 in 17 Australians will be diagnosed with melanoma before the age of 85\n", "- More than 90% of melanoma can be successfully treated with surgery if detected early\n", "\n", "Source: https://melanomapatients.org.au/about-melanoma/melanoma-facts/"]}, {"cell_type": "markdown", "metadata": {"id": "si5Twho8djYx"}, "source": ["![\"Drawing\"](\"https://github.com/SaschaMet/melanoma-classification/blob/master/images/melanoma-impact.jpg?raw=1\")
\n", "\n", "Source: https://impactmelanoma.org/wp-content/uploads/2018/11/Standard-Infographic_0.jpg"]}, {"cell_type": "markdown", "metadata": {"id": "e-AFobcxdjYx"}, "source": ["## Setup"]}, {"cell_type": "code", "metadata": {"id": "E06dG0gndjYy"}, "source": ["import os\n", "import json\n", "import random\n", "import warnings\n", "import itertools\n", "import numpy as np\n", "import pandas as pd\n", "from tqdm import tqdm\n", "import tensorflow as tf\n", "from pathlib import Path\n", "from tensorflow import keras\n", "import matplotlib.pyplot as plt\n", "from keras.optimizers import Adam\n", "from datetime import datetime, date\n", "from tensorflow.keras import layers\n", "import tensorflow.keras.backend as K\n", "from keras.applications.vgg16 import VGG16\n", "from pandas_profiling import ProfileReport\n", "from sklearn.model_selection import train_test_split\n", "from tensorflow.keras.models import Sequential, Model\n", "from keras.preprocessing.image import ImageDataGenerator\n", "from keras.callbacks import ModelCheckpoint, EarlyStopping\n", "from sklearn.metrics import roc_curve, auc, precision_recall_curve, plot_precision_recall_curve, confusion_matrix"], "execution_count": 2, "outputs": []}, {"cell_type": "code", "metadata": {"id": "JnvR9SrjdjYy", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "1bc1ff6f-760d-4cff-bfd8-9885ae28ebb7"}, "source": ["SEED = 1\n", "EPOCHS = 40\n", "BATCH_SIZE = 64\n", "NUM_CLASSES = 2\n", "VERBOSE_LEVEL = 2\n", "SAVE_OUTPUT = True\n", "IMG_SIZE = (224, 224)\n", "INPUT_SHAPE = (224, 224, 3)\n", "\n", "CWD = os.getcwd()\n", "warnings.filterwarnings('ignore')\n", "\n", "GOOGLE_COLAB = False\n", "if CWD == \"/content\":\n", " %matplotlib inline\n", " GOOGLE_COLAB = True\n", " print ('Running in colab:', GOOGLE_COLAB)"], "execution_count": 3, "outputs": []}, {"cell_type": "code", "metadata": {"id": "QH9nAk3sdjYz", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "4b0bf073-c3d4-403e-e6e9-0cc39ef9591f"}, "source": ["# Tensorflow execution optimizations\n", "# Source: https://www.tensorflow.org/guide/mixed_precision & https://www.tensorflow.org/xla\n", "MIXED_PRECISION = True\n", "XLA_ACCELERATE = True\n", "GPUS = 0\n", "\n", "GPUS = len(tf.config.experimental.list_physical_devices('GPU'))\n", "if GPUS == 0:\n", " DEVICE = 'CPU'\n", "else:\n", " DEVICE = 'GPU'\n", " if MIXED_PRECISION:\n", " from tensorflow.keras.mixed_precision import experimental as mixed_precision\n", " policy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\n", " mixed_precision.set_policy(policy)\n", " print('Mixed precision enabled')\n", " if XLA_ACCELERATE:\n", " tf.config.optimizer.set_jit(True)\n", " print('Accelerated Linear Algebra enabled')\n", "\n", "print(\"Tensorflow version \" + tf.__version__)"], "execution_count": 4, "outputs": []}, {"cell_type": "code", "metadata": {"id": "7QrMRrKjdjYz", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "10df2198-94f3-462f-e656-f1d18fe28bd3"}, "source": ["BASE_PATH = '/kaggle/input/siim-isic-melanoma-classification'\n", "PATH_TO_IMAGES = '/kaggle/input/siim-isic-melanoma-classification/jpeg' \n", "IMAGE_TYPE = \".jpg\"\n", "\n", "# check on which system we are\n", "if os.path.exists(CWD + '/data'):\n", " BASE_PATH = os.path.join(CWD, 'data')\n", " PATH_TO_IMAGES = BASE_PATH\n", " IMAGE_TYPE = \".png\"\n", " print(\"change BASE_PATH to \", BASE_PATH)\n", "\n", "elif GOOGLE_COLAB:\n", " CWD = \"/content/melanoma-classification\"\n", " BASE_PATH = os.path.join(CWD, 'data')\n", " PATH_TO_IMAGES = BASE_PATH\n", " IMAGE_TYPE = \".png\"\n", " print(\"change BASE_PATH to \", BASE_PATH)"], "execution_count": 5, "outputs": []}, {"cell_type": "code", "metadata": {"id": "Ui2-3AEcdjY0", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "8957352d-9e3d-45fa-8159-e8d95ae03cc8"}, "source": ["\"\"\" Helper function to seed everything for getting reproducible results\n", "\"\"\"\n", "def seed_all(seed):\n", " print(\"Set seed\")\n", " random.seed(seed)\n", " np.random.seed(seed)\n", " os.environ['PYTHONHASHSEED'] = str(seed)\n", " os.environ['TF_KERAS'] = str(seed)\n", " os.environ['TF_DETERMINISTIC_OPS'] = str(seed)\n", " os.environ['TF_CUDNN_DETERMINISTIC'] = str(seed)\n", " tf.random.set_seed(seed)\n", " tf.experimental.numpy.random.seed(seed)\n", " \n", "seed_all(SEED)"], "execution_count": 6, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "E7Vq9jDVdjY3"}, "source": ["## Loading the data"]}, {"cell_type": "code", "metadata": {"id": "mvQWw7NtdjY3"}, "source": ["\"\"\" Helper function to validate the image paths\n", "\n", " Parameters:\n", " file_path (string): Path to the image \n", "\n", " Returns:\n", " The file path if the file exists, otherwise false if the file does not exist\n", "\n", "\"\"\"\n", "def check_image(file_path):\n", " img_file = Path(file_path)\n", " if img_file.is_file():\n", " return file_path\n", " return False"], "execution_count": 7, "outputs": []}, {"cell_type": "code", "metadata": {"id": "kA5weeYJdjY4"}, "source": ["\"\"\" Helper function to get the train dataset\n", "\"\"\"\n", "def get_train_data():\n", " # read the data from the train.csv file\n", " train = pd.read_csv(os.path.join(BASE_PATH, 'train.csv'))\n", " # add the image_path to the train set\n", " train['image_path'] = train['image_name'].apply(lambda x: PATH_TO_IMAGES + \"/train/\" + x + IMAGE_TYPE)\n", " # check if the we have an image \n", " train['image_path'] = train.apply(lambda row : check_image(row['image_path']), axis = 1)\n", " # if we do not have an image we will not include the data\n", " train = train[train['image_path'] != False]\n", " print(\"valid rows in train\", train.shape[0])\n", " return train\n", "\n", "\"\"\" Helper function to get the test dataset\n", "\"\"\"\n", "def get_test_data():\n", " test = pd.read_csv(os.path.join(BASE_PATH, 'test.csv'))\n", " test['image_path'] = test['image_name'].apply(lambda x: PATH_TO_IMAGES + \"/test/\" + x + IMAGE_TYPE)\n", " test['image_path'] = test.apply(lambda row : check_image(row['image_path']), axis = 1)\n", " test = test[test['image_path'] != False]\n", " print(\"valid rows in test\", test.shape[0])\n", " return test\n"], "execution_count": 8, "outputs": []}, {"cell_type": "code", "metadata": {"id": "kdk3Ha_rdjY4", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "5ccae5ce-4f94-4c08-ab80-b69aee63b62e"}, "source": ["train = get_train_data()\n", "test = get_test_data()"], "execution_count": 9, "outputs": []}, {"cell_type": "code", "metadata": {"id": "SAukZCdpdjY4", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "60aea1cf-1c86-403f-ea29-b727710daea1"}, "source": ["train.dtypes"], "execution_count": 10, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "-xS3V7IedjY5"}, "source": ["Train Dataset:\n", "- image name: the filename for the specific image\n", "- patient_id: unique patient id\n", "- sex: gender of the patient\n", "- age_approx: age of the patient\n", "- anatom_site_general_challenge: location of the scan site\n", "- diagnosis: information about the diagnosis\n", "- benign_malignant: indicates if the scan result is malignant or benign\n", "- target: 0 for benign and 1 for malignant\n", "- image_path: path to the image"]}, {"cell_type": "code", "metadata": {"id": "X0ZADtmddjY5", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "08836bc4-d435-4751-f80f-53ef670404cf"}, "source": ["test.dtypes"], "execution_count": 11, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "YPt_jwVidjY5"}, "source": ["Test Dataset Consists Of:\n", "- image name: the filename for the specific image\n", "- patient_id: unique patient id\n", "- sex: gender of the patient\n", "- age_approx: age of the patient\n", "- anatom_site_general_challenge: location of the scan site\n", "- image_path: path to the image"]}, {"cell_type": "markdown", "metadata": {"id": "M09KJFyadjY6"}, "source": ["## EDA"]}, {"cell_type": "markdown", "metadata": {"id": "MXkucGoxdjZt"}, "source": ["Pandas profiling is a Python module with which you can do an exploratory data analysis with just a few lines of code. Unfortunately, it does not work in every environment, so this has to be checked beforehand.\n", "\n", "If you want to see this package's output, you can open the output.html file in your browser."]}, {"cell_type": "code", "metadata": {"id": "JegtHmIg0_ft", "outputId": "ee6bb9bf-e142-4f07-bc8a-5665cc93a865", "colab": {"base_uri": "https://localhost:8080/"}}, "source": ["try:\n", " t = train[['image_name', 'patient_id', 'sex', 'age_approx', 'anatom_site_general_challenge', 'diagnosis', 'benign_malignant', 'target']]\n", " profile = ProfileReport(t, explorative=True)\n", " # output the report to html\n", " profile.to_file(\"output.html\")\n", " # show the output inline\n", " profile\n", "except:\n", " print(\"Pandas Profiling does not work in this environment\")"], "execution_count": 12, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "ezInl9iUdjZu"}, "source": ["### Check for missing values"]}, {"cell_type": "code", "metadata": {"id": "C7CSiXIbdjZu", "colab": {"base_uri": "https://localhost:8080/", "height": 328}, "outputId": "fe3c2689-44e2-445b-9ed9-bd2fd581c09c"}, "source": ["\"\"\" Helper function check a dataframe for missing values\n", "\n", " Parameters:\n", " df (dataframe): The dataframe to check\n", "\n", " Returns:\n", " A dataframe with the number of missing and zero values for each column in percent\n", "\n", "\"\"\"\n", "def check_for_missing_and_null(df):\n", " null_df = pd.DataFrame({'columns': df.columns, \n", " 'percent_null': df.isnull().sum() * 100 / len(df), \n", " 'percent_zero': df.isin([0]).sum() * 100 / len(df),\n", " 'total_zero': df.isnull().sum() * 100 / len(df) + df.isin([0]).sum() * 100 / len(df),\n", " })\n", " return null_df\n", "\n", "check_for_missing_and_null(train)"], "execution_count": 13, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "5yaIn2dQdjZu"}, "source": ["There is a small portion of missing values for age and sex, as well as for the anatom_site_general_challenge column. \n", "\n", "The target column consists of 98 % zero values. This means we have a highly imbalanced dataset."]}, {"cell_type": "markdown", "metadata": {"id": "gCB_FFkYdjZx"}, "source": ["### Removing missing values\n", "\n", "To do the EDA, I will remove the dataset's missing values because we will not lose much information. Later, when we prepare the dataset for training, I will add these missing values again."]}, {"cell_type": "code", "metadata": {"id": "TuszpnbgdjZy"}, "source": ["train = train.dropna()"], "execution_count": 14, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "Dpro3gGYdjZy"}, "source": ["### Target distribution"]}, {"cell_type": "code", "metadata": {"id": "UowDHn02djZy", "colab": {"base_uri": "https://localhost:8080/", "height": 527}, "outputId": "fd60d213-4ecf-4df7-c646-2d7f91273f60"}, "source": ["plt.figure(figsize = (8,6))\n", "x = plt.bar([\"Melanoma\",\"Benign\"],[len(train[train.target==1]), len(train[train.target==0])])"], "execution_count": 15, "outputs": []}, {"cell_type": "code", "metadata": {"id": "LnZ0oqxvgKL8", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "dc6ceac8-bbe9-4418-cb95-7aa56c1c137f"}, "source": ["benign_cases = train[train.target == 0]\n", "melanoma_cases = train[train.target == 1]\n", "\n", "print(\"Benign Cases\", len(benign_cases))\n", "print(\"Melanoma Cases\", len(melanoma_cases))\n", "print(\" \")\n", "print(\"There are only\", len(melanoma_cases), \"malignant cases in the dataset. This is very important to know, because this has implications on how to perpare the dataset for training the machine learning model.\")"], "execution_count": 16, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "65LkV7UQdjZz"}, "source": ["### Gender distribution"]}, {"cell_type": "code", "metadata": {"id": "bsBh6hYpdjZz", "colab": {"base_uri": "https://localhost:8080/", "height": 544}, "outputId": "d5d968d4-554e-4c97-85af-b6e4bf631cb8"}, "source": ["female = train[train.sex == \"female\"]\n", "male = train[train.sex == \"male\"]\n", "plt.figure(figsize = (8,6))\n", "x = plt.bar(\n", " [\"Female\",\"Male\"],\n", " [len(female), len(male)]\n", ")\n", "print('There are', len(female), 'female patients in the dataset and', len(male), 'male patients.')"], "execution_count": 17, "outputs": []}, {"cell_type": "code", "metadata": {"id": "A-pyTo-R0_fz", "outputId": "7b8e46e2-b651-4235-af94-e0be8e2e9b13", "colab": {"base_uri": "https://localhost:8080/", "height": 527}}, "source": ["benign_cases_female = train[train.target==0][train.sex == \"female\"]\n", "malignant_cases_female = train[train.target==1][train.sex == \"female\"]\n", "\n", "benign_cases_male = train[train.target==0][train.sex == \"male\"]\n", "malignant_cases_male = train[train.target==1][train.sex == \"male\"]\n", "\n", "plt.figure(figsize = (8,6))\n", "x = plt.bar(\n", " [\"Benign & Female\",\"Malignant & Female\", \"Benign & Male\",\"Malignant & Male\"],\n", " [len(benign_cases_female), len(malignant_cases_female), len(benign_cases_male), len(malignant_cases_male)]\n", ")"], "execution_count": 18, "outputs": []}, {"cell_type": "code", "metadata": {"id": "prEJvzLwdjZ1", "colab": {"base_uri": "https://localhost:8080/", "height": 173}, "outputId": "ab58980f-18ca-42f1-a152-275dce1e745b"}, "source": ["grouped_df_by_sex = train.groupby(['target','sex'])['benign_malignant'].count().to_frame().reset_index()\n", "grouped_df_by_sex"], "execution_count": 19, "outputs": []}, {"cell_type": "code", "metadata": {"id": "Su2PnvF6ixFI", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "9ac244ed-1308-4b3b-a8c4-9029150205e5"}, "source": ["f_m = train[train.target == 1][train.sex == \"female\"]\n", "m_m = train[train.target == 1][train.sex == \"male\"]\n", "\n", "print(\"There are\", len(m_m) ,\"malignant male cases in the dataset compared to\", len(f_m) ,\"female cases.\")"], "execution_count": 20, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "noATRLdYdjZ7"}, "source": ["### Age distribution"]}, {"cell_type": "code", "metadata": {"id": "VJ947xNOdjZ7"}, "source": ["# create ten age bins, from 0 to 100\n", "age_bins = np.arange(0, 100, 10)\n", "\n", "\"\"\" Helper function to return the age bin for a specific age\n", "\n", " Parameters:\n", " age (int)\n", "\n", " Returns:\n", " age bin (int)\n", "\"\"\"\n", "def add_age_bin(age):\n", " for idx, val in enumerate(age_bins):\n", " if age < val:\n", " return idx\n", "\n", "# add the age bins to the train df\n", "train['age_bin'] = train.apply(lambda row : add_age_bin(row['age_approx']), axis = 1)"], "execution_count": 21, "outputs": []}, {"cell_type": "code", "metadata": {"id": "eDLo_EZYdjZ7", "colab": {"base_uri": "https://localhost:8080/", "height": 527}, "outputId": "1de61ad8-6271-4d42-b302-1c21ed924f9f"}, "source": ["plt.figure(figsize=(8,6))\n", "plt.hist( train.age_bin, bins = 20)\n", "plt.show()"], "execution_count": 22, "outputs": []}, {"cell_type": "code", "metadata": {"id": "_nkOM4msdjZ8", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "0748bc5a-ba7e-49b7-a9d2-96532f8d4330"}, "source": ["print(\"The mean age of a patient in the dataset is\", round(np.mean(train.age_approx, 0)))"], "execution_count": 23, "outputs": []}, {"cell_type": "code", "metadata": {"id": "836oCQZodjZ8", "colab": {"base_uri": "https://localhost:8080/", "height": 527}, "outputId": "95909d95-a70a-4c8c-9ba0-9feeaf6a9b19"}, "source": ["plt.figure(figsize=(8,6))\n", "plt.hist( train[train.target==1].age_bin, bins = 20)\n", "plt.show()"], "execution_count": 24, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "iXKLO5frdjZ9"}, "source": ["The age distributions follows a normal distribution. If we look only at the malignant cases however we can see, that the distribution seems to be wider. "]}, {"cell_type": "code", "metadata": {"id": "7UBG-TCVdjZ9", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "30240b1d-8462-4ea3-a744-92b065f9dd59"}, "source": ["def get_ratio_by_age_bin(age_bin):\n", " total = train[train['age_bin'] == age_bin]\n", " malignant = train[train['age_bin'] == age_bin][train['target'] == 1]\n", " return round((len(malignant) / len(total)) * 100, 2)\n", " \n", "for age_bin in [2,3,4,5,6,7,8]:\n", " print(\"Ratio malignant / total cases for age_bin\", age_bin, \"=\" , get_ratio_by_age_bin(age_bin))\n"], "execution_count": 25, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "6H3jKX8sdjZ-"}, "source": ["There are indeed more malignant cases at the ends of the age distribution."]}, {"cell_type": "markdown", "metadata": {"id": "QIsSW6Z7djZ-"}, "source": ["### Anatom Site General Challenge distribution"]}, {"cell_type": "code", "metadata": {"id": "ubTnh6V-djZ-", "colab": {"base_uri": "https://localhost:8080/", "height": 547}, "outputId": "87831388-f508-444d-b098-e71fb2600f30"}, "source": ["anatom_site = list(train.anatom_site_general_challenge.unique())\n", "anatom_site = [x for x in anatom_site if str(x) != 'nan']\n", "\n", "anatom_site_value_counts = []\n", "for x in anatom_site:\n", " y = train[train['anatom_site_general_challenge'] == x]\n", " anatom_site_value_counts.append(len(y))\n", "\n", "y_pos = np.arange(len(anatom_site))\n", "plt.figure(figsize=(8,6))\n", "plt.bar(y_pos, anatom_site_value_counts, align='center')\n", "plt.xticks(y_pos, anatom_site)\n", "plt.ylabel('# of rows')\n", "plt.title('Anatom Site General Challenge')\n", "\n", "plt.show()"], "execution_count": 26, "outputs": []}, {"source": ["Most often a lesion was found in the torso area, followed by the lower and upper extremity."], "cell_type": "markdown", "metadata": {}}, {"cell_type": "markdown", "metadata": {"id": "8bR7V1xWdjZ_"}, "source": ["### Diagnosis distribution"]}, {"cell_type": "code", "metadata": {"id": "hdhrURJ0djZ_"}, "source": ["diagnosis = list(train.diagnosis.unique())\n", "diagnosis = [x for x in diagnosis if str(x) != 'unknown']\n", "\n", "diagnosis_value_counts = []\n", "for x in diagnosis:\n", " y = train[train['diagnosis'] == x]\n", " diagnosis_value_counts.append(len(y))"], "execution_count": 27, "outputs": []}, {"cell_type": "code", "metadata": {"id": "CDNIXujZdjZ_", "colab": {"base_uri": "https://localhost:8080/", "height": 499}, "outputId": "c1b0fe99-7314-4b38-fa86-0b18f64c0399"}, "source": ["labels = diagnosis\n", "sizes = diagnosis_value_counts\n", "plt.figure(figsize=(8,6))\n", "patches, texts = plt.pie(sizes, shadow=True, startangle=90)\n", "plt.legend(patches, labels, loc=\"best\")\n", "plt.axis('equal')\n", "plt.show()"], "execution_count": 28, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "FW-F5CQIdjZ_"}, "source": ["The main finding in the dataset is \"nevus\". Nevus is a nonspecific medical term for a visible, circumscribed, chronic lesion of the skin (e.g. a \"birthmark\"). The second most common finding was melanoma.\n", "\n", "\n", "Source: https://en.wikipedia.org/wiki/Nevus"]}, {"cell_type": "markdown", "metadata": {"id": "XrP2muWNdjZ_"}, "source": ["## Images from the dataset"]}, {"cell_type": "code", "metadata": {"id": "MBHUu1ITdjaA", "colab": {"base_uri": "https://localhost:8080/", "height": 1000}, "outputId": "b9f73fd6-4a13-48ec-d0a5-c50bb5576bf2"}, "source": ["plt.figure(figsize=(16, 16))\n", "for i in range(9):\n", " plt.subplot(3, 3, i + 1)\n", " img_path = train.iloc[i].image_path\n", " img = plt.imread(img_path)\n", " plt.imshow(img, cmap='gray')\n", " plt.axis('off')\n", "plt.tight_layout() "], "execution_count": 29, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "oG7NweRPdjaA"}, "source": ["## Histograms of pixel intensities\n", "\n", "Histograms of pixel intensities show how frequently various color values occur in the image, i.e., frequency of pixels intensity values. In a black & white color space, pixel values range from 0 to 255, where 0 stands for black and 255 stands for white. Analysis of a histogram can help us understand the brightness, contrast, and intensity distribution of an image."]}, {"cell_type": "code", "metadata": {"id": "kkHJulH4djaA"}, "source": ["\"\"\" Helper function to plot the pixel intensitiy distribution for rgb images\n", "\n", " Parameters:\n", " image_path (str) The path to the image\n", " image_title (str) The title of the plot\n", "\n", " Returns:\n", " Null\n", "\"\"\"\n", "def print_image_dist(image_path, image_title):\n", " f = plt.figure(figsize=(16,8))\n", " f.add_subplot(1,2, 1)\n", "\n", " raw_image = plt.imread(image_path)\n", " plt.imshow(raw_image, cmap='gray')\n", " plt.colorbar()\n", " plt.title(image_title)\n", " print(f\"Image dimensions: {raw_image.shape[0],raw_image.shape[1]}\")\n", " print(f\"Maximum pixel value : {raw_image.max():.1f} ; Minimum pixel value:{raw_image.min():.1f}\")\n", " print(f\"Mean value of the pixels : {raw_image.mean():.1f} ; Standard deviation : {raw_image.std():.1f}\")\n", "\n", " f.add_subplot(1,2, 2)\n", "\n", " _ = plt.hist(raw_image[:, :, 0].ravel(), bins = 256, color = 'red', alpha = 0.5)\n", " _ = plt.hist(raw_image[:, :, 1].ravel(), bins = 256, color = 'Green', alpha = 0.5)\n", " _ = plt.hist(raw_image[:, :, 2].ravel(), bins = 256, color = 'Blue', alpha = 0.5)\n", " _ = plt.xlabel('Intensity Value')\n", " _ = plt.ylabel('Count')\n", " _ = plt.legend(['Red_Channel', 'Green_Channel', 'Blue_Channel'])\n", " plt.show()"], "execution_count": 30, "outputs": []}, {"cell_type": "code", "metadata": {"id": "IRSvCWdSdjaB", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "b0e04ddd-56eb-4bc5-d7fe-af6e9cbd31b0"}, "source": ["benign_img = train[train['benign_malignant'] == \"benign\"].iloc[0]\n", "benign_img"], "execution_count": 31, "outputs": []}, {"cell_type": "code", "metadata": {"id": "sPQvvb4BdjaC", "colab": {"base_uri": "https://localhost:8080/", "height": 757}, "outputId": "ce558077-8df6-4de1-f7ff-41832dde84a8"}, "source": ["print_image_dist(benign_img['image_path'], 'Benign Image')"], "execution_count": 32, "outputs": []}, {"source": ["This image shows a benign finding located in the head/neck area. The mean value of the pixels is 0.7. The standard deviation is 0.2. The distribution of the RGB-Channels follows a normal distribution. The Red Channel is overrepresented in this image."], "cell_type": "markdown", "metadata": {}}, {"cell_type": "code", "metadata": {"id": "ZtMCWuRKdjaD", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "ee5fa9f9-7394-4596-e838-09c2f1b509bc"}, "source": ["malignant_img = train[train['benign_malignant'] == \"malignant\"].iloc[0]\n", "malignant_img"], "execution_count": 33, "outputs": []}, {"cell_type": "code", "metadata": {"id": "LxZv0uhKdjaE", "colab": {"base_uri": "https://localhost:8080/", "height": 757}, "outputId": "93f4346c-516f-4a90-fc33-e56511419db9"}, "source": ["print_image_dist(malignant_img['image_path'], 'Malignant Image')"], "execution_count": 34, "outputs": []}, {"source": ["This image shows a malignant finding located in the upper extremity area. The mean value of the pixels is 0.7. The standard deviation is 0.1. The distribution, as well as the over representative Red-Channel, is similar to the benign image."], "cell_type": "markdown", "metadata": {}}, {"source": ["Based on these two images, we can already see how hard it is to identify a melanoma correctly."], "cell_type": "markdown", "metadata": {}}, {"cell_type": "markdown", "metadata": {"id": "NTOLu_PxdjaE"}, "source": ["## Data preparation\n", "\n", "Because we removed some values from the dataset for the EDA, we load the train and test set again."]}, {"cell_type": "code", "metadata": {"id": "DP1lSkATdjaE", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "f2ea5e6c-9ce5-4ced-d522-883f5f7aa5b3"}, "source": ["train = get_train_data()\n", "test = get_test_data()"], "execution_count": 35, "outputs": []}, {"cell_type": "code", "metadata": {"id": "l40Yikt8djaE"}, "source": ["# reduce amount of data when running on a cpu\n", "if DEVICE == 'CPU':\n", " print(\"reduce the amount of data because of cpu a runtime\")\n", " # take 30% of the available data\n", " train = train.sample(int(train.shape[0] * 0.3))\n", " EPOCHS = 5\n", " SAVE_OUTPUT = False"], "execution_count": 36, "outputs": []}, {"cell_type": "code", "metadata": {"id": "tG54suCTdjaF", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "ad62324d-efa0-4ed9-9813-f9b1edc473ab"}, "source": ["# getting dummy variables for gender\n", "sex_dummies = pd.get_dummies(train['sex'], prefix='sex', dtype=\"int\")\n", "train = pd.concat([train, sex_dummies], axis=1)\n", "\n", "sex_dummies = pd.get_dummies(test['sex'], prefix='sex', dtype=\"int\")\n", "test = pd.concat([test, sex_dummies], axis=1)\n", "\n", "# getting dummy variables for anatom_site_general_challenge\n", "anatom_dummies = pd.get_dummies(train['anatom_site_general_challenge'], prefix='anatom', dtype=\"int\")\n", "train = pd.concat([train, anatom_dummies], axis=1)\n", "\n", "anatom_dummies = pd.get_dummies(test['anatom_site_general_challenge'], prefix='anatom', dtype=\"int\")\n", "test = pd.concat([test, anatom_dummies], axis=1)\n", "\n", "# getting dummy variables for target column\n", "target_dummies = pd.get_dummies(train['target'], prefix='target', dtype=\"int\")\n", "train = pd.concat([train, target_dummies], axis=1)\n", "\n", "# dropping not useful columns\n", "train.drop(['sex','diagnosis','benign_malignant','anatom_site_general_challenge'], axis=1, inplace=True)\n", "test.drop(['sex','anatom_site_general_challenge'], axis=1, inplace=True)\n", "\n", "# replace missing age values wiht the mean age\n", "train['age_approx'] = train['age_approx'].fillna(int(np.mean(train['age_approx'])))\n", "test['age_approx'] = test['age_approx'].fillna(int(np.mean(test['age_approx'])))\n", "\n", "# convert age to int\n", "train['age_approx'] = train['age_approx'].astype('int')\n", "test['age_approx'] = test['age_approx'].astype('int')\n", "\n", "print(\"rows in train\", train.shape[0])\n", "print(\"rows in test\", test.shape[0])"], "execution_count": 37, "outputs": []}, {"cell_type": "code", "metadata": {"id": "jBNW9t1MdjaI", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "b18733b1-ae80-4491-bcc9-9f386f3652eb"}, "source": ["train.dtypes"], "execution_count": 38, "outputs": []}, {"cell_type": "code", "metadata": {"id": "fI23iTg3djaJ", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "a87e8628-c115-4d52-d184-6e28441dab66"}, "source": ["test.dtypes"], "execution_count": 39, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "bamRcA-xdjaJ"}, "source": ["### Balance the dataset\n", "\n", "Because we have a highly imbalanced dataset we need to balance it."]}, {"cell_type": "code", "metadata": {"id": "3Ayqx5OTdjaJ", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "4deb5bc6-f858-4c91-fb14-9c33f3a05d66"}, "source": ["# 1 means 50 / 50 => equal amount of positive and negative cases in Training\n", "# 4 = 20%; 8 = ~11%; 12 = ~8%\n", "balance = 1\n", "p_inds = train[train.target == 1].index.tolist()\n", "np_inds = train[train.target == 0].index.tolist()\n", "\n", "np_sample = random.sample(np_inds, balance * len(p_inds))\n", "train = train.loc[p_inds + np_sample]\n", "print(\"Samples in train\", train['target'].sum()/len(train))\n", "print(\"Remaining rows in train set\", len(train))"], "execution_count": 40, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "1lWAdFggdjaK"}, "source": ["### Patient Overlap\n", "\n", "Important to note is that there are patients with multiple images taken in both train and test datasets.\n", "We, therefore, need to check that the same patient images do not appear in the training and test set.\n"]}, {"cell_type": "code", "metadata": {"id": "IcKg60KjdjaK", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "4183b2fc-fbe8-4c97-da18-704149b89861"}, "source": ["print(\"Max number of images from one patient in the train set:\", np.max(train.patient_id.value_counts()))\n", "print(\"Max number of images from one patient in the test set:\", np.max(test.patient_id.value_counts()))"], "execution_count": 41, "outputs": []}, {"cell_type": "code", "metadata": {"id": "55Kb4-q0djaK", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "8cd14d5f-501c-4f49-b171-4897a11daffa"}, "source": ["# get the unique patient ids from the test and training set\n", "ids_train = set(train.patient_id.values)\n", "ids_test = set(test.patient_id.values)\n", "\n", "print(\"There are\", len(ids_train), \"unique patients in the training set\")\n", "print(\"There are\", len(ids_test), \"unique patients in the test set\")\n", "\n", "# Identify patient overlap by looking at the intersection between the sets\n", "patient_overlap = list(ids_train.intersection(ids_test))\n", "n_overlap = len(patient_overlap)\n", "print(\"There are\", n_overlap, \"patients in both the training and test sets\")"], "execution_count": 42, "outputs": []}, {"cell_type": "code", "metadata": {"id": "bIdtddiJdjaL"}, "source": ["\"\"\" Helper function to create a train and a validation dataset\n", "\n", " Parameters:\n", " df (dataframe): The dataframe to split\n", " test_size (int): Size of the validation set\n", " classToPredict: The target column\n", "\n", " Returns:\n", " train_data (dataframe)\n", " val_data (dataframe)\n", "\"\"\"\n", "def create_splits(df, test_size, classToPredict):\n", " train_data, val_data = train_test_split(df, test_size = test_size, random_state = SEED, stratify = df[classToPredict])\n", " return train_data, val_data"], "execution_count": 43, "outputs": []}, {"cell_type": "code", "metadata": {"id": "wdrJ0CbSdjaN"}, "source": ["\"\"\" Helper function to plot the history of a tensorflow model\n", "\n", " Parameters:\n", " history (history object): The history from a tf model\n", " timestamp (string): The timestamp of the function execution\n", "\n", " Returns:\n", " Null\n", "\"\"\"\n", "def save_history(history, timestamp):\n", " f = plt.figure()\n", " f.set_figwidth(15)\n", "\n", " f.add_subplot(1, 2, 1)\n", " plt.plot(history['val_loss'], label='val loss')\n", " plt.plot(history['loss'], label='train loss')\n", " plt.legend()\n", " plt.title(\"Modell Loss\")\n", "\n", " f.add_subplot(1, 2, 2)\n", " plt.plot(history['val_accuracy'], label='val accuracy')\n", " plt.plot(history['accuracy'], label='train accuracy')\n", " plt.legend()\n", " plt.title(\"Modell Accuracy\")\n", "\n", " if SAVE_OUTPUT:\n", " plt.savefig(\"./\" + timestamp + \"-history.png\")\n", " with open(\"./\" + timestamp + \"-history.json\", 'w') as f:\n", " json.dump(history, f)"], "execution_count": 44, "outputs": []}, {"cell_type": "code", "metadata": {"id": "VUFWVx4hdjaN"}, "source": ["\"\"\" Helper function to plot the auc curve\n", "\n", " Parameters:\n", " t_y (array): True binary labels\n", " p_y (array): Target scores\n", "\n", " Returns:\n", " Null\n", "\"\"\"\n", "def plot_auc(t_y, p_y):\n", " fpr, tpr, thresholds = roc_curve(t_y, p_y, pos_label=1)\n", " fig, c_ax = plt.subplots(1,1, figsize = (8, 8))\n", " c_ax.plot(fpr, tpr, label = '%s (AUC:%0.2f)' % ('Target', auc(fpr, tpr)))\n", " c_ax.plot([0, 1], [0, 1], color='navy', lw=1, linestyle='--')\n", " c_ax.legend()\n", " c_ax.set_xlabel('False Positive Rate')\n", " c_ax.set_ylabel('True Positive Rate')"], "execution_count": 45, "outputs": []}, {"source": ["## Data augmentation"], "cell_type": "markdown", "metadata": {}}, {"cell_type": "code", "metadata": {"id": "0DkseDQSdjaN"}, "source": ["\"\"\" Factory function to create a training image data generator\n", "\n", "Parameters:\n", " df (dataframe): Training dataframe \n", "\n", "Returns:\n", " Image Data Generator function\n", "\"\"\"\n", "def get_training_gen(df):\n", " ## prepare images for training\n", " train_idg = ImageDataGenerator(\n", " rescale = 1 / 255.0,\n", " horizontal_flip = True, \n", " vertical_flip = True, \n", " height_shift_range = 0.15, \n", " width_shift_range = 0.15,\n", " shear_range=0.15,\n", " rotation_range = 90, \n", " zoom_range = 0.20,\n", " fill_mode='nearest'\n", " )\n", "\n", " train_gen = train_idg.flow_from_dataframe(\n", " seed=SEED,\n", " dataframe=df,\n", " directory=None,\n", " x_col='image_path',\n", " y_col=['target_0','target_1'],\n", " class_mode='raw',\n", " shuffle=True,\n", " target_size=IMG_SIZE,\n", " batch_size=BATCH_SIZE,\n", " validate_filenames = False\n", " )\n", "\n", " return train_gen"], "execution_count": 46, "outputs": []}, {"cell_type": "code", "metadata": {"id": "IoYfVIj1djaN"}, "source": ["\"\"\" Factory function to create a validation image data generator\n", "\n", "Parameters:\n", " df (dataframe): Validation dataframe \n", "\n", "Returns:\n", " Image Data Generator function\n", "\"\"\"\n", "def get_validation_gen(df):\n", " ## prepare images for validation\n", " val_idg = ImageDataGenerator(rescale=1. / 255.0)\n", " val_gen = val_idg.flow_from_dataframe(\n", " seed=SEED,\n", " dataframe=df,\n", " directory=None,\n", " x_col='image_path',\n", " y_col=['target_0','target_1'],\n", " class_mode='raw',\n", " shuffle=False,\n", " target_size=IMG_SIZE,\n", " batch_size=BATCH_SIZE,\n", " validate_filenames = False\n", " )\n", "\n", " return val_gen"], "execution_count": 47, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "Y_EECRkFdjaN"}, "source": ["### Images returned from the ImageDataGenerator"]}, {"cell_type": "code", "metadata": {"tags": [], "id": "GeRvqJ-qdjaO", "colab": {"base_uri": "https://localhost:8080/", "height": 1000}, "outputId": "08f1a46e-540f-46b4-de53-07d7fe435219"}, "source": ["train_gen = get_training_gen(train)\n", "t_x, t_y = next(train_gen)\n", "fig, m_axs = plt.subplots(4, 4, figsize = (16, 16))\n", "for (c_x, c_y, c_ax) in zip(t_x, t_y, m_axs.flatten()):\n", " c_ax.imshow(c_x, cmap = 'bone')\n", " if c_y[0] == 1: \n", " c_ax.set_title('MALIGNANT')\n", " else:\n", " c_ax.set_title('BENIGN')\n", " c_ax.axis('off')"], "execution_count": 48, "outputs": []}, {"source": ["The Image Data Generator function returns these transformed images.\n", "\n", "The Keras ImageDataGenerator class works by:\n", "- Accepting a batch of images used for training.\n", "- Taking this batch and applying a series of random transformations to each image in the batch (including random rotation, resizing, shearing, etc.).\n", "- Replacing and returning the original batch with the new, randomly transformed batch.\n", "\n", "Source: https://www.pyimagesearch.com/2019/07/08/keras-imagedatagenerator-and-data-augmentation/"], "cell_type": "markdown", "metadata": {}}, {"cell_type": "markdown", "metadata": {"id": "67EjcGWPdjaO"}, "source": ["## Transfer Learning\n", "\n", "Conventional machine learning and deep learning algorithms, so far, have been traditionally designed to work in isolation. These algorithms are trained to solve specific tasks. The models have to be rebuilt from scratch once the feature-space distribution changes. Transfer learning is the idea of overcoming the isolated learning paradigm and utilizing knowledge acquired for one task to solve related ones. "]}, {"cell_type": "markdown", "metadata": {"id": "fkqvRu-0djaO"}, "source": ["\n", "\n", " "]}, {"cell_type": "markdown", "metadata": {"id": "g-vCqePSD4iI"}, "source": ["Traditional learning is isolated and occurs purely based on specific tasks, datasets, and training separate isolated models on them. No knowledge is retained, which can be transferred from one model to another. In transfer learning, you can leverage knowledge (features, weights, etc.) from previously trained models for training newer models and even tackle problems like having less data for the more recent task."]}, {"cell_type": "markdown", "metadata": {"id": "oLBLoI_mEp3q"}, "source": ["**Fine Tuning Off-the-shelf Pre-trained Models**\n", "\n", "This is a more involved technique, where we do not just replace the final layer (for classification/regression), but we also selectively retrain some of the previous layers. \n", "\n", "\n", "\n", " \n", "\n"]}, {"cell_type": "markdown", "metadata": {"id": "TXTzfRx8EmSV"}, "source": ["Source: https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a"]}, {"cell_type": "code", "metadata": {"id": "5aWRaNHUdjaO"}, "source": ["\"\"\" Helper function which returns a VGG16 model\n", "\"\"\"\n", "def load_pretrained_model():\n", " base_model = VGG16(\n", " input_shape=INPUT_SHAPE,\n", " include_top=False,\n", " weights='imagenet'\n", " )\n", " \n", " # freeze the first 15 layers of the base model. All other layers are trainable.\n", " for layer in base_model.layers[0:15]:\n", " layer.trainable = False\n", "\n", " for idx, layer in enumerate(base_model.layers):\n", " print(\"layer\", idx + 1, \":\", layer.name, \"is trainable:\", layer.trainable)\n", "\n", " return base_model\n", "\n", "\n", "\"\"\" Helper function which returns a tensorflow model\n", "\"\"\"\n", "def create_model():\n", " print(\"create model\")\n", "\n", " # Create a new sequentail model and add the pretrained model\n", " model = Sequential()\n", "\n", " # Add the pretrained model\n", " model.add(load_pretrained_model()) \n", "\n", " # Add a flatten layer to prepare the ouput of the cnn layer for the next layers\n", " model.add(layers.Flatten())\n", "\n", " # Add a dense (aka. fully-connected) layer. \n", " # Add a dropout-layer which may prevent overfitting and improve generalization ability to unseen data.\n", " model.add(layers.Dense(128, activation='relu'))\n", " model.add(layers.Dropout(0.3))\n", "\n", " model.add(layers.Dense(64, activation='relu'))\n", " model.add(layers.Dropout(0.3))\n", "\n", " model.add(layers.Dense(32, activation='relu'))\n", "\n", " # Use the Sigmoid activation function for binary predictions, softmax for n-classes\n", " # We use the softmax function, because we have two classes (target_0 & target_1)\n", " model.add(layers.Dense(NUM_CLASSES, activation='softmax'))\n", "\n", " return model"], "execution_count": 49, "outputs": []}, {"cell_type": "code", "metadata": {"id": "4tiogvTddjaO", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "6e60b24b-a310-44b6-d066-eac8e234a19f"}, "source": ["model = create_model()\n", "model.summary()"], "execution_count": 50, "outputs": []}, {"cell_type": "code", "metadata": {"id": "7DjCz-uXdjaP"}, "source": ["# get the current timestamp. This timestamp is used to save the model data with a unique name\n", "now = datetime.now()\n", "today = date.today()\n", "current_time = now.strftime(\"%H:%M:%S\")\n", "timestamp = str(today) + \"_\" + str(current_time)"], "execution_count": 51, "outputs": []}, {"cell_type": "code", "metadata": {"id": "xWdRhc_mdjaQ"}, "source": ["callback_list = []\n", "\n", "# if the model does not improve for 10 epochs, stop the training\n", "stop_early = EarlyStopping(monitor='val_loss', mode='auto', patience=10)\n", "callback_list.append(stop_early)\n", "\n", "# if the output of the model should be saved, create a checkpoint callback function\n", "if SAVE_OUTPUT:\n", " # set the weight path for saving the model\n", " weight_path = \"./\" + timestamp + \"-model.hdf5\"\n", " # create the model checkpoint callback to save the model wheights to a file\n", " checkpoint = ModelCheckpoint(\n", " weight_path,\n", " save_weights_only=True,\n", " verbose=VERBOSE_LEVEL,\n", " save_best_only=True,\n", " monitor='val_loss',\n", " overwrite=True,\n", " mode='auto',\n", " )\n", " # append the checkpoint callback to the callback list\n", " callback_list.append(checkpoint)"], "execution_count": 52, "outputs": []}, {"source": ["## Model training"], "cell_type": "markdown", "metadata": {}}, {"cell_type": "code", "metadata": {"id": "zxj3bFJPdjaR", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "fbc3976a-1fbb-40fc-90a2-dd521dc63d7f"}, "source": ["# create a training and validation dataset from the train df\n", "train_df, val_df = create_splits(train, 0.2, 'target')\n", "\n", "print(\"rows in train_df\", train_df.shape[0])\n", "print(\"rows in val_df\", val_df.shape[0])\n", "\n", "# because we do not need the target column anymore we can drop it\n", "train_df.drop(['target'], axis=1, inplace=True)\n", "val_df.drop(['target'], axis=1, inplace=True)\n", "\n", "# call the generator functions\n", "train_gen = get_training_gen(train_df)\n", "val_gen = get_validation_gen(val_df)\n", "valX, valY = val_gen.next()"], "execution_count": 53, "outputs": []}, {"cell_type": "code", "metadata": {"tags": [], "id": "3JvcUnNLdjab", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "ca64560f-232f-4206-c64a-c4674c5035da"}, "source": ["LEARNING_RATE = 1e-4\n", "OPTIMIZER = Adam(lr=LEARNING_RATE)\n", "LOSS = 'binary_crossentropy'\n", "METRICS = [\n", " 'accuracy', \n", " 'AUC'\n", "] \n", "\n", "model.compile(\n", " loss=LOSS,\n", " metrics=METRICS,\n", " optimizer=OPTIMIZER,\n", ")\n", "\n", "# when on a cpu, do not save the model data\n", "if DEVICE == 'CPU':\n", " print(\"fit model on cpu\")\n", " history = model.fit(\n", " train_gen, \n", " epochs=EPOCHS, \n", " verbose=VERBOSE_LEVEL,\n", " validation_data=(valX, valY)\n", " )\n", "else:\n", " print(\"fit model on gpu\")\n", " history = model.fit(\n", " train_gen, \n", " epochs=EPOCHS, \n", " verbose=VERBOSE_LEVEL,\n", " callbacks=callback_list, \n", " validation_data=(valX, valY),\n", " )"], "execution_count": 54, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "VT9WHX7ze3sz"}, "source": ["## Model Evaluation"]}, {"cell_type": "code", "metadata": {"id": "Bn1YGzsedjac", "colab": {"base_uri": "https://localhost:8080/", "height": 509}, "outputId": "4923d9ec-9d98-4e40-c1eb-3d927c6cda9e"}, "source": ["# plot model history\n", "save_history(history.history, timestamp)"], "execution_count": 55, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "0NlzfWYk0_f_"}, "source": ["From the accuracy plot, we can see that the model stops learning after epoch 22. We can also see that the model has not yet over-learned the training dataset. We can see that the model has comparable performance on both train and validation datasets from the loss plot. The model achieved the lowest loss at the 19th epoch.\n", "\n", "However, the plots suggest that our model has difficulty generalizing, as the validation curves vary widely in some cases."]}, {"cell_type": "code", "metadata": {"id": "7PGfEYGqdjac", "colab": {"base_uri": "https://localhost:8080/", "height": 718}, "outputId": "62f7716d-73f8-4093-9fcf-f6fd7686725f"}, "source": ["# plot the auc\n", "y_t = [] # true labels\n", "y_p = [] # predictions\n", "\n", "# iterate over the validation df and make a prediction for each image\n", "for i in tqdm(range(val_df.shape[0])):\n", " y_true = val_df.iloc[i].target_1\n", " image_path = val_df.iloc[i].image_path\n", "\n", " img = keras.preprocessing.image.load_img(image_path, target_size=IMG_SIZE)\n", " img = keras.preprocessing.image.img_to_array(img)\n", " img = img / 255\n", " img_array = tf.expand_dims(img, 0)\n", " y_pred = model.predict(img_array)\n", " y_pred = tf.nn.softmax(y_pred)[0].numpy()[1]\n", "\n", " y_t.append(y_true)\n", " y_p.append(y_pred)\n", "\n", "plot_auc(y_t, y_p)"], "execution_count": 56, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "0xTrPH_D0_f_"}, "source": ["AUC - ROC curve is a performance measurement for the classification problem at various threshold settings. ROC is a probability curve, and AUC represents the degree or measure of separability. It tells how much the model is capable of distinguishing between classes. Higher the AUC, the better the model is at predicting 0s as 0s and 1s as 1s. By analogy, the Higher the AUC, the better the model is at distinguishing between patients with the disease and no disease. The ROC curve is plotted with TPR against the FPR, where TPR is on the y-axis and FPR is on the x-axis.\n", "\n", "Source: https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5"]}, {"cell_type": "markdown", "metadata": {"id": "uINcuIl7oN6y"}, "source": ["### F1 Score Calculation\n", "\n", "The F1 score is the harmonic mean of precision and recall. In a statistical analysis of binary classification, the F-score is a measure of a test's accuracy. It is calculated from the precision and recall of the test, where the precision is the number of correctly identified positive results divided by the number of all positive outcomes, including those not identified correctly. The recall is the number of correctly identified positive results divided by the number of all samples that should have been identified as positive.\n", "\n", "The highest possible value of an F-score is 1, indicating perfect precision and recall, and the lowest potential value is 0 if either the precision or the recall is zero.\n", "\n", "Source: https://en.wikipedia.org/wiki/F-score"]}, {"cell_type": "code", "metadata": {"id": "4M-vmev8n2EQ", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "90da6820-5de2-4677-9747-c98feed85dcd"}, "source": ["\"\"\" Helper function to calculate the F1 Score\n", "\n", " Parameters:\n", " prec (int): precision\n", " recall (int): recall\n", "\n", " Returns:\n", " f1 score (int)\n", "\"\"\"\n", "def calc_f1(prec, recall):\n", " return 2*(prec*recall)/(prec+recall) if recall and prec else 0\n", "\n", "# calculate the precision, recall and the thresholds\n", "precision, recall, thresholds = precision_recall_curve(y_t, y_p)\n", "\n", "# calculate the f1 score\n", "f1score = [calc_f1(precision[i],recall[i]) for i in range(len(thresholds))]\n", "\n", "# get the index from the highest f1 score\n", "idx = np.argmax(f1score)\n", "\n", "# get the precision, recall, threshold and the f1score\n", "precision = round(precision[idx], 4)\n", "recall = round(recall[idx], 4)\n", "threshold = round(thresholds[idx], 4)\n", "f1score = round(f1score[idx], 4)\n", "\n", "print('Precision:', precision)\n", "print('Recall:', recall)\n", "print('Threshold:', threshold)\n", "print('F1 Score:', f1score)"], "execution_count": 57, "outputs": []}, {"cell_type": "code", "metadata": {"id": "lvpxEUE-AQW5"}, "source": ["\"\"\" Helper function turn the model predictions into a binary (0,1) format\n", "\n", " Parameters:\n", " pred (float): Model prediction\n", "\n", " Returns:\n", " binary prediction (int)\n", "\"\"\"\n", "def pred_to_binary(pred):\n", " if pred < threshold:\n", " return 0\n", " else:\n", " return 1\n", "\n", "y_pred_binary = [pred_to_binary(x) for x in y_p]"], "execution_count": 58, "outputs": []}, {"cell_type": "code", "metadata": {"id": "asa6b_y9AD4y", "colab": {"base_uri": "https://localhost:8080/"}, "outputId": "5f807a95-12de-448f-bc2c-583298d614a2"}, "source": ["# create a confusion matrix\n", "cm = confusion_matrix(y_t, y_pred_binary)\n", "cm"], "execution_count": 59, "outputs": []}, {"cell_type": "code", "metadata": {"id": "YyiE5vMSzIpW", "colab": {"base_uri": "https://localhost:8080/", "height": 557}, "outputId": "73ce3f0a-a4dc-4e33-e0a5-13759dee17d7"}, "source": ["\"\"\" Helper function to plot a confusion matrix\n", "\n", " Parameters:\n", " cm (confusion matrix)\n", "\n", " Returns:\n", " Null\n", "\"\"\"\n", "def plot_confusion_matrix(cm, labels):\n", " plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)\n", " plt.title('Confusion Matrix')\n", " plt.colorbar()\n", " tick_marks = np.arange(len(labels))\n", " plt.xticks(tick_marks, labels, rotation=55)\n", " plt.yticks(tick_marks, labels)\n", "\n", " thresh = cm.max() / 2.\n", " for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n", " plt.text(j, i, format(cm[i, j], 'd'), horizontalalignment=\"center\", color=\"white\" if cm[i, j] > thresh else \"black\")\n", "\n", " plt.ylabel('True label')\n", " plt.xlabel('Predicted label')\n", " plt.tight_layout()\n", "\n", "cm_plot_label =['benign', 'malignant']\n", "plot_confusion_matrix(cm, cm_plot_label)"], "execution_count": 60, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "M3t77b-A0_gA"}, "source": ["The model predicted 191 images correclty, but failed on 43."]}, {"cell_type": "markdown", "metadata": {"id": "iV0hJPkxe6ct"}, "source": ["## Inference"]}, {"cell_type": "code", "metadata": {"id": "iHP7Lb3Odjac", "colab": {"base_uri": "https://localhost:8080/", "height": 459}, "outputId": "96bd8ef3-d03c-445d-ea27-d267b8147f1d"}, "source": ["# Show a prediction for a random image\n", "image_path = test.iloc[0].image_path\n", "img = keras.preprocessing.image.load_img(image_path, target_size=IMG_SIZE)\n", "img = keras.preprocessing.image.img_to_array(img)\n", "img = img / 255\n", "img_array = tf.expand_dims(img, 0)\n", "\n", "prediction = np.max(tf.nn.softmax(model.predict(img_array)[0])[1])\n", "print(\"Chance of beeing malignant: {:.2f} %\".format(prediction))\n", "\n", "finding = \"Diagnosis: BENIGN\"\n", "if prediction > threshold:\n", " finding = \"Diagnosis: MALIGNANT\"\n", "\n", "x = plt.figure(figsize=(5,5))\n", "x = plt.imshow(img)\n", "x = plt.title(finding)\n", "x = plt.axis(\"off\")"], "execution_count": 61, "outputs": []}, {"cell_type": "markdown", "metadata": {"id": "kX5wJVU6zMS2"}, "source": ["## Discussion\n", "\n", "We achieved an F1-Score of 0.817. Based on a study from Han SS, Moon IJ, Lim W, et al. \"Keratinocytic Skin Cancer Detection on the Face Using Region-Based Convolutional Neural Network\" the F1-score of a professional dermatologist is 0.835. Compared to this result, our neural network performed slightly worse. It should be noted, however, that only facial skin cancer was considered in this study. The F1-Score of professional dermatologists on a more realistic dataset like this one could therefore differ.\n", "\n", "\n", "Source: https://pubmed.ncbi.nlm.nih.gov/31799995/"]}, {"cell_type": "markdown", "metadata": {"id": "t_-SwTO3OMU0"}, "source": ["![\"Clinical](\"https://github.com/SaschaMet/melanoma-classification/blob/master/images/clinical_relevance.png?raw=1\")
\n", "\n", "Source: https://www.udacity.com/course/ai-for-healthcare-nanodegree--nd320"]}, {"cell_type": "markdown", "metadata": {"id": "Pgj2fLtWOPz7"}, "source": ["Precision and Recall are of particular interest to the clinical applicability of the model. A model with high precision has increased confidence in a positive result. It is, therefore, better suited in confirming a diagnosis. A model with high Recall, on the other hand, is most confident when the test is negative. Such a model is better used for prioritization tasks (e.g., which lesions should be looked at first).\n", "\n", "The precision of this model is 0.8136. The recall is 0.8205. This neural network should be better suited for prioritization tasks than for confirming a diagnosis. But, because we achieved an F1-Score comparable to professional dermatologists, plus precision and recall are almost the same, the model could also be useful in, e.g., confirming a dermatologist's diagnosis."]}, {"cell_type": "markdown", "metadata": {"id": "qYl9K5mG0_gB"}, "source": ["### Cut-Off Thresholds\n", "\n", "The output of our CNN's last layer will output a probability that an image belongs to a given class (target_0 or target_1). Changing the threshold for this classification will transform the true positive, false positive, false negative, and true negative rates. This will, in turn, change the precision and recall of our model. We could, for example, change the threshold so that our precision increases. This, however, has the result that our recall changes and probably even decreases. One metric is optimized at the expense of another. Because of this, we used the F1-Score as a final metric because the F1-Score combines both precision and recall. (It is the harmonic mean of precision and recall.)"]}, {"cell_type": "markdown", "metadata": {"id": "eNoVI-2v0_gB"}, "source": ["### Kaggle Leaderboard\n", "\n", "When predicting the images from the provided test set, the model achieves a private score of 0.8041 and a public score of 0.8247. This results in a place on the leaderboard at around 2.700.\n", "\n", "The best model on the private leaderboard of the Kaggle competition achieved a score of 0.9490.\n", "\n", "All submissions are evaluated on the area under the ROC curve between the predicted probability and the observed target."]}, {"cell_type": "markdown", "metadata": {"id": "t6iybwN4OiWn"}, "source": ["## How to further improve the model\n", "\n", "Based on the placement on the leaderboard, you can already see that there is still a lot of potential for optimization. The following possibilities could be addressed, for example:\n", "\n", "- Try different pre-trained models: https://keras.io/api/applications\n", "- Hyperparameter tuning: https://blog.tensorflow.org/2020/01/hyperparameter-tuning-with-keras-tuner.html\n", "- Get more training data: https://www.kaggle.com/wanderdust/skin-lesion-analysis-toward-melanoma-detection\n", "- Experiment with different loss functions: https://www.tensorflow.org/addons/api_docs/python/tfa/losses/SigmoidFocalCrossEntropy\n", "- Improve the data augmentation, e.g. by removing body hair: https://www.kaggle.com/vatsalparsaniya/melanoma-hair-remove"]}, {"cell_type": "markdown", "metadata": {"id": "iHPE_dwq0_gC"}, "source": ["However, even with these options, it will be challenging to get a top ranking on the leaderboard. For example, the winning team consists of three Kaggle Grandmasters, all of whom work at NVIDIA. In an interview, they also mentioned that one of their most significant advantages was the abundant resources (e.g., GPUs) they received from NVIDIA.\n", "\n", "Source: https://www.youtube.com/watch?v=L1QKTPb6V_I"]}, {"cell_type": "code", "metadata": {"id": "xV0j-1PE0_gC", "outputId": "84981ccd-312a-4fb7-d05b-89f5ffd9324c", "colab": {"base_uri": "https://localhost:8080/"}}, "source": ["if SAVE_OUTPUT:\n", " # save the model to a json file\n", " model_json = model.to_json()\n", " with open(\"./\" + timestamp + \"-model.json\", \"w\") as json_file:\n", " json_file.write(model_json)\n", "\n", " # create the submission.csv file\n", " data=[]\n", " for i in tqdm(range(test.shape[0])):\n", " image_path = test.iloc[i].image_path\n", " image_name = test.iloc[i].image_name\n", " img = keras.preprocessing.image.load_img(image_path, target_size=IMG_SIZE)\n", " img = keras.preprocessing.image.img_to_array(img)\n", " img = img / 255\n", " img_array = tf.expand_dims(img, 0)\n", " y_pred = model.predict(img_array)\n", " y_pred = tf.nn.softmax(y_pred)[0].numpy()[1]\n", " data.append([image_name, y_pred])\n", "\n", " sub_df = pd.DataFrame(data, columns = ['image_name', 'target']) \n", " sub_df.to_csv(\"./submission.csv\", index=False)\n", "\n", " sub_df.head()"], "execution_count": 62, "outputs": []}]}