{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Tom Segal's Entry For the Competition \"SIIM-ISIC Melanoma Classification\" ##"},{"metadata":{},"cell_type":"markdown","source":"This is my third ML project.\n\nThe first project was the MNIST digit recognizer, 10 labels for 1-color images.\n\nThe second project was flower identification, >100 labels for 3-color images.\n\nIn this project tumors are differiented into malignant (melanoma-inducing) and benign.\n\nIn this project, downsampling is used in order to even the label distribution. An unconventional loss function, Focal Loss, is used in order to compensate for the remaining difference in the distribution.\n\nProject overview https://www.kaggle.com/c/siim-isic-melanoma-classification\n\ndata https://www.kaggle.com/c/siim-isic-melanoma-classification/data\n\n\n\nthis project relies on research of the following notebooks:\n\nhttps://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579\n\nhttps://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once\n\nhttps://www.kaggle.com/ibtesama/siim-baseline-keras-vgg16\n\n\nadditional references:\n\nhttps://pypi.org/project/focal-loss/\n\n\nI acknowledge and appreciate the support of the kaggle community.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install focal-loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## install missing libraries ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nprint(\"tensorflow version: \" + tf.__version__)\nfrom kaggle_datasets import KaggleDatasets\nimport pandas as pd\nimport os\nimport matplotlib.pyplot as plt\nimport cv2\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom keras.applications.vgg16 import VGG16\nfrom keras.applications import DenseNet201\nfrom keras.layers import Flatten, Dense\nfrom keras.models import Model\nfrom keras.optimizers import Adam\nfrom tensorflow.python.keras import backend\nfrom focal_loss import BinaryFocalLoss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## define constants ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"random_state = 19 # using a constant random seed makes the results more consistent and helps comparing between them.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## read the data as panda dataframe for examination ##"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Get the path of the Current System (GCS)\nGCS_PATH = KaggleDatasets().get_gcs_path(\"siim-isic-melanoma-classification\")\n# get the train data in dataframe format for quick examination of the data\ndataframe_train = pd.read_csv(\"../input/siim-isic-melanoma-classification/train.csv\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## examine the data ##"},{"metadata":{},"cell_type":"markdown","source":"view the first few dataframe entries"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataframe_train.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"plot some samples"},{"metadata":{"trusted":true},"cell_type":"code","source":"# image_paths = GCS_PATH_TRAIN + \"\\\\\" + dataframe_train[\"image_name\"]+\".jpg\" # \\\\ because \\ is an escape character\n# image_paths = \"../input/siim-isic-melanoma-classification/jpeg/train/\" + dataframe_train[\"image_name\"] + \".jpg\" \nimage_paths = \"../input/jpeg-melanoma-256x256/train/\" + dataframe_train[\"image_name\"] + \".jpg\" \nf, ax = plt.subplots(3, 5, figsize = (10,6))\nfor i in range(15):\n    #print(image_paths[i])\n    img = cv2.imread(image_paths[i])\n    #print(img.shape)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # the default cv2 format is BGR\n    ax[i//5, i%5].imshow(img)\n    ax[i//5, i%5].axis(\"off\")\nplt.show()\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"examine the label distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataframe_train[\"target\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"584/32542 = ~1.8% of the tumors are malignant, such that the labels are highly unbalanced.\n\nthis can be treated using oversampling or undersampling."},{"metadata":{},"cell_type":"markdown","source":"## undersampling ##"},{"metadata":{},"cell_type":"markdown","source":"only a portion of the benign tumor training data will be used in order to make the labels more balanced."},{"metadata":{"trusted":true},"cell_type":"code","source":"downsampling = 1000\n# sample 1000 benign samples and merge them together with all of the malignant samples\ndataframe_train_benign_downsampled = dataframe_train[dataframe_train[\"target\"]==0].sample(downsampling)\ndataframe_train_malignant = dataframe_train[dataframe_train[\"target\"]==1]\n# join the two parts together. Note that now the two sample types are not mixed anymore in the data\n# but appear in two blocks.\ndataframe_train_downsampled = pd.concat([dataframe_train_benign_downsampled, dataframe_train_malignant])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"show benign tumor samples"},{"metadata":{"trusted":true},"cell_type":"code","source":"image_paths = [\"../input/jpeg-melanoma-256x256/train/\" + dataframe_train_benign_downsampled[\"image_name\"].values[i] + \".jpg\" for i in range(downsampling)]\nf, ax = plt.subplots(3, 5, figsize = (10,6))\nfor i in range(15):\n    #print(image_paths[i])\n    img = cv2.imread(image_paths[i])\n    #print(img.shape)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # the default cv2 format is BGR\n\n    ax[i//5, i%5].imshow(img)\n    ax[i//5, i%5].axis(\"off\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"show malignant tumor samples"},{"metadata":{"trusted":true},"cell_type":"code","source":"image_paths = [\"../input/jpeg-melanoma-256x256/train/\" + dataframe_train_malignant[\"image_name\"].values[i] + \".jpg\" for i in range(dataframe_train_malignant.shape[0])]\n\n#print(image_paths[1])\nf, ax = plt.subplots(3, 5, figsize = (10,6))\nfor i in range(15):\n    #print(image_paths[i])\n    img = cv2.imread(image_paths[i])\n    #print(img.shape)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # the default cv2 format is BGR\n    ax[i//5, i%5].imshow(img)\n    ax[i//5, i%5].axis(\"off\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## simplify the data  ##"},{"metadata":{},"cell_type":"markdown","source":"create datasets with only the images and the labels"},{"metadata":{"trusted":true},"cell_type":"code","source":"image_paths = [\"../input/jpeg-melanoma-256x256/train/\" + dataframe_train_benign_downsampled[\"image_name\"].values[i] + \".jpg\" for i in range(downsampling)]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dataframe_train_labels = []\ndataframe_train_images = []\nfor i in range(dataframe_train_downsampled.shape[0]):\n    dataframe_train_labels.append(dataframe_train_downsampled[\"target\"].values[i])\n    dataframe_train_images.append(\"../input/jpeg-melanoma-256x256/train/\" + dataframe_train_downsampled[\"image_name\"].values[i] + \".jpg\")\n    \n# create a dataframe from the columns\nnparray_train_reduced_tuples = zip(dataframe_train_images, dataframe_train_labels)\n# dataframe_train_reduced = pd.DataFrame(np.array([dataframe_train_labels, dataframe_train_images]), columns = [\"label\",\"image\"])\ndataframe_train_reduced = pd.DataFrame(nparray_train_reduced_tuples, columns = [\"image\",\"label\"])\n# dataframe_train_reduced = pd.DataFrame(np.array([dataframe_train_labels, dataframe_train_images]))\ndataframe_train_reduced.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## split the data into train and validation ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train, x_val, y_train, y_val = train_test_split(dataframe_train_reduced[\"image\"], dataframe_train_reduced[\"label\"],\n                                                 test_size = 0.2, random_state = random_state)\ndataframe_train_split = pd.DataFrame(zip(x_train,y_train), columns = [\"image\",\"label\"])\ndataframe_val = pd.DataFrame(zip(x_val,y_val), columns = [\"image\",\"label\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## data normaliztion & augmentation ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"gen_train = ImageDataGenerator(\n    rescale = 1./255, # rescale the images (RGB [0,255])\n    width_shift_range = 0.15, height_shift_range = 0.15, # randomly shift the pictures by 15% in both axes\n    horizontal_flip = True, vertical_flip = True, # randomly flip the images in both axes\n)\ntrain_generator = gen_train.flow_from_dataframe(dataframe_train_reduced, x_col = \"image\", y_col = \"label\",\n                                               target_size = (256,256), batch_size = 8,\n                                               shuffle = True, # important as mentioned above\n                                               class_mode = \"raw\")\nval_generator = gen_train.flow_from_dataframe(dataframe_val, x_col = \"image\", y_col = \"label\",\n                                               target_size = (256,256), batch_size = 8,\n                                               shuffle = True, # not sure if important\n                                               class_mode = \"raw\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## define the model ##"},{"metadata":{},"cell_type":"markdown","source":"for the model a pretrained VGG16 model will be used, pre-weighted on \"imagenet\", with a flatten and a dense model placed on top.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = VGG16(weights = \"imagenet\",\n             include_top = False, # because a new top will be added to match the dimensions of this dataset\n             input_shape = (256,256,3))\nx = Flatten()(model.output)\noutput = Dense(1,activation = \"sigmoid\")(x)\nmodel = Model(model.input, output)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## compile the model ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(loss = \"binary_crossentropy\", metrics = [tf.keras.metrics.AUC()], optimizer = Adam(lr=0.00001))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## train the model ##"},{"metadata":{},"cell_type":"markdown","source":"batch_size = 8\nsteps_per_epoch = dataframe_train_reduced.shape[0] // batch_size\nepochs = 3\nvalidation_steps = dataframe_val.shape[0] // batch_size\nhistory = model.fit_generator(train_generator, steps_per_epoch = steps_per_epoch, epochs = epochs,\n                    validation_data = val_generator, validation_steps = validation_steps)"},{"metadata":{},"cell_type":"markdown","source":"## results ##"},{"metadata":{},"cell_type":"markdown","source":"Epoch 1/3\n198/198 [==============================] - 2103s 11s/step - loss: 0.5735 - auc_1: 0.7306 - val_loss: 0.5575 - val_auc_1: 0.8149\nEpoch 2/3\n198/198 [==============================] - 2106s 11s/step - loss: 0.5137 - auc_1: 0.8001 - val_loss: 0.4799 - val_auc_1: 0.8388\nEpoch 3/3\n198/198 [==============================] - 2104s 11s/step - loss: 0.4836 - auc_1: 0.8234 - val_loss: 0.4836 - val_auc_1: 0.8667"},{"metadata":{},"cell_type":"markdown","source":"## using focal loss instead of crossentropy ##"},{"metadata":{},"cell_type":"markdown","source":"https://arxiv.org/abs/1708.02002\n\n\"We propose to address this class imbalance by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples. Our novel Focal Loss focuses training on a sparse set of hard examples and prevents the vast number of easy negatives from overwhelming the detector during training.\""},{"metadata":{"trusted":true},"cell_type":"code","source":"def focal_loss(alpha=0.25,gamma=2.0):\n    def focal_crossentropy(y_true, y_pred):\n        bce = backend.binary_crossentropy(y_true, y_pred)\n        \n        y_pred = backend.clip(y_pred, backend.epsilon(), 1.- backend.epsilon())\n        p_t = (y_true*y_pred) + ((1-y_true)*(1-y_pred))\n        \n        alpha_factor = 1\n        modulating_factor = 1\n\n        alpha_factor = y_true*alpha + ((1-alpha)*(1-y_true))\n        modulating_factor = backend.pow((1-p_t), gamma)\n\n        # compute the final loss and return\n        return backend.mean(alpha_factor*modulating_factor*bce, axis=-1)\n    return focal_crossentropy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(loss=BinaryFocalLoss(gamma=2), metrics = [tf.keras.metrics.AUC()], optimizer = Adam(lr=0.00001))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"batch_size = 8\nsteps_per_epoch = dataframe_train_reduced.shape[0] // batch_size\nepochs = 3\nvalidation_steps = dataframe_val.shape[0] // batch_size\nhistory2 = model.fit_generator(train_generator, steps_per_epoch = steps_per_epoch, epochs = epochs,\n                    validation_data = val_generator, validation_steps = validation_steps)"},{"metadata":{},"cell_type":"markdown","source":"## results ##"},{"metadata":{},"cell_type":"markdown","source":"Epoch 1/3\n198/198 [==============================] - 2109s 11s/step - loss: 0.1244 - auc_6: 0.8345 - val_loss: 0.0998 - val_auc_6: 0.8968\nEpoch 2/3\n198/198 [==============================] - 2109s 11s/step - loss: 0.1140 - auc_6: 0.8596 - val_loss: 0.1055 - val_auc_6: 0.8924\nEpoch 3/3\n198/198 [==============================] - 2113s 11s/step - loss: 0.1112 - auc_6: 0.8663 - val_loss: 0.1087 - val_auc_6: 0.8911\n\nbetter than with the crossentropy loss metric"},{"metadata":{},"cell_type":"markdown","source":"## using DenseNet201 instead of VGG16 ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"model2 = DenseNet201(weights = \"imagenet\",\n             include_top = False, # because a new top will be added to match the dimensions of this dataset\n             input_shape = (256,256,3))\nx = Flatten()(model2.output)\noutput = Dense(1,activation = \"sigmoid\")(x)\nmodel2 = Model(model2.input, output)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model2.compile(loss=BinaryFocalLoss(gamma=2), metrics = [tf.keras.metrics.AUC()], optimizer = Adam(lr=0.00001))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 8\nsteps_per_epoch = dataframe_train_reduced.shape[0] // batch_size\nepochs = 3\nvalidation_steps = dataframe_val.shape[0] // batch_size\nhistory3 = model2.fit_generator(train_generator, steps_per_epoch = steps_per_epoch, epochs = epochs,\n                    validation_data = val_generator, validation_steps = validation_steps)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## results ##"},{"metadata":{},"cell_type":"markdown","source":"almost twice as fast.\n\nEpoch 1/3\n198/198 [==============================] - 1194s 6s/step - loss: 0.2557 - auc_2: 0.6912 - val_loss: 0.2462 - val_auc_2: 0.6968\nEpoch 2/3\n198/198 [==============================] - 1223s 6s/step - loss: 0.2170 - auc_2: 0.7629 - val_loss: 0.1782 - val_auc_2: 0.8301\nEpoch 3/3\n198/198 [==============================] - 1197s 6s/step - loss: 0.1912 - auc_2: 0.8044 - val_loss: 0.1833 - val_auc_2: 0.8421"},{"metadata":{},"cell_type":"markdown","source":"## results summary ##"},{"metadata":{},"cell_type":"markdown","source":"so far the best result is that of VGG16 with focal loss\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## submit the results ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = [] # the test predictions will be stored here\n# read the test csv file and obtain the image paths from it\ndataframe_test = pd.read_csv(\"../input/siim-isic-melanoma-classification/test.csv\")\ntest_image_paths = [\"../input/jpeg-melanoma-256x256/test/\" + image_name + \".jpg\" for image_name in dataframe_test[\"image_name\"]]\nprint(test_image_paths[5])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true,"collapsed":true},"cell_type":"code","source":"predictions = [] # the test predictions will be stored here\n# read the test csv file and obtain the image paths from it\ndataframe_test = pd.read_csv(\"../input/siim-isic-melanoma-classification/test.csv\")\ntest_image_paths = [\"../input/jpeg-melanoma-256x256/test/\" + image_name + \".jpg\" for image_name in dataframe_test[\"image_name\"]]\n# go over the image paths, load their respective images, make a prediction for them and save the predictions\ni=0\nfor test_image_path in test_image_paths:\n    img = cv2.imread(test_image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    img = np.reshape(img,(1,256,256,3))\n    predictions.append(model.predict(img))\n    if i%100 == 0:\n        print(\"finished \" + str(i) + \" out of \" + str(len(test_image_paths)))\n    i += 1\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.read_csv(\"../input/siim-isic-melanoma-classification/sample_submission.csv\")\nsubmission[\"target\"] = predictions\nsubmission.to_csv(\"submission.csv\", index = False)\n\nsubmission.head(30)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}