{"cells":[{"metadata":{},"cell_type":"markdown","source":"# SIIM - ISIC - Melanoma Classification","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Melanoma is the deadliest form of skin cancer. The goal of this competition is to develop a model which can classifiy benign and malignant skin lesions. Let's first understand some basic statistics and charachteristics about Melanoma through this [video](https://www.youtube.com/watch?v=DS9eIhcuWEM) \n[![video](https://img.youtube.com/vi/DS9eIhcuWEM/0.jpg)](https://www.youtube.com/watch?v=DS9eIhcuWEM)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n# import os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"path = '/kaggle/input/siim-isic-melanoma-classification'\ndf_train = pd.read_csv(f\"{path}/train.csv\")\ndf_test = pd.read_csv(f\"{path}/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## EDA","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Train/Test ratio: {df_train.shape[0]/df_test.shape[0]}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Train set is 3 times as bigger as test set","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Train Male-female ratio: {df_train.loc[df_train['sex']=='male'].shape[0]/df_train.loc[df_train['sex']=='female'].shape[0]}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Test Male-female ratio: {df_test.loc[df_test['sex']=='male'].shape[0]/df_test.loc[df_test['sex']=='female'].shape[0]}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have slightly more males in test than in train but I think that is fine because train set is also 3 times bigger than test set","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"num_malignant_males = df_train.loc[(df_train['target']==1) & (df_train['sex']=='male'), :].shape[0]\nnum_malignant_females = df_train.loc[(df_train['target']==1) & (df_train['sex']=='female'), :].shape[0]\nprint(f\"Male ratio in Malignant: {num_malignant_males/(num_malignant_males + num_malignant_females)}\")\n\nnum_benign_males = df_train.loc[(df_train['target']==0) & (df_train['sex']=='male'), :].shape[0]\nnum_benign_females = df_train.loc[(df_train['target']==0) & (df_train['sex']=='female'), :].shape[0]\nprint(f\"Male ratio in Benign: {num_benign_males/(num_benign_males + num_benign_females)}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So number of males in malignant cases are more","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(6,6))\nsns.countplot(x='sex', data=df_train.loc[df_train['target']==1], ax=ax, palette={'male': '#5db1e4','female': '#fb9ed6'})\nax.set_title('Sex Distribution in Malignant')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(6,6))\nsns.countplot(x='sex', \n              data=df_train.loc[df_train['target']==0], \n              ax=ax, \n              palette={'male': '#5db1e4','female': '#fb9ed6'})\nax.set_title('Sex Distribution in Benign')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,9))\nsns.countplot(x='age_approx', \n              hue='sex', \n              data=df_train.loc[df_train['target']==1], \n              ax=ax, \n              palette={'male': '#5db1e4','female': '#fb9ed6'})\nax.set_title('Age-Sex Distribution in Malignant cases')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,9))\nsns.countplot(x='age_approx', \n              hue='sex', \n              data=df_train.loc[df_train['target']==0], \n              ax=ax, \n              palette={'male': '#5db1e4','female': '#fb9ed6'})\nax.set_title('Age-Sex Distribution in Benign cases')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Not accurately but it seems that men are more likely to get Melanoma especially for age group of above 60 number of malignant cases in men are marginally high","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,9))\nsns.countplot(x='anatom_site_general_challenge', \n              hue='sex', \n              data=df_train.loc[df_train['target']==1], \n              ax=ax, \n              palette={'male': '#5db1e4','female': '#fb9ed6'})\nax.set_title('Site-Sex Distribution in Malignant cases')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,9))\nsns.countplot(x='anatom_site_general_challenge', \n              hue='sex', \n              data=df_train.loc[df_train['target']==0], \n              ax=ax, \n              palette={'male': '#5db1e4','female': '#fb9ed6'})\nax.set_title('Site-Sex Distribution in Benign cases')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So in general torso is where people get skin patches. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(6,6))\nsns.countplot(x='diagnosis', \n              data=df_train.loc[df_train['target']==1], \n              ax=ax)\nax.set_title('Diagnosis Distribution in Malignant cases')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(24,6))\nsns.countplot(x='diagnosis', \n              data=df_train.loc[df_train['target']==0], \n              ax=ax)\nax.set_title('Diagnosis Distribution in Benign cases')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It will be interesting to see Nevus, seborrheic keratosis etc and malanoma if we can find some clue","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Benign images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\n# benign images only\nimages = df_train.loc[df_train['target']==0, 'image_name'].values\n\n# Extract 9 random images from it\nrandom_images = [np.random.choice(images+'.jpg') for i in range(9)]\n\n# Location of the image dir\nimg_dir = path +'/jpeg/train'\n\nprint('Display Random Benign Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n    plt.subplot(3, 3, i + 1)\n    img = plt.imread(os.path.join(img_dir, random_images[i]))\n    plt.title(str(df_train.loc[df_train['image_name']==random_images[i].split('.')[0], 'benign_malignant'].item()))\n    plt.imshow(img, cmap='gray')\n    plt.axis('off')\n    \n# Adjust subplot parameters to give specified padding\nplt.tight_layout()   ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Malignant images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\n# malignant images only\nimages = df_train.loc[df_train['target']==1, 'image_name'].values\n\n# Extract 9 random images from it\nrandom_images = [np.random.choice(images+'.jpg') for i in range(9)]\n\n# Location of the image dir\nimg_dir = path +'/jpeg/train'\n\nprint('Display Random Malignant Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n    plt.subplot(3, 3, i + 1)\n    img = plt.imread(os.path.join(img_dir, random_images[i]))\n    plt.title(str(df_train.loc[df_train['image_name']==random_images[i].split('.')[0], 'benign_malignant'].item()))\n    plt.imshow(img, cmap='gray')\n    plt.axis('off')\n    \n# Adjust subplot parameters to give specified padding\nplt.tight_layout()   ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets see types of benign","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\n# benign-unknown images only\nimages = df_train.loc[(df_train['target']==0) & (df_train['diagnosis']=='unknown'), 'image_name'].values\n\n# Extract 9 random images from it\nrandom_images = [np.random.choice(images+'.jpg') for i in range(9)]\n\n# Location of the image dir\nimg_dir = path +'/jpeg/train'\n\nprint('Display Random Bening - unknown diagnosed Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n    plt.subplot(3, 3, i + 1)\n    img = plt.imread(os.path.join(img_dir, random_images[i]))\n    plt.title('benign-' + str(df_train.loc[df_train['image_name']==random_images[i].split('.')[0], 'diagnosis'].item()))\n    plt.imshow(img, cmap='gray')\n    plt.axis('off')\n    \n# Adjust subplot parameters to give specified padding\nplt.tight_layout()   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\n# benign nevus images only\nimages = df_train.loc[(df_train['target']==0) & (df_train['diagnosis']=='nevus'), 'image_name'].values\n\n# Extract 9 random images from it\nrandom_images = [np.random.choice(images+'.jpg') for i in range(9)]\n\n# Location of the image dir\nimg_dir = path +'/jpeg/train'\n\nprint('Display Random  Benign - nevus diagnosed Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n    plt.subplot(3, 3, i + 1)\n    img = plt.imread(os.path.join(img_dir, random_images[i]))\n    plt.title('benign-' + str(df_train.loc[df_train['image_name']==random_images[i].split('.')[0], 'diagnosis'].item()))\n    plt.imshow(img, cmap='gray')\n    plt.axis('off')\n    \n# Adjust subplot parameters to give specified padding\nplt.tight_layout()   ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The video said if the the shape is not symmetric and if the circumference is rugged it might be mealnoma. We might make some feature around this but lets keep this for later","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Modelling","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"IMG_SIZE = 100","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import cv2\nX = []\ny = []\nfor img in tqdm(os.listdir(f'{path}/jpeg/train')):\n    img_array = cv2.imread(os.path.join(f'{path}/jpeg/train', img))\n    img_array = cv2.resize(img_array, (IMG_SIZE, IMG_SIZE))\n    img_array = cv2.cvtColor(img_array, cv2.COLOR_BGR2RGB)\n    X.append(img_array.flatten())\n    y.append(df_train.loc[df_train['image_name'] == img.split('.')[0], 'target'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.DataFrame(data=X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport tensorflow as tf","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tfrecord_location = f'{path}/tfrecords'\nfilename = os.path.join(tfrecord_location)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = tf.data.TFRecordDataset(filename)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"filenames = f'{path}/tfrecords/train00-2071.tfrec'\nraw_dataset = tf.data.TFRecordDataset(filenames)\nraw_dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for raw_record in raw_dataset.take(1):\n    example = tf.train.Example()\n    example.ParseFromString(raw_record.numpy())\n    print(example)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tf.io.parse_single_example(example_proto, feature_description)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_tfrecord(serialized_example):\n    feature_description = { 'image': tf.FixedLenFeature([], tf.string),\n               'label': tf.FixedLenFeature([], tf.int64)\n           }\n    example = tf.io.parse_single_example(serialized_example, feature_description)\n    \n    image = tf.io.parse_tensor(example['image'], out_type = float)\n    image_shape = [example['height'], example['width'], example['depth']]\n    image = tf.reshape(image, image_shape)\n    \n    return image, example['label']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tfrecord_dataset = tf.data.TFRecordDataset(tfrecord_location)\nparsed_dataset = tfrecord_dataset.map(read_tfrecord)\n\nplt.figure(figsize=(10,10))\nfor i, data in enumerate(parsed_dataset.take(9)):\n    img = tf.keras.preprocessing.image.array_to_img(data[0])\n    plt.subplot(3,3,i+1)\n    plt.imshow(img)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import glob\nreader = tf.TFRecordReader()\nfilenames = glob.glob('/tfrecords/train*')\nfilename_queue = tf.train.string_input_producer(\n   filenames)\n_, serialized_example = reader.read(filename_queue)\nfeature_set = { 'image': tf.FixedLenFeature([], tf.string),\n               'label': tf.FixedLenFeature([], tf.int64)\n           }\n           \nfeatures = tf.parse_single_example( serialized_example, features= feature_set )\nlabel = features['label']\n \nwith tf.Session() as sess:\n    print(sess.run([image,label]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode(serialized_example):\n  \"\"\"\n  Parses an image and label from the given `serialized_example`.\n  It is used as a map function for `dataset.map`\n  \"\"\"\n  IMAGE_SIZE = 28\n  IMAGE_PIXELS = IMAGE_SIZE * IMAGE_SIZE\n  \n  # 1. define a parser\n  features = tf.parse_single_example(\n      serialized_example,\n      # Defaults are not specified since both keys are required.\n      features={\n          'image_raw': tf.FixedLenFeature([], tf.string),\n          'label': tf.FixedLenFeature([], tf.int64),\n      })\n\n  # 2. Convert the data\n  image = tf.decode_raw(features['image_raw'], tf.uint8)\n  label = tf.cast(features['label'], tf.int32)\n  # 3. reshape\n  image.set_shape((IMAGE_PIXELS))\n  return image, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = dataset.map(decode)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Inspiration\n\nhttps://www.kaggle.com/parulpandey/melanoma-classification-eda-starter","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}