\n\n\n
\n\n# 1. Introduction โถ\n\n### 1.1 What is Melanoma? Stats and Facts:\n* [Melanoma is the least common but the most deadly skin cancer, accounting for only about 1% of all cases, but the vast majority of skin cancer death.](https://www.aimatmelanoma.org/about-melanoma/melanoma-stats-facts-and-figures/)\n* Melanoma is the third most common cancer among men and women ages 20-39.\n* In the U.S., melanoma continues to be \n * the fifth most common cancer in men of all age groups\n * the sixth most common cancer in women of all age groups\n* The worldโs highest incidence of melanoma is in Australia and New Zealand (more than twice as high as in North America)\n\n\n### 1.2 What we need to do? Data and Overview:\n> The purpose is to correctly identify the **benign** and **malignant** cases. A *benign* tumor is a tumor that DOES NOT invade its surrounding tissue or spread around the body. A *malignant* tumor is a tumor that MAY invade its surrounding tissue or spread around the body.\n:max_bytes(150000):strip_icc():format(webp)/514240-article-img-malignant-vs-benign-tumor2111891f-54cc-47aa-8967-4cd5411fdb2f-5a2848f122fa3a0037c544be.png)
\n\n> Data: DICOM Files split in Train (33,126 observations) and Test (10,982 observations)\n
\n\n### 1.3 Metrics of Evaluation. Area under the ROC curve:\n* [The ROC curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings.](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)\n\n# 2. Libraries ๐","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Regular Imports\nimport os\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport matplotlib.image as mpimg\nfrom tabulate import tabulate\nimport missingno as msno \nfrom IPython.display import display_html\nfrom PIL import Image\nimport gc\nimport cv2\n\nimport pydicom # for DICOM images\nfrom skimage.transform import resize\n\n# SKLearn\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.preprocessing import OneHotEncoder\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n# Set Color Palettes for the notebook\ncolors_nude = ['#e0798c','#65365a','#da8886','#cfc4c4','#dfd7ca']\nsns.palplot(sns.color_palette(colors_nude))\n\n# Set Style\nsns.set_style(\"whitegrid\")\nsns.despine(left=True, bottom=True)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true,"_kg_hide-output":false},"cell_type":"code","source":"list(os.listdir('../input/siim-isic-melanoma-classification'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. CSV Files - Train๐ + Test๐","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Directory\ndirectory = '../input/siim-isic-melanoma-classification'\n\n# Import the 2 csv s\ntrain_df = pd.read_csv(directory + '/train.csv')\ntest_df = pd.read_csv(directory + '/test.csv')\n\nprint('Train has {:,} rows and Test has {:,} rows.'.format(len(train_df), len(test_df)))\n\n# Change columns names\nnew_names = ['dcm_name', 'ID', 'sex', 'age', 'anatomy', 'diagnosis', 'benign_malignant', 'target']\ntrain_df.columns = new_names\ntest_df.columns = new_names[:5]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df1_styler = train_df.head().style.set_table_attributes(\"style='display:inline'\").set_caption('Head Train Data')\ndf2_styler = test_df.head().style.set_table_attributes(\"style='display:inline'\").set_caption('Head Test Data')\n\ndisplay_html(df1_styler._repr_html_() + df2_styler._repr_html_(), raw=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.1 Missing Values โ\n\nLet's first visualize the missing values.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"f, (ax1, ax2) = plt.subplots(1, 2, figsize = (16, 6))\n\nmsno.matrix(train_df, ax = ax1, color=(207/255, 196/255, 171/255), fontsize=10)\nmsno.matrix(test_df, ax = ax2, color=(218/255, 136/255, 130/255), fontsize=10)\n\nax1.set_title('Train Missing Values Map', fontsize = 16)\nax2.set_title('Test Missing Values Map', fontsize = 16);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Train**:\n1. `sex`: 65 missing values (0.2% of total data)\n2. `age`: 68 missing values (correspond with `sex` missingness)\n3. `anatomy`: 527 missing values (1.59% of total data)\n\n**Test**:\n1. `anatomy`: 351 missing values (3.1% of total data)\n\nLet's take them 1 by 1 and deal with em.\n\n### Train: SEX Variable\n\n> All missing values are *benign* and the majority of the patients have the Melanoma in the Lower Extremity, Upper Extremity and Torso. All values for `diagnosis` are unknown. Therefore, we'll use the most predominant gender that appears in these values to impute the missing values.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"# Data\nnan_sex = train_df[train_df['sex'].isna() == True]\nis_sex = train_df[train_df['sex'].isna() == False]\n\n# Figure\nf, (ax1, ax2) = plt.subplots(1, 2, figsize = (16, 6))\n\na = sns.countplot(nan_sex['anatomy'], ax = ax1, palette=colors_nude)\nb = sns.countplot(is_sex['anatomy'], ax = ax2, palette=colors_nude)\nax1.set_title('NAN Gender: Anatomy', fontsize=16)\nax2.set_title('Rest Gender: Anatomy', fontsize=16)\n\na.set_xticklabels(a.get_xticklabels(), rotation=35, ha=\"right\")\nb.set_xticklabels(b.get_xticklabels(), rotation=35, ha=\"right\")\nsns.despine(left=True, bottom=True);\n\n# Benign/ Malignant check\nprint('Out of 65 NAN values, {} are benign and 0 malignant.'.format(nan_sex['benign_malignant'].value_counts()[0]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check how many are males and how many females\nanatomy = ['lower extremity', 'upper extremity', 'torso']\ntrain_df[(train_df['anatomy'].isin(anatomy)) & (train_df['target'] == 0)]['sex'].value_counts()\n\n# Impute the missing values with male\ntrain_df['sex'].fillna(\"male\", inplace = True) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train: AGE Variable\n> The distributions and values are very similar with the missingness patern in `sex` variable. So, we'll impute in the same manner. The *mean* and *median* of `age` variable has the same value of 50, while the *mode* is at 45. The distribution is normal, so we'll use the MEDIAN to impute.","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"# Data\nnan_age = train_df[train_df['age'].isna() == True]\nis_age = train_df[train_df['age'].isna() == False]\n\n# Figure\nf, (ax1, ax2) = plt.subplots(1, 2, figsize = (16, 6))\n\na = sns.countplot(nan_age['anatomy'], ax = ax1, palette=colors_nude)\nb = sns.countplot(is_age['anatomy'], ax = ax2, palette=colors_nude)\nax1.set_title('NAN age: Anatomy', fontsize=16)\nax2.set_title('Rest age: Anatomy', fontsize=16)\n\na.set_xticklabels(a.get_xticklabels(), rotation=35, ha=\"right\")\nb.set_xticklabels(b.get_xticklabels(), rotation=35, ha=\"right\")\nsns.despine(left=True, bottom=True);\n\n# Benign/ Malignant check\nprint('Out of 68 NAN values, {} are benign and 0 malignant.'.format(nan_age['benign_malignant'].value_counts()[0]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check the mean age\nanatomy = ['lower extremity', 'upper extremity', 'torso']\nmedian = train_df[(train_df['anatomy'].isin(anatomy)) & (train_df['target'] == 0) & (train_df['sex'] == 'male')]['age'].median()\nprint('Median is:', median)\n\n# Impute the missing values with male\ntrain_df['age'].fillna(median, inplace = True) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train: ANATOMY Variable\n> First, we need to keep in mind that between the missing data there are 9 malignant cases, so we should treat the imputation separate for both benign and malignant. In terms of `age` and `gender`, both missing and not missing data seem to behave about the same. However, the most frequent anatomy for both benign and malignant is *torso*, so we'll impute this value.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"anatomy = train_df.copy()\nanatomy['flag'] = np.where(train_df['anatomy'].isna()==True, 'missing', 'not_missing')\n\n# Figure\nf, (ax1, ax2) = plt.subplots(1, 2, figsize = (16, 6))\n\nsns.countplot(anatomy['flag'], hue=anatomy['sex'], ax=ax1, palette=colors_nude)\n\nsns.distplot(anatomy[anatomy['flag'] == 'missing']['age'], \n hist=False, rug=True, label='Missing', ax=ax2, \n color=colors_nude[2], kde_kws=dict(linewidth=4))\nsns.distplot(anatomy[anatomy['flag'] == 'not_missing']['age'], \n hist=False, rug=True, label='Not Missing', ax=ax2, \n color=colors_nude[3], kde_kws=dict(linewidth=4))\n\nax1.set_title('Gender for Anatomy', fontsize=16)\nax2.set_title('Age Distribution for Anatomy', fontsize=16)\nsns.despine(left=True, bottom=True);\n\n# Benign - malignant\nben_mal = anatomy[anatomy['flag'] == 'missing']['benign_malignant'].value_counts()\nprint('From all missing values, {} are benign and {} malignant.'.format(ben_mal[0], ben_mal[1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Impute for anatomy\ntrain_df['anatomy'].fillna('torso', inplace = True) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Test: ANATOMY Variable\n> The majority of the people with missing `anatomy` have 70 yo, so we'll use the anatomy with the biggest frequency for age 70.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"anatomy = test_df.copy()\nanatomy['flag'] = np.where(test_df['anatomy'].isna()==True, 'missing', 'not_missing')\n\n# Figure\nf, (ax1, ax2) = plt.subplots(1, 2, figsize = (16, 6))\n\nsns.countplot(anatomy['flag'], hue=anatomy['sex'], ax=ax1, palette=colors_nude)\n\nsns.distplot(anatomy[anatomy['flag'] == 'missing']['age'],\n hist=False, rug=True, label='Missing', ax=ax2, \n color=colors_nude[2], kde_kws=dict(linewidth=4, bw=0.1))\n\nsns.distplot(anatomy[anatomy['flag'] == 'not_missing']['age'], \n hist=False, rug=True, label='Not Missing', ax=ax2, \n color=colors_nude[3], kde_kws=dict(linewidth=4, bw=0.1))\n\nax1.set_title('Gender for Anatomy', fontsize=16)\nax2.set_title('Age Distribution for Anatomy', fontsize=16)\nsns.despine(left=True, bottom=True);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Select most frequent anatomy for age 70\nvalue = test_df[test_df['age'] == 70]['anatomy'].value_counts().reset_index()['index'][0]\n\n# Impute the value\ntest_df['anatomy'].fillna(value, inplace = True) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Note: Before continuing, let's save the clean files with imputations.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Save the files\ntrain_df.to_csv('train_clean.csv', index=False)\ntest_df.to_csv('test_clean.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.2 EDA - Let's take a look ๐\n\n### Target Variable:\n1. Very HIGH class imbalance. We need to take this in consideration when Modeling.\n2. Age distribution:\n * Benign: follows a normal distribution\n * Malignant: a little skewed to the left, with the peak oriented towards higher age values.","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"# Figure\nf, (ax1, ax2) = plt.subplots(1, 2, figsize = (16, 6))\n\na = sns.countplot(data = train_df, x = 'benign_malignant', palette=colors_nude[2:4],\n ax=ax1)\nb = sns.distplot(a = train_df[train_df['target']==0]['age'], ax=ax2, color=colors_nude[2], \n hist=False, rug=True, kde_kws=dict(linewidth=4), label='Benign')\nc = sns.distplot(a = train_df[train_df['target']==1]['age'], ax=ax2, color=colors_nude[3], \n hist=False, rug=True, kde_kws=dict(linewidth=4), label='Malignant')\n\nfor p in a.patches:\n a.annotate(format(p.get_height(), ','), \n (p.get_x() + p.get_width() / 2., \n p.get_height()), ha = 'center', va = 'center', \n xytext = (0, 4), textcoords = 'offset points')\n \nax1.set_title('Frequency for Target Variable', fontsize=16)\nax2.set_title('Age Distribution the Target types', fontsize=16)\nsns.despine(left=True, bottom=True);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Target and Genders:\n1. There are more males than females in the dataset\n2. However, the percentages are ~ the same","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\na = sns.countplot(data=train_df, x='benign_malignant', hue='sex', palette=colors_nude)\n\nfor p in a.patches:\n a.annotate(format(p.get_height(), ','), \n (p.get_x() + p.get_width() / 2., \n p.get_height()), ha = 'center', va = 'center', \n xytext = (0, 4), textcoords = 'offset points')\n\nplt.title('Gender split by Target Variable', fontsize=16)\nsns.despine(left=True, bottom=True);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Anatomy and Diagnosis:\n> Diagnosis: 'cafe-au-lait macule' and 'atypical melanocytic proliferation' appear only once in the data, so the observations will be deleted.\n\n1. Anatomy: Most of the melanomas are in the *torso* and *lower extremities* of the body\n2. Diagnosis: For most patients, the diagnosis is unknown, but there are ~ 17% that have some kind of diagnosis available.","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"# Delete 'atypical melanocytic proliferation','cafe-au-lait macule'\n# train_df = train_df[~train_df['diagnosis'].isin(['atypical melanocytic proliferation','cafe-au-lait macule'])]\n\n# Figure\nf, (ax1, ax2) = plt.subplots(1, 2, figsize = (16, 6))\n\na = sns.countplot(train_df['anatomy'], ax=ax1, palette = colors_nude)\nb = sns.countplot(train_df['diagnosis'], ax=ax2, palette = colors_nude)\n\na.set_xticklabels(a.get_xticklabels(), rotation=35, ha=\"right\")\nb.set_xticklabels(b.get_xticklabels(), rotation=35, ha=\"right\")\n\nfor p in a.patches:\n a.annotate(format(p.get_height(), ','), \n (p.get_x() + p.get_width() / 2., \n p.get_height()), ha = 'center', va = 'center', \n xytext = (0, 4), textcoords = 'offset points')\n \nfor p in b.patches:\n b.annotate(format(p.get_height(), ','), \n (p.get_x() + p.get_width() / 2., \n p.get_height()), ha = 'center', va = 'center', \n xytext = (0, 4), textcoords = 'offset points')\n \nax1.set_title('Anatomy Frequencies', fontsize=16)\nax2.set_title('Diagnosis Frequencies', fontsize=16)\nsns.despine(left=True, bottom=True);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Anatomy and Target\n> Note: Distributions are about the same shape for both benign and malignant cases.","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\na = sns.countplot(data=train_df, x='benign_malignant', hue='anatomy', palette=colors_nude)\n\nfor p in a.patches:\n a.annotate(format(p.get_height(), ','), \n (p.get_x() + p.get_width() / 2., \n p.get_height()), ha = 'center', va = 'center', \n xytext = (0, 4), textcoords = 'offset points')\n\nplt.title('Anatomy split by Target Variable', fontsize=16)\nsns.despine(left=True, bottom=True);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Diagnosis and Target","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"# Figure\nf, (ax1, ax2) = plt.subplots(1, 2, figsize = (16, 6))\n\na = sns.countplot(train_df[train_df['target']==0]['diagnosis'], ax=ax1, palette = colors_nude)\nb = sns.countplot(train_df[train_df['target']==1]['diagnosis'], ax=ax2, palette = colors_nude)\n\na.set_xticklabels(a.get_xticklabels(), rotation=35, ha=\"right\")\nb.set_xticklabels(b.get_xticklabels(), rotation=35, ha=\"right\")\n\nfor p in a.patches:\n a.annotate(format(p.get_height(), ','), \n (p.get_x() + p.get_width() / 2., \n p.get_height()), ha = 'center', va = 'center', \n xytext = (0, 4), textcoords = 'offset points')\n \nfor p in b.patches:\n b.annotate(format(p.get_height(), ','), \n (p.get_x() + p.get_width() / 2., \n p.get_height()), ha = 'center', va = 'center', \n xytext = (0, 4), textcoords = 'offset points')\n \nax1.set_title('Benign cases: Diagnosis view', fontsize=16)\nax2.set_title('Malignant cases: Diagnosis view', fontsize=16)\nsns.despine(left=True, bottom=True);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Test Dataset Overview\n> Distributions look ~ the same as in Train Data.","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"# Figure\nf, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize = (16, 6))\n\na = sns.countplot(test_df['sex'], palette=colors_nude, ax=ax1)\nb = sns.countplot(test_df['anatomy'], ax=ax2, palette = colors_nude)\nc = sns.distplot(a = test_df['age'], ax=ax3, color=colors_nude[3], \n hist=False, rug=True, kde_kws=dict(linewidth=4))\n\nfor p in a.patches:\n a.annotate(format(p.get_height(), ','), \n (p.get_x() + p.get_width() / 2., \n p.get_height()), ha = 'center', va = 'center', \n xytext = (0, 4), textcoords = 'offset points')\n \nfor p in b.patches:\n b.annotate(format(p.get_height(), ','), \n (p.get_x() + p.get_width() / 2., \n p.get_height()), ha = 'center', va = 'center', \n xytext = (0, 4), textcoords = 'offset points')\n \nb.set_xticklabels(b.get_xticklabels(), rotation=35, ha=\"right\")\n\nax1.set_title('Test: Gender Frequencies', fontsize=16)\nax2.set_title('Test: Anatomy Frequencies', fontsize=16)\nax3.set_title('Test: Age Distribution', fontsize=16)\nsns.despine(left=True, bottom=True);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Patients\n> **Important to notice** that there are patients with multiple images taken, in BOTH Train and Test datasets.","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"# Count the number of images per ID\npatients_count_train = train_df.groupby(by='ID')['dcm_name'].count().reset_index()\npatients_count_test = test_df.groupby(by='ID')['dcm_name'].count().reset_index()\n\n# Figure\nf, (ax1, ax2) = plt.subplots(1, 2, figsize = (16, 6))\n\na = sns.distplot(patients_count_train['dcm_name'], kde=False, bins=50, \n ax=ax1, color=colors_nude[0], hist_kws={'alpha': 1})\nb = sns.distplot(patients_count_test['dcm_name'], kde=False, bins=50, \n ax=ax2, color=colors_nude[1], hist_kws={'alpha': 1})\n \nax1.set_title('Train: Images per Patient Distribution', fontsize=16)\nax2.set_title('Test: Images per Patient Distribution', fontsize=16)\nsns.despine(left=True, bottom=True);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Note: Before continuing, let's save the clean files again.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Save the files\ntrain_df.to_csv('train_clean.csv', index=False)\ntest_df.to_csv('test_clean.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Preprocess .csv files ๐\n\n## 4.1 Add Image Path\nThis will help access the images in the feature.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# === DICOM ===\n# Create the paths\npath_train = directory + '/train/' + train_df['dcm_name'] + '.dcm'\npath_test = directory + '/test/' + test_df['dcm_name'] + '.dcm'\n\n# Append to the original dataframes\ntrain_df['path_dicom'] = path_train\ntest_df['path_dicom'] = path_test\n\n# === JPEG ===\n# Create the paths\npath_train = directory + '/jpeg/train/' + train_df['dcm_name'] + '.jpg'\npath_test = directory + '/jpeg/test/' + test_df['dcm_name'] + '.jpg'\n\n# Append to the original dataframes\ntrain_df['path_jpeg'] = path_train\ntest_df['path_jpeg'] = path_test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2 One Hot Encoding\nTransforming all categorical features un numerical.\n> Note1: `sex`, `anatomy`, `diagnosis` need to be encoded.\n\n> Note2: `benign_malignant` column will be dropped, as the information is already in the `target` column.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# === TRAIN ===\nto_encode = ['sex', 'anatomy', 'diagnosis']\nencoded_all = []\n\nlabel_encoder = LabelEncoder()\n\nfor column in to_encode:\n encoded = label_encoder.fit_transform(train_df[column])\n encoded_all.append(encoded)\n \ntrain_df['sex'] = encoded_all[0]\ntrain_df['anatomy'] = encoded_all[1]\ntrain_df['diagnosis'] = encoded_all[2]\n\nif 'benign_malignant' in train_df.columns : train_df.drop(['benign_malignant'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# === TEST ===\nto_encode = ['sex', 'anatomy']\nencoded_all = []\n\nlabel_encoder = LabelEncoder()\n\nfor column in to_encode:\n encoded = label_encoder.fit_transform(test_df[column])\n encoded_all.append(encoded)\n \ntest_df['sex'] = encoded_all[0]\ntest_df['anatomy'] = encoded_all[1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Save the files before continuing.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Save the files\ntrain_df.to_csv('train_clean.csv', index=False)\ntest_df.to_csv('test_clean.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. The Images ๐ธ\n\nThere are 2 types of images containing the same information:\n1. `.dcm` files: [DICOM files](https://en.wikipedia.org/wiki/DICOM). It's saved in the \"Digital Imaging and Communications in Medicine\" format. It contains an image from a medical scan, such as an ultrasound or MRI + information about the patient.\n2. `.jpeg` files: the DICOM files converted into .jpeg format\n3. `.tfrec` files: [The TFRecord file format is a simple record-oriented binary format for ML training data.](https://docs.databricks.com/applications/deep-learning/data-prep/tfrecords-to-tensorflow.html#:~:text=The%20TFRecord%20file%20format%20is,part%20of%20an%20input%20pipeline.)\n\n## 5.1 Sanity Check\n> Check if images in `.dcm` and `.jpeg` format have the same number of observations as in `train_df` and `test_df`.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Train .dcm number of images:', len(list(os.listdir('../input/siim-isic-melanoma-classification/train'))), '\\n' +\n 'Test .dcm number of images:', len(list(os.listdir('../input/siim-isic-melanoma-classification/test'))), '\\n' +\n 'Train .jpeg number of images:', len(list(os.listdir('../input/siim-isic-melanoma-classification/jpeg/train'))), '\\n' +\n 'Test .jpeg number of images:', len(list(os.listdir('../input/siim-isic-melanoma-classification/jpeg/test'))), '\\n' +\n '-----------------------', '\\n' +\n 'There is the same number of images as in train/ test .csv datasets')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Image shapes?\nAlso, let's look at the size of the images (to not overload the memory, we'll check 100 different images). They are pretty different, so we'll need to deal with this in the augmentations part.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"shapes_train = []\n\nfor k, path in enumerate(train_df['path_jpeg']):\n image = Image.open(path)\n shapes_train.append(image.size)\n \n if k >= 100: break\n \nshapes_train = pd.DataFrame(data = shapes_train, columns = ['H', 'W'], dtype='object')\nshapes_train['Size'] = '[' + shapes_train['H'].astype(str) + ', ' + shapes_train['W'].astype(str) + ']'","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (16, 6))\n\na = sns.countplot(shapes_train['Size'], palette=colors_nude)\n\nfor p in a.patches:\n a.annotate(format(p.get_height(), ','), \n (p.get_x() + p.get_width() / 2., \n p.get_height()), ha = 'center', va = 'center', \n xytext = (0, 4), textcoords = 'offset points')\n \nplt.title('100 Images Shapes', fontsize=16)\nsns.despine(left=True, bottom=True);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.2 DICOM Images\n\n### Malignant vs Benign Images\n\nLet's look at the difference between *malignant* and *benign* melanomas.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_images(data, n = 5, rows=1, cols=5, title='Default'):\n plt.figure(figsize=(16,4))\n\n for k, path in enumerate(data['path_dicom'][:n]):\n image = pydicom.read_file(path)\n image = image.pixel_array\n \n # image = resize(image, (200, 200), anti_aliasing=True)\n\n plt.suptitle(title, fontsize = 16)\n plt.subplot(rows, cols, k+1)\n plt.imshow(image)\n plt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Show Benign Samples\nshow_images(train_df[train_df['target'] == 0], n=10, rows=2, cols=5, title='Benign Sample')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Show Malignant Samples\nshow_images(train_df[train_df['target'] == 1], n=10, rows=2, cols=5, title='Malignant Sample')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6. Class Imbalance โ\n\nThis is a **very** important topic in this classification problem, as the 2 classes we are dealing with are highly imbalanced, with 98% of the data being *benign* and only 2% of the data being *malignant*.\n\n
\n\nThis is also the kind of problem where you **DON'T** want to have False Negatives. It's waaayyy worse to tell a patient they don't have cancer when they actually do, than to tell em they do have it and they actually don't. So, having balanced classes is *crucial*.\n\n### We can do:\n* **Oversampling**: of the minority class, increasing the number of images through aougmentations\n* **Understampling**: of the majority class (we shall see how the process is going)\n\n> 
\n\n### Other things to keep in mind:\n\n#1: Different skin tones. Might need to find something that levels that.
\n#2: Different lightings in the image.
\n#3: Different sizes of the images. We need to resize them.
\n\n\n## 6.1 B&W View ๐ค๐ค","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=2, ncols=6, figsize=(16,6))\nplt.suptitle(\"B&W\", fontsize = 16)\n\nfor i in range(0, 2*6):\n data = pydicom.read_file(train_df['path_dicom'][i])\n image = data.pixel_array\n \n # Transform to B&W\n # The function converts an input image from one color space to another.\n image = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)\n image = cv2.resize(image, (200,200))\n \n x = i // 6\n y = i % 6\n axes[x, y].imshow(image, cmap=plt.cm.bone) \n axes[x, y].axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.2 Ben Graham: greyscale + Gaussian Blur ๐ค\n> **Note: this idea is taken from [SIIM: EDA, Augmentations + Model (SeResNet + UNet)](https://www.kaggle.com/nxrprime/siim-eda-augmentations-model-seresnet-unet) notebook**\n\n`cv2.GaussiaBlur()`: The function convolves the source image with the specified Gaussian kernel.\n\n### #1. Without Gaussian Blur","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=2, ncols=6, figsize=(16,6))\nplt.suptitle(\"Without Gaussian Blur\", fontsize = 16)\n\nfor i in range(0, 2*6):\n data = pydicom.read_file(train_df['path_dicom'][i])\n image = data.pixel_array\n \n # Transform to B&W\n # The function converts an input image from one color space to another.\n image = cv2.cvtColor(image, cv2.COLOR_RGB2HSV)\n image = cv2.resize(image, (200,200))\n \n x = i // 6\n y = i % 6\n axes[x, y].imshow(image, cmap=plt.cm.bone) \n axes[x, y].axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### #2. With Gaussian Blur","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=2, ncols=6, figsize=(16,6))\nplt.suptitle(\"With Gaussian Blur\", fontsize = 16)\n\nfor i in range(0, 2*6):\n data = pydicom.read_file(train_df['path_dicom'][i])\n image = data.pixel_array\n \n # Transform to B&W\n # The function converts an input image from one color space to another.\n image = cv2.cvtColor(image, cv2.COLOR_RGB2HSV)\n image = cv2.resize(image, (200,200))\n image=cv2.addWeighted(image, 4, cv2.GaussianBlur(image, (0,0) ,256/10), -4, 128)\n \n x = i // 6\n y = i % 6\n axes[x, y].imshow(image, cmap=plt.cm.bone) \n axes[x, y].axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.3 Hue, Saturation, Brightness โ","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=2, ncols=6, figsize=(16,6))\nplt.suptitle(\"Hue, Saturation, Brightness\", fontsize = 16)\n\nfor i in range(0, 2*6):\n data = pydicom.read_file(train_df['path_dicom'][i])\n image = data.pixel_array\n \n # Transform to B&W\n # The function converts an input image from one color space to another.\n image = cv2.cvtColor(image, cv2.COLOR_RGB2HLS)\n image = cv2.resize(image, (200,200))\n \n x = i // 6\n y = i % 6\n axes[x, y].imshow(image, cmap=plt.cm.bone) \n axes[x, y].axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.4 LUV Color Space ๐จ","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=2, ncols=6, figsize=(16,6))\nplt.suptitle(\"LUV Color Space\", fontsize = 16)\n\nfor i in range(0, 2*6):\n data = pydicom.read_file(train_df['path_dicom'][i])\n image = data.pixel_array\n \n # Transform to B&W\n # The function converts an input image from one color space to another.\n image = cv2.cvtColor(image, cv2.COLOR_RGB2LUV)\n image = cv2.resize(image, (200,200))\n \n x = i // 6\n y = i % 6\n axes[x, y].imshow(image, cmap=plt.cm.bone) \n axes[x, y].axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.5 Torchvision.transforms ๐ฉน\n\nIt's a library that goes hand in hand with `PyTorch` and it's easily used to augment data. Let's demonstrate.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Necessary Imports\nimport torch\nfrom torch.utils.data import DataLoader, Dataset\nimport torchvision.transforms as transforms\nimport torchvision","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Select a small sample of the .jpeg image paths\nimage_list = train_df.sample(12)['path_jpeg']\nimage_list = image_list.reset_index()['path_jpeg']\n\n# Show the sample\nplt.figure(figsize=(16,6))\nplt.suptitle(\"Original View\", fontsize = 16)\n \nfor k, path in enumerate(image_list):\n image = mpimg.imread(path)\n \n plt.subplot(2, 6, k+1)\n plt.imshow(image)\n plt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create PyTorch Dataset Object\nclass DatasetExample(Dataset):\n def __init__(self, image_list, transforms=None):\n self.image_list = image_list\n self.transforms = transforms\n \n # To get item's length\n def __len__(self):\n return (len(self.image_list))\n \n # For indexing\n def __getitem__(self, i):\n # Read in image\n image = plt.imread(self.image_list[i])\n image = Image.fromarray(image).convert('RGB') \n image = np.asarray(image).astype(np.uint8)\n if self.transforms is not None:\n image = self.transforms(image)\n \n return torch.tensor(image, dtype=torch.float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Predefined Show Images Function\ndef show_transform(image, title=\"Default\"):\n plt.figure(figsize=(16,6))\n plt.suptitle(title, fontsize = 16)\n \n # Unnormalize\n image = image / 2 + 0.5 \n npimg = image.numpy()\n npimg = np.clip(npimg, 0., 1.)\n plt.imshow(np.transpose(npimg, (1, 2, 0)))\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### #1. Crop โ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Transform\ntransform = transforms.Compose([\n transforms.ToPILImage(),\n transforms.Resize((300, 300)),\n transforms.CenterCrop((100, 100)),\n transforms.ToTensor(),\n transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),\n ])\n\n# Create the dataset\npytorch_dataset = DatasetExample(image_list=image_list, transforms=transform)\npytorch_dataloader = DataLoader(dataset=pytorch_dataset, batch_size=12, shuffle=True)\n\n# Select the data\nimages = next(iter(pytorch_dataloader))\n \n# show images\nshow_transform(torchvision.utils.make_grid(images, nrow=6), title=\"Crop\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### #2. ColorJitter ๐ซ\nRandomly change the brightness, contrast and saturation of an image.","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"# Transform\ntransform = transforms.Compose([\n transforms.ToPILImage(),\n transforms.Resize((300, 300)),\n transforms.ColorJitter(brightness=0.7, contrast=0.7, saturation=0.7, hue=0.5),\n transforms.ToTensor(),\n transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),\n ])\n\n# Create the dataset\npytorch_dataset = DatasetExample(image_list=image_list, transforms=transform)\npytorch_dataloader = DataLoader(dataset=pytorch_dataset, batch_size=12, shuffle=True)\n\n# Select the data\nimages = next(iter(pytorch_dataloader))\n \n# show images\nshow_transform(torchvision.utils.make_grid(images, nrow=6), title=\"Color Jitter\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### #3. RandomGreyscale ๐\nRandomly convert image to grayscale with a probability of p (default 0.1).","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"# Transform\ntransform = transforms.Compose([\n transforms.ToPILImage(),\n transforms.Resize((300, 300)),\n transforms.RandomGrayscale(p=0.7),\n transforms.ToTensor(),\n transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),\n ])\n\n# Create the dataset\npytorch_dataset = DatasetExample(image_list=image_list, transforms=transform)\npytorch_dataloader = DataLoader(dataset=pytorch_dataset, batch_size=12, shuffle=True)\n\n# Select the data\nimages = next(iter(pytorch_dataloader))\n \n# show images\nshow_transform(torchvision.utils.make_grid(images, nrow=6), title=\"Random Greyscale\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### #4. RandomVerticalFlip ๐๐\nVertically flip the given PIL Image randomly with a given probability.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Transform\ntransform = transforms.Compose([\n transforms.ToPILImage(),\n transforms.Resize((300, 300)),\n transforms.RandomVerticalFlip(p=0.7),\n transforms.ToTensor(),\n transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),\n ])\n\n# Create the dataset\npytorch_dataset = DatasetExample(image_list=image_list, transforms=transform)\npytorch_dataloader = DataLoader(dataset=pytorch_dataset, batch_size=12, shuffle=True)\n\n# Select the data\nimages = next(iter(pytorch_dataloader))\n \n# show images\nshow_transform(torchvision.utils.make_grid(images, nrow=6), title=\"Random Vertical Flip\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7. Hair Removal โ\n\nAs you may have noticed, all images are for light skin colors, so no preprocessing in this area is required. However, *hair* removal might be a good augmentation that will help the model perform better.\n> [Body hair removal thread here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/165582)\n\n> [Notebook here](https://www.kaggle.com/vatsalparsaniya/melanoma-hair-remove)\n\n*Note: White hair seems to not be able to be removed**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def hair_remove(image):\n # convert image to grayScale\n grayScale = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)\n\n # kernel for morphologyEx\n kernel = cv2.getStructuringElement(1,(17,17))\n\n # apply MORPH_BLACKHAT to grayScale image\n blackhat = cv2.morphologyEx(grayScale, cv2.MORPH_BLACKHAT, kernel)\n\n # apply thresholding to blackhat\n _,threshold = cv2.threshold(blackhat,10,255,cv2.THRESH_BINARY)\n\n # inpaint with original image and threshold image\n final_image = cv2.inpaint(image,threshold,1,cv2.INPAINT_TELEA)\n\n return final_image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Select a small sample of the .jpeg image paths\n# We select some hairy photos on purpose\nhairy_photos = train_df[train_df[\"sex\"] == 1].reset_index().iloc[[12, 14, 17, 22, 33, 34]]\nimage_list = hairy_photos['path_jpeg']\nimage_list = image_list.reset_index()['path_jpeg']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Show the Augmented Images\nplt.figure(figsize=(16,3))\nplt.suptitle(\"Original Hairy Images\", fontsize = 16)\n \nfor k, path in enumerate(image_list):\n image = mpimg.imread(path)\n image = cv2.resize(image,(300, 300))\n \n plt.subplot(1, 6, k+1)\n plt.imshow(image)\n plt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Show the sample\nplt.figure(figsize=(16,3))\nplt.suptitle(\"Non Hairy Images\", fontsize = 16)\n \nfor k, path in enumerate(image_list):\n image = mpimg.imread(path)\n image = cv2.resize(image,(300, 300))\n image = hair_remove(image)\n \n plt.subplot(1, 6, k+1)\n plt.imshow(image)\n plt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# To Be Continued\n> [**Next step: Models ๐**](https://www.kaggle.com/andradaolteanu/melanoma-competiton-augment-effnetb2-lb-0-91)\n\n \nIf you found this helpful, upvote!
\nCheers!
\n\n\n# References:\n* [About Melanoma](https://www.aimatmelanoma.org/about-melanoma/melanoma-stats-facts-and-figures/)\n* [ROC Curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)\n* [DICOM Wiki](https://en.wikipedia.org/wiki/DICOM)\n* [TF Records Tensorflow](https://docs.databricks.com/applications/deep-learning/data-prep/tfrecords-to-tensorflow.html#:~:text=The%20TFRecord%20file%20format%20is,part%20of%20an%20input%20pipeline.)","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}
#1: Different skin tones. Might need to find something that levels that.
\n#2: Different lightings in the image.
\n#3: Different sizes of the images. We need to resize them.
\nIf you found this helpful, upvote!
\nCheers!
\n