{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Melanoma Image Classification"},{"metadata":{},"cell_type":"markdown","source":"### Objective: \nThe purpose of this project is to identify Melanoma cases in images of skin lesions. In particular, we need to create a model that predicts the probability whether the lesion is malignant or benign."},{"metadata":{},"cell_type":"markdown","source":"### Data Overview:\n**The dataset consists of images in :**\n- DIOCOM format\n- JPEG format in JPEG directory\n- TFRecord format in tfrecords directory\n- A metadata comprising of train, test and submission file in CSV format."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true,"collapsed":true},"cell_type":"code","source":"# Import libraries \nimport numpy as np \nimport pandas as pd\nimport missingno as msno\nfrom scipy import stats\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Suppress warnings \nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load the data \ntrain = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\ntest = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')\nsample_submission = pd.read_csv('../input/siim-isic-melanoma-classification/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data cleaning"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Rename the columns \nnew_names = ['image_name', 'patient_ID', 'sex', 'age', 'anatomy', 'diagnosis', 'benign_malignant', 'target']\ntrain.columns = new_names\ntest.columns = new_names[:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Print a concise summary of the DataFrame\nprint('Train:', train.info(),'\\n')\nprint('Test:', test.info())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Print the dimensionality of the DataFrame\nprint('Train:', train.shape)\nprint('Test :',test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Print the columns' names \nprint('Train:', train.columns,'\\n')\nprint('Test:', test.columns)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Observe the index values\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# The data type of each column\nprint('Train:', train.dtypes,'\\n')\nprint('Test:', test.dtypes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# The unique IDs\nprint(f\"The total patient IDs are {train['patient_ID'].count()}, from those the unique IDs are {train['patient_ID'].value_counts().shape[0]} \")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The number of unique patients is less than the total number of patients. Which means, several patients have multiple records."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Total number of images in the dataset(train+test)\nprint(\"Total images in Train set: \",train['image_name'].count())\nprint(\"Total images in Test set: \",test['image_name'].count())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Missing Data:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Train:', train.isna().sum(),'\\n')\nprint('Test:', test.isna().sum())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Missing data visualization\n\nf, (ax1, ax2) = plt.subplots(1, 2, figsize = (16, 6))\n\nmsno.matrix(train, ax = ax1, fontsize=10)\nmsno.matrix(test, ax = ax2, fontsize=10)\n\nax1.set_title('Train Missing Values', fontsize = 16)\nax2.set_title('Test Missing Values', fontsize = 16);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Train**:\n- `sex`: 65 missing values (0.2% of the total data)\n- `age`: 68 missing values (correspond with sex missingness)\n- `anatomy`: 527 missing values (1.59% of the total data)\n\n**Test**:\n- `anatomy`: 351 missing values (3.1% of the total data)"},{"metadata":{},"cell_type":"markdown","source":"## Dealing with the missing data"},{"metadata":{},"cell_type":"markdown","source":"### Train: SEX Variable"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['sex'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['sex'].mode()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Impute the missing values with the mode \ntrain['sex'].fillna(train['sex'].mode()[0], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train: AGE Variable"},{"metadata":{"trusted":true},"cell_type":"code","source":"age_median = int(train['age'].median())\nprint('Median:', age_median)\nage_mean = int(train['age'].mean())\nprint('Mean:', age_mean)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The mean and median of `age` variable has the same value of 50, while the mode is 45. The distribution is normal, so we'll use the MEDIAN to impute. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Impute the missing values with the median\ntrain['age'] = train['age'].fillna(age_median)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train & Test: ANATOMY Variable"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fill the missing data in anatomy with 'unknown'\ntrain['anatomy'].fillna('unknown', inplace=True)\ntest['anatomy'].fillna('unknown', inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Exploration"},{"metadata":{"trusted":true},"cell_type":"code","source":"# The sex in each dataset\nfig, ax = plt.subplots(1,2,figsize=(15,5));\n#train\nsns.countplot(train['sex'], ax=ax[0]);\nax[0].set_title('sex count of Train');\n#test\nsns.countplot(test['sex'], ax=ax[1]);\nax[1].set_title('sex count of Test');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Gender vs Target\ntarget_sex = train.groupby(['target','sex'])['benign_malignant'].count().to_frame().reset_index()\ntarget_sex.style.background_gradient() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.catplot(x='target',y='benign_malignant', hue='sex',data=target_sex,kind='bar')\nplt.ylabel('Count')\nplt.xlabel('benign:0 vs malignant:1');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Age distribution\nfig, ax = plt.subplots(1,2,figsize=(17,6))\n#train\nsns.countplot(train['age'], ax=ax[0]);\nlabel = ax[0].get_xticklabels();\nax[0].set_xticklabels(label, rotation=50);\nax[0].set_title('Train age distribution');\n#test\nsns.countplot(test['age'], ax=ax[1]);\nlabel = ax[1].get_xticklabels();\nax[1].set_xticklabels(label, rotation=50);\nax[1].set_title('Test age distribution');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Count The anatomy location imaged site for each dataset\nfig, ax = plt.subplots(1,2, figsize=(17,6));\n#train\nsns.countplot(train['anatomy'].sort_values(), ax=ax[0]);\nlabel = ax[0].get_xticklabels();\nax[0].set_xticklabels(label, rotation=50);\nax[0].set_title('Train imaged site');\n#test\nsns.countplot(test['anatomy'].sort_values(), ax=ax[1]);\nlabel = ax[1].get_xticklabels();\nax[1].set_xticklabels(label, rotation=50);\nax[1].set_title('Test imaged site');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# sex vs. anatomy\nsex_anatomy = train.groupby(['sex','anatomy'])['benign_malignant'].count().to_frame().reset_index();\nsns.catplot(x='anatomy',y='benign_malignant', hue='sex',data=sex_anatomy,kind='bar');\nplt.xlabel('Location of imaged site');\nplt.xticks(rotation=90,fontsize='10');\nplt.ylabel('Count of melanoma cases');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Count benign and malignant\nfig, ax = plt.subplots(figsize=(10,6))\nsns.countplot(train['benign_malignant']);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Images Visualization"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualizing a random selection of images\n\nimages = train['image_name'].values\n\n# Extract 9 random images from it\nrandom_images = [np.random.choice(images+'.jpg') for i in range(9)]\n\n# Location of the image dirictory \nIMAGE_PATH = \"../input/siim-isic-melanoma-classification/\"\nimg_dir = IMAGE_PATH+'/jpeg/train'\n\nprint('Display Random Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n    plt.subplot(3, 3, i + 1)\n    img = plt.imread(os.path.join(img_dir, random_images[i]))\n    plt.imshow(img, cmap='gray')\n    plt.axis('off')\n    \n# Adjust subplot parameters to give specified padding\nplt.tight_layout()   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualizing Images with benign lesions\n\nbenign = train[train['benign_malignant']=='benign']\nmalignant = train[train['benign_malignant']=='malignant']\n\nimages = benign['image_name'].values\nrandom_images = [np.random.choice(images+'.jpg') for i in range(9)]\nimg_dir = IMAGE_PATH+'/jpeg/train'\n\nprint('Display benign Images')\n\nplt.figure(figsize=(10,8))\nfor i in range(9):\n    plt.subplot(3, 3, i + 1)\n    img = plt.imread(os.path.join(img_dir, random_images[i]))\n    plt.imshow(img, cmap='gray')\n    plt.axis('off')\n    \nplt.tight_layout() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualizing Images with Malignant lesions\n\nimages = malignant['image_name'].values\nrandom_images = [np.random.choice(images+'.jpg') for i in range(9)]\nimg_dir = IMAGE_PATH+'/jpeg/train'\n\nprint('Display malignant Images')\n\nplt.figure(figsize=(10,8))\nfor i in range(9):\n    plt.subplot(3, 3, i + 1)\n    img = plt.imread(os.path.join(img_dir, random_images[i]))\n    plt.imshow(img, cmap='gray')\n    plt.axis('off')\n    \nplt.tight_layout()  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}