{"cells":[{"metadata":{},"cell_type":"markdown","source":"### About the Data\nThe dataset was generated by the International Skin Imaging Collaboration (ISIC) and images are from the following sources: Hospital Clínic de Barcelona, Medical University of Vienna, Memorial Sloan Kettering Cancer Center, Melanoma Institute Australia, The University of Queensland, and the University of Athens Medical School.\n\n### Columns\n* image_name - unique identifier, points to filename of related DICOM image\n* patient_id - unique patient identifier\n* sex - the sex of the patient (when unknown, will be blank)\n* age_approx - approximate patient age at time of imaging\n* anatom_site_general_challenge - location of imaged site\n* diagnosis - detailed diagnosis information (train only)\n* benign_malignant - indicator of malignancy of imaged lesion\n* target - binarized version of the target variable","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Let's go Exploring\n![](https://img.freepik.com/free-vector/explorer-with-backpack-background_23-2148159527.jpg?size=626&ext=jpg)","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Importing necessary libraries for visualizations","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from os import listdir\nimport matplotlib.pyplot as plt\n%matplotlib inline  \n# To store resultimg plots/graphs in the notebook document below the respective code cells\n\n!pip install chart_studio\nimport plotly.express as px\nimport chart_studio.plotly as py\nimport plotly.graph_objs as go\nfrom plotly.offline import iplot\nimport cufflinks\n#Required to apply plotly\ncufflinks.go_offline()\ncufflinks.set_config_file(world_readable=True, theme='pearl')\n\nimport seaborn as sns\nsns.set(style='whitegrid')\n\nimport pydicom\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nplt.style.use('fivethirtyeight')\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Color_Palette for beautiful visualizations","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"orange_black = [\n    '#fdc029', '#df861d', '#FF6347', '#aa3d01', '#a30e15', '#800000', '#171820'\n]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Content:","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"print(os.listdir('../input/siim-isic-melanoma-classification/'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Tabular Dataframes","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_PATH = '../input/siim-isic-melanoma-classification/'\n\ntrain_df = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\ntest_df = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')\n\n#Training data\nprint('Number of Training examples', train_df.shape[0])\nprint('Number of Test examples', test_df.shape[0])\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"z = train_df.groupby(['benign_malignant'])['target'].count().to_frame()\nz.style.background_gradient(cmap='Oranges')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analysing Missing values","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#missing values\n\nprint('----train_df-----')\nprint(train_df.info());\n\nprint('\\n')\nprint('----test_df-----')\nprint(test_df.info());","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Clearly we have somemissing values","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Total images in training set ',train_df['image_name'].count())\nprint('Total images in test set ',test_df['image_name'].count())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"There are total {train_df['patient_id'].count()} patient ids, out of which {train_df['patient_id'].value_counts().count()} are unique\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"=> we have multiple images for unique patients","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"columns = train_df.columns.tolist()\ncolumns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualizing Distributions","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Target Distributions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"z = train_df['target'].value_counts().to_frame()\nz.style.background_gradient(cmap='Oranges')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#interactive plots\ntrain_df['target'].value_counts(normalize=True).iplot(kind='bar',\n                                                     yTitle='Percentage',\n                                                     linecolor='black',\n                                                     opacity=0.7,\n                                                     color='orange',\n                                                     bargap=0.8,\n                                                     gridcolor='white',\n                                                     title='[INTERACTIVE] Target value distribution from training set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Gender distribution","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"z = train_df['sex'].value_counts().to_frame()\nz.style.background_gradient(cmap='Oranges')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#interactive plots\ntrain_df['sex'].value_counts(normalize=True).iplot(kind='bar',\n                                                     yTitle='Percentage',\n                                                     linecolor='black',\n                                                     opacity=0.7,\n                                                     color='purple',\n                                                     bargap=0.8,\n                                                     gridcolor='white',\n                                                     title='[INTERACTIVE] Sex column distribution from training set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Count value for male/female for each target category","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"z = train_df.groupby(['target','sex'])['benign_malignant'].count().to_frame().reset_index()\nz.style.background_gradient(cmap='Oranges')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.catplot(x='target', y='benign_malignant',hue='sex', data=z, kind='bar',palette=orange_black);\nplt.xlabel('Benign:0   Malignant:1')\nplt.ylabel('Count')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Location of imaged site on body","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"z = train_df['anatom_site_general_challenge'].value_counts(normalize=True).sort_values(ascending=True).to_frame()\nz.style.background_gradient(cmap='Oranges')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['anatom_site_general_challenge'].value_counts(normalize=True).sort_values().iplot(kind='barh',\n                                                                                          xTitle='Percentage',\n                                                                                          linecolor='black',\n                                                                                          opacity=0.7,\n                                                                                          color='orange',\n                                                                                          theme='pearl',\n                                                                                          bargap=0.2,\n                                                                                          gridcolor='white',\n                                                                                          title='[INTERACTIVE] Distribution of imaged site of intrerest from training set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Location of imaged site of interest w.r.t gender","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"z = train_df.groupby(['sex', 'anatom_site_general_challenge'])['benign_malignant'].count().to_frame().reset_index()\nz.style.background_gradient(cmap='Oranges')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.catplot(x='anatom_site_general_challenge', y='benign_malignant', hue='sex', data=z, kind='bar', palette=orange_black)\nplt.gcf().set_size_inches(12,8)\nplt.xlabel('Location of imaged site')\nplt.xticks(rotation=45, fontsize=15)\nplt.ylabel('# of Melanoma cases')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Age distributions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(15,8))\n\nsns.distplot(train_df.age_approx,ax=axes[0], label='Train', color='#fdc029')\nsns.distplot(test_df.age_approx,ax=axes[0], label='Test', color='#171820')\naxes[0].set_title('Age distribution in train/test sets')\naxes[0].legend()\n\n\nsns.distplot(train_df[train_df.sex=='female'].age_approx,ax=axes[1], label='Female', color='#fdc029')\nsns.distplot(train_df[train_df.sex=='male'].age_approx,ax=axes[1], label='Female', color='#171820')\naxes[1].set_title('Age distribution w.r.t gender')\naxes[1].legend()\n\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Distribution of Diagnosis column in train","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"z = train_df['diagnosis'].value_counts().sort_values().to_frame()\nz.style.background_gradient(cmap='Oranges')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['diagnosis'].value_counts(normalize=True).sort_values().iplot(kind='barh',\n                                                          xTitle='Percentage',\n                                                          linecolor='black',\n                                                          opacity=0.7,\n                                                          color='orange',\n                                                          theme='pearl',\n                                                          bargap=0.2,\n                                                          gridcolor='white',\n                                                          title='[INTERACTIVE] Distribution of Diagnosis column from training set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **Kernel Density Estimation plots** : \nGives the normal/gaussian distribution of data points conditioning on required features.[seaborn-kdeplots](https://www.journaldev.com/40204/seaborn-kdeplot)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(16,8))\n# kdeplot of age_approx for benign target\nsns.kdeplot(train_df.loc[train_df['target'] == 0, 'age_approx'],ax=axes[0], label='Benign',color='g', shade=True)\n\n# kdeplot of age_approx for Malignant target\nsns.kdeplot(train_df.loc[train_df['target'] == 1, 'age_approx'],ax=axes[0], label='Malignant',color='b', shade=True)\n\naxes[0].set_xlabel('Age in years')\naxes[0].set_ylabel('Density')\naxes[0].set_title('Age Distribution [Benign/Malignant]')\n\n# kdeplot of age_approx for male gender\nsns.kdeplot(train_df.loc[train_df['sex'] == 'male', 'age_approx'],ax=axes[1], label='male',color='g', shade=True)\n\n# kdeplot of age_approx for female gender\nsns.kdeplot(train_df.loc[train_df['sex'] == 'female', 'age_approx'],ax=axes[1], label='female',color='b', shade=True)\n\naxes[1].set_xlabel('Gender')\naxes[1].set_ylabel('Density')\naxes[1].set_title('Age Distribution [Male/Female]')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Histogram distribution of RBG colors in random images\nIn a RGB color space, pixel values range from 0 to 255 where 0 stands for black and 255 stands for white. We will look at histogram of each target category","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Location of the image dir\nimg_dir = IMAGE_PATH+'jpeg/train'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Benign category","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Benign\nbenign = train_df[train_df['benign_malignant']=='benign']\n\nf = plt.figure(figsize=(16,8))\nf.add_subplot(1,2,1)\n\nsample_img = benign['image_name'][0]+'.jpg'\nimage = plt.imread(os.path.join(img_dir, sample_img))\nplt.imshow(image, cmap='gray')\nplt.colorbar()\nplt.title('Benign Image')\nprint(f\"Image dimensions {image.shape}\")\nprint(f\"Maximum pixel value {image.max():.2f}; Minimum pixel value {image.min():.2f}\")\nprint(f\"Mean value of the pixels : {image.mean():.2f} ; Standard deviation : {image.std():.2f}\")\n\nf.add_subplot(1,2,2)\n\n_ = plt.hist(image[:,:,0].ravel(), bins = 256, color = 'red', alpha = 0.5)\n_ = plt.hist(image[:,:,1].ravel(), bins = 256, color = 'green', alpha = 0.5)\n_ = plt.hist(image[:,:,2].ravel(), bins = 256, color = 'blue', alpha = 0.5)\n_ = plt.xlabel('Intensity Values')\n_ = plt.ylabel('Count')\n_ = plt.legend(['red_channel','green_channel','blue_channel'])\nplt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Malignant category","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Malignant\nmalignant = train_df[train_df['benign_malignant']=='malignant']\n\nf = plt.figure(figsize=(16,8))\nf.add_subplot(1,2,1)\n\nsample_img = malignant['image_name'][91]+'.jpg'\nimage = plt.imread(os.path.join(img_dir, sample_img))\nplt.imshow(image, cmap='gray')\nplt.colorbar()\nplt.title('Malignant Image')\nprint(f\"Image dimensions {image.shape}\")\nprint(f\"Maximum pixel value {image.max():.2f}; Minimum pixel value {image.min():.2f}\")\nprint(f\"Mean value of the pixels : {image.mean():.2f}; Standard deviation : {image.std():.2f}\")\n\nf.add_subplot(1,2,2)\n\n_ = plt.hist(image[:,:,0].ravel(), bins = 256, color = 'red', alpha = 0.5) \n_ = plt.hist(image[:,:,1].ravel(), bins = 256, color = 'green', alpha = 0.5)\n_ = plt.hist(image[:,:,2].ravel(), bins = 256, color = 'blue', alpha = 0.5)\n_ = plt.xlabel('Intensity Values')\n_ = plt.ylabel('Count')\n_ = plt.legend(['red_channel','green_channel','blue_channel'])\nplt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is my very first exploratory data analysis for a featured competition. I took help extensively from the following two amazying kernels. Do check them out for even better insights.\n\n[Parul Pandey's kernel](https://www.kaggle.com/parulpandey/melanoma-classification-eda-starter)\n\n[Ertuğrul Demir's kernel](https://www.kaggle.com/datafan07/starter-analysis-of-melanoma-metadata-and-images)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# If you like my kernel, do upvote :)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}