{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Description:\n\nSkin cancer is the most prevalent type of cancer. Melanoma, specifically, is responsible for 75% of skin cancer deaths, despite being the least common skin cancer. The American Cancer Society estimates over 100,000 new melanoma cases will be diagnosed in 2020. It's also expected that almost 7,000 people will die from the disease. As with other cancers, early and accurate detection—potentially aided by data science—can make treatment more effective.\n\nCurrently, dermatologists evaluate every one of a patient's moles to identify outlier lesions or “ugly ducklings” that are most likely to be melanoma. Existing AI approaches have not adequately considered this clinical frame of reference. Dermatologists could enhance their diagnostic accuracy if detection algorithms take into account “contextual” images within the same patient to determine which images represent a melanoma. If successful, classifiers would be more accurate and could better support dermatological clinic work.\n\nAs the leading healthcare organization for informatics in medical imaging, the Society for Imaging Informatics in Medicine (SIIM)'s mission is to advance medical imaging informatics through education, research, and innovation in a multi-disciplinary community. SIIM is joined by the International Skin Imaging Collaboration (ISIC), an international effort to improve melanoma diagnosis. The ISIC Archive contains the largest publicly available collection of quality-controlled dermoscopic images of skin lesions.\n\nIn this competition, you’ll identify melanoma in images of skin lesions. In particular, you’ll use images within the same patient and determine which are likely to represent a melanoma. Using patient-level contextual information may help the development of image analysis tools, which could better support clinical dermatologists.\n\nMelanoma is a deadly disease, but if caught early, most melanomas can be cured with minor surgery. Image analysis tools that automate the diagnosis of melanoma will improve dermatologists' diagnostic accuracy. Better detection of melanoma has the opportunity to positively impact millions of people.\n\n## Evaluation\nSubmissions are evaluated on area under the ROC curve between the predicted probability and the observed target.\n\nAn ROC curve (receiver operating characteristic curve) is a graph showing the performance of a classification model at all classification thresholds. This curve plots two parameters:\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Data:\n\n## What should I expect the data format to be?\nThe images are provided in DICOM format. This can be accessed using commonly-available libraries like `pydicom`, and contains both image and metadata. It is a commonly used medical imaging data format.\n\nImages are also provided in `JPEG` and `TFRecord` format (in the jpeg and tfrecords directories, respectively). Images in TFRecord format have been resized to a uniform 1024x1024.\n\nMetadata is also provided outside of the DICOM format, in CSV files. See the `Columns` section for a description.\n\n## What am I predicting?\nYou are predicting a binary target for each image. Your model should predict the probability (floating point) between 0.0 and 1.0 that the lesion in the image is malignant (the target). In the training data, train.csv, the value 0 denotes benign, and 1 indicates malignant.\n\n## Files\n1. train.csv - the training set\n2. test.csv - the test set\n3. sample_submission.csv - a sample submission file in the correct format\n\n## Columns\n1. image_name - unique identifier, points to filename of related DICOM image\n2. patient_id - unique patient identifier\n3. sex - the sex of the patient (when unknown, will be blank)\n4. age_approx - approximate patient age at time of imaging\n5. anatom_site_general_challenge - location of imaged site\n6. diagnosis - detailed diagnosis information (train only)\n7. benign_malignant - indicator of malignancy of imaged lesion\n8. target - binarized version of the target variable\n\nDownload dataset from [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/data)\n\nor use kaggle API and run following command \n\n kaggle competitions download -c siim-isic-melanoma-classification\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## What is Melanoma?\n\nMelanoma is a type of skin cancer that develops when melanocytes (the cells that give the skin its tan or brown color) start to grow out of control. Cancer starts when cells in the body begin to grow out of control. Cells in nearly any part of the body can become cancer, and can then spread to other areas of the body. \n\nBut melanoma is more dangerous because it’s much more likely to spread to other parts of the body if not caught and treated early.\n\nThe stage of a cancer at diagnosis will indicate how far it has already spread and what kind of treatment will be suitable.\n\nThis method of assigning a stage to melanoma describes the cancer in five stages, from 0 to 4:\n\n**Stage 0:** The cancer is only present in the outermost layer of skin. Doctors refer to this stage as “melanoma in situ.”\n\n**Stage 1:** The cancer is up to 2 millimeters (mm) thick. It has not yet spread to lymph nodes or other sites, and it may or may not be ulcerated.\n\n**Stage 2:** The cancer is at least 1 mm thick but may be thicker than 4 mm. It may or may not be ulcerated, and it has not yet spread to lymph nodes or other sites.\n\n**Stage 3:** The cancer has spread to one or more lymph nodes or nearby lymphatic channels but not distant sites. The original cancer may no longer be visible. If it is visible, it may be thicker than 4 mm and also ulcerated.\n\n**Stage 4:** The cancer has spread to distant lymph nodes or organs, such as the brain, lungs, or liver.\n\n","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# importing libraries\nimport os\nimport pandas as pd\nimport numpy as np\nfrom sklearn.preprocessing import LabelEncoder\nimport missingno as msno\nfrom PIL import Image\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport seaborn as sns\nsns.set(style=\"whitegrid\")\n\n#bokeh\nfrom bokeh.models import ColumnDataSource, HoverTool, Panel, FactorRange\nfrom bokeh.plotting import figure\nfrom bokeh.io import output_notebook, show, output_file\nfrom bokeh.palettes import Spectral6\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Setup Directory and Files path","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# set up directory and files path\n\nbase_dir = \"../input/siim-isic-melanoma-classification/\"\ntrain_csv = os.path.join(base_dir + \"train.csv\")\ntest_csv = os.path.join(base_dir + \"test.csv\")\njpeg_train_images = os.path.join(base_dir + \"jpeg/train\")\njpeg_test_images = os.path.join(base_dir + \"jpeg/test\")\n\ntrain_df = pd.read_csv(train_csv)\ntest_df = pd.read_csv(test_csv)\n\ntrain_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We are predicting a binary target for each image. Model should predict the probability (floating point) between 0.0 and 1.0 that the lesion in the image is malignant (the target). So, let's check how many images ara benign and how many are malignant in training dataframe.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[\"benign_malignant\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have more examples from benign than malignant. So, we need to perform oversampling or undersampling","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Benign:** Benign tumors are normal cells that divide and grow too much, but do not interfere with the function of normal cells around them.\n\n**Malignant:** Malignant tumors are overgrowths of abnormal cells (cancer) that divide without control and order.\nThey do not stop growing, even when they come into contact with nearby cells.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"benign = train_df[train_df['benign_malignant']=='benign']\nmalignant = train_df[train_df['benign_malignant']=='malignant']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extract 9 random images from benign lesions\nrandom_images = [np.random.choice((benign['image_name'].values)+'.jpg') for i in range(9)]\n\nprint('Display benign Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n plt.subplot(3, 3, i + 1)\n img = plt.imread(os.path.join(jpeg_train_images, random_images[i]))\n plt.imshow(img, cmap='gray')\n plt.axis('off')\n \n# Adjust subplot parameters to give specified padding\nplt.tight_layout() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extract 9 random images from malignant lesions\nrandom_images = [np.random.choice((malignant['image_name'].values)+'.jpg') for i in range(9)]\n\nprint('Display malignant Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n plt.subplot(3, 3, i + 1)\n img = plt.imread(os.path.join(jpeg_train_images, random_images[i]))\n plt.imshow(img, cmap='gray')\n plt.axis('off')\n \n# Adjust subplot parameters to give specified padding\nplt.tight_layout() ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Exploration","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# check for missing values\nprint(train_df.isnull().any())\n\nmsno.matrix(train_df, color=(207/255, 196/255, 171/255), fontsize=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are missing values in `sex`, `age_approx`, and `anatom_site_general_challenge` column. Let's check how many missing values we have","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Number of missing values in sex column\nprint(\"Number of missing values in sex column is {}\".format(train_df.shape[0] - train_df['sex'].count()))\nprint(\"--------------------------------------------------\")\n# Number of missing values in age_approx column\nprint(\"Number of missing values in age_approx column is {}\".format(train_df.shape[0] - train_df['age_approx'].count()))\nprint(\"--------------------------------------------------\")\n# Number of missing values in anatom_site_general_challenge column\nprint(\"Number of missing values in anatom_site_general_challenge column is {}\".format(train_df.shape[0] - train_df['anatom_site_general_challenge'].count()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(test_df.isnull().any())\n\nmsno.matrix(train_df, color=(207/255, 196/255, 171/255), fontsize=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are missing values in `anatom_site_general_challenge` column in test dataset. Let's check how many missing values we have","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Number of missing values in anatom_site_general_challenge column\nprint(\"Number of missing values in anatom_site_general_challenge column is {}\".format(test_df.shape[0] - test_df['anatom_site_general_challenge'].count()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Total number of training and testing images\nprint(\"Total images in Train set:\", train_df[\"image_name\"].count())\nprint(\"Total images in Test set:\", test_df[\"image_name\"].count())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, we have 75-25 distribution for train-test images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# unique number of patients\nprint(\"Total patients ids are {}\".format(train_df[\"patient_id\"].count()))\nprint(\"Unique patients ids are {}\".format(len(train_df[\"patient_id\"].unique())))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Total number of patient ids are much larger than unique patient that means we have multiple records of same patient. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# exploring the target column\ntrain_df[\"target\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This function will plot different type of histogram with Bokeh. It takes dataframe, column for which we want \n# histogram, color palate, bins for axes and title and return histogram\n\n# For more information on how histograms work follow this blog\n# https://towardsdatascience.com/interactive-histograms-with-bokeh-202b522265f3\n\ndef hist_hover(dataframe, column, colors=[\"#94c8d8\", \"#ea5e51\"], bins=30, title=''):\n hist, edges = np.histogram(dataframe[column], bins = bins)\n \n hist_df = pd.DataFrame({column: hist,\n \"left\": edges[:-1],\n \"right\": edges[1:]})\n hist_df[\"interval\"] = [\"%d to %d\" % (left, right) for left,\n right in zip(hist_df[\"left\"], hist_df[\"right\"])]\n \n src = ColumnDataSource(hist_df)\n plot = figure(plot_height = 400, plot_width = 600,\n title = title,\n x_axis_label = column,\n y_axis_label = \"Count\") \n plot.quad(bottom = 0, top = column,left = \"left\",\n right = \"right\", source = src, fill_color = colors[0],\n line_color = \"#35838d\", fill_alpha = 0.7,\n hover_fill_alpha = 0.7, hover_fill_color = colors[1])\n \n hover = HoverTool(tooltips = [('Interval', '@interval'), ('Count', str(\"@\" + column))])\n plot.add_tools(hover)\n output_notebook()\n show(plot)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# histogram of Target column in training set\nhist_hover(train_df, 'target', bins=3, title='Distribution of the Target column in the training set')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Gender wise Distribution of target in traing set\n\nSex = [\"Female\", \"Male\"]\nTarget = ['0', '1']\n\ng = train_df.groupby([\"target\", \"sex\"]).size()\nmale = list(g[0].values)\nfemale = list(g[1].values)\n\ndata = {'Sex':Sex,\n 'Male':male,\n 'Female':female}\n\nx = [(sex, target) for sex in Sex for target in Target]\ncounts = sum(zip(data['Male'], data['Female']), ())\n\nsource = ColumnDataSource(data=dict(x=x, counts=counts, color=Spectral6))\n\np = figure(x_range=FactorRange(*x), plot_height=400, plot_width=800, title=\"Location of Image site with respect of sex\",\n tools=\"hover, pan, box_zoom, wheel_zoom, reset, save\", tooltips= (\"@x: @counts\"))\n\np.vbar(x='x', top='counts', width=0.9, color='color', source=source)\n\np.xgrid.grid_line_color = None\np.legend.orientation = \"horizontal\"\np.legend.location = \"top_center\"\n\nshow(p)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, we have more Male patients than Female in both target category.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# location of image anatom site\ntrain_df[\"anatom_site_general_challenge\"].value_counts(sort=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Distribution of anatom site general challenge column in training set\n\nCategories = [\"torso\", \"lower extremity\", \"upper extremity\", \"head/neck\", \"palms/soles\", \"oral/genital\"]\ncounts = list(train_df[\"anatom_site_general_challenge\"].value_counts(sort=True))\n\nsource = ColumnDataSource(data=dict(Categories=Categories, counts=counts, color=Spectral6))\n\np = figure(x_range=Categories, y_range=(0,22000), plot_height=300, title=\"Distribution of the anatom_site_general_challenge in the training set\",\n tools=\"hover, pan, box_zoom, wheel_zoom, reset, save\", tooltips= (\"@Categories: @counts\"))\n\np.vbar(x='Categories', top='counts', width=0.9, color='color', legend_field=\"Categories\", source=source)\n\np.xgrid.grid_line_color = None\np.legend.orientation = \"horizontal\"\np.legend.location = \"top_center\"\n\nshow(p)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have maximum examples from torso and minimum from oral/genital but let's see distribution with respect to gender","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Gender wise distribution of anatom site column in training set\nprint(train_df.groupby([\"sex\", \"anatom_site_general_challenge\"]).size())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Gender wise distribution of anatom site column in training set\nCategories = [\"head/neck\", \"lower extremity\", \"oral/genital\", \"palms/soles\", \"torso\", \"upper extremity\"]\nSex = [\"Male\", \"Female\"]\n\ng = train_df.groupby([\"sex\", \"anatom_site_general_challenge\"]).size()\nmale = list(g.male.values)\nfemale = list(g.female.values)\n\ndata = {'Categories':Categories,\n 'Male':male,\n 'Female':female}\n\nx = [(categories, sex) for categories in Categories for sex in Sex]\ncounts = sum(zip(data['Male'], data['Female']), ())\n\nsource = ColumnDataSource(data=dict(x=x, counts=counts, color=Spectral6))\n\np = figure(x_range=FactorRange(*x), plot_height=400, plot_width=800, title=\"Location of Image site with respect of sex\",\n tools=\"hover, pan, box_zoom, wheel_zoom, reset, save\", tooltips= (\"@x: @counts\"))\n\np.vbar(x='x', top='counts', width=0.9, color='color', source=source)\n\np.xgrid.grid_line_color = None\np.legend.orientation = \"horizontal\"\np.legend.location = \"top_center\"\n\nshow(p)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We still have maximum examples from torso and minimum from oral/genital in both gender type. Distribution are also same in both scenario. But, in torso Male has more cases while in lower extremity Female has more cases. Let's visualise images from torso and lower extremity.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extract 9 random images from malignant lesions with growth in torso\n\ntorsomale = train_df[(train_df['benign_malignant']=='malignant') & (train_df['anatom_site_general_challenge'] == 'torso') & (train_df['sex'] == 'male')]\n\nrandom_images = [np.random.choice((torsomale['image_name'].values)+'.jpg') for i in range(9)]\n\nprint('Display malignant torso Images with Male')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n plt.subplot(3, 3, i + 1)\n img = plt.imread(os.path.join(jpeg_train_images, random_images[i]))\n plt.imshow(img, cmap='gray')\n plt.axis('off')\n \n# Adjust subplot parameters to give specified padding\nplt.tight_layout() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extract 9 random images from malignant lesions with growth in lower extremity\n\nlowerextremity = train_df[(train_df['benign_malignant']=='malignant') & (train_df['anatom_site_general_challenge'] == 'lower extremity') & (train_df['sex'] == 'female')]\n\nrandom_images = [np.random.choice((lowerextremity['image_name'].values)+'.jpg') for i in range(9)]\n\nprint('Display malignant lower extremity Images with Female')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n plt.subplot(3, 3, i + 1)\n img = plt.imread(os.path.join(jpeg_train_images, random_images[i]))\n plt.imshow(img, cmap='gray')\n plt.axis('off')\n \n# Adjust subplot parameters to give specified padding\nplt.tight_layout() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Distribution of age_approx column in training set\n# we have missing values in age_approx so we need to fix that before plotting histograms\n\ntraining_df = train_df\ntraining_df['age_approx'].fillna(45.0, inplace=True) # 45 is mode of age_approx\ntraining_df['age_approx'].isnull().any()\nhist_hover(training_df, 'age_approx', title='Age Distribution of patients')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Age column has normal distribution for training data where we have less number of patients in younger age or starting age and older age or ending age while more number of patients in average age or middle age. Let's visualize images with all three types of age gap.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"hist_hover(test_df, 'age_approx', title='Age Distribution of patients')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We do not have normal distribution here but same more number of patients in middle age.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extract 9 random images from malignant lesions with age less than 21\n\nyoungerage = train_df[(train_df['benign_malignant']=='malignant') & (train_df['age_approx'] <= 21)]\n\nrandom_images = [np.random.choice((youngerage['image_name'].values)+'.jpg') for i in range(9)]\n\nprint('Display malignant younger age Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n plt.subplot(3, 3, i + 1)\n img = plt.imread(os.path.join(jpeg_train_images, random_images[i]))\n plt.imshow(img, cmap='gray')\n plt.axis('off')\n \n# Adjust subplot parameters to give specified padding\nplt.tight_layout() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extract 9 random images from malignant lesions with age gap 45-48\n\nmiddleage = train_df[(train_df['benign_malignant']=='malignant') & (train_df['age_approx'] >= 45) & (train_df['age_approx'] <= 48)]\n\nrandom_images = [np.random.choice((middleage['image_name'].values)+'.jpg') for i in range(9)]\n\nprint('Display malignant middle age Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n plt.subplot(3, 3, i + 1)\n img = plt.imread(os.path.join(jpeg_train_images, random_images[i]))\n plt.imshow(img, cmap='gray')\n plt.axis('off')\n \n# Adjust subplot parameters to give specified padding\nplt.tight_layout() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extract 9 random images from malignant lesions with age greater than 78\n\noldage = train_df[(train_df['benign_malignant']=='malignant') & (train_df['age_approx'] >= 78)]\n\nrandom_images = [np.random.choice((oldage['image_name'].values)+'.jpg') for i in range(9)]\n\nprint('Display malignant old age Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n plt.subplot(3, 3, i + 1)\n img = plt.imread(os.path.join(jpeg_train_images, random_images[i]))\n plt.imshow(img, cmap='gray')\n plt.axis('off')\n \n# Adjust subplot parameters to give specified padding\nplt.tight_layout() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# distribution of diagnosis column in training set\ntrain_df['diagnosis'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Gender wise distribution of diagnosis column in training set\n\nCategories = [\"unknown\", \"nevus\", \"melanoma\", \"seborrheic keratosis\", \"lentigo NOS\", \"lichenoid keratosis\", \n \"solar lentigo\", \"cafe-au-lait macule\", \"atypical melanocytic proliferation\"]\ncounts = list(train_df[\"diagnosis\"].value_counts())\n\nsource = ColumnDataSource(data=dict(Categories=Categories, counts=counts, color=Spectral6))\n\np = figure(x_range=Categories, y_range=(0,300), plot_width=800, plot_height=300, title=\"Distribution of the diagnosis in the training set\",\n tools=\"hover, pan, box_zoom, wheel_zoom, reset, save\", tooltips= (\"@Categories: @counts\"))\n\np.vbar(x='Categories', top='counts', width=0.9, color='color', legend_field=\"Categories\", source=source)\n\np.xgrid.grid_line_color = None\np.legend.orientation = \"horizontal\"\np.legend.location = \"top_center\"\nshow(p)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have 27124 maximum number of patients with unknown diagnosis and 584 only with melanoma which we are predicting. Let's visualize images of both","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extract 9 random images with unknown diagnosis\n\nunknown = train_df[train_df['diagnosis'] == 'unknown']\n\nrandom_images = [np.random.choice((unknown['image_name'].values)+'.jpg') for i in range(9)]\n\nprint('Display unknown diagnosis Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n plt.subplot(3, 3, i + 1)\n img = plt.imread(os.path.join(jpeg_train_images, random_images[i]))\n plt.imshow(img, cmap='gray')\n plt.axis('off')\n \n# Adjust subplot parameters to give specified padding\nplt.tight_layout() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extract 9 random images with melanoma diagnosis\n\nmelanoma = train_df[train_df['diagnosis'] == 'melanoma']\n\nrandom_images = [np.random.choice((melanoma['image_name'].values)+'.jpg') for i in range(9)]\n\nprint('Display melanoma diagnosis Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n plt.subplot(3, 3, i + 1)\n img = plt.imread(os.path.join(jpeg_train_images, random_images[i]))\n plt.imshow(img, cmap='gray')\n plt.axis('off')\n \n# Adjust subplot parameters to give specified padding\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Label Encoding\n\nwe have sex, anatom_site_general_challenge, diagnosis column with categorical values so we need to encode them first. But, first we need to handle missing values.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['sex'].fillna(\"male\", inplace = True)\ntrain_df['age_approx'].fillna(50, inplace = True)\ntrain_df['anatom_site_general_challenge'].fillna('torso', inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"categorical = ['sex', 'anatom_site_general_challenge', 'diagnosis']\n\nlabel_encoder = LabelEncoder()\n\nfor column in categorical:\n train_df[column] = label_encoder.fit_transform(train_df[column])\n \n# we do not need benign_malignant column as information is already present in target\ntrain_df.drop(['benign_malignant'], axis = 1, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['anatom_site_general_challenge'].fillna('torso', inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"categorical = ['sex', 'anatom_site_general_challenge']\n\nlabel_encoder = LabelEncoder()\n\nfor column in categorical:\n test_df[column] = label_encoder.fit_transform(test_df[column])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From visualization we can see that we have images with different shapes. So. let's visualize images shape.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"images_shape = []\n\nfor k, image_name in enumerate(train_df['image_name']):\n image = Image.open(jpeg_train_images + \"/\" + image_name + '.jpg')\n images_shape.append(image.size)\n\nimages_shape_df = pd.DataFrame(data = images_shape, columns = ['H', 'W'], dtype='object')\nimages_shape_df['Size'] = '[' + images_shape_df['H'].astype(str) + ',' + images_shape_df['W'].astype(str) + ']'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images_shape_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"We have {} types of different shapes in training images\".format(len(list(images_shape_df['Size'].unique()))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Distribution of shapes in training set\n\n# We have 88 types of unique shapes but many of them contain only few samples. so we will plot only 10 with \n# highest number of samples\n\nCategories = list(images_shape_df['Size'].value_counts().keys())[0:10]\ncounts = list(images_shape_df['Size'].value_counts().values)[0:10]\n\nsource = ColumnDataSource(data=dict(Categories=Categories, counts=counts, color=Spectral6))\n\np = figure(x_range=Categories, y_range=(0,22000), plot_width = 1000, plot_height=300, title=\"Images shape in training set\",\n tools=\"hover, pan, box_zoom, wheel_zoom, reset, save\", tooltips= (\"@Categories: @counts\"))\n\np.vbar(x='Categories', top='counts', width=0.9, color='color', legend_field=\"Categories\", source=source)\n\np.xgrid.grid_line_color = None\np.legend.orientation = \"horizontal\"\np.legend.location = \"top_center\"\n\nshow(p)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Maximum 14703 images have shape of 6000x4000","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"If you like this please upvote","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}