{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"## Context    \n> * Skin cancer is the most prevalent type of cancer.    \n> * Melanoma, specifically, is responsible for 75% of skin cancer deaths, despite being the least common skin cancer.    \n> * The American Cancer Society estimates over 100,000 new melanoma cases will be diagnosed in 2020.    \n> * It's also expected that almost 7,000 people will die from the disease.    \n> * As with other cancers, early and accurate detection—potentially aided by data science—can make treatment more effective. \n   \n## Task    \n> * In this competition, you’ll identify melanoma in images of skin lesions.    \n> * In particular, you’ll use images within the same patient and determine which are likely to represent a melanoma.    \n> * Using patient-level contextual information may help the development of image analysis tools, which could better support clinical dermatologists.    \n   \n## What should I expect the data format to be?    \n> * The images are provided in DICOM format.    \n> * This can be accessed using commonly-available libraries like pydicom, and contains both image and metadata.    \n> * It is a commonly used medical imaging data format.    \n   \n   \n> * Images are also provided in JPEG and TFRecord format (in the jpeg and tfrecords directories, respectively).    \n> * Images in TFRecord format have been resized to a uniform 1024x1024.    \n   \n   \n> * Metadata is also provided outside of the DICOM format, in CSV files.    \n> * See the Columns section for a description. \n   \n## What am I predicting?    \n> * You are predicting a binary target for each image.    \n> * Your model should predict the probability (floating point) between 0.0 and 1.0 that the lesion in the image is malignant (the target).    \n> * In the training data, train.csv, the value 0 denotes benign, and 1 indicates malignant.    ","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"## Libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport glob\n\nimport numpy as np \nimport pandas as pd \n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport matplotlib.image as mpimg\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix, classification_report\n\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator, load_img\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Dense, Flatten, Dropout","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"! ls ../input/siim-isic-melanoma-classification","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true},"cell_type":"markdown","source":"### Files    \n> * train.csv - the training set    \n> * test.csv - the test set    \n> * sample_submission.csv - a sample submission file in the correct format \n   \n### Columns    \n> * **image_name** - unique identifier, points to filename of related DICOM image    \n> * **patient_id** - unique patient identifier    \n> * **sex** - the sex of the patient (when unknown, will be blank)    \n> * **age_approx** - approximate patient age at time of imaging    \n> * **anatom_site_general_challenge** - location of imaged site    \n> * **diagnosis** - detailed diagnosis information (train only)    \n> * **benign_malignant** - indicator of malignancy of imaged lesion    \n> * **target** - binarized version of the target variable    ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Read dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = pd.read_csv('../input/siim-isic-melanoma-classification/sample_submission.csv')\nsample_submission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Explore datasets","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('No. of images in the train dataset :', train.shape[0])\nprint('No. of unique patients in train dataset :', train['patient_id'].nunique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('No. of images in the test dataset :', test.shape[0])\nprint('No. of unique patients in test dataset :', test['patient_id'].nunique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Value counts","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def count_hbar(col, title, pal='Dark2'):\n    df = pd.DataFrame(train[col].value_counts()).reset_index()\n    \n    sns.set_style('whitegrid')\n\n    sns.barplot(data=df, x=col, y='index', palette=pal)\n    \n    for ind, row in df.iterrows():\n        plt.text(row[col]+500, ind, row[col])\n        \n    \n    sns.despine(bottom=True)\n    plt.title(title)\n    plt.xlabel('')\n    plt.ylabel('')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"count_hbar('sex', 'Gender', ['royalblue', 'deeppink'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"count_hbar('anatom_site_general_challenge', 'Anatomy site of the mole')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"count_hbar('diagnosis', 'Diagnosis  of the mole')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"count_hbar('benign_malignant', 'Benign or Malignant', ['dimgray', 'orangered'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### EDA","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"temp = train.groupby('patient_id').agg({'sex':max, 'age_approx':np.mean}).reset_index()\n\nplt.figure(figsize=(12, 5))\nsns.kdeplot(temp[temp['sex']=='male']['age_approx'], label='Male', shade=True, color='royalblue')\nsns.kdeplot(temp[temp['sex']=='female']['age_approx'], label='Female', shade=True, color='deeppink')\nplt.title('Age distribution Male and Female patients', \n          loc='left', fontsize=16)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = pd.DataFrame(train.groupby(['anatom_site_general_challenge'])['target'].mean()) \\\n        .sort_values('target', ascending=False) \\\n        .reset_index() \ndf['target'] = round(df['target'], 4)\n\nplt.figure(figsize=(12, 5))\nsns.set_style('darkgrid')\n\nsns.barplot(data=df, x='target', y='anatom_site_general_challenge', palette='Set2')\n\nfor ind, row in df.iterrows():\n    plt.text(row['target']+0.0001, ind+0.1, row['target'])\n\nsns.despine(bottom=True)\nplt.title('Probability of mole being a Malignant one wrt to it\\'s possition on the human body', \n          loc='left', fontsize=16)\nplt.xlabel('')\nplt.ylabel('')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Images","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_images(diagnosis, title, n):\n    temp = train[train['diagnosis']==diagnosis]\n    img_ids = ['../input/siim-isic-melanoma-classification/jpeg/train/'+i+'.jpg' for i in temp['image_name'].sample(n)]\n\n    fig, ax = plt.subplots(figsize=(24, 5))\n    fig.suptitle(title, fontsize=24)\n    for ind, img in enumerate(img_ids[:n]):\n        plt.subplot(1, 5, ind+1)\n        image = plt.imread(img) # read image\n        plt.axis('off')\n        plt.imshow(image)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_image(diagnosis, title):\n    temp = train[train['diagnosis']==diagnosis]\n    img_ids = ['../input/siim-isic-melanoma-classification/jpeg/train/'+i+'.jpg' for i in temp['image_name']]\n    \n    plt.figure(figsize = (4, 4))\n    image = plt.imread(img_ids[0])\n    plt.axis('off')\n    plt.title(title, fontsize=16)\n    plt.imshow(image)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_images('melanoma', 'Melanoma', 5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_images('seborrheic keratosis', 'Seborrheic Keratosis', 5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_images('lichenoid keratosis', 'Lichenoid Keratosis', 5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_images('lentigo NOS', 'Lentigo NOS', 5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_images('solar lentigo', 'Solar Lentigo', 5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_image('cafe-au-lait macule', 'Cafe-au-lait Macule')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_image('atypical melanocytic proliferation', 'Atypical Melanocytic Proliferation')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}