{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<div align=\"center\">\n<font size=\"6\"> SIIM-ISIC Melanoma Classification  </font>  \n</div> \n\n\n<div align=\"center\">\n<font size=\"4\"> Identify melanoma in lesion images  </font>  \n</div> ","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"<img align=\"left\" src=\"https://raw.githubusercontent.com/kabartay/kaggle-siim-isic-melanoma-classification/master/materials/logo.png\" data-canonical-src=\"https://raw.githubusercontent.com/kabartay/kaggle-siim-isic-melanoma-classification/master/materials/logo.png\" width=\"280\" height=\"280\" />\n\nSkin cancer is the most prevalent type of cancer. **Melanoma**, specifically, is responsible for **75%** of skin cancer deaths, despite being the least common skin cancer. The American Cancer Society estimates over 100,000 new melanoma cases will be diagnosed in 2020. It's also expected that almost 7,000 people will die from the disease. As with other cancers, early and accurate detection—potentially aided by data science—can make treatment more effective.\n\nCurrently, dermatologists evaluate every one of a patient's moles to identify outlier lesions or “ugly ducklings” that are most likely to be melanoma. Existing AI approaches have not adequately considered this clinical frame of reference. Dermatologists could enhance their diagnostic accuracy if detection algorithms take into account “contextual” images within the same patient to determine which images represent a melanoma. If successful, classifiers would be more accurate and could better support dermatological clinic work.\n\nAs the leading healthcare organization for informatics in medical imaging, the [Society for Imaging Informatics in Medicine (SIIM)](https://siim.org/)'s mission is to advance medical imaging informatics through education, research, and innovation in a multi-disciplinary community. SIIM is joined by the [International Skin Imaging Collaboration (ISIC)](https://www.isic-archive.com/), an international effort to improve melanoma diagnosis. The ISIC Archive contains the largest publicly available collection of quality-controlled dermoscopic images of skin lesions.\n\nIn this competition, you’ll identify melanoma in images of skin lesions. In particular, you’ll use images within the same patient and determine which are likely to represent a melanoma. Using patient-level contextual information may help the development of image analysis tools, which could better support clinical dermatologists.\n\nMelanoma is a deadly disease, but if caught early, most melanomas can be cured with minor surgery. Image analysis tools that automate the diagnosis of melanoma will improve dermatologists' diagnostic accuracy. Better detection of melanoma has the opportunity to positively impact millions of people.","metadata":{}},{"cell_type":"markdown","source":"<img align=\"left\" src=\"https://raw.githubusercontent.com/kabartay/kaggle-siim-isic-melanoma-classification/master/materials/melanoma.png\" data-canonical-src=\"https://raw.githubusercontent.com/kabartay/kaggle-siim-isic-melanoma-classification/master/materials/melanoma.png\" width=\"1200\" height=\"450\" />\n","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        #print(os.path.join(dirname, filename)) # otherwise too long\n        continue\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"execution":{"iopub.status.busy":"2021-05-27T16:38:40.437431Z","iopub.execute_input":"2021-05-27T16:38:40.437861Z","iopub.status.idle":"2021-05-27T16:41:07.106338Z","shell.execute_reply.started":"2021-05-27T16:38:40.437807Z","shell.execute_reply":"2021-05-27T16:41:07.105451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport json\nfrom pathlib import Path\n\nimport matplotlib.pyplot as plt\n%matplotlib inline","metadata":{"execution":{"iopub.status.busy":"2021-05-27T16:41:07.107996Z","iopub.execute_input":"2021-05-27T16:41:07.108361Z","iopub.status.idle":"2021-05-27T16:41:07.117071Z","shell.execute_reply.started":"2021-05-27T16:41:07.108316Z","shell.execute_reply":"2021-05-27T16:41:07.116115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_path = Path('/kaggle/input/siim-isic-melanoma-classification/')\ntrain_path = data_path / 'train'\ntest_path = data_path / 'test'\nprint(\"training_path\", train_path)\nprint(\"test_path\", test_path)","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls /kaggle/input/siim-isic-melanoma-classification/","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Images","metadata":{}},{"cell_type":"code","source":"#!ls /kaggle/input/siim-isic-melanoma-classification/jpeg/train","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#!ls /kaggle/input/siim-isic-melanoma-classification/jpeg/test","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_path = Path('/kaggle/input/siim-isic-melanoma-classification/')\nim_train_path = data_path / 'jpeg' / 'train'\nim_test_path = data_path / 'jpeg' / 'test'\nprint(\"train_path: \", im_train_path)\nprint(\"test_path:  \", im_test_path)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.python.keras.preprocessing.image import load_img, img_to_array\n\nfrom keras import models, regularizers, layers, optimizers, losses, metrics\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.utils import np_utils, to_categorical\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.preprocessing import image","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def image_show(im_num,im_folder,im_size):\n    \"\"\"\n    MO: Show melanoma images.\n    \"\"\"\n    im_ind = 'ISIC'\n    im_name = '{}_{}'.format(im_ind,im_num)\n    if im_folder=='train':\n        im_dir = im_train_path\n    elif im_folder=='test':\n        im_dir = im_test_path\n    im_path = str(im_dir)+'/'+str(im_name)+'.jpg'\n    im_path\n    \n    #from tf.keras.preprocessing.image.load_img\n    img = image.load_img(im_path, target_size=(im_size, im_size)) #target_size=(224, 224)\n    imgplot = plt.imshow(img)\n    print(im_ind,\"Image Number:\", im_num)\n    plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (10,10))\nimage_show(5225336,'train',224)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (10,10))\nimage_show(5224960,'test',224)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls /kaggle/input/siim-isic-melanoma-classification/test/ISIC_5224960.dcm","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#import dicom\n\nimport pydicom\nfrom pydicom.data import get_testdata_files\n\nprint(__doc__)\n\nPathDicom = '/kaggle/input/siim-isic-melanoma-classification/'\nlstFilesDCM = []  # create an empty list\nfor dirName, subdirList, fileList in os.walk(PathDicom):\n    for filename in fileList:\n        if \".dcm\" in filename.lower():  # check whether the file's DICOM\n            lstFilesDCM.append(os.path.join(dirName,filename))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(lstFilesDCM[0])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"RefDs = pydicom.dcmread(lstFilesDCM[0])\nRefDs","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get ref file\nRefDs = pydicom.dcmread(lstFilesDCM[0])\n\n# Load dimensions based on the number of rows, columns, and slices (along the Z axis)\nConstPixelDims = (int(RefDs.Rows), int(RefDs.Columns), len(lstFilesDCM))\nprint(ConstPixelDims)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pat_name = RefDs.PatientName\ndisplay_name = pat_name.family_name + \", \" + pat_name.given_name\nprint(\"Patient's name...:\", display_name)\nprint(\"Patient id.......:\", RefDs.PatientID)\nprint(\"Modality.........:\", RefDs.Modality)\nprint(\"Study Date.......:\", RefDs.StudyDate)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if 'PixelData' in RefDs:\n    rows = int(RefDs.Rows)\n    cols = int(RefDs.Columns)\n    print(\"Image size.......: {rows:d} x {cols:d}, {size:d} bytes\".format(rows=rows, cols=cols, size=len(RefDs.PixelData)))\n    if 'PixelSpacing' in RefDs:\n        print(\"Pixel spacing....:\", RefDs.PixelSpacing) \n\n        \n# use .get() if not sure the item exists, and want a default value if missing\nprint(\"Slice location...:\", RefDs.get('SliceLocation', \"(missing)\"))\n\n# plot the image using matplotlib\nplt.figure(figsize = (10,10))\nplt.imshow(RefDs.pixel_array, cmap=plt.cm.bone)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Explore tables","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(data_path / 'train.csv')\ntest  = pd.read_csv(data_path / 'test.csv')\nsub   = pd.read_csv(data_path / 'sample_submission.csv')\n\ntrain.shape, test.shape, sub.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.isna().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['sex'] = train['sex'].fillna('na')\ntrain['age_approx'] = train['age_approx'].fillna(0)\ntrain['anatom_site_general_challenge'] = train['anatom_site_general_challenge'].fillna('na')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.isna().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.isna().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test['anatom_site_general_challenge'] = test['anatom_site_general_challenge'].fillna('na')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.isna().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['sex'].value_counts().plot(kind='bar')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test['sex'].value_counts().plot(kind='bar')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['sex'].isna().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['age_approx'].value_counts().plot(kind='bar')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test['age_approx'].value_counts().plot(kind='bar')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['diagnosis'].value_counts().plot(kind='bar')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Melanoma is rare, <2%","metadata":{}},{"cell_type":"code","source":"train['diagnosis'].value_counts()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Diagnosis                             Percent\\n-----------------------------------------------')\nprint((train['diagnosis'].value_counts() / train['diagnosis'].value_counts().sum() ) *100)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## In one plot and save","metadata":{}},{"cell_type":"code","source":"fig, axs = plt.subplots(4,2, figsize=(13,20))\n\n# left train, right test\n\ntrain['sex'].value_counts().plot(kind='bar', legend=True, ax=axs[0,0])\ntest['sex'].value_counts().plot(kind='bar', legend=True, ax=axs[0,1])\n\ntrain['age_approx'].value_counts().plot(kind='bar', legend=True, ax=axs[1,0])\ntest['age_approx'].value_counts().plot(kind='bar', legend=True, ax=axs[1,1])\n\ntrain['age_approx'].hist(bins=90, ax=axs[2,0])\ntest['age_approx'].hist(bins=90, ax=axs[2,1])\naxs[2,0].set_xlabel('Age')\naxs[2,1].set_xlabel('Age')\n\ntrain['anatom_site_general_challenge'].value_counts().plot(kind='bar', legend=True, ax=axs[3,0])\ntest['anatom_site_general_challenge'].value_counts().plot(kind='bar', legend=True, ax=axs[3,1])\n\n\nplt.savefig('data_sex_age_anatom.png',dpi=100)\n\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}