{"cells":[{"metadata":{},"cell_type":"markdown","source":"![https://healthcare-in-europe.com/media/story_section_text/18136/image-01-nci-vol-9872-300_hires.jpg](http://healthcare-in-europe.com/media/story_section_text/18136/image-01-nci-vol-9872-300_hires.jpg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"[https://healthcare-in-europe.com/media/story_section_text/18136/image-01-nci-vol-9872-300_hires.jpg](http://healthcare-in-europe.com/media/story_section_text/18136/image-01-nci-vol-9872-300_hires.jpg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Melanoma is a skin cancer that arises from skin cell called a melanocyte,which makes a pigment that gives skin its color. Melanomas typicallyy occur in the skin but rarely occur in the mouth, intestines or eye.Melanoma is more likely to grow and spread than the more common types of skin Cancer.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Most melanomas can be cured with minor surgery. Image analysis tools that automate the diagnosis of melanoma will imporove dermatologists .","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"SIIM and ISIC will kick off the Melanoma Classification Challenge in the weeks. \n> > The International Skin Imaging Collaboration(ISIC) is delighted to collaborate with SIIM for its fifth annual grand challenge for skin cancer identification. This year's challenge will test the diagnostic impact of clinical context by providing multiple images of different lesions from individual patients.It will also be the first time we have incorporated data standards by using DICOM to encode the images.\nIn this challenge, we develop image analysis tools to enable that the automated diagnosis of melanoma using patient-level contextual information, a process more similar to a clinincal workflow. Standard-based healthcare APIs will be used to reduce the interoperability barriers to clinical implementation .","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"#Define data path\npath=\"../input/siim-isic-melanoma-classification/\"\ntrain=pd.read_csv(path+'train.csv')\ntest=pd.read_csv(path+'test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Training data\ndisplay('Training data shape:',train.shape)\ndisplay(train.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Test data\ndisplay('Test data shape:',test.shape)\ndisplay(test.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(['benign_malignant']).count()['sex'].to_frame()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(['age_approx']).count()['benign_malignant'].to_frame()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(['anatom_site_general_challenge']).count()['sex'].to_frame()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(['age_approx']).count()['anatom_site_general_challenge'].to_frame()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Exploration","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Missing values**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display('Train Set')\ndisplay(train.info())\ndisplay('-------------')\ndisplay('Test Set')\ndisplay(test.info())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are missing values in some of the columns.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Unique IDs**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display(\"Total patients ids are :\", train['patient_id'].count())\ndisplay('Unique ids are:',train['patient_id'].value_counts().shape[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The number of Unique patients is less than the total number of patients. This means that patients have multiple records.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"col=train.keys()\ncol=list(col)\ndisplay(col)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Exploring Target Columns**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display(train['target'].value_counts(normalize=True))\ndisplay(sns.countplot(train['target']))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Gender Wise  Distribution**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display(train['sex'].value_counts(normalize=True))\ndisplay(sns.countplot(train['sex']))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Benign_malignant Distribution**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display(train['benign_malignant'].value_counts(normalize=True))\ndisplay(sns.countplot(train['benign_malignant']))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Anatom Site Distribution**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display(train['anatom_site_general_challenge'].value_counts(normalize=True).sort_values())\ndisplay(sns.countplot(train['anatom_site_general_challenge']))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Diagnosis Distribution**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display(train['diagnosis'].value_counts())\nsns.countplot(train['diagnosis'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Gender Vs Target","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"z=train.groupby(['target','sex'])['benign_malignant'].count().to_frame().reset_index()\nz.style.background_gradient(cmap='Blues')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.catplot(x='target',y='benign_malignant',hue='sex',data=z,kind='bar')\nplt.ylabel('Count')\nplt.xlabel('benign:0 vs malignant:1')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Imaged w.r.t gender**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"z=train.groupby(['anatom_site_general_challenge','sex','target'])['benign_malignant'].count().to_frame().reset_index()\nz.style.background_gradient(cmap='Blues')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"z1=train.groupby(['sex','anatom_site_general_challenge'])['benign_malignant'].count().to_frame().reset_index()\nz1.style.background_gradient(cmap='Reds')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.catplot(x='anatom_site_general_challenge',y='benign_malignant', hue='sex',data=z1,kind='bar')\nplt.gcf().set_size_inches(10,8)\nplt.xlabel('location of imaged site')\nplt.xticks(rotation=45,fontsize='10', horizontalalignment='right')\nplt.ylabel('count of melanoma cases')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"z2=train.groupby(['sex','anatom_site_general_challenge','target'])['benign_malignant'].count().to_frame().reset_index()\nz2.style.background_gradient(cmap='Reds')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.catplot(x='anatom_site_general_challenge',y='benign_malignant', hue='target',data=z2,kind='bar')\nplt.gcf().set_size_inches(10,8)\nplt.xlabel('location of imaged site')\nplt.xticks(rotation=45,fontsize='10', horizontalalignment='right')\nplt.ylabel('count of melanoma cases')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Diagnosis w.r.t gender**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"z3=train.groupby(['anatom_site_general_challenge','sex','target','diagnosis'])['benign_malignant'].count().to_frame().reset_index()\nz3.style.background_gradient(cmap='Blues')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.catplot(x='diagnosis',y='benign_malignant', hue='sex',data=z3,kind='bar')\nplt.gcf().set_size_inches(10,8)\nplt.xlabel('Diagnosis')\nplt.xticks(rotation=45,fontsize='10', horizontalalignment='right')\nplt.ylabel('count of melanoma cases')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.catplot(x='anatom_site_general_challenge',y='benign_malignant', hue='diagnosis',data=z3,kind='bar')\nplt.gcf().set_size_inches(10,8)\nplt.xlabel('location of imaged site')\nplt.xticks(rotation=45,fontsize='10', horizontalalignment='right')\nplt.ylabel('count of melanoma cases')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Distribution of Ages w.r.t target**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# KDE plot of age that were diagnosed as benign\nsns.kdeplot(train.loc[train['target'] == 0, 'age_approx'], label = 'Benign',shade=True)\n\n# KDE plot of age that were diagnosed as malignant\nsns.kdeplot(train.loc[train['target'] == 1, 'age_approx'], label = 'Malignant',shade=True)\n\n# Labeling of plot\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Distribution of Ages w.r.t gender**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# KDE plot of age that were diagnosed as benign\nsns.kdeplot(train.loc[train['sex'] == 'male', 'age_approx'], label = 'Male',shade=True)\n\n# KDE plot of age that were diagnosed as malignant\nsns.kdeplot(train.loc[train['sex'] == 'female', 'age_approx'], label = 'Female',shade=True)\n\n# Labeling of plot\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Patient Overlap**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We need to check that the same patient lesion images shouldn't appear in training and test set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extract patient id's for the training set\nids_train = train['patient_id'].values\n# Extract patient id's for the validation set\nids_test = test['patient_id'].values\n\n# Create a \"set\" datastructure of the training set id's to identify unique id's\nids_train_set = set(ids_train)\ndisplay(f'There are {len(ids_train_set)} unique Patient IDs in the training set')\n# Create a \"set\" datastructure of the validation set id's to identify unique id's\nids_test_set = set(ids_test)\ndisplay(f'There are {len(ids_test_set)} unique Patient IDs in the training set')\n\n# Identify patient overlap by looking at the intersection between the sets\npatient_overlap = list(ids_train_set.intersection(ids_test_set))\nn_overlap = len(patient_overlap)\ndisplay(f'There are {n_overlap} Patient IDs in both the training and test sets')\ndisplay('')\ndisplay(f'These patients are in both the training and test datasets:')\ndisplay(f'{patient_overlap}')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualizing Images","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Visualizing a random selection of images**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"images = train['image_name'].values\n\n# Extract 9 random images from it\nrandom_images = [np.random.choice(images+'.jpg') for i in range(9)]\n\n# Location of the image dir\nimg_dir = path+'/jpeg/train'\n\nprint('Display Random Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n    plt.subplot(3, 3, i + 1)\n    img = plt.imread(os.path.join(img_dir, random_images[i]))\n    plt.imshow(img, cmap='gray')\n    plt.axis('off')\n    \n# Adjust subplot parameters to give specified padding\nplt.tight_layout()   ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"JPEG format images vary in sizes.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Visualizing Images with benign lesions**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"benign = train[train['benign_malignant']=='benign']\nmalignant = train[train['benign_malignant']=='malignant']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images = benign['image_name'].values\n\n# Extract 9 random images from it\nrandom_images = [np.random.choice(images+'.jpg') for i in range(9)]\n\n# Location of the image dir\nimg_dir = path+'/jpeg/train'\n\nprint('Display benign Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n    plt.subplot(3, 3, i + 1)\n    img = plt.imread(os.path.join(img_dir, random_images[i]))\n    plt.imshow(img, cmap='gray')\n    plt.axis('off')\n    \n# Adjust subplot parameters to give specified padding\nplt.tight_layout() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images = malignant['image_name'].values\n\n# Extract 9 random images from it\nrandom_images = [np.random.choice(images+'.jpg') for i in range(9)]\n\n# Location of the image dir\nimg_dir = path+'/jpeg/train'\n\nprint('Display Malignant Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n    plt.subplot(3, 3, i + 1)\n    img = plt.imread(os.path.join(img_dir, random_images[i]))\n    plt.imshow(img, cmap='gray')\n    plt.axis('off')\n    \n# Adjust subplot parameters to give specified padding\nplt.tight_layout() ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Histograms are a graphical representation showing how frequently various color values occur in the image i.e frequency of pixels intensity values.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16,8))\nf.add_subplot(1,2, 1)\n\nsample_img = benign['image_name'][3]+'.jpg'\nraw_image = plt.imread(os.path.join(img_dir, sample_img))\nplt.imshow(raw_image, cmap='gray')\nplt.colorbar()\nplt.title('Benign Image')\nprint(f\"Image dimensions:  {raw_image.shape[0],raw_image.shape[1]}\")\nprint(f\"Maximum pixel value : {raw_image.max():.1f} ; Minimum pixel value:{raw_image.min():.1f}\")\nprint(f\"Mean value of the pixels : {raw_image.mean():.1f} ; Standard deviation : {raw_image.std():.1f}\")\n\nf.add_subplot(1,2, 2)\n\n#_ = plt.hist(raw_image.ravel(),bins = 256, color = 'orange',)\n_ = plt.hist(raw_image[:, :, 0].ravel(), bins = 256, color = 'red', alpha = 0.5)\n_ = plt.hist(raw_image[:, :, 1].ravel(), bins = 256, color = 'Green', alpha = 0.5)\n_ = plt.hist(raw_image[:, :, 2].ravel(), bins = 256, color = 'Blue', alpha = 0.5)\n_ = plt.xlabel('Intensity Value')\n_ = plt.ylabel('Count')\n_ = plt.legend(['Red_Channel', 'Green_Channel', 'Blue_Channel'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Malignant Category**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16,8))\nf.add_subplot(1,2, 1)\n\nsample_img = malignant['image_name'][235]+'.jpg'\nraw_image = plt.imread(os.path.join(img_dir, sample_img))\nplt.imshow(raw_image, cmap='gray')\nplt.colorbar()\nplt.title('Malignant Image')\nprint(f\"Image dimensions:  {raw_image.shape[0],raw_image.shape[1]}\")\nprint(f\"Maximum pixel value : {raw_image.max():.1f} ; Minimum pixel value:{raw_image.min():.1f}\")\nprint(f\"Mean value of the pixels : {raw_image.mean():.1f} ; Standard deviation : {raw_image.std():.1f}\")\n\nf.add_subplot(1,2, 2)\n\n#_ = plt.hist(raw_image.ravel(),bins = 256, color = 'orange',)\n_ = plt.hist(raw_image[:, :, 0].ravel(), bins = 256, color = 'red', alpha = 0.5)\n_ = plt.hist(raw_image[:, :, 1].ravel(), bins = 256, color = 'Green', alpha = 0.5)\n_ = plt.hist(raw_image[:, :, 2].ravel(), bins = 256, color = 'Blue', alpha = 0.5)\n_ = plt.xlabel('Intensity Value')\n_ = plt.ylabel('Count')\n_ = plt.legend(['Red_Channel', 'Green_Channel', 'Blue_Channel'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocessing DIOCOM files\n\nDigital Imaging and Communications in Medicine(DICOM) is the standard for communication and mangement of medical imaging information and related data.\n\nDICOM images have the extension dcm.A Dicom file has two parts:the header and the dataset.The header contains information on the encapsulated dataset.It consist of a File Preamble, a DICOM prefix, and the FIle Meta Elements.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install pydicom","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pydicom\nprint(pydicom.__version__)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef show_dcm_info(dataset):\n    print(\"Filename.........:\", file_path)\n    print(\"Storage type.....:\", dataset.SOPClassUID)\n    print()\n\n    pat_name = dataset.PatientName\n    display_name = pat_name.family_name + \", \" + pat_name.given_name\n    print(\"Patient's name......:\", display_name)\n    print(\"Patient id..........:\", dataset.PatientID)\n    print(\"Patient's Age.......:\", dataset.PatientAge)\n    print(\"Patient's Sex.......:\", dataset.PatientSex)\n    print(\"Modality............:\", dataset.Modality)\n    print(\"Body Part Examined..:\", dataset.BodyPartExamined)\n   \n    \n    \n    if 'PixelData' in dataset:\n        rows = int(dataset.Rows)\n        cols = int(dataset.Columns)\n        print(\"Image size.......: {rows:d} x {cols:d}, {size:d} bytes\".format(\n            rows=rows, cols=cols, size=len(dataset.PixelData)))\n        if 'PixelSpacing' in dataset:\n            print(\"Pixel spacing....:\", dataset.PixelSpacing)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_pixel_array(dataset, figsize=(5,5)):\n    plt.figure(figsize=figsize)\n    plt.grid(False)\n    plt.imshow(dataset.pixel_array)\n    plt.show()\n    \ni = 1\nnum_to_plot = 5\nfor file_name in os.listdir('../input/siim-isic-melanoma-classification/train/'):\n        file_path = os.path.join('../input/siim-isic-melanoma-classification/train/',file_name)\n        dataset = pydicom.dcmread(file_path)\n        show_dcm_info(dataset)\n        plot_pixel_array(dataset)\n    \n        if i >= num_to_plot:\n            break\n    \n        i += 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Extracting DIOCOM files information in a dataframe**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# source: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154658\nfolder='train'\nPATH='../input/siim-isic-melanoma-classification/'\n\ndef extract_DICOM_attributes(folder):\n    images = list(os.listdir(os.path.join(PATH, folder)))\n    df = pd.DataFrame()\n    for image in images:\n        image_name = image.split(\".\")[0]\n        dicom_file_path = os.path.join(PATH,folder,image)\n        dicom_file_dataset = pydicom.read_file(dicom_file_path)\n        study_date = dicom_file_dataset.StudyDate\n        modality = dicom_file_dataset.Modality\n        age = dicom_file_dataset.PatientAge\n        sex = dicom_file_dataset.PatientSex\n        body_part_examined = dicom_file_dataset.BodyPartExamined\n        patient_orientation = dicom_file_dataset.PatientOrientation\n        photometric_interpretation = dicom_file_dataset.PhotometricInterpretation\n        rows = dicom_file_dataset.Rows\n        columns = dicom_file_dataset.Columns\n\n        df = df.append(pd.DataFrame({'image_name': image_name, \n                        'dcm_modality': modality,'dcm_study_date':study_date, 'dcm_age': age, 'dcm_sex': sex,\n                        'dcm_body_part_examined': body_part_examined,'dcm_patient_orientation': patient_orientation,\n                        'dcm_photometric_interpretation': photometric_interpretation,\n                        'dcm_rows': rows, 'dcm_columns': columns}, index=[0]))\n    return df\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"extract_DICOM_attributes('train')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}