{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":20270,"databundleVersionId":1222630,"sourceType":"competition"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"SIIM-ISIC Melanoma Classification -EDA
\n
\n\n\nhttps://impactmelanoma.org/wp-content/uploads/2018/11/Standard-Infographic_0.jpg\n\nMelanoma is a skin cancer that arises from a skin cell called a melanocyte, which makes a the pigment (melanin) that gives your skin its color.1 Melanoma can appear in different ways, most commonly as a new spot on the skin or as an already existing mole that changes in color, size, or shape. While considered the most dangerous type of skin cancer because of its ability to rapidly spread throughout the body, melanoma is generally very treatable if found early.2\nhttps://www.verywellhealth.com/what-is-melanoma-514215\n\nThe[ Society for Imaging Informatics in Medicine (SIIM)](https://siim.org/page/about_siim) is the leading healthcare professional organization for those interested in the current and future use of informatics in medical imaging. The society's mission is to advance medical imaging informatics across the enterprise through education, research, and innovation in a multi-disciplinary community. The [International Skin Imaging Collaboration or \nISIC](https://siim.org/page/about_siim) Melanoma Project is an academia and industry partnership designed to facilitate the application of digital skin imaging to help reduce melanoma mortality\n\nThe overarching goal of the ISIC Melanoma Project is to support efforts to reduce melanoma-related deaths and unnecessary biopsies by improving the accuracy and efficiency of melanoma early detection since when recognized and treated in its earliest stages, melanoma is readily curable\n\n## Objective\n\nThe objective of this competition is to identify melanoma in images of skin lesions. In particular, we need to use images within the same patient and determine which are likely to represent a melanoma. In other words, we need to create a model which should predict the probability whether the lesion in the image is malignantor benign.Value 0 denotes benign, and 1 indicates malignant\n\n## Dataset\nThe dataset consists of images in :\n* DIOCOM format\n* JPEG format in JPEG directory\n* TFRecord format in tfrecords directory\n\nAdditionally, there is a metadata comprising of train, test and submission file in CSV format.\n\n## Understanding the Evaluation Metric\n\nFor this particluar problem, our submissions will be evaluated using **area under the ROC curve**. An ROC curve (receiver operating characteristic curve) is a graph showing the performance of a classification model at all classification thresholds. This curve plots two parameters:\n\n\n\nAn ROC curve plots TPR vs. FPR at different classification thresholds. Lowering the classification threshold classifies more items as positive, thus increasing both False Positives and True Positives. The following figure shows a typical ROC curve.\n\n\n\nsource: https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2025-10-02T11:41:32.816528Z","iopub.execute_input":"2025-10-02T11:41:32.817058Z","iopub.status.idle":"2025-10-02T11:41:32.83046Z","shell.execute_reply.started":"2025-10-02T11:41:32.817021Z","shell.execute_reply":"2025-10-02T11:41:32.82913Z"}}},{"cell_type":"code","source":"import os\nfrom tqdm import tqdm\nfrom os import listdir\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n#plotly\n!pip install chart_studio\nimport plotly.express as px\nimport chart_studio.plotly as py\nimport plotly.graph_objs as go\nfrom plotly.offline import iplot\nimport cufflinks\ncufflinks.go_offline()\ncufflinks.set_config_file(world_readable=True, theme='pearl')\n\nimport seaborn as sns\nsns.set(style=\"whitegrid\")\n\n\n#pydicom\nimport pydicom\n\n# Suppress warnings \nimport warnings\nwarnings.filterwarnings('ignore')\n\n\n# Settings for pretty nice plots\nplt.style.use('fivethirtyeight')\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-03T05:22:00.301684Z","iopub.execute_input":"2025-10-03T05:22:00.302119Z","iopub.status.idle":"2025-10-03T05:22:04.74917Z","shell.execute_reply.started":"2025-10-03T05:22:00.302076Z","shell.execute_reply":"2025-10-03T05:22:04.747835Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# List files available\nprint(os.listdir(\"../input/siim-isic-melanoma-classification\"))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-02T11:42:36.209795Z","iopub.execute_input":"2025-10-02T11:42:36.210266Z","iopub.status.idle":"2025-10-02T11:42:36.217138Z","shell.execute_reply.started":"2025-10-02T11:42:36.210225Z","shell.execute_reply":"2025-10-02T11:42:36.216166Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Defining data path\nIMAGE_PATH = \"../input/siim-isic-melanoma-classification/\"\n\ntrain_df = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\ntest_df = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')\n\n\n#Training data\nprint('Training data shape: ', train_df.shape)\ntrain_df.head(5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-02T11:42:36.361458Z","iopub.execute_input":"2025-10-02T11:42:36.362376Z","iopub.status.idle":"2025-10-02T11:42:36.507191Z","shell.execute_reply.started":"2025-10-02T11:42:36.362341Z","shell.execute_reply":"2025-10-02T11:42:36.506266Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df.groupby(['benign_malignant']).count()['sex'].to_frame()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-02T11:42:37.346171Z","iopub.execute_input":"2025-10-02T11:42:37.34656Z","iopub.status.idle":"2025-10-02T11:42:37.379489Z","shell.execute_reply.started":"2025-10-02T11:42:37.346515Z","shell.execute_reply":"2025-10-02T11:42:37.378358Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 3. Data Exploration\n\n## Missing Values","metadata":{}},{"cell_type":"code","source":"# Null values and Data types\nprint('Train Set')\nprint(train_df.info())\nprint('-------------')\nprint('Test Set')\nprint(test_df.info())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-02T11:42:41.085535Z","iopub.execute_input":"2025-10-02T11:42:41.085885Z","iopub.status.idle":"2025-10-02T11:42:41.126811Z","shell.execute_reply.started":"2025-10-02T11:42:41.085861Z","shell.execute_reply":"2025-10-02T11:42:41.125599Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"There are some missing values in some of the columns. We shall deal with them later.","metadata":{}},{"cell_type":"markdown","source":"## Total Number of images","metadata":{}},{"cell_type":"code","source":"# Total number of images in the dataset(train+test)\nprint(\"Total images in Train set: \",train_df['image_name'].count())\nprint(\"Total images in Test set: \",test_df['image_name'].count())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-02T11:43:06.264146Z","iopub.execute_input":"2025-10-02T11:43:06.264504Z","iopub.status.idle":"2025-10-02T11:43:06.27346Z","shell.execute_reply.started":"2025-10-02T11:43:06.264479Z","shell.execute_reply":"2025-10-02T11:43:06.272468Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Unique IDs ","metadata":{}},{"cell_type":"code","source":"print(f\"The total patient ids are {train_df['patient_id'].count()}, from those the unique ids are {train_df['patient_id'].value_counts().shape[0]} \")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-02T11:43:21.657151Z","iopub.execute_input":"2025-10-02T11:43:21.657524Z","iopub.status.idle":"2025-10-02T11:43:21.668683Z","shell.execute_reply.started":"2025-10-02T11:43:21.6575Z","shell.execute_reply":"2025-10-02T11:43:21.667639Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The number of unique patients is less than the total number of patients. This means that, patients have multiple records.","metadata":{}},{"cell_type":"code","source":"columns = train_df.keys()\ncolumns = list(columns)\nprint(columns)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-02T11:43:34.704872Z","iopub.execute_input":"2025-10-02T11:43:34.705208Z","iopub.status.idle":"2025-10-02T11:43:34.71039Z","shell.execute_reply.started":"2025-10-02T11:43:34.705177Z","shell.execute_reply":"2025-10-02T11:43:34.709309Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Exploring the Target column","metadata":{}},{"cell_type":"code","source":"train_df['target'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-02T11:45:43.123366Z","iopub.execute_input":"2025-10-02T11:45:43.123757Z","iopub.status.idle":"2025-10-02T11:45:43.13458Z","shell.execute_reply.started":"2025-10-02T11:45:43.123732Z","shell.execute_reply":"2025-10-02T11:45:43.133397Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df['target'].value_counts(normalize=True).iplot(kind='bar',\n yTitle='Percentage', \n linecolor='black', \n opacity=0.7,\n color='red',\n theme='pearl',\n bargap=0.8,\n gridcolor='white',\n \n title='Distribution of the Target column in the training set')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-02T11:46:14.589156Z","iopub.execute_input":"2025-10-02T11:46:14.589451Z","iopub.status.idle":"2025-10-02T11:46:14.630499Z","shell.execute_reply.started":"2025-10-02T11:46:14.589432Z","shell.execute_reply":"2025-10-02T11:46:14.629515Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Gender wise distribution\n","metadata":{}},{"cell_type":"code","source":"train_df['sex'].value_counts(normalize=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-02T11:46:21.882334Z","iopub.execute_input":"2025-10-02T11:46:21.882696Z","iopub.status.idle":"2025-10-02T11:46:21.893189Z","shell.execute_reply.started":"2025-10-02T11:46:21.88267Z","shell.execute_reply":"2025-10-02T11:46:21.892092Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df['sex'].value_counts(normalize=True).iplot(kind='bar',\n yTitle='Percentage', \n linecolor='black', \n opacity=0.7,\n color='green',\n theme='pearl',\n bargap=0.8,\n gridcolor='white',\n \n title='Distribution of the Sex column in the training set')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-02T11:46:27.509657Z","iopub.execute_input":"2025-10-02T11:46:27.509956Z","iopub.status.idle":"2025-10-02T11:46:27.557498Z","shell.execute_reply.started":"2025-10-02T11:46:27.509928Z","shell.execute_reply":"2025-10-02T11:46:27.556365Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Gender vs Target","metadata":{}},{"cell_type":"code","source":"z=train_df.groupby(['target','sex'])['benign_malignant'].count().to_frame().reset_index()\nz.style.background_gradient(cmap='Reds') ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-02T11:46:59.372138Z","iopub.execute_input":"2025-10-02T11:46:59.372451Z","iopub.status.idle":"2025-10-02T11:46:59.502115Z","shell.execute_reply.started":"2025-10-02T11:46:59.372433Z","shell.execute_reply":"2025-10-02T11:46:59.50076Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.catplot(x='target',y='benign_malignant', hue='sex',data=z,kind='bar')\nplt.ylabel('Count')\nplt.xlabel('benign:0 vs malignant:1')","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.826Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Location of imaged site","metadata":{}},{"cell_type":"code","source":"train_df['anatom_site_general_challenge'].value_counts(normalize=True).sort_values()","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.826Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df['anatom_site_general_challenge'].value_counts(normalize=True).sort_values().iplot(kind='barh',\n xTitle='Percentage', \n linecolor='black', \n opacity=0.7,\n color='#FB8072',\n theme='pearl',\n bargap=0.2,\n gridcolor='white',\n title='Distribution of the imaged site in the training set')","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.826Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Location of imaged site w.r.t gender","metadata":{}},{"cell_type":"code","source":"\nz1=train_df.groupby(['sex','anatom_site_general_challenge'])['benign_malignant'].count().to_frame().reset_index()\nz1.style.background_gradient(cmap='Reds')\nsns.catplot(x='anatom_site_general_challenge',y='benign_malignant', hue='sex',data=z1,kind='bar')\nplt.gcf().set_size_inches(10,8)\nplt.xlabel('location of imaged site')\nplt.xticks(rotation=45,fontsize='10', horizontalalignment='right')\nplt.ylabel('count of melanoma cases')\n","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.826Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Age Distribution of patients","metadata":{}},{"cell_type":"code","source":"train_df['age_approx'].iplot(kind='hist',bins=30,color='orange',xTitle='Age distribution',yTitle='Count')","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.826Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Visualising Age KDEs\nSummarizing the data with Density plots to see where the mass of the data is located. [A kernel density estimate plot](https://chemicalstatistician.wordpress.com/2013/06/09/exploratory-data-analysis-kernel-density-estimation-in-r-on-ozone-pollution-data-in-new-york-and-ozonopolis/) shows the distribution of a single variable and can be thought of as a smoothed histogram (it is created by computing a kernel, usually a Gaussian, at each data point and then averaging all the individual kernels to develop a single smooth curve). We will use the seaborn kdeplot for this graph.\n\n### Distribution of Ages w.r.t Target","metadata":{}},{"cell_type":"code","source":"# KDE plot of age that were diagnosed as benign\nsns.kdeplot(train_df.loc[train_df['target'] == 0, 'age_approx'], label = 'Benign',shade=True)\n\n# KDE plot of age that were diagnosed as malignant\nsns.kdeplot(train_df.loc[train_df['target'] == 1, 'age_approx'], label = 'Malignant',shade=True)\n\n# Labeling of plot\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages');","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.826Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n### Distribution of Ages w.r.t gender","metadata":{}},{"cell_type":"code","source":"# KDE plot of age that were diagnosed as benign\nsns.kdeplot(train_df.loc[train_df['sex'] == 'male', 'age_approx'], label = 'Male',shade=True)\n\n# KDE plot of age that were diagnosed as malignant\nsns.kdeplot(train_df.loc[train_df['sex'] == 'female', 'age_approx'], label = 'Female',shade=True)\n\n# Labeling of plot\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages');\n","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.827Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Distribution of Diagnosis","metadata":{}},{"cell_type":"code","source":"train_df['diagnosis'].value_counts()","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.827Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df['diagnosis'].value_counts(normalize=True).sort_values().iplot(kind='barh',\n xTitle='Percentage', \n linecolor='black', \n opacity=0.7,\n color='blue',\n theme='pearl',\n bargap=0.2,\n gridcolor='white',\n title='Distribution in the training set')","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.827Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Patient Overlap \nWe need to check that the the same patient lesion images shouldn't appear in both training and test set.","metadata":{}},{"cell_type":"code","source":"# Extract patient id's for the training set\nids_train = train_df.patient_id.values\n# Extract patient id's for the validation set\nids_test = test_df.patient_id.values\n\n# Create a \"set\" datastructure of the training set id's to identify unique id's\nids_train_set = set(ids_train)\nprint(f'There are {len(ids_train_set)} unique Patient IDs in the training set')\n# Create a \"set\" datastructure of the validation set id's to identify unique id's\nids_test_set = set(ids_test)\nprint(f'There are {len(ids_test_set)} unique Patient IDs in the training set')\n\n# Identify patient overlap by looking at the intersection between the sets\npatient_overlap = list(ids_train_set.intersection(ids_test_set))\nn_overlap = len(patient_overlap)\nprint(f'There are {n_overlap} Patient IDs in both the training and test sets')\nprint('')\nprint(f'These patients are in both the training and test datasets:')\nprint(f'{patient_overlap}')","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.827Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 4. Visualising Images : JPEG\n\n## Visualizing a random selection of images","metadata":{}},{"cell_type":"code","source":"images = train_df['image_name'].values\n\n# Extract 9 random images from it\nrandom_images = [np.random.choice(images+'.jpg') for i in range(9)]\n\n# Location of the image dir\nimg_dir = IMAGE_PATH+'/jpeg/train'\n\nprint('Display Random Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n plt.subplot(3, 3, i + 1)\n img = plt.imread(os.path.join(img_dir, random_images[i]))\n plt.imshow(img, cmap='gray')\n plt.axis('off')\n \n# Adjust subplot parameters to give specified padding\nplt.tight_layout() ","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.827Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We do see that the JPEG format images vary in sizes","metadata":{}},{"cell_type":"markdown","source":"## Visualizing Images with benign lesions","metadata":{}},{"cell_type":"code","source":"benign = train_df[train_df['benign_malignant']=='benign']\nmalignant = train_df[train_df['benign_malignant']=='malignant']","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.827Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"images = benign['image_name'].values\n\n# Extract 9 random images from it\nrandom_images = [np.random.choice(images+'.jpg') for i in range(9)]\n\n# Location of the image dir\nimg_dir = IMAGE_PATH+'/jpeg/train'\n\nprint('Display benign Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n plt.subplot(3, 3, i + 1)\n img = plt.imread(os.path.join(img_dir, random_images[i]))\n plt.imshow(img, cmap='gray')\n plt.axis('off')\n \n# Adjust subplot parameters to give specified padding\nplt.tight_layout() ","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.827Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Visualizing Images with Malignant lesions","metadata":{}},{"cell_type":"code","source":"images = malignant['image_name'].values\n\n# Extract 9 random images from it\nrandom_images = [np.random.choice(images+'.jpg') for i in range(9)]\n\n# Location of the image dir\nimg_dir = IMAGE_PATH+'/jpeg/train'\n\nprint('Display malignant Images')\n\n# Adjust the size of your images\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n plt.subplot(3, 3, i + 1)\n img = plt.imread(os.path.join(img_dir, random_images[i]))\n plt.imshow(img, cmap='gray')\n plt.axis('off')\n \n# Adjust subplot parameters to give specified padding\nplt.tight_layout() ","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.827Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Histograms\n\nHistograms are a graphical representation showing how frequently various color values occur in the image i.e frequency of pixels intensity values. In a RGB color space, pixel values range from 0 to 255 where 0 stands for black and 255 stands for white. Analysis of a histogram can help us understand thee brightness, contrast and intensity distribution of an image. Now let's look at the histogram of a random selected sample from each category.\n\n### Benign category","metadata":{}},{"cell_type":"code","source":"f = plt.figure(figsize=(16,8))\nf.add_subplot(1,2, 1)\n\nsample_img = benign['image_name'][0]+'.jpg'\nraw_image = plt.imread(os.path.join(img_dir, sample_img))\nplt.imshow(raw_image, cmap='gray')\nplt.colorbar()\nplt.title('Benign Image')\nprint(f\"Image dimensions: {raw_image.shape[0],raw_image.shape[1]}\")\nprint(f\"Maximum pixel value : {raw_image.max():.1f} ; Minimum pixel value:{raw_image.min():.1f}\")\nprint(f\"Mean value of the pixels : {raw_image.mean():.1f} ; Standard deviation : {raw_image.std():.1f}\")\n\nf.add_subplot(1,2, 2)\n\n#_ = plt.hist(raw_image.ravel(),bins = 256, color = 'orange',)\n_ = plt.hist(raw_image[:, :, 0].ravel(), bins = 256, color = 'red', alpha = 0.5)\n_ = plt.hist(raw_image[:, :, 1].ravel(), bins = 256, color = 'Green', alpha = 0.5)\n_ = plt.hist(raw_image[:, :, 2].ravel(), bins = 256, color = 'Blue', alpha = 0.5)\n_ = plt.xlabel('Intensity Value')\n_ = plt.ylabel('Count')\n_ = plt.legend(['Red_Channel', 'Green_Channel', 'Blue_Channel'])\nplt.show()","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.827Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Malignant category","metadata":{}},{"cell_type":"code","source":"f = plt.figure(figsize=(16,8))\nf.add_subplot(1,2, 1)\n\nsample_img = malignant['image_name'][235]+'.jpg'\nraw_image = plt.imread(os.path.join(img_dir, sample_img))\nplt.imshow(raw_image, cmap='gray')\nplt.colorbar()\nplt.title('Malignant Image')\nprint(f\"Image dimensions: {raw_image.shape[0],raw_image.shape[1]}\")\nprint(f\"Maximum pixel value : {raw_image.max():.1f} ; Minimum pixel value:{raw_image.min():.1f}\")\nprint(f\"Mean value of the pixels : {raw_image.mean():.1f} ; Standard deviation : {raw_image.std():.1f}\")\n\nf.add_subplot(1,2, 2)\n\n#_ = plt.hist(raw_image.ravel(),bins = 256, color = 'orange',)\n_ = plt.hist(raw_image[:, :, 0].ravel(), bins = 256, color = 'red', alpha = 0.5)\n_ = plt.hist(raw_image[:, :, 1].ravel(), bins = 256, color = 'Green', alpha = 0.5)\n_ = plt.hist(raw_image[:, :, 2].ravel(), bins = 256, color = 'Blue', alpha = 0.5)\n_ = plt.xlabel('Intensity Value')\n_ = plt.ylabel('Count')\n_ = plt.legend(['Red_Channel', 'Green_Channel', 'Blue_Channel'])\nplt.show()","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.827Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 5 Preprocessing DIOCOM files \n[Digital Imaging and Communications in Medicine (DICOM)](https://en.wikipedia.org/wiki/DICOM) is the standard for the communication and management of medical imaging information and related data.DICOM is most commonly used for storing and transmitting medical images enabling the integration of medical imaging devices such as scanners, servers, workstations, printers, network hardware, and picture archiving and communication systems (PACS) from multiple manufacturers\n\nDICOM images have the extension dcm. A DICOM file has two parts: the header and the dataset. The header contains information on the encapsulated dataset. It consists of a File Preamble, a DICOM prefix, and the File Meta Elements.\nFortunately we have a library in Python called Pydicom which can be used to read the DIOCOM files.pydicom makes it easy to read these complex files into natural pythonic structures for easy manipulation. Modified datasets can be written again to DICOM format files.\n\nThere is very nice [kernel](https://www.kaggle.com/schlerp/getting-to-know-dicom-and-the-data) from a competition couple of years ago which serves as a great introduction to DIOCOM image files.I have borrowed the below mentioned code from there.\nKernel: https://www.kaggle.com/schlerp/getting-to-know-dicom-and-the-data\n","metadata":{}},{"cell_type":"code","source":"print (pydicom.__version__)","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.827Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# https://www.kaggle.com/schlerp/getting-to-know-dicom-and-the-data\ndef show_dcm_info(dataset):\n print(\"Filename.........:\", file_path)\n print(\"Storage type.....:\", dataset.SOPClassUID)\n print()\n\n pat_name = dataset.PatientName\n display_name = pat_name.family_name + \", \" + pat_name.given_name\n print(\"Patient's name......:\", display_name)\n print(\"Patient id..........:\", dataset.PatientID)\n print(\"Patient's Age.......:\", dataset.PatientAge)\n print(\"Patient's Sex.......:\", dataset.PatientSex)\n print(\"Modality............:\", dataset.Modality)\n print(\"Body Part Examined..:\", dataset.BodyPartExamined)\n \n \n \n if 'PixelData' in dataset:\n rows = int(dataset.Rows)\n cols = int(dataset.Columns)\n print(\"Image size.......: {rows:d} x {cols:d}, {size:d} bytes\".format(\n rows=rows, cols=cols, size=len(dataset.PixelData)))\n if 'PixelSpacing' in dataset:\n print(\"Pixel spacing....:\", dataset.PixelSpacing)","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.827Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plot_pixel_array(dataset, figsize=(5,5)):\n plt.figure(figsize=figsize)\n plt.grid(False)\n plt.imshow(dataset.pixel_array)\n plt.show()\n \ni = 1\nnum_to_plot = 5\nfor file_name in os.listdir('../input/siim-isic-melanoma-classification/train/'):\n file_path = os.path.join('../input/siim-isic-melanoma-classification/train/',file_name)\n dataset = pydicom.dcmread(file_path)\n show_dcm_info(dataset)\n plot_pixel_array(dataset)\n \n if i >= num_to_plot:\n break\n \n i += 1","metadata":{"trusted":true,"execution":{"execution_failed":"2025-10-02T12:00:04.827Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Extracting DIOCOM files information in a dataframe\n\n[Gabriel Preda](https://www.kaggle.com/gpreda) has shared the following code in the [discussion forum](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154658) which let's you easily extract the relevant information from the diocom files and store it in a dataframe.","metadata":{}},{"cell_type":"code","source":"# source: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154658\nfolder='train'\nPATH='../input/siim-isic-melanoma-classification/'\n\ndef extract_DICOM_attributes(folder):\n images = list(os.listdir(os.path.join(PATH, folder)))\n df = pd.DataFrame()\n for image in tqdm(images):\n image_name = image.split(\".\")[0]\n dicom_file_path = os.path.join(PATH,folder,image)\n dicom_file_dataset = pydicom.dcmread(dicom_file_path)\n study_date = dicom_file_dataset.StudyDate\n modality = dicom_file_dataset.Modality\n age = dicom_file_dataset.PatientAge\n sex = dicom_file_dataset.PatientSex\n body_part_examined = dicom_file_dataset.BodyPartExamined\n patient_orientation = dicom_file_dataset.PatientOrientation\n photometric_interpretation = dicom_file_dataset.PhotometricInterpretation\n rows = dicom_file_dataset.Rows\n columns = dicom_file_dataset.Columns\n\n df = pd.concat([\n df,pd.DataFrame(\n {\n 'image_name': image_name, \n 'dcm_modality': modality,\n 'dcm_study_date':study_date, \n 'dcm_age': age, \n 'dcm_sex': sex, \n 'dcm_body_part_examined': body_part_examined,\n 'dcm_patient_orientation': patient_orientation, \n 'dcm_photometric_interpretation': photometric_interpretation, \n 'dcm_rows': rows, \n 'dcm_columns': columns\n }, \n \n index=[0],\n )\n ])\n return df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-03T05:22:22.126855Z","iopub.execute_input":"2025-10-03T05:22:22.127209Z","iopub.status.idle":"2025-10-03T05:22:22.136186Z","shell.execute_reply.started":"2025-10-03T05:22:22.127184Z","shell.execute_reply":"2025-10-03T05:22:22.135172Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"extract_DICOM_attributes('train')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-03T05:22:22.426055Z","iopub.execute_input":"2025-10-03T05:22:22.426409Z","iopub.status.idle":"2025-10-03T05:30:19.634549Z","shell.execute_reply.started":"2025-10-03T05:22:22.426384Z","shell.execute_reply":"2025-10-03T05:30:19.632324Z"}},"outputs":[],"execution_count":null}]}