{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":10338,"databundleVersionId":862042,"sourceType":"competition"}],"dockerImageVersionId":30786,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Took help from this [Notebook](https://www.kaggle.com/code/umerfarooq807/exploratory-data-analysis)","metadata":{}},{"cell_type":"markdown","source":"I recently started exploring computer vision tasks and came across a competition focused on medical data. As someone who isn’t familiar with this field, I’ve been reading various kernels to understand the processes better. I’d like to share some key points to focus on when working with image processing tasks in the medical domain.\n\n\n****Do checkout above notebook also****\n","metadata":{}},{"cell_type":"markdown","source":"The main objective of this competition is to detect bounding boxes that indicate the presence of pneumonia in chest radiographs. These radiographs are 2D grayscale images. The bounding box indicates where pneumonia is detected within the image, defined by its position (x, y) and its dimensions (width and height).","metadata":{}},{"cell_type":"markdown","source":"## Import libraries","metadata":{}},{"cell_type":"code","source":"import os \nimport pydicom \nimport numpy as np \nimport pandas as pd\nimport pylab\nfrom IPython.display import display, HTML","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-10-18T09:02:18.010543Z","iopub.execute_input":"2024-10-18T09:02:18.010937Z","iopub.status.idle":"2024-10-18T09:02:18.683856Z","shell.execute_reply.started":"2024-10-18T09:02:18.010901Z","shell.execute_reply":"2024-10-18T09:02:18.682660Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Import Data","metadata":{}},{"cell_type":"markdown","source":"#### Dataset Structure\n\n- **`stage_2_train_images/`**: Contains all the training images in DICOM format.\n- **`stage_2_test_images/`**: Contains all the testing images in DICOM format.\n- **`stage_2_detailed_class_info.csv`**: CSV file containing detailed labels (explored further below).\n- **`stage_2_detailed_train_labels.csv`**: CSV file containing training set patient IDs and labels, including bounding boxes.\n","metadata":{}},{"cell_type":"code","source":"BASE_INPUT_DIR = \"/kaggle/input/rsna-pneumonia-detection-challenge/\"\nTRAIN_IMG_DIR = BASE_INPUT_DIR + \"stage_2_train_images\"\nTEST_IMG_DIR = BASE_INPUT_DIR +  \"stage_2_test_images\"\nTRAIN_LABELS_CSV = BASE_INPUT_DIR + \"stage_2_train_labels.csv\"\nCLASS_INFO_CSV = BASE_INPUT_DIR + \"stage_2_detailed_class_info.csv\"","metadata":{"execution":{"iopub.status.busy":"2024-10-18T09:02:46.994871Z","iopub.execute_input":"2024-10-18T09:02:46.995303Z","iopub.status.idle":"2024-10-18T09:02:47.000616Z","shell.execute_reply.started":"2024-10-18T09:02:46.995263Z","shell.execute_reply":"2024-10-18T09:02:46.999551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_imgs_path = [os.path.join(TRAIN_IMG_DIR, i) for i in os.listdir(TRAIN_IMG_DIR)]\ndisplay(HTML(f\"<strong>Number of images present in the train directory is {len(train_imgs_path)}</strong>\"))","metadata":{"execution":{"iopub.status.busy":"2024-10-18T09:02:48.075794Z","iopub.execute_input":"2024-10-18T09:02:48.076227Z","iopub.status.idle":"2024-10-18T09:02:51.824334Z","shell.execute_reply.started":"2024-10-18T09:02:48.076188Z","shell.execute_reply":"2024-10-18T09:02:51.823165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_imgs_path = [os.path.join(TEST_IMG_DIR, i) for i in os.listdir(TEST_IMG_DIR)]\ndisplay(HTML(f\"<strong>number of images present in the test directory is {len(test_imgs_path)}</strong>\"))","metadata":{"execution":{"iopub.status.busy":"2024-10-18T09:03:33.984489Z","iopub.execute_input":"2024-10-18T09:03:33.984907Z","iopub.status.idle":"2024-10-18T09:03:34.822783Z","shell.execute_reply.started":"2024-10-18T09:03:33.984858Z","shell.execute_reply":"2024-10-18T09:03:34.821627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploring datasets","metadata":{}},{"cell_type":"code","source":"train_labels = pd.read_csv(TRAIN_LABELS_CSV)\nprint(\"shape of the dataframe is \", train_labels.shape)\nprint(train_labels.head())","metadata":{"execution":{"iopub.status.busy":"2024-10-18T09:08:52.027424Z","iopub.execute_input":"2024-10-18T09:08:52.028484Z","iopub.status.idle":"2024-10-18T09:08:52.110298Z","shell.execute_reply.started":"2024-10-18T09:08:52.028430Z","shell.execute_reply":"2024-10-18T09:08:52.109081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels['Target'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2024-10-18T09:09:25.070210Z","iopub.execute_input":"2024-10-18T09:09:25.070650Z","iopub.status.idle":"2024-10-18T09:09:25.085633Z","shell.execute_reply.started":"2024-10-18T09:09:25.070603Z","shell.execute_reply":"2024-10-18T09:09:25.084370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the **`stage_2_train_labels.csv`** file, you will find **`patientId`** along with their corresponding details. The **`Target`** column is binary, containing values of `0` and `1`, which indicate the absence and presence of pneumonia, respectively.\n\n\n\nWhen pneumonia is present (i.e., when **`Target`** is `1`), the file also provides bounding box information, including the values for **`x`**, **`y`**, **`width`**, and **`height`**.\n\nsimilary when there is no pneumonia all the other fields contains **`NaN`**\n","metadata":{}},{"cell_type":"markdown","source":"> **Important Note:**  \n> If pneumonia is detected in various locations, there may be **more than one bounding box** associated with a single patient. This is a crucial point to consider during analysis.\n","metadata":{}},{"cell_type":"markdown","source":"## Exploring DICOM Images","metadata":{}},{"cell_type":"markdown","source":"**DICOM (Digital Imaging and Communications in Medicine)** is a standard file format used for storing and transmitting medical images and related data.\n\nDICOM files contain not only the actual image data but also metadata, which includes important patient information (like name and ID), study details, imaging parameters, and equipment specifics.","metadata":{}},{"cell_type":"code","source":"# Read the DICOM file from stage_2_train_images folder\npatientId = train_labels['patientId'][0]\ndcm_file = os.path.join(TRAIN_IMG_DIR, patientId + '.dcm')\ndcm_data = pydicom.dcmread(dcm_file)\nprint(dcm_data)\n","metadata":{"execution":{"iopub.status.busy":"2024-10-18T09:22:03.580532Z","iopub.execute_input":"2024-10-18T09:22:03.581410Z","iopub.status.idle":"2024-10-18T09:22:03.603951Z","shell.execute_reply.started":"2024-10-18T09:22:03.581365Z","shell.execute_reply":"2024-10-18T09:22:03.602686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"im = dcm_data.pixel_array\nprint(type(im))\nprint(im.dtype)\nprint(im.shape)","metadata":{"execution":{"iopub.status.busy":"2024-10-18T07:00:45.742544Z","iopub.execute_input":"2024-10-18T07:00:45.742986Z","iopub.status.idle":"2024-10-18T07:00:45.779205Z","shell.execute_reply.started":"2024-10-18T07:00:45.742943Z","shell.execute_reply":"2024-10-18T07:00:45.777888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pylab.imshow(im, cmap=pylab.cm.gist_gray)\npylab.axis('off')","metadata":{"execution":{"iopub.status.busy":"2024-10-18T07:01:51.403410Z","iopub.execute_input":"2024-10-18T07:01:51.403840Z","iopub.status.idle":"2024-10-18T07:01:51.707753Z","shell.execute_reply.started":"2024-10-18T07:01:51.403785Z","shell.execute_reply":"2024-10-18T07:01:51.706435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualizing Images with Bouding boxes","metadata":{}},{"cell_type":"code","source":"def parse_data(df):\n    extract_box = lambda row: [row['y'], row['x'], row['height'], row['width']]\n    parsed = {}\n    for n, row in df.iterrows():\n        # --- Initialize patient entry into parsed \n        pid = row['patientId']\n        if pid not in parsed:\n            parsed[pid] = {\n                'dicom': os.path.join(TRAIN_IMG_DIR, pid + '.dcm'),\n                'label': row['Target'],\n                'boxes': []}\n        if parsed[pid]['label'] == 1:\n            parsed[pid]['boxes'].append(extract_box(row))\n\n    return parsed","metadata":{"execution":{"iopub.status.busy":"2024-10-18T09:30:29.336991Z","iopub.execute_input":"2024-10-18T09:30:29.338070Z","iopub.status.idle":"2024-10-18T09:30:29.345300Z","shell.execute_reply.started":"2024-10-18T09:30:29.338025Z","shell.execute_reply":"2024-10-18T09:30:29.343861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"parsed = parse_data(train_labels)","metadata":{"execution":{"iopub.status.busy":"2024-10-18T09:30:30.941304Z","iopub.execute_input":"2024-10-18T09:30:30.941715Z","iopub.status.idle":"2024-10-18T09:30:32.970900Z","shell.execute_reply.started":"2024-10-18T09:30:30.941676Z","shell.execute_reply":"2024-10-18T09:30:32.969980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(parsed['00436515-870c-4b36-a041-de91049b9ab4'])","metadata":{"execution":{"iopub.status.busy":"2024-10-18T09:30:33.439285Z","iopub.execute_input":"2024-10-18T09:30:33.439707Z","iopub.status.idle":"2024-10-18T09:30:33.445265Z","shell.execute_reply.started":"2024-10-18T09:30:33.439665Z","shell.execute_reply":"2024-10-18T09:30:33.444177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def Draw(data):\n    img_path = data['dicom']\n    img = pydicom.dcmread(img_path).pixel_array\n    img = np.stack([img] * 3, axis=2) # converts the image into 3 dimenssions\n    \n    for box in data['boxes']:\n        rgb = np.floor(np.random.rand(3) * 256).astype('int') # this lines gives 3 random numbers\n        img = Overlay(img=img, box=box, rgb=rgb, stroke=6)\n        \n    pylab.imshow(img, cmap=pylab.cm.gist_gray)\n    pylab.axis('off')\n    \ndef Overlay(img, box, rgb, stroke=1):\n    box = [int(i) for i in box]\n    y1, x1, height, width = box\n    y2 = y1 + height\n    x2 = x1 + width\n    img[y1:y1 + stroke, x1:x2] = rgb  # Top edge\n    img[y2:y2 + stroke, x1:x2] = rgb  # Bottom edge\n    img[y1:y2, x1:x1 + stroke] = rgb   # Left edge\n    img[y1:y2, x2:x2 + stroke] = rgb   # Right edge\n    return img","metadata":{"execution":{"iopub.status.busy":"2024-10-18T09:56:12.601033Z","iopub.execute_input":"2024-10-18T09:56:12.601893Z","iopub.status.idle":"2024-10-18T09:56:12.611251Z","shell.execute_reply.started":"2024-10-18T09:56:12.601846Z","shell.execute_reply":"2024-10-18T09:56:12.609894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# draw(parsed['00436515-870c-4b36-a041-de91049b9ab4'])\nDraw(parsed['00436515-870c-4b36-a041-de91049b9ab4'])","metadata":{"execution":{"iopub.status.busy":"2024-10-18T09:56:14.278635Z","iopub.execute_input":"2024-10-18T09:56:14.279373Z","iopub.status.idle":"2024-10-18T09:56:14.660059Z","shell.execute_reply.started":"2024-10-18T09:56:14.279331Z","shell.execute_reply":"2024-10-18T09:56:14.659022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## About Labels","metadata":{}},{"cell_type":"markdown","source":"In the **`stage_2_detailed_class_info.csv`** file, you will find two key columns: **`patientID`** and **`class`**.\n\n- For patients diagnosed with pneumonia, the **`class`** is labeled as **Lung Opacity**.\n- For patients without a bounding box, there are two possible categories:\n  - **Normal**\n  - **No Lung Opacity / Not Normal**\n\n","metadata":{}},{"cell_type":"code","source":"class_info = pd.read_csv(CLASS_INFO_CSV)\nprint(\"shape of the dataframe is \", class_info.shape)\nprint(class_info.head())\nprint(class_info['class'].value_counts())","metadata":{"execution":{"iopub.status.busy":"2024-10-18T10:13:40.117153Z","iopub.execute_input":"2024-10-18T10:13:40.117978Z","iopub.status.idle":"2024-10-18T10:13:40.165755Z","shell.execute_reply.started":"2024-10-18T10:13:40.117937Z","shell.execute_reply":"2024-10-18T10:13:40.164597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"summary = {}\nfor n, row in class_info.iterrows():\n    if row['class'] not in summary:\n        summary[row['class']] = 0\n    summary[row['class']] += 1\n    \nprint(summary)","metadata":{"execution":{"iopub.status.busy":"2024-10-18T07:18:30.665450Z","iopub.execute_input":"2024-10-18T07:18:30.665904Z","iopub.status.idle":"2024-10-18T07:18:32.499678Z","shell.execute_reply.started":"2024-10-18T07:18:30.665862Z","shell.execute_reply":"2024-10-18T07:18:32.498467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"PatientId is : {class_info['patientId'][0]} | class is {class_info['class'][0]}\")\nDraw(parsed[class_info['patientId'][0]])","metadata":{"execution":{"iopub.status.busy":"2024-10-18T10:26:56.914788Z","iopub.execute_input":"2024-10-18T10:26:56.915410Z","iopub.status.idle":"2024-10-18T10:26:57.309638Z","shell.execute_reply.started":"2024-10-18T10:26:56.915358Z","shell.execute_reply":"2024-10-18T10:26:57.307311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"PatientId is : {class_info['patientId'][4]} | class is {class_info['class'][4]}\")\nDraw(parsed[class_info['patientId'][4]])","metadata":{"execution":{"iopub.status.busy":"2024-10-18T10:28:47.537315Z","iopub.execute_input":"2024-10-18T10:28:47.537735Z","iopub.status.idle":"2024-10-18T10:28:47.924428Z","shell.execute_reply.started":"2024-10-18T10:28:47.537696Z","shell.execute_reply":"2024-10-18T10:28:47.923325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}