{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"\n# print(\"\\n... IMPORTS STARTING ...\\n\")\n# print(\"\\n\\tVERSION INFORMATION\")\n# # Machine Learning and Data Science Imports\n# import tensorflow as tf; print(f\"\\t\\t– TENSORFLOW VERSION: {tf.__version__}\");\n# import tensorflow_addons as tfa; print(f\"\\t\\t– TENSORFLOW ADDONS VERSION: {tfa.__version__}\");\n\n\n# # Built In Imports\n# from kaggle_datasets import KaggleDatasets\n# from collections import Counter\n# from datetime import datetime\n# from glob import glob\n# import warnings\n# import requests\n# import imageio\n# import IPython\n# import urllib\n# import zipfile\n# import pickle\n# import random\n# import shutil\n# import string\n# import math\n# import time\n# import gzip\n# import ast\n# import sys\n# import io\n# import gc\n# import re\n\n# # Visualization Imports\n# from matplotlib.colors import ListedColormap\n# import matplotlib.patches as patches\n# import plotly.graph_objects as go\n# import plotly.express as px\n# import seaborn as sns\n# import matplotlib; print(f\"\\t\\t– MATPLOTLIB VERSION: {matplotlib.__version__}\");\n# import plotly\n# import PIL\n    \n# print(\"\\n\\n... IMPORTS COMPLETE ...\\n\")\n\n","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":41.017298,"end_time":"2021-02-05T02:18:25.770952","exception":false,"start_time":"2021-02-05T02:17:44.753654","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-03T20:19:41.779024Z","iopub.execute_input":"2021-07-03T20:19:41.779397Z","iopub.status.idle":"2021-07-03T20:19:41.784500Z","shell.execute_reply.started":"2021-07-03T20:19:41.779367Z","shell.execute_reply":"2021-07-03T20:19:41.783238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Background\n\nIn this competition, we are identifying and localizing COVID-19 abnormalities on chest radiographs. <br>**This is an object detection and classification problem.**\n\nFor each test image, you will be predicting a bounding box and class for all findings. \n* If you predict that there are no findings, you should create a prediction of **`\"none 1 0 0 1 1\"`** \n    * \"none\" is the class ID for no finding, and this provides a one-pixel bounding box with a confidence of 1.0\n\n\nTo make a prediction of one of the above labels, create a prediction string similar to the \"none\" class above: \n* i.e. **`atypical 1 0 0 1 1`**\n\n---\n\n**MESSAGE FROM THE COMPETITION HOST ON LABEL AND BBOX DETAILS:**\n\nIn this challenge, the chest radiographs (CXRs) were categorized using a specific grading schema, based on a published paper:\n\n[**Litmanovich DE, Chung M, Kirkbride RR, Kicska G, Kanne JP. Review of chest radiograph findings of COVID-19 pneumonia and suggested reporting language. Journal of thoracic imaging. 2020 Nov 14;35(6):354-60.**](https://journals.lww.com/thoracicimaging/Fulltext/2020/11000/Review_of_Chest_Radiograph_Findings_of_COVID_19.4.aspx)\n\nPer the grading schema, chest radiographs are classified into one of four categories, which are mutually exclusive:\n\n1. **Typical Appearance**: Multifocal bilateral, peripheral opacities with rounded morphology, lower lung–predominant distribution\n2. **Indeterminate Appearance**: Absence of typical findings AND unilateral, central or upper lung predominant distribution\n3. **Atypical Appearance**: Pneumothorax, pleural effusion, pulmonary edema, lobar consolidation, solitary lung nodule or mass, diffuse tiny nodules, cavity\n4. **Negative for Pneumonia**: No lung opacities\n\nBounding boxes were placed on lung opacities, whether typical or indeterminate. Bounding boxes were also placed on some atypical findings including solitary lobar consolidation, nodules/masses, and cavities. Bounding boxes were not placed on pleural effusions, or pneumothoraces. No bounding boxes were placed for the negative for pneumonia category.\n\nIn cases of multiple adjacent opacities, we opted for one large bounding box, rather than multiple adjacent smaller boxes, to improve consistency in the labeling.\n\nAnnotators did have access to the COVID status for each patient, but were asked to adhere to the grading system above irrespective of the status. As such, some patients who were COVID negative still had chest radiographs with typical appearances. Similarly, some patients who were COVID positive had atypical appearances, or were negative for pneumonia (no lung opacities), because the grading system is based off the chest radiographic findings alone.\n\nThe goal in this challenge is to determine the appropriate category for each radiograph, as well as localize the lung opacities with a bounding box prediction.\n\n---\n\nThe images are in DICOM format, which means they contain additional data that might be useful for visualizing and classifying.\nNote that the images are in **DICOM** format, which means they contain additional data that might be useful for visualizing and classifying.\n\n![Example Radiographs](https://i.imgur.com/QWmbhXx.png)\n\n<br>\n\n<b style=\"text-decoration: underline; font-family: Verdana;\">DATASET INFORMATION</b>\n\nThe **train dataset** comprises **`6,334`** chest scans in **DICOM** format, which were de-identified to protect patient privacy. \n\nNote that all images are stored in paths with the form **`study/series/image`**. \n* The **`study`** ID here relates directly to the study-level predictions\n* the **`image`** ID is the ID used for image-level predictions\n\nThe **test dataset** is of roughly the same scale as the training dataset. \n* As this is a kernels only competition we shsould plan accordingly\n* i.e. we should be able to infer on the entirety of the training dataset within the submission kernel\n\n<br>\n\n<b style=\"text-decoration: underline; font-family: Verdana;\">DATA FILES</b>\n> **`train_study_level.csv`**\n> * **`id`** - unique study identifier\n> * **Negative for Pneumonia** - **`1`** if the study is negative for pneumonia, **`0`** otherwise\n> * **Typical Appearance** - **`1`** if the study has this appearance, **`0`** otherwise\n> * **Indeterminate Appearance**  - **`1`** if the study has this appearance, **`0`** otherwise\n> * **Atypical Appearance**  - **`1`** if the study has this appearance, **`0`** otherwise\n\n> **`train_image_level.csv`**\n> * **`id`** - unique image identifier\n> * **`boxes`** - bounding boxes in easily-readable dictionary format\n> * **`label`** - the correct prediction label for the provided bounding boxes","metadata":{"papermill":{"duration":0.04413,"end_time":"2021-02-05T02:18:25.953034","exception":false,"start_time":"2021-02-05T02:18:25.908904","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"<br>\n\n<h2 style=\"font-family: Verdana; font-size: 20px; font-style: normal; font-weight: normal; text-decoration: none; text-transform: none; letter-spacing: 2px; color: darkred; background-color: #ffffff;\">1.2  THE GOAL</h2>\n\n---\n\nIn this competition, you’ll identify and localize COVID-19 abnormalities on chest radiographs. In particular, you'll categorize the radiographs as one of a possible **`4`** categories. \n\nIn this competition, we are making predictions at both a study (multi-image) and image level.\n* **`negative for pneumonia`** or **`typical`**, **`indeterminate`**, or **`atypical`** \n     \nYou'll work with a dataset consisting of **`8,781`** scans that have been annotated by experienced radiologists. You can train your model with **`6,334`** independently-labeled images and you will be evaluated on a test set of **`2,447`** images. \n\nThe challenge uses the standard PASCAL VOC 2010 mean Average Precision (mAP) at IoU > 0.5.\n* Note that the linked document describes VOC 2012, which differs in some minor ways (e.g. there is no concept of \"difficult\" classes in VOC 2010). The P/R curve and AP calculations remain the same.\n\n","metadata":{"papermill":{"duration":0.044853,"end_time":"2021-02-05T02:18:26.044977","exception":false,"start_time":"2021-02-05T02:18:26.000124","status":"completed"},"tags":[]}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" ### ADDITIONAL INFORMATION ON ABNORMALITIES\n\nNegative for Pneumonia =  No lung opacities\n\nTypical Appearance = Multifocal bilateral, peripheral opacities with rounded morphology, lower lung–predominant distribution\n\nIndeterminate Appearance = Absence of typical findings AND unilateral, central or upper lung predominant distribution\n\nAtypical Appearance = Pneumothorax, pleural effusion, pulmonary edema, lobar consolidation, solitary lung nodule or mass, diffuse tiny nodules, cavity","metadata":{"papermill":{"duration":0.046066,"end_time":"2021-02-05T02:18:26.137054","exception":false,"start_time":"2021-02-05T02:18:26.090988","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# TRAIN_CSV_PATH = \"/kaggle/input/siim-covid19-updated-train-labels/updated_train_labels.csv\"\n# SS_CSV_PATH = \"/kaggle/input/siim-covid19-updated-train-labels/updated_sample_submission.csv\"\n\n# print(\"\\n\\nCOMBINED AND EXPLODED TRAIN DATAFRAME\\n\\n\")\n# train_df = pd.read_csv(TRAIN_CSV_PATH)\n# display(train_df)\n\n# print(\"\\n\\nSAMPLE SUBMISSION DATAFRAME\\n\\n\")\n# ss_df = pd.read_csv(SS_CSV_PATH)\n# display(ss_df)\n\n","metadata":{"papermill":{"duration":0.717244,"end_time":"2021-02-05T02:18:26.987446","exception":false,"start_time":"2021-02-05T02:18:26.270202","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-19T11:58:08.426627Z","iopub.execute_input":"2021-06-19T11:58:08.427068Z","iopub.status.idle":"2021-06-19T11:58:08.441907Z","shell.execute_reply.started":"2021-06-19T11:58:08.427027Z","shell.execute_reply":"2021-06-19T11:58:08.440847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### This work is adaped from existing notebooks of:\n\nhttps://www.kaggle.com/dschettler8845/covid-detection-studies-with-multiple-images-viz\n\nhttps://www.kaggle.com/c/siim-covid19-detection/discussion/240878\n\nhttps://www.kaggle.com/c/siim-covid19-detection/discussion/246597\n\nfor getting started and also see multi-image cases.\n\n-----------------------\n\nhttps://www.kaggle.com/jaideepvalani/basic-exploration-eda-duplicate-nonduplicate\n\nhttps://www.kaggle.com/devanshchowdhury/eda-understand-data\n\nfor further EDA on study/image level rows\n\n\n-----------------------\n\nhttps://www.kaggle.com/aleksandramowio/getting-started-simple-eda\n\nto read image data/voxels\n\n\n\n-----------------------\n\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:21:15.064989Z","iopub.execute_input":"2021-07-03T19:21:15.065395Z","iopub.status.idle":"2021-07-03T19:21:15.074921Z","shell.execute_reply.started":"2021-07-03T19:21:15.065360Z","shell.execute_reply":"2021-07-03T19:21:15.073683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df = pd.read_csv(\"../input/siim-covid19-detection/train_study_level.csv\")\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:21:18.001645Z","iopub.execute_input":"2021-07-03T19:21:18.002078Z","iopub.status.idle":"2021-07-03T19:21:18.025665Z","shell.execute_reply.started":"2021-07-03T19:21:18.002040Z","shell.execute_reply":"2021-07-03T19:21:18.024696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:21:20.253696Z","iopub.execute_input":"2021-07-03T19:21:20.254100Z","iopub.status.idle":"2021-07-03T19:21:20.284281Z","shell.execute_reply.started":"2021-07-03T19:21:20.254066Z","shell.execute_reply":"2021-07-03T19:21:20.283000Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df.sum()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:21:23.725881Z","iopub.execute_input":"2021-07-03T19:21:23.726238Z","iopub.status.idle":"2021-07-03T19:21:23.755707Z","shell.execute_reply.started":"2021-07-03T19:21:23.726208Z","shell.execute_reply":"2021-07-03T19:21:23.754566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df.describe()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:21:24.365201Z","iopub.execute_input":"2021-07-03T19:21:24.365570Z","iopub.status.idle":"2021-07-03T19:21:24.401909Z","shell.execute_reply.started":"2021-07-03T19:21:24.365537Z","shell.execute_reply":"2021-07-03T19:21:24.400911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df[~study_df.id.str.contains(\"study\")]","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:21:27.539553Z","iopub.execute_input":"2021-07-03T19:21:27.539953Z","iopub.status.idle":"2021-07-03T19:21:27.556271Z","shell.execute_reply.started":"2021-07-03T19:21:27.539923Z","shell.execute_reply":"2021-07-03T19:21:27.555099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df[\"id\"] = study_df[\"id\"].str.replace(\"_study\", \"\")\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:21:32.476745Z","iopub.execute_input":"2021-07-03T19:21:32.477191Z","iopub.status.idle":"2021-07-03T19:21:32.488734Z","shell.execute_reply.started":"2021-07-03T19:21:32.477160Z","shell.execute_reply":"2021-07-03T19:21:32.487392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:21:34.419048Z","iopub.execute_input":"2021-07-03T19:21:34.419419Z","iopub.status.idle":"2021-07-03T19:21:34.430824Z","shell.execute_reply.started":"2021-07-03T19:21:34.419387Z","shell.execute_reply":"2021-07-03T19:21:34.429531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:21:35.785389Z","iopub.execute_input":"2021-07-03T19:21:35.785773Z","iopub.status.idle":"2021-07-03T19:21:35.790470Z","shell.execute_reply.started":"2021-07-03T19:21:35.785728Z","shell.execute_reply":"2021-07-03T19:21:35.789230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_absolute_file_paths(directory):\n    all_abs_file_paths = []\n    for dirpath,_,filenames in os.walk(directory):\n        for f in filenames:\n            all_abs_file_paths.append(os.path.abspath(os.path.join(dirpath, f)))\n    return all_abs_file_paths","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:21:37.984318Z","iopub.execute_input":"2021-07-03T19:21:37.984712Z","iopub.status.idle":"2021-07-03T19:21:37.993469Z","shell.execute_reply.started":"2021-07-03T19:21:37.984680Z","shell.execute_reply":"2021-07-03T19:21:37.992360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tqdm.notebook import tqdm; tqdm.pandas();\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:21:38.677596Z","iopub.execute_input":"2021-07-03T19:21:38.678014Z","iopub.status.idle":"2021-07-03T19:21:38.684982Z","shell.execute_reply.started":"2021-07-03T19:21:38.677980Z","shell.execute_reply":"2021-07-03T19:21:38.683974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df[\"study_dir\"] = \"/kaggle/input/siim-covid19-detection/train/\"+study_df[\"id\"]\nstudy_df[\"images_per_study\"] = study_df.study_dir.progress_apply(lambda x: len(get_absolute_file_paths(x)))\n# study_df[\"images_per_study\"] = study_df.study_dir.apply(lambda x: len(get_absolute_file_paths(x)))\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:21:55.073730Z","iopub.execute_input":"2021-07-03T19:21:55.074182Z","iopub.status.idle":"2021-07-03T19:22:22.357249Z","shell.execute_reply.started":"2021-07-03T19:21:55.074127Z","shell.execute_reply":"2021-07-03T19:22:22.355868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:22:22.359271Z","iopub.execute_input":"2021-07-03T19:22:22.359725Z","iopub.status.idle":"2021-07-03T19:22:22.374500Z","shell.execute_reply.started":"2021-07-03T19:22:22.359679Z","shell.execute_reply":"2021-07-03T19:22:22.373293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# study_df.images_per_study.describe()\nstudy_df.images_per_study.value_counts()\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:22:22.376397Z","iopub.execute_input":"2021-07-03T19:22:22.376731Z","iopub.status.idle":"2021-07-03T19:22:22.394653Z","shell.execute_reply.started":"2021-07-03T19:22:22.376699Z","shell.execute_reply":"2021-07-03T19:22:22.393501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### where is train data?","metadata":{}},{"cell_type":"code","source":"row = study_df.iloc[0]","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:25:09.981133Z","iopub.execute_input":"2021-07-03T19:25:09.981562Z","iopub.status.idle":"2021-07-03T19:25:09.987395Z","shell.execute_reply.started":"2021-07-03T19:25:09.981527Z","shell.execute_reply":"2021-07-03T19:25:09.986375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"row","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:25:10.869194Z","iopub.execute_input":"2021-07-03T19:25:10.869939Z","iopub.status.idle":"2021-07-03T19:25:10.878562Z","shell.execute_reply.started":"2021-07-03T19:25:10.869891Z","shell.execute_reply":"2021-07-03T19:25:10.877169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ! ls /kaggle/input/siim-covid19-detection/train/ | grep 00086460a852\n! ls /kaggle/input/siim-covid19-detection/train/00086460a852/9e8302230c91/65761e66de9f.dcm\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:25:13.327078Z","iopub.execute_input":"2021-07-03T19:25:13.327449Z","iopub.status.idle":"2021-07-03T19:25:14.116603Z","shell.execute_reply.started":"2021-07-03T19:25:13.327416Z","shell.execute_reply":"2021-07-03T19:25:14.115648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### visualise a dicom file ?","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### study can have multi images.","metadata":{}},{"cell_type":"code","source":"multiple_images_per_study_df = study_df[study_df.images_per_study>1].reset_index(drop=True)\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:25:46.758537Z","iopub.execute_input":"2021-07-03T19:25:46.759188Z","iopub.status.idle":"2021-07-03T19:25:46.772117Z","shell.execute_reply.started":"2021-07-03T19:25:46.759122Z","shell.execute_reply":"2021-07-03T19:25:46.770985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"multiple_images_per_study_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:25:48.643878Z","iopub.execute_input":"2021-07-03T19:25:48.644531Z","iopub.status.idle":"2021-07-03T19:25:48.658303Z","shell.execute_reply.started":"2021-07-03T19:25:48.644468Z","shell.execute_reply":"2021-07-03T19:25:48.657231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"multiple_images_per_study_df.shape","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:25:57.360137Z","iopub.execute_input":"2021-07-03T19:25:57.360500Z","iopub.status.idle":"2021-07-03T19:25:57.366387Z","shell.execute_reply.started":"2021-07-03T19:25:57.360470Z","shell.execute_reply":"2021-07-03T19:25:57.365261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:26:35.857045Z","iopub.execute_input":"2021-07-03T19:26:35.857423Z","iopub.status.idle":"2021-07-03T19:26:35.861339Z","shell.execute_reply.started":"2021-07-03T19:26:35.857390Z","shell.execute_reply":"2021-07-03T19:26:35.860510Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df = pd.read_csv(\"/kaggle/input/siim-covid19-detection/train_image_level.csv\")\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:26:36.455931Z","iopub.execute_input":"2021-07-03T19:26:36.456440Z","iopub.status.idle":"2021-07-03T19:26:36.508081Z","shell.execute_reply.started":"2021-07-03T19:26:36.456396Z","shell.execute_reply":"2021-07-03T19:26:36.507228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:26:38.499843Z","iopub.execute_input":"2021-07-03T19:26:38.500388Z","iopub.status.idle":"2021-07-03T19:26:38.515090Z","shell.execute_reply.started":"2021-07-03T19:26:38.500336Z","shell.execute_reply":"2021-07-03T19:26:38.513885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_image_ids = image_df.id.str.replace(\"_image\", \"\")\nbbox_image_ids = image_df.dropna().id.str.replace(\"_image\", \"\")","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:26:42.397325Z","iopub.execute_input":"2021-07-03T19:26:42.397688Z","iopub.status.idle":"2021-07-03T19:26:42.421525Z","shell.execute_reply.started":"2021-07-03T19:26:42.397657Z","shell.execute_reply":"2021-07-03T19:26:42.420177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# all_image_ids\nlen(all_image_ids)","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:28:56.728069Z","iopub.execute_input":"2021-07-03T19:28:56.728666Z","iopub.status.idle":"2021-07-03T19:28:56.734628Z","shell.execute_reply.started":"2021-07-03T19:28:56.728614Z","shell.execute_reply":"2021-07-03T19:28:56.733650Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_image_ids[:10]","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:33:08.508042Z","iopub.execute_input":"2021-07-03T19:33:08.508455Z","iopub.status.idle":"2021-07-03T19:33:08.519532Z","shell.execute_reply.started":"2021-07-03T19:33:08.508419Z","shell.execute_reply":"2021-07-03T19:33:08.518527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# bbox_image_ids\nlen(bbox_image_ids)","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:29:03.660181Z","iopub.execute_input":"2021-07-03T19:29:03.660776Z","iopub.status.idle":"2021-07-03T19:29:03.666033Z","shell.execute_reply.started":"2021-07-03T19:29:03.660717Z","shell.execute_reply":"2021-07-03T19:29:03.664867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bbox_image_ids[:10]","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:33:17.038460Z","iopub.execute_input":"2021-07-03T19:33:17.038864Z","iopub.status.idle":"2021-07-03T19:33:17.046388Z","shell.execute_reply.started":"2021-07-03T19:33:17.038828Z","shell.execute_reply":"2021-07-03T19:33:17.045265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### for this need to read dicom files ","metadata":{}},{"cell_type":"code","source":"import numpy as np; print(f\"\\t\\t– NUMPY VERSION: {np.__version__}\");\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:27:10.618571Z","iopub.execute_input":"2021-07-03T19:27:10.619138Z","iopub.status.idle":"2021-07-03T19:27:10.624831Z","shell.execute_reply.started":"2021-07-03T19:27:10.619072Z","shell.execute_reply":"2021-07-03T19:27:10.623780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pydicom\nfrom pydicom import dcmread\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:27:11.217456Z","iopub.execute_input":"2021-07-03T19:27:11.218091Z","iopub.status.idle":"2021-07-03T19:27:11.544561Z","shell.execute_reply.started":"2021-07-03T19:27:11.218037Z","shell.execute_reply":"2021-07-03T19:27:11.543717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Installs\n# !cp /kaggle/input/gdcm-conda-install/gdcm.tar .\n# !tar -xvzf gdcm.tar\n# !conda install --offline ./gdcm/gdcm-2.8.9-py37h71b2a6d_0.tar.bz2\n# !rm -rf ./gdcm.tar\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T20:19:56.573406Z","iopub.execute_input":"2021-07-03T20:19:56.573814Z","iopub.status.idle":"2021-07-03T20:19:56.578477Z","shell.execute_reply.started":"2021-07-03T20:19:56.573770Z","shell.execute_reply":"2021-07-03T20:19:56.577184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\ndef dicom2array(path, voi_lut=True, fix_monochrome=True):\n    \"\"\" Convert dicom file to numpy array \n    \n    Args:\n        path (str): Path to the dicom file to be converted\n        voi_lut (bool): Whether or not VOI LUT is available\n        fix_monochrome (bool): Whether or not to apply monochrome fix\n        \n    Returns:\n        Numpy array of the respective dicom file \n        \n    \"\"\"\n    # Use the pydicom library to read the dicom file\n    dicom = pydicom.read_file(path)\n    \n    # VOI LUT (if available by DICOM device) is used to \n    # transform raw DICOM data to \"human-friendly\" view\n    if voi_lut:\n        data = apply_voi_lut(dicom.pixel_array, dicom)\n    else:\n        data = dicom.pixel_array\n        \n    # The XRAY may look inverted\n    #   - If we want to fix this we can\n    if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n        data = np.amax(data) - data\n    \n    # Normalize the image array and return\n    data = data - np.min(data)\n    data = data / np.max(data)\n    data = (data * 255).astype(np.uint8)\n    return data\n'''\n\n\ndef dicom2array_2(fname, target_size=512, use_clahe=True, clip_limit=2., grid_size=(8,8)):\n    dicom = pydicom.dcmread(fname)\n    \n    try:\n        data = dicom.pixel_array\n    except Exception as err:\n        print('exception seen=', err)\n        data = None\n    \n#     data = apply_voi_lut(dicom.pixel_array, dicom)\n#     im = data - np.min(data)\n#     im = 255. * im / np.max(im)\n    \n#     if dicom.PhotometricInterpretation == \"MONOCHROME1\": # check for inverted image\n#         im = 255. - im\n    \n#     if use_clahe:\n#         clahe = cv2.createCLAHE(clipLimit=clip_limit, tileGridSize=grid_size)\n#         climg = clahe.apply(im.astype('uint8'))\n#         img = Image.fromarray(climg.astype('uint8'), 'L')\n#     else:\n#         img = Image.fromarray(im.astype('uint8'), 'L')\n#     org_size = img.size\n    \n#     if max(img.size) > target_size:\n#         img.thumbnail((target_size, target_size), Image.ANTIALIAS)\n    \n#     return np.asarray(img)\n    return data\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:42:28.139845Z","iopub.execute_input":"2021-07-03T19:42:28.140513Z","iopub.status.idle":"2021-07-03T19:42:28.147251Z","shell.execute_reply.started":"2021-07-03T19:42:28.140466Z","shell.execute_reply":"2021-07-03T19:42:28.146408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import cv2\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:41:51.659440Z","iopub.execute_input":"2021-07-03T19:41:51.659819Z","iopub.status.idle":"2021-07-03T19:41:51.664203Z","shell.execute_reply.started":"2021-07-03T19:41:51.659786Z","shell.execute_reply":"2021-07-03T19:41:51.663084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from PIL import Image\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T19:41:51.883015Z","iopub.execute_input":"2021-07-03T19:41:51.883584Z","iopub.status.idle":"2021-07-03T19:41:51.887260Z","shell.execute_reply.started":"2021-07-03T19:41:51.883550Z","shell.execute_reply":"2021-07-03T19:41:51.886437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for ix, case in multiple_images_per_study_df.head(20).iterrows():\n    print('new case= \\n ', case)\n    dir_path = case.study_dir\n    image_paths = get_absolute_file_paths(dir_path)\n    \n    selected = []\n    rejected = []\n    \n    if len(image_paths) <= 4:\n        plt.figure(figsize=(18,4))\n        print('\\n dir_path=', dir_path)\n        \n        study_id = dir_path.rsplit('/', 1)[1]\n        print('\\n study_id=', study_id)\n        \n        \n        plt.suptitle(f\"\\n\\nSTUDY: {study_id}\", fontsize=16, fontweight=\"bold\")\n        print('\\n image_paths=', image_paths)\n        \n        for j, x in enumerate(image_paths):\n            print(j, x)\n            if any(True for xx in all_image_ids if xx in x):\n                title = \"\\nINCLUDED IN IMAGE LEVEL\"\n                if any(True for xx in bbox_image_ids if xx in x):\n                    title += \" - W/ BBOX!!!\"\n                    selected.append(x)\n                    \n            else:\n                title = \"xxxxxxxxxxxxxxxxxxxxxxx\"\n                \n            data = dicom2array_2(x)\n            \n            if data is not None:\n                plt.subplot(1,4,j+1)\n                plt.imshow(data)\n                plt.axis(False)\n                plt.title(title, fontweight=\"bold\")\n                \n        print('\\n selected=', selected)\n        selected_img = selected[0]\n        selected_img_id = selected_img.split('/')[-1]\n        selected_img_id = selected_img_id.replace('.dcm', '_image')\n        print('selected_img_id=', selected_img_id)\n\n        docs_img_selected = image_df[(image_df.StudyInstanceUID == study_id) & (image_df.id == selected_img_id)] \n        print('\\n docs_img_selected =\\n', docs_img_selected.iloc[0])\n        \n            \n    else:\n        print('not processing this right now')\n    \n    '''\n    elif len(image_paths)<=8:\n        plt.figure(figsize=(18,8))\n        plt.suptitle(f\"\\n\\nSTUDY: {dir_path.rsplit('/', 1)[1]}\", fontsize=16, fontweight=\"bold\")\n        for j, x in enumerate(image_paths):\n            if any(True for xx in all_image_ids if xx in x):\n                title = \"\\nINCLUDED IN IMAGE LEVEL\"\n                if any(True for xx in bbox_image_ids if xx in x):\n                    title += \" - W/ BBOX!!!\"\n            else:\n                title = \"xxxxxxxxxxxxxxxxxxxxxxx\"\n            plt.subplot(2,4,j+1)\n            plt.imshow(dicom2array_2(x))\n            plt.axis(False)\n            plt.title(title, fontweight=\"bold\")\n    else:\n        plt.figure(figsize=(18,12))\n        plt.suptitle(f\"\\n\\nSTUDY: {dir_path.rsplit('/', 1)[1]}\", fontsize=16, fontweight=\"bold\")\n        for j, x in enumerate(image_paths):\n            if any(True for xx in all_image_ids if xx in x):\n                title = \"\\nINCLUDED IN IMAGE LEVEL\"\n                if any(True for xx in bbox_image_ids if xx in x):\n                    title += \" - W/ BBOX!!!\"\n            else:\n                title = \"xxxxxxxxxxxxxxxxxxxxxxx\"\n            plt.subplot(3,4,j+1)\n            plt.imshow(dicom2array_2(x))\n            plt.title(title, fontweight=\"bold\")\n            plt.axis(False)\n    '''\n\n    plt.tight_layout()\n    plt.show()\n#     break\n\n    print('===============================\\n')","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","papermill":{"duration":0.083595,"end_time":"2021-02-05T02:18:27.209879","exception":false,"start_time":"2021-02-05T02:18:27.126284","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-03T20:15:40.810158Z","iopub.execute_input":"2021-07-03T20:15:40.810533Z","iopub.status.idle":"2021-07-03T20:15:56.256485Z","shell.execute_reply.started":"2021-07-03T20:15:40.810500Z","shell.execute_reply":"2021-07-03T20:15:56.254631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### viz  another image","metadata":{}},{"cell_type":"code","source":"study_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:15.553968Z","iopub.execute_input":"2021-06-19T11:58:15.554273Z","iopub.status.idle":"2021-06-19T11:58:15.568617Z","shell.execute_reply.started":"2021-06-19T11:58:15.554239Z","shell.execute_reply":"2021-06-19T11:58:15.567387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nfor dir_path in study_df.study_dir.values:\n    print('dir_path=', dir_path)\n    image_paths = get_absolute_file_paths(dir_path)\n    if len(image_paths)<=4:\n        plt.figure(figsize=(18,4))\n        plt.suptitle(f\"\\n\\nSTUDY: {dir_path.rsplit('/', 1)[1]}\", fontsize=16, fontweight=\"bold\")\n        for j, x in enumerate(image_paths):\n            print(j,x)\n            if any(True for xx in all_image_ids if xx in x):\n                title = \"\\nINCLUDED IN IMAGE LEVEL\"\n                if any(True for xx in bbox_image_ids if xx in x):\n                    title += \" - W/ BBOX!!!\"\n            else:\n                title = \"xxxxxxxxxxxxxxxxxxxxxxx\"\n            plt.subplot(1,4,j+1)\n            plt.imshow(dicom2array_2(x))\n            plt.axis(False)\n            plt.title(title, fontweight=\"bold\")\n    elif len(image_paths)<=8:\n        plt.figure(figsize=(18,8))\n        plt.suptitle(f\"\\n\\nSTUDY: {dir_path.rsplit('/', 1)[1]}\", fontsize=16, fontweight=\"bold\")\n        for j, x in enumerate(image_paths):\n            if any(True for xx in all_image_ids if xx in x):\n                title = \"\\nINCLUDED IN IMAGE LEVEL\"\n                if any(True for xx in bbox_image_ids if xx in x):\n                    title += \" - W/ BBOX!!!\"\n            else:\n                title = \"xxxxxxxxxxxxxxxxxxxxxxx\"\n            plt.subplot(2,4,j+1)\n            plt.imshow(dicom2array_2(x))\n            plt.axis(False)\n            plt.title(title, fontweight=\"bold\")\n    else:\n        plt.figure(figsize=(18,12))\n        plt.suptitle(f\"\\n\\nSTUDY: {dir_path.rsplit('/', 1)[1]}\", fontsize=16, fontweight=\"bold\")\n        for j, x in enumerate(image_paths):\n            if any(True for xx in all_image_ids if xx in x):\n                title = \"\\nINCLUDED IN IMAGE LEVEL\"\n                if any(True for xx in bbox_image_ids if xx in x):\n                    title += \" - W/ BBOX!!!\"\n            else:\n                title = \"xxxxxxxxxxxxxxxxxxxxxxx\"\n            plt.subplot(3,4,j+1)\n            plt.imshow(dicom2array_2(x))\n            plt.title(title, fontweight=\"bold\")\n            plt.axis(False)\n    \n    plt.tight_layout()\n    plt.show()\n    break","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:15.570375Z","iopub.execute_input":"2021-06-19T11:58:15.570663Z","iopub.status.idle":"2021-06-19T11:58:15.864779Z","shell.execute_reply.started":"2021-06-19T11:58:15.570635Z","shell.execute_reply":"2021-06-19T11:58:15.861411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### So for each test study, possible labels:\n\n'Negative for Pneumonia' - 1676 have this - 27%\n\n\n'Typical Appearance' - 2855 have this - 47%\n\n\n'Indeterminate Appearance' - 1049 have this - 17% \n\n'Atypical Appearance' - 474 have this - 7%\n\n\n#### Overall Multi-class(4), unbalanced classification problem\n\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Is it  a Multilabel Multiclass problem?\n\n","metadata":{}},{"cell_type":"code","source":"study_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:15.866157Z","iopub.execute_input":"2021-06-19T11:58:15.866551Z","iopub.status.idle":"2021-06-19T11:58:15.879054Z","shell.execute_reply.started":"2021-06-19T11:58:15.866519Z","shell.execute_reply":"2021-06-19T11:58:15.878305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df.columns","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:15.880184Z","iopub.execute_input":"2021-06-19T11:58:15.880564Z","iopub.status.idle":"2021-06-19T11:58:15.892139Z","shell.execute_reply.started":"2021-06-19T11:58:15.880535Z","shell.execute_reply":"2021-06-19T11:58:15.891316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df['all_sum'] = study_df.apply(lambda row: row['Negative for Pneumonia'] + \n                                     row['Typical Appearance'] + row['Indeterminate Appearance']\n                                    + row['Atypical Appearance'], axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:15.89329Z","iopub.execute_input":"2021-06-19T11:58:15.893656Z","iopub.status.idle":"2021-06-19T11:58:16.074653Z","shell.execute_reply.started":"2021-06-19T11:58:15.893625Z","shell.execute_reply":"2021-06-19T11:58:16.073509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:16.076006Z","iopub.execute_input":"2021-06-19T11:58:16.076316Z","iopub.status.idle":"2021-06-19T11:58:16.090111Z","shell.execute_reply.started":"2021-06-19T11:58:16.076285Z","shell.execute_reply":"2021-06-19T11:58:16.089099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df['all_sum'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:16.092018Z","iopub.execute_input":"2021-06-19T11:58:16.092436Z","iopub.status.idle":"2021-06-19T11:58:16.106821Z","shell.execute_reply.started":"2021-06-19T11:58:16.092401Z","shell.execute_reply":"2021-06-19T11:58:16.105957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Is it  a Multilabel Multiclass problem?\n\nNo","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### dig bit more into imagelevel df","metadata":{}},{"cell_type":"code","source":"image_df.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:16.107854Z","iopub.execute_input":"2021-06-19T11:58:16.108258Z","iopub.status.idle":"2021-06-19T11:58:16.115902Z","shell.execute_reply.started":"2021-06-19T11:58:16.108228Z","shell.execute_reply":"2021-06-19T11:58:16.114826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:16.117086Z","iopub.execute_input":"2021-06-19T11:58:16.117356Z","iopub.status.idle":"2021-06-19T11:58:16.13398Z","shell.execute_reply.started":"2021-06-19T11:58:16.117331Z","shell.execute_reply":"2021-06-19T11:58:16.133184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df[['image_id','image_type']]= image_df['id'].str.split('_',expand=True)","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:16.135336Z","iopub.execute_input":"2021-06-19T11:58:16.135907Z","iopub.status.idle":"2021-06-19T11:58:16.347819Z","shell.execute_reply.started":"2021-06-19T11:58:16.135874Z","shell.execute_reply":"2021-06-19T11:58:16.346685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:16.349152Z","iopub.execute_input":"2021-06-19T11:58:16.349666Z","iopub.status.idle":"2021-06-19T11:58:16.370943Z","shell.execute_reply.started":"2021-06-19T11:58:16.349629Z","shell.execute_reply":"2021-06-19T11:58:16.370075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df.StudyInstanceUID.describe()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:16.37228Z","iopub.execute_input":"2021-06-19T11:58:16.372668Z","iopub.status.idle":"2021-06-19T11:58:16.391426Z","shell.execute_reply.started":"2021-06-19T11:58:16.37264Z","shell.execute_reply":"2021-06-19T11:58:16.390636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df.image_id.describe()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:16.392735Z","iopub.execute_input":"2021-06-19T11:58:16.393239Z","iopub.status.idle":"2021-06-19T11:58:16.415517Z","shell.execute_reply.started":"2021-06-19T11:58:16.393209Z","shell.execute_reply":"2021-06-19T11:58:16.41452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df.image_type.describe()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:16.417043Z","iopub.execute_input":"2021-06-19T11:58:16.417403Z","iopub.status.idle":"2021-06-19T11:58:16.434498Z","shell.execute_reply.started":"2021-06-19T11:58:16.417369Z","shell.execute_reply":"2021-06-19T11:58:16.433605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### so we have 6.3k images belonging to 6k cases\n\n1 case has 9 images","metadata":{}},{"cell_type":"code","source":"image_df[image_df.StudyInstanceUID == '0fd2db233deb']","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:16.435585Z","iopub.execute_input":"2021-06-19T11:58:16.436129Z","iopub.status.idle":"2021-06-19T11:58:16.465905Z","shell.execute_reply.started":"2021-06-19T11:58:16.436097Z","shell.execute_reply":"2021-06-19T11:58:16.464742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df[image_df.StudyInstanceUID == '0fd2db233deb'].iloc[3].label","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:00:11.090768Z","iopub.execute_input":"2021-06-19T12:00:11.09116Z","iopub.status.idle":"2021-06-19T12:00:11.101269Z","shell.execute_reply.started":"2021-06-19T12:00:11.091128Z","shell.execute_reply":"2021-06-19T12:00:11.100264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df[image_df.StudyInstanceUID == '0fd2db233deb'].iloc[3].label","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df[image_df.StudyInstanceUID == '0fd2db233deb'].iloc[3].label.split()\n","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:22:20.670302Z","iopub.execute_input":"2021-06-19T12:22:20.670703Z","iopub.status.idle":"2021-06-19T12:22:20.681341Z","shell.execute_reply.started":"2021-06-19T12:22:20.67067Z","shell.execute_reply":"2021-06-19T12:22:20.679868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df[image_df.StudyInstanceUID == '0fd2db233deb'].iloc[3].boxes","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:13:19.71002Z","iopub.execute_input":"2021-06-19T12:13:19.710399Z","iopub.status.idle":"2021-06-19T12:13:19.719047Z","shell.execute_reply.started":"2021-06-19T12:13:19.710371Z","shell.execute_reply":"2021-06-19T12:13:19.718075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eval(image_df[image_df.StudyInstanceUID == '0fd2db233deb'].iloc[3].boxes)","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:16:22.972402Z","iopub.execute_input":"2021-06-19T12:16:22.972739Z","iopub.status.idle":"2021-06-19T12:16:22.984026Z","shell.execute_reply.started":"2021-06-19T12:16:22.97271Z","shell.execute_reply":"2021-06-19T12:16:22.982614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df[study_df.id == '0fd2db233deb']","metadata":{"execution":{"iopub.status.busy":"2021-06-19T11:58:16.467438Z","iopub.execute_input":"2021-06-19T11:58:16.467741Z","iopub.status.idle":"2021-06-19T11:58:16.483095Z","shell.execute_reply.started":"2021-06-19T11:58:16.467712Z","shell.execute_reply":"2021-06-19T11:58:16.482293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df[study_df.id == '0fd2db233deb'].iloc[0].study_dir","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:38:33.189816Z","iopub.execute_input":"2021-06-19T12:38:33.190266Z","iopub.status.idle":"2021-06-19T12:38:33.200525Z","shell.execute_reply.started":"2021-06-19T12:38:33.190231Z","shell.execute_reply":"2021-06-19T12:38:33.199722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### so opacity is one form of indeterminate appearance?\n\nyes - Bounding boxes were placed on lung opacities, whether typical or indeterminate","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### how many image labels are possible?\n\nso far seen opacity and none\n","metadata":{}},{"cell_type":"code","source":"all_labels = list(image_df.label)\nall_labels_list = [k.split(' ') for k in all_labels]\n","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:05:48.097231Z","iopub.execute_input":"2021-06-19T12:05:48.097734Z","iopub.status.idle":"2021-06-19T12:05:48.11029Z","shell.execute_reply.started":"2021-06-19T12:05:48.097688Z","shell.execute_reply":"2021-06-19T12:05:48.109119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# all_labels_list[:3]","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:05:07.416993Z","iopub.execute_input":"2021-06-19T12:05:07.417555Z","iopub.status.idle":"2021-06-19T12:05:07.421137Z","shell.execute_reply.started":"2021-06-19T12:05:07.417508Z","shell.execute_reply":"2021-06-19T12:05:07.420106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"possible_lens = [len(k) for k in all_labels_list]","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:06:18.745706Z","iopub.execute_input":"2021-06-19T12:06:18.746134Z","iopub.status.idle":"2021-06-19T12:06:18.75124Z","shell.execute_reply.started":"2021-06-19T12:06:18.7461Z","shell.execute_reply":"2021-06-19T12:06:18.750456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"possible_values = set([k[0] for k in all_labels_list])","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:10:21.258063Z","iopub.execute_input":"2021-06-19T12:10:21.2586Z","iopub.status.idle":"2021-06-19T12:10:21.266932Z","shell.execute_reply.started":"2021-06-19T12:10:21.258561Z","shell.execute_reply":"2021-06-19T12:10:21.265771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"possible_values","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:10:22.248727Z","iopub.execute_input":"2021-06-19T12:10:22.249106Z","iopub.status.idle":"2021-06-19T12:10:22.255137Z","shell.execute_reply.started":"2021-06-19T12:10:22.249068Z","shell.execute_reply":"2021-06-19T12:10:22.253888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"d = {12:0, 6:0, 18:0, 24:0, 30:0, 48:0}\nfor k in possible_lens:\n#     print(k)\n    d[k] += 1\n","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:08:10.864854Z","iopub.execute_input":"2021-06-19T12:08:10.865209Z","iopub.status.idle":"2021-06-19T12:08:10.870472Z","shell.execute_reply.started":"2021-06-19T12:08:10.865168Z","shell.execute_reply":"2021-06-19T12:08:10.869792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"d","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:08:14.655365Z","iopub.execute_input":"2021-06-19T12:08:14.655731Z","iopub.status.idle":"2021-06-19T12:08:14.66098Z","shell.execute_reply.started":"2021-06-19T12:08:14.655704Z","shell.execute_reply":"2021-06-19T12:08:14.660298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### so bbox/label distribution\n\nalmost 50% have 1 bbox (can also be none), \n\n50% have 2 bbox (should not be none)\n\nno other label possible","metadata":{}},{"cell_type":"code","source":"image_df['boxes'].fillna(0,inplace=True)","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:12:47.930089Z","iopub.execute_input":"2021-06-19T12:12:47.930444Z","iopub.status.idle":"2021-06-19T12:12:47.937976Z","shell.execute_reply.started":"2021-06-19T12:12:47.930417Z","shell.execute_reply":"2021-06-19T12:12:47.936755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:12:48.801743Z","iopub.execute_input":"2021-06-19T12:12:48.802159Z","iopub.status.idle":"2021-06-19T12:12:48.816441Z","shell.execute_reply.started":"2021-06-19T12:12:48.802129Z","shell.execute_reply":"2021-06-19T12:12:48.815282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df['len_boxes'] = image_df['boxes'].apply(lambda row: len(eval(row)) if row else 0)","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:17:11.544335Z","iopub.execute_input":"2021-06-19T12:17:11.544737Z","iopub.status.idle":"2021-06-19T12:17:11.667831Z","shell.execute_reply.started":"2021-06-19T12:17:11.544705Z","shell.execute_reply":"2021-06-19T12:17:11.666722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:17:14.321587Z","iopub.execute_input":"2021-06-19T12:17:14.32199Z","iopub.status.idle":"2021-06-19T12:17:14.337228Z","shell.execute_reply.started":"2021-06-19T12:17:14.321951Z","shell.execute_reply":"2021-06-19T12:17:14.3362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df.len_boxes.value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:17:35.217158Z","iopub.execute_input":"2021-06-19T12:17:35.217518Z","iopub.status.idle":"2021-06-19T12:17:35.226674Z","shell.execute_reply.started":"2021-06-19T12:17:35.217488Z","shell.execute_reply":"2021-06-19T12:17:35.224733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df['label_type'] = image_df.label.str.split(expand=True)[0]","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:20:35.347874Z","iopub.execute_input":"2021-06-19T12:20:35.348338Z","iopub.status.idle":"2021-06-19T12:20:35.438539Z","shell.execute_reply.started":"2021-06-19T12:20:35.348301Z","shell.execute_reply":"2021-06-19T12:20:35.437348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:20:44.648894Z","iopub.execute_input":"2021-06-19T12:20:44.649299Z","iopub.status.idle":"2021-06-19T12:20:44.670872Z","shell.execute_reply.started":"2021-06-19T12:20:44.649264Z","shell.execute_reply":"2021-06-19T12:20:44.669922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_df.label_type.value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:23:11.704456Z","iopub.execute_input":"2021-06-19T12:23:11.705076Z","iopub.status.idle":"2021-06-19T12:23:11.717728Z","shell.execute_reply.started":"2021-06-19T12:23:11.705034Z","shell.execute_reply":"2021-06-19T12:23:11.716318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### similar + finer bbox info  now","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### look at 2 dfs together?","metadata":{}},{"cell_type":"code","source":"image_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:25:18.976096Z","iopub.execute_input":"2021-06-19T12:25:18.976485Z","iopub.status.idle":"2021-06-19T12:25:18.995131Z","shell.execute_reply.started":"2021-06-19T12:25:18.976454Z","shell.execute_reply":"2021-06-19T12:25:18.993709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:25:27.551204Z","iopub.execute_input":"2021-06-19T12:25:27.551591Z","iopub.status.idle":"2021-06-19T12:25:27.566815Z","shell.execute_reply.started":"2021-06-19T12:25:27.551558Z","shell.execute_reply":"2021-06-19T12:25:27.565514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(image_df.columns,study_df.columns)","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:24:59.452196Z","iopub.execute_input":"2021-06-19T12:24:59.452644Z","iopub.status.idle":"2021-06-19T12:24:59.460152Z","shell.execute_reply.started":"2021-06-19T12:24:59.452609Z","shell.execute_reply":"2021-06-19T12:24:59.458727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_study_df = pd.merge(image_df, \n                          study_df, \n                          left_on='StudyInstanceUID',\n                          right_on='id')","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:26:15.245539Z","iopub.execute_input":"2021-06-19T12:26:15.245947Z","iopub.status.idle":"2021-06-19T12:26:15.280588Z","shell.execute_reply.started":"2021-06-19T12:26:15.245913Z","shell.execute_reply":"2021-06-19T12:26:15.279484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_study_df.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:26:30.359369Z","iopub.execute_input":"2021-06-19T12:26:30.359759Z","iopub.status.idle":"2021-06-19T12:26:30.366019Z","shell.execute_reply.started":"2021-06-19T12:26:30.359713Z","shell.execute_reply":"2021-06-19T12:26:30.365042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_study_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:26:35.019358Z","iopub.execute_input":"2021-06-19T12:26:35.01971Z","iopub.status.idle":"2021-06-19T12:26:35.039643Z","shell.execute_reply.started":"2021-06-19T12:26:35.019681Z","shell.execute_reply":"2021-06-19T12:26:35.038659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# image_study_df['study_id'] = image_study_df['id_y']","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:29:03.332469Z","iopub.execute_input":"2021-06-19T12:29:03.333101Z","iopub.status.idle":"2021-06-19T12:29:03.338524Z","shell.execute_reply.started":"2021-06-19T12:29:03.333053Z","shell.execute_reply":"2021-06-19T12:29:03.337064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_study_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:28:01.153774Z","iopub.execute_input":"2021-06-19T12:28:01.154208Z","iopub.status.idle":"2021-06-19T12:28:01.177133Z","shell.execute_reply.started":"2021-06-19T12:28:01.154163Z","shell.execute_reply":"2021-06-19T12:28:01.175857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_study_df[image_study_df['StudyInstanceUID']=='0fd2db233deb']","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:29:23.302373Z","iopub.execute_input":"2021-06-19T12:29:23.30296Z","iopub.status.idle":"2021-06-19T12:29:23.33686Z","shell.execute_reply.started":"2021-06-19T12:29:23.302924Z","shell.execute_reply":"2021-06-19T12:29:23.335863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### so if any1 image in case is opaque, complete case becomes opaque/indeterminate","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### how to read an image data","metadata":{}},{"cell_type":"code","source":"image_study_df[image_study_df['StudyInstanceUID']=='0fd2db233deb'].iloc[3]","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:40:06.05094Z","iopub.execute_input":"2021-06-19T12:40:06.051311Z","iopub.status.idle":"2021-06-19T12:40:06.063043Z","shell.execute_reply.started":"2021-06-19T12:40:06.051279Z","shell.execute_reply":"2021-06-19T12:40:06.06151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_study_df[image_study_df['StudyInstanceUID']=='0fd2db233deb'].iloc[3].study_dir","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:40:27.576424Z","iopub.execute_input":"2021-06-19T12:40:27.576871Z","iopub.status.idle":"2021-06-19T12:40:27.587386Z","shell.execute_reply.started":"2021-06-19T12:40:27.576827Z","shell.execute_reply":"2021-06-19T12:40:27.585938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_study_df[image_study_df['StudyInstanceUID']=='0fd2db233deb'].iloc[3].image_id","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:40:43.886932Z","iopub.execute_input":"2021-06-19T12:40:43.887348Z","iopub.status.idle":"2021-06-19T12:40:43.897633Z","shell.execute_reply.started":"2021-06-19T12:40:43.887316Z","shell.execute_reply":"2021-06-19T12:40:43.896285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_dir = '/kaggle/input/siim-covid19-detection/train/0fd2db233deb'","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:38:50.784049Z","iopub.execute_input":"2021-06-19T12:38:50.784459Z","iopub.status.idle":"2021-06-19T12:38:50.788395Z","shell.execute_reply.started":"2021-06-19T12:38:50.784427Z","shell.execute_reply":"2021-06-19T12:38:50.787615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! ls  /kaggle/input/siim-covid19-detection/train/0fd2db233deb/","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:43:25.68096Z","iopub.execute_input":"2021-06-19T12:43:25.68158Z","iopub.status.idle":"2021-06-19T12:43:26.421641Z","shell.execute_reply.started":"2021-06-19T12:43:25.681531Z","shell.execute_reply":"2021-06-19T12:43:26.420236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! ls -r /kaggle/input/siim-covid19-detection/train/0fd2db233deb/*/","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:42:13.925868Z","iopub.execute_input":"2021-06-19T12:42:13.926253Z","iopub.status.idle":"2021-06-19T12:42:14.664583Z","shell.execute_reply.started":"2021-06-19T12:42:13.92622Z","shell.execute_reply":"2021-06-19T12:42:14.663339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! ls /kaggle/input/siim-covid19-detection/train/0fd2db233deb/9025f953c3d2/26f643772090.dcm","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:44:14.672581Z","iopub.execute_input":"2021-06-19T12:44:14.673028Z","iopub.status.idle":"2021-06-19T12:44:15.411912Z","shell.execute_reply.started":"2021-06-19T12:44:14.67299Z","shell.execute_reply":"2021-06-19T12:44:15.410484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3-level folder hierarchy here\n\nstudy_id/folder?/image_id.dcm","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path = '/kaggle/input/siim-covid19-detection/train/0fd2db233deb/9025f953c3d2/26f643772090.dcm'","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:44:28.826567Z","iopub.execute_input":"2021-06-19T12:44:28.826969Z","iopub.status.idle":"2021-06-19T12:44:28.831431Z","shell.execute_reply.started":"2021-06-19T12:44:28.826928Z","shell.execute_reply":"2021-06-19T12:44:28.830538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"    # Use the pydicom library to read the dicom file\n    dicom = pydicom.read_file(path)\n","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:46:02.252699Z","iopub.execute_input":"2021-06-19T12:46:02.253084Z","iopub.status.idle":"2021-06-19T12:46:02.923667Z","shell.execute_reply.started":"2021-06-19T12:46:02.253055Z","shell.execute_reply":"2021-06-19T12:46:02.922516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = dicom.pixel_array\n","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:46:19.873495Z","iopub.execute_input":"2021-06-19T12:46:19.873974Z","iopub.status.idle":"2021-06-19T12:46:19.88854Z","shell.execute_reply.started":"2021-06-19T12:46:19.873936Z","shell.execute_reply":"2021-06-19T12:46:19.8877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:46:25.103782Z","iopub.execute_input":"2021-06-19T12:46:25.104345Z","iopub.status.idle":"2021-06-19T12:46:25.112109Z","shell.execute_reply.started":"2021-06-19T12:46:25.10431Z","shell.execute_reply":"2021-06-19T12:46:25.1111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:46:34.958657Z","iopub.execute_input":"2021-06-19T12:46:34.959299Z","iopub.status.idle":"2021-06-19T12:46:34.965906Z","shell.execute_reply.started":"2021-06-19T12:46:34.959239Z","shell.execute_reply":"2021-06-19T12:46:34.964857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"            plt.imshow(dicom2array_2(x))\n            plt.axis(False)\n            plt.title(title, fontweight=\"bold\")\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.imshow(data, cmap='gray')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:47:56.60054Z","iopub.execute_input":"2021-06-19T12:47:56.601032Z","iopub.status.idle":"2021-06-19T12:47:57.69047Z","shell.execute_reply.started":"2021-06-19T12:47:56.600995Z","shell.execute_reply":"2021-06-19T12:47:57.6895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# this can help with increasing the contrast\nfrom skimage import exposure\nequ_img = exposure.equalize_hist(data)\nplt.imshow(equ_img, cmap='gray')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:49:09.794857Z","iopub.execute_input":"2021-06-19T12:49:09.795281Z","iopub.status.idle":"2021-06-19T12:49:12.091222Z","shell.execute_reply.started":"2021-06-19T12:49:09.795246Z","shell.execute_reply":"2021-06-19T12:49:12.089972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### also add bboxes now","metadata":{}},{"cell_type":"code","source":"image_study_df[image_study_df['StudyInstanceUID']=='0fd2db233deb'].iloc[3]","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:50:52.581487Z","iopub.execute_input":"2021-06-19T12:50:52.581916Z","iopub.status.idle":"2021-06-19T12:50:52.593186Z","shell.execute_reply.started":"2021-06-19T12:50:52.581878Z","shell.execute_reply":"2021-06-19T12:50:52.591886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eval(image_study_df[image_study_df['StudyInstanceUID']=='0fd2db233deb'].iloc[3].boxes)","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:51:09.39407Z","iopub.execute_input":"2021-06-19T12:51:09.394447Z","iopub.status.idle":"2021-06-19T12:51:09.403919Z","shell.execute_reply.started":"2021-06-19T12:51:09.394413Z","shell.execute_reply":"2021-06-19T12:51:09.402655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_study_df[image_study_df['StudyInstanceUID']=='0fd2db233deb'].iloc[3].label","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:53:06.230636Z","iopub.execute_input":"2021-06-19T12:53:06.231208Z","iopub.status.idle":"2021-06-19T12:53:06.24066Z","shell.execute_reply.started":"2021-06-19T12:53:06.231166Z","shell.execute_reply":"2021-06-19T12:53:06.239533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"box1 = image_study_df[image_study_df['StudyInstanceUID']=='0fd2db233deb'].iloc[3].label.split()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:54:37.521784Z","iopub.execute_input":"2021-06-19T12:54:37.522222Z","iopub.status.idle":"2021-06-19T12:54:37.531042Z","shell.execute_reply.started":"2021-06-19T12:54:37.522188Z","shell.execute_reply":"2021-06-19T12:54:37.529691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"box1","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:54:42.901817Z","iopub.execute_input":"2021-06-19T12:54:42.902353Z","iopub.status.idle":"2021-06-19T12:54:42.90897Z","shell.execute_reply.started":"2021-06-19T12:54:42.902309Z","shell.execute_reply":"2021-06-19T12:54:42.908152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img = data","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:55:45.646807Z","iopub.execute_input":"2021-06-19T12:55:45.647327Z","iopub.status.idle":"2021-06-19T12:55:45.650703Z","shell.execute_reply.started":"2021-06-19T12:55:45.647296Z","shell.execute_reply":"2021-06-19T12:55:45.650014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img = cv2.rectangle(img,(int(float(box1[2])), int(float(box1[3]))), \n                    (int(float(box1[4])), int(float(box1[5]))),\n                    color=(200, 0, 0), thickness=15)","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:55:45.85409Z","iopub.execute_input":"2021-06-19T12:55:45.854482Z","iopub.status.idle":"2021-06-19T12:55:45.859945Z","shell.execute_reply.started":"2021-06-19T12:55:45.85445Z","shell.execute_reply":"2021-06-19T12:55:45.859148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(img, cmap='gray')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:55:46.119103Z","iopub.execute_input":"2021-06-19T12:55:46.119661Z","iopub.status.idle":"2021-06-19T12:55:47.11339Z","shell.execute_reply.started":"2021-06-19T12:55:46.119622Z","shell.execute_reply":"2021-06-19T12:55:47.112424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"equ_img = exposure.equalize_hist(data)\n\nequ_img = cv2.rectangle(equ_img,(int(float(box1[2])), int(float(box1[3]))), \n                    (int(float(box1[4])), int(float(box1[5]))),\n                    color=(200, 0, 0), thickness=15)","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:58:22.448317Z","iopub.execute_input":"2021-06-19T12:58:22.448742Z","iopub.status.idle":"2021-06-19T12:58:23.141845Z","shell.execute_reply.started":"2021-06-19T12:58:22.448706Z","shell.execute_reply":"2021-06-19T12:58:23.140556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(equ_img, cmap='gray')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T12:58:23.415855Z","iopub.execute_input":"2021-06-19T12:58:23.416292Z","iopub.status.idle":"2021-06-19T12:58:24.309272Z","shell.execute_reply.started":"2021-06-19T12:58:23.416254Z","shell.execute_reply":"2021-06-19T12:58:24.308227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### these image might not be of same size - and require resizing","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### images without boxes","metadata":{}},{"cell_type":"code","source":"image_study_df.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:00:42.983223Z","iopub.execute_input":"2021-06-19T13:00:42.983628Z","iopub.status.idle":"2021-06-19T13:00:42.989845Z","shell.execute_reply.started":"2021-06-19T13:00:42.983596Z","shell.execute_reply":"2021-06-19T13:00:42.989052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# image_study_df[image_study_df.boxes == 0].head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:05:37.024379Z","iopub.execute_input":"2021-06-19T13:05:37.024727Z","iopub.status.idle":"2021-06-19T13:05:37.029508Z","shell.execute_reply.started":"2021-06-19T13:05:37.024699Z","shell.execute_reply":"2021-06-19T13:05:37.027905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# image_study_df[image_study_df.boxes == 0].shape","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:05:39.98183Z","iopub.execute_input":"2021-06-19T13:05:39.982249Z","iopub.status.idle":"2021-06-19T13:05:39.986307Z","shell.execute_reply.started":"2021-06-19T13:05:39.982216Z","shell.execute_reply":"2021-06-19T13:05:39.985498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# image_study_df[image_study_df.boxes != 0].shape","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:06:17.272885Z","iopub.execute_input":"2021-06-19T13:06:17.273534Z","iopub.status.idle":"2021-06-19T13:06:17.276846Z","shell.execute_reply.started":"2021-06-19T13:06:17.273498Z","shell.execute_reply":"2021-06-19T13:06:17.275841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# image_study_df[image_study_df.boxes != 0].head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:06:19.337675Z","iopub.execute_input":"2021-06-19T13:06:19.338079Z","iopub.status.idle":"2021-06-19T13:06:19.341528Z","shell.execute_reply.started":"2021-06-19T13:06:19.338034Z","shell.execute_reply":"2021-06-19T13:06:19.340878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### There are 4294 images with boxes and 2040 images without any boxes.","metadata":{}},{"cell_type":"code","source":"nobox_docs = image_study_df[image_study_df.boxes == 0]","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:02:50.806236Z","iopub.execute_input":"2021-06-19T13:02:50.806631Z","iopub.status.idle":"2021-06-19T13:02:50.815617Z","shell.execute_reply.started":"2021-06-19T13:02:50.806596Z","shell.execute_reply":"2021-06-19T13:02:50.814278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"box_docs = image_study_df[image_study_df.boxes != 0]","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:05:53.676769Z","iopub.execute_input":"2021-06-19T13:05:53.677148Z","iopub.status.idle":"2021-06-19T13:05:53.686483Z","shell.execute_reply.started":"2021-06-19T13:05:53.677117Z","shell.execute_reply":"2021-06-19T13:05:53.685256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nobox_docs.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:02:56.545704Z","iopub.execute_input":"2021-06-19T13:02:56.546134Z","iopub.status.idle":"2021-06-19T13:02:56.552704Z","shell.execute_reply.started":"2021-06-19T13:02:56.546078Z","shell.execute_reply":"2021-06-19T13:02:56.551545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"box_docs.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:05:59.883422Z","iopub.execute_input":"2021-06-19T13:05:59.883774Z","iopub.status.idle":"2021-06-19T13:05:59.890154Z","shell.execute_reply.started":"2021-06-19T13:05:59.883727Z","shell.execute_reply":"2021-06-19T13:05:59.88934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nobox_docs.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:03:19.987114Z","iopub.execute_input":"2021-06-19T13:03:19.987504Z","iopub.status.idle":"2021-06-19T13:03:20.008706Z","shell.execute_reply.started":"2021-06-19T13:03:19.987473Z","shell.execute_reply":"2021-06-19T13:03:20.007617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"box_docs.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:06:09.232579Z","iopub.execute_input":"2021-06-19T13:06:09.232937Z","iopub.status.idle":"2021-06-19T13:06:09.253512Z","shell.execute_reply.started":"2021-06-19T13:06:09.232908Z","shell.execute_reply":"2021-06-19T13:06:09.252833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nobox_docs.study_id.describe()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:03:48.174927Z","iopub.execute_input":"2021-06-19T13:03:48.175327Z","iopub.status.idle":"2021-06-19T13:03:48.18803Z","shell.execute_reply.started":"2021-06-19T13:03:48.175297Z","shell.execute_reply":"2021-06-19T13:03:48.187064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nobox_docs['Negative for Pneumonia'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:04:54.163579Z","iopub.execute_input":"2021-06-19T13:04:54.164022Z","iopub.status.idle":"2021-06-19T13:04:54.17349Z","shell.execute_reply.started":"2021-06-19T13:04:54.163986Z","shell.execute_reply":"2021-06-19T13:04:54.17205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nobox_docs['Typical Appearance'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:07:14.844977Z","iopub.execute_input":"2021-06-19T13:07:14.845336Z","iopub.status.idle":"2021-06-19T13:07:14.853402Z","shell.execute_reply.started":"2021-06-19T13:07:14.845307Z","shell.execute_reply":"2021-06-19T13:07:14.852577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"box_docs['Negative for Pneumonia'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:06:30.879819Z","iopub.execute_input":"2021-06-19T13:06:30.88031Z","iopub.status.idle":"2021-06-19T13:06:30.888643Z","shell.execute_reply.started":"2021-06-19T13:06:30.880279Z","shell.execute_reply":"2021-06-19T13:06:30.887382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"box_docs['Typical Appearance'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-06-19T13:07:20.270556Z","iopub.execute_input":"2021-06-19T13:07:20.271119Z","iopub.status.idle":"2021-06-19T13:07:20.279879Z","shell.execute_reply.started":"2021-06-19T13:07:20.271084Z","shell.execute_reply":"2021-06-19T13:07:20.278549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### if you have a box, you can't be negative for pneumonia","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### TODO:\n\nplay with few more examples to see opacity vs. none\n\nthink of classifier","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}