{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n
\n \n
\n
","metadata":{}},{"cell_type":"markdown","source":"

\n
\n
\n SIIM-FISABIO-RSNA COVID-19 Detection - Basic EDA\n
\n

","metadata":{}},{"cell_type":"markdown","source":"# 1. Introduction\n> In this competition, you’ll identify and localize COVID-19 abnormalities on chest radiographs. In particular, you'll categorize the radiographs as negative for pneumonia or typical, indeterminate, or atypical for COVID-19. we need to make predictions at both a `study` (multi-image) and `image` level.\n\n## 1.1. Study-level labels\n> Studies in the test set may contain more than one label.\n\n> `negative`, `typical`, `indeterminate`, `atypical`\n\n## 1.2. Image-level labels\n> Images in the test set may contain more than one object.\n\n> you must predict a `class ID` of \"opacity\", a `confidence score`, and `bounding box` in format xmin ymin xmax ymax.\n(`none` is the class ID for \"No finding\".)\n\n## 1.3. Evaluation Metrics\n> the standard PASCAL VOC 2010 `mean Average Precision (mAP)` at IoU > `0.5`\n\n> Here is no concept of `difficult` classes in VOC 2010(Perhaps VOC 2012 will have that concept.).\n\n You can see more details in [mAP understanding with code and its tips](https://www.kaggle.com/its7171/map-understanding-with-code-and-its-tips) for understanding it.","metadata":{}},{"cell_type":"markdown","source":"# 2. Importing the libraries📗","metadata":{}},{"cell_type":"code","source":"!pip install gdcm","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nfrom os import listdir\nimport pandas as pd\nimport numpy as np\nimport glob\nfrom skimage import exposure\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n# pydicom\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nfrom fastai.imports import *\nfrom fastai.medical.imaging import *\n\nimport cv2\n\n# color\nfrom colorama import Fore, Back, Style\n\nimport seaborn as sns\nsns.set(style=\"whitegrid\")\n\n# plotly\nimport plotly.express as px\nimport plotly\n\n# Suppress warnings \nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Settings for pretty nice plots\nplt.style.use('fivethirtyeight')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Reading the csv📚","metadata":{}},{"cell_type":"code","source":"# List files available\nlist(os.listdir(\"../input/siim-covid19-detection\"))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"IMAGE_PATH = \"../input/siim-covid19-detection/\"\n\ntrain_study_df = pd.read_csv('../input/siim-covid19-detection/train_study_level.csv')\ntrain_image_df = pd.read_csv('../input/siim-covid19-detection/train_image_level.csv')\n\nprint(Fore.YELLOW + 'Train study df shape: ',Style.RESET_ALL,train_study_df.shape)\nprint(Fore.YELLOW + 'Train image df shape: ',Style.RESET_ALL,train_image_df.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(train_study_df.head(5))\ndisplay(train_image_df.head(5))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission = pd.read_csv('../input/siim-covid19-detection/sample_submission.csv')\nprint(Fore.YELLOW + 'Sample submission df shape: ',Style.RESET_ALL,sample_submission.shape)\ndisplay(sample_submission)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For each test study, we should make a determination within the following labels.\n\n- `Negative for Pneumonia`\n\n- `Typical Appearance`\n\n- `Indeterminate Appearance`\n\n- `Atypical Appearance`\n\nFor each test image, we will be predicting a bounding box and class for all findings.\n- e.g. `none 1 0 0 1 1`","metadata":{}},{"cell_type":"markdown","source":"# 4. Basic Data Exploration 🏕️","metadata":{}},{"cell_type":"markdown","source":"## General Info","metadata":{}},{"cell_type":"code","source":"# Null values and Data types\nprint(Fore.BLUE + 'Train Study Set !!',Style.RESET_ALL)\nprint(train_study_df.info())\nprint('-------------')\nprint(Fore.YELLOW + 'Train Image Set !!',Style.RESET_ALL)\nprint(train_image_df.info())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Missing values","metadata":{}},{"cell_type":"code","source":"train_study_df.isnull().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image_df.isnull().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`2040` : It seems missing bounding boxes.","metadata":{}},{"cell_type":"markdown","source":"## 4.1 Basic Data Exploration for `Study level` 📖","metadata":{}},{"cell_type":"code","source":"train_study_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(train_study_df['Negative for Pneumonia'].value_counts())\ndisplay(train_study_df['Typical Appearance'].value_counts())\ndisplay(train_study_df['Indeterminate Appearance'].value_counts())\ndisplay(train_study_df['Atypical Appearance'].value_counts())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are `4` labels in `train study level.csv`","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(20, 6))\nsns.kdeplot(train_study_df['Negative for Pneumonia'], shade=True)\nsns.kdeplot(train_study_df['Typical Appearance'], shade=True)\nsns.kdeplot(train_study_df['Indeterminate Appearance'], shade=True)\nsns.kdeplot(train_study_df['Atypical Appearance'], shade=True)\n\n# Labeling of plot\nplt.xlabel('Labels for each study'); plt.ylabel('Density'); plt.title('Distribution of Study level');","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Distribution of Negative for Pneumonia","metadata":{}},{"cell_type":"code","source":"fig = px.bar(train_study_df, \n x=train_study_df['Negative for Pneumonia'].value_counts().index, \n y=train_study_df['Negative for Pneumonia'].value_counts(),\n color=['0','1'])\nfig.update_traces(marker_line_color='rgb(8,48,107)',\n marker_line_width=1.5, opacity=0.6)\nfig.update_layout(title_text=\"Distribution of Negative for Pneumonia\")\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Distribution of Typical Appearance","metadata":{}},{"cell_type":"code","source":"fig = px.bar(train_study_df, \n x=train_study_df['Typical Appearance'].value_counts().index, \n y=train_study_df['Typical Appearance'].value_counts(),\n color=['0','1'])\nfig.update_traces(marker_line_color='rgb(8,48,107)',\n marker_line_width=1.5, opacity=0.6)\nfig.update_layout(title_text=\"Distribution of Typical Appearance\")\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Distribution of Indeterminate Appearance","metadata":{}},{"cell_type":"code","source":"fig = px.bar(train_study_df, \n x=train_study_df['Indeterminate Appearance'].value_counts().index, \n y=train_study_df['Indeterminate Appearance'].value_counts(),\n color=['0','1'])\nfig.update_traces(marker_line_color='rgb(8,48,107)',\n marker_line_width=1.5, opacity=0.6)\nfig.update_layout(title_text=\"Distribution of Indeterminate Appearance\")\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Distribution of Indeterminate Appearance","metadata":{}},{"cell_type":"code","source":"fig = px.bar(train_study_df, \n x=train_study_df['Atypical Appearance'].value_counts().index, \n y=train_study_df['Atypical Appearance'].value_counts(),\n color=['0','1'])\nfig.update_traces(marker_line_color='rgb(8,48,107)',\n marker_line_width=1.5, opacity=0.6)\nfig.update_layout(title_text=\"Distribution of Indeterminate Appearance\")\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2 Basic Data Exploration for `Image level` ✅","metadata":{}},{"cell_type":"code","source":"train_image_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`StudyInstanceUID` seems same `id` in `study level`.\n- https://www.kaggle.com/yujiariyasu/plot-3positive-classes","metadata":{}},{"cell_type":"code","source":"# merge study csv\ntrain_study_df['StudyInstanceUID'] = train_study_df['id'].apply(lambda x: x.replace('_study', ''))\ndel train_study_df['id']\ntrain = train_image_df.merge(train_study_df, on='StudyInstanceUID')\ntrain.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In `image level` df, Most of the data is related to the `bboxes`.\n(Exactly `labels`, `confedence scores`, `bboxes`)\n\nWhen looking at the `.dcm` images, we will use that information to draw the bboxes.","metadata":{}},{"cell_type":"markdown","source":"# 5. Visualising Images : DICOM 🗺️ ","metadata":{}},{"cell_type":"code","source":"dataset_dir = Path('../input/siim-covid19-detection')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nhttps://www.kaggle.com/trungthanhnguyen0502/eda-vinbigdata-chest-x-ray-abnormalities\n'''\ndef dicom2array(path, voi_lut=True, fix_monochrome=True):\n dicom = pydicom.read_file(path)\n # VOI LUT (if available by DICOM device) is used to\n # transform raw DICOM data to \"human-friendly\" view\n if voi_lut:\n data = apply_voi_lut(dicom.pixel_array, dicom)\n else:\n data = dicom.pixel_array\n # depending on this value, X-ray may look inverted - fix that:\n if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\n data = data - np.min(data)\n data = data / np.max(data)\n data = (data * 255).astype(np.uint8)\n return data\n \n \ndef plot_img(img, size=(7, 7), is_rgb=True, title=\"\", cmap='gray'):\n plt.figure(figsize=size)\n plt.imshow(img, cmap=cmap)\n plt.suptitle(title)\n plt.show()\n\n\ndef plot_imgs(imgs, cols=2, size=7, is_rgb=True, title=\"\", cmap='gray', img_size=(500,500)):\n rows = len(imgs)//cols + 1\n fig = plt.figure(figsize=(cols*size, rows*size))\n for i, img in enumerate(imgs):\n if img_size is not None:\n img = cv2.resize(img, img_size)\n fig.add_subplot(rows, cols, i+1)\n plt.imshow(img, cmap=cmap)\n plt.suptitle(title)\n plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dicom_paths = get_dicom_files(dataset_dir/'train')\nimgs = [dicom2array(path) for path in dicom_paths[:2]]\nplot_imgs(imgs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# preprocess - equalize histogram\n\nimgs = [exposure.equalize_hist(img) for img in imgs]\nplot_imgs(imgs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Visualizing Bounding Box 📦","metadata":{}},{"cell_type":"markdown","source":"For visualizing BBox, I will use `class_id`, [`x_min`, `y_min`, `x_max`, `y_max`] in `label` column. Please see following EDA.\n - https://www.kaggle.com/yujiariyasu/plot-3positive-classes\n - https://www.kaggle.com/tanlikesmath/siim-covid-19-detection-a-simple-eda","metadata":{}},{"cell_type":"markdown","source":"class_id\n- `Typical Appearance`, `Indeterminate Appearance`, `Atypical Appearance`, or `negative`","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"we already know that `2048` bboxes seems missing in `train`.","metadata":{}},{"cell_type":"code","source":"train.isnull().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class_names = ['Typical Appearance', 'Indeterminate Appearance', 'Atypical Appearance']","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nhttps://www.kaggle.com/yujiariyasu/plot-3positive-classes\n'''\n\nlabel2color = {\n '[1, 0, 0]': [255,0,0], # Typical Appearance\n '[0, 1, 0]': [0,255,0], # Indeterminate Appearance\n '[0, 0, 1]': [0,0,255], # Atypical Appearance\n}\n\ndef plot_bboxes_with_label(label_name, n):\n print('Typical Appearance: ' + Fore.RED + 'Red',Style.RESET_ALL)\n print('Indeterminate Appearance: ' + Fore.GREEN + 'Green',Style.RESET_ALL)\n print('Atypical Appearance: ' + Fore.BLUE + 'Blue',Style.RESET_ALL)\n \n imgs = []\n thickness = 2\n scale = 5\n \n if label_name == 'Negative for Pneumonia':\n flag = 0\n else:\n flag = 1\n \n for _, row in train[train[label_name]==flag].iloc[:n].iterrows():\n study_id = row['StudyInstanceUID']\n img_path = glob.glob(f'{dataset_dir}/train/{study_id}/*/*')[0]\n img = dicom2array(path=img_path)\n img = cv2.resize(img, None, fx=1/scale, fy=1/scale)\n img = np.stack([img, img, img], axis=-1)\n\n claz = row[class_names].values\n color = label2color[str(claz.tolist())]\n\n bboxes = []\n bbox = []\n for i, l in enumerate(row['label'].split(' ')):\n '''\n Add comments for others\n '''\n # ================================================================\n # labels, confidence, x_min, y_min, width, height == 6 or NaN == 1\n # ================================================================\n if (i % 6 == 0) | (i % 6 == 1):\n continue\n bbox.append(float(l)/scale)\n if i % 6 == 5:\n bboxes.append(bbox)\n bbox = []\n for box in bboxes:\n img = cv2.rectangle(\n img,\n (int(box[0]), int(box[1])),\n (int(box[2]), int(box[3])),\n color, thickness\n )\n img = cv2.resize(img, (512,512))\n imgs.append(img)\n\n plot_imgs(imgs, cmap=None)\n \n del img, imgs, bbox, bboxes","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualizing `Negative for Pneumonia`","metadata":{}},{"cell_type":"code","source":"plot_bboxes_with_label('Negative for Pneumonia', 8)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualizing `Typical Appearance`","metadata":{}},{"cell_type":"code","source":"plot_bboxes_with_label('Typical Appearance', 8)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualizing `Indeterminate Appearance`","metadata":{}},{"cell_type":"code","source":"plot_bboxes_with_label('Indeterminate Appearance', 8)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 7. Etc. - Pandas Profiling 🌤️","metadata":{}},{"cell_type":"code","source":"train_study_df = pd.read_csv('../input/siim-covid19-detection/train_study_level.csv')\ntrain_image_df = pd.read_csv('../input/siim-covid19-detection/train_image_level.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pandas_profiling import ProfileReport\nprofile_study = ProfileReport(train_study_df, title=\"Pandas Profiling Report - Train Study df\")\nprofile_study.to_widgets()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"profile_image = ProfileReport(train_image_df, title=\"Pandas Profiling Report - Train Image df\")\nprofile_image.to_widgets()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## If this kernel is useful, please upvote!","metadata":{}},{"cell_type":"markdown","source":"# WORK IN PROGRESS...","metadata":{}},{"cell_type":"markdown","source":"# References\n\n- https://www.kaggle.com/piantic/osic-pulmonary-fibrosis-progression-basic-eda\n- https://www.kaggle.com/dschettler8845/visual-in-depth-eda-vinbigdata-competition-data\n- https://www.kaggle.com/bjoernholzhauer/eda-dicom-reading-vinbigdata-chest-x-ray\n- https://www.kaggle.com/yujiariyasu/plot-3positive-classes\n- https://www.kaggle.com/tanlikesmath/siim-covid-19-detection-a-simple-eda\n- https://www.kaggle.com/jeongyoonlee/siim-covid-19-detection-eda-wip","metadata":{}}]}