{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Intro\nWelcome to the [SIIM-FISABIO-RSNA COVID-19 Detection](https://www.kaggle.com/c/siim-covid19-detection/data) compedition.\n\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/26680/logos/header.png)\n\nFor handling chest-x-ray data we recommend [this notebook](https://www.kaggle.com/drcapa/chest-x-ray-starter).\n\n<span style=\"color: royalblue;\">Please vote the notebook up if it helps you. Feel free to leave a comment above the notebook. Thank you. </span>","metadata":{}},{"cell_type":"markdown","source":"# Libraries","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib\nimport pydicom as dicom\nimport cv2\nimport ast\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Path","metadata":{}},{"cell_type":"code","source":"path = '/kaggle/input/siim-covid19-detection/'\nos.listdir(path)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Data","metadata":{}},{"cell_type":"code","source":"train_image = pd.read_csv(path+'train_image_level.csv')\ntrain_study = pd.read_csv(path+'train_study_level.csv')\nsamp_subm = pd.read_csv(path+'sample_submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Overview","metadata":{}},{"cell_type":"code","source":"print('Number train images samples:', len(train_image))\nprint('Number train study samples:', len(train_study))\nprint('Number test samples:', len(samp_subm))","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The train image-level metadata, with one row for each image, including both correct labels and any bounding boxes in a dictionary format.","metadata":{}},{"cell_type":"code","source":"train_image.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The train study-level metadata, with one row for each study, including correct labels.","metadata":{}},{"cell_type":"code","source":"train_study.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Read DCM File\nWe consider the first train sample.\n\nAll images are stored in paths with the form **study/series/image**.","metadata":{}},{"cell_type":"code","source":"# Define image path of the example\npath_train = path+'train/'+train_image.loc[0, 'StudyInstanceUID']+'/'+'81456c9c5423'+'/'\n# Extract image name of the example\nimg_id = train_image.loc[0, 'id'].replace('_image', '.dcm')\n# Load dicom file\ndata_file = dicom.dcmread(path_train+img_id)\n# Extract image data of the dicom file\nimg = data_file.pixel_array","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Print meta data of the image:","metadata":{}},{"cell_type":"code","source":"print(data_file)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Image shape:","metadata":{}},{"cell_type":"code","source":"print('Image shape:', img.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Bounding Boxes:","metadata":{}},{"cell_type":"code","source":"boxes = ast.literal_eval(train_image.loc[0, 'boxes'])\nboxes","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Plot the image of the chest-x-ray with the bounding boxes:","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 1, figsize=(20, 4))\n\nfor box in boxes:\n    p = matplotlib.patches.Rectangle((box['x'], box['y']), box['width'], box['height'],\n                                     ec='r', fc='none', lw=2.)\n    ax.add_patch(p)\nax.imshow(img, cmap='gray')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Show Examples\nWe plot some examples with the chest-x-ray image, the bounding boxes and the label:","metadata":{}},{"cell_type":"code","source":"fig, axs = plt.subplots(3, 3, figsize=(20, 20))\nfig.subplots_adjust(hspace = .1, wspace=.1)\naxs = axs.ravel()\n\nfor row in range(9):\n    study = train_image.loc[row, 'StudyInstanceUID']\n    path_in = path+'train/'+study+'/'\n    folder = os.listdir(path_in)\n    path_file = path_in+folder[0]\n    filename = os.listdir(path_file)[0]\n    file_id = filename.split('.')[0]\n    \n    data_file = dicom.dcmread(path_file+'/'+file_id+'.dcm')\n    img = data_file.pixel_array\n    if (train_image.loc[row, 'boxes']!=train_image.loc[row, 'boxes']) == False:\n        boxes = ast.literal_eval(train_image.loc[row, 'boxes'])\n    \n        for box in boxes:\n            p = matplotlib.patches.Rectangle((box['x'], box['y']), box['width'], box['height'],\n                                     ec='r', fc='none', lw=2.)\n            axs[row].add_patch(p)\n    axs[row].imshow(img, cmap='gray')\n    axs[row].set_title(train_image.loc[row, 'label'].split(' ')[0])\n    axs[row].set_xticklabels([])\n    axs[row].set_yticklabels([])","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering\nThere are 3 labels possible:\n* none: no abnormalities on chest radiographs\n* simple opacity: abnormalities on one side\n* double opacity: abnormalities on both sides\n\nSo we can define 3 catgories:","metadata":{}},{"cell_type":"code","source":"label_dict = {0: 'none', 1: 'simple_opacity', 2: 'double_opacity'}","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def split_label(s):\n    split_string = s.split(' ')\n    if len(split_string)==6 and 'none' in split_string:\n        return 0\n    elif len(split_string)==6 and 'opacity' in split_string:\n        return 1\n    else:\n        return 2","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image['category'] = train_image['label'].apply(split_label)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA","metadata":{}},{"cell_type":"markdown","source":"We consider on the distribution of the three categories:","metadata":{}},{"cell_type":"code","source":"train_image['category'].value_counts().sort_index().rename(label_dict).plot.bar(rot=0, color='orange', alpha=0.6, grid=True, figsize=(8,4), fontsize=16)\nplt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next we have a look on the study-level metadata:\n* Negative for Pneumonia - 1 if the study is negative for pneumonia, 0 otherwise\n* Typical Appearance - 1 if the study has this appearance, 0 otherwise\n* Indeterminate Appearance  - 1 if the study has this appearance, 0 otherwise\n* Atypical Appearance  - 1 if the study has this appearance, 0 otherwise","metadata":{}},{"cell_type":"code","source":"train_study.sum()[1:].plot.bar(rot=45, color='orange', alpha=0.6, grid=True, figsize=(8,4), fontsize=12)\nplt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Generator\nTo load the data on demand we use a data generator.\n\n*Coming soon*","metadata":{}},{"cell_type":"markdown","source":"# Export","metadata":{}},{"cell_type":"code","source":"samp_subm.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}