{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# SIIM-FISABIO-RSNA COVID-19 Detection: A Simple and Easy EDA\nIn this competition, we are used DICOM images of chest X-ray radiographs, and to identify and localize COVID-19 abnormalities. This is important because typical diagnosis of COVID-19 requires molecular testing (polymerase chain reaction) requires several hours, while chest radiographs can be obtained in minutes, but it is hard to distinguish between COVID-19 pneumonia and other other viral and bacterial pneumonias. Therefore, in this competition, be hope to develop AI that that eventually help radiologists diagnose the millions of COVID-19 patients more confidently and quickly.\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"## Import the Depedencies","metadata":{}},{"cell_type":"code","source":"import pandas as pd \nimport numpy as np\nimport os\nimport pydicom\nimport glob\nfrom tqdm.notebook import tqdm\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nimport matplotlib.pyplot as plt \nfrom skimage import exposure\nimport cv2\nimport warnings\nfrom path import Path\nfrom fastai.vision.all import *\nfrom fastai.medical.imaging import *\nwarnings.filterwarnings(\"ignore\")","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:18:54.731889Z","iopub.execute_input":"2021-05-27T17:18:54.732233Z","iopub.status.idle":"2021-05-27T17:18:54.739921Z","shell.execute_reply.started":"2021-05-27T17:18:54.732204Z","shell.execute_reply":"2021-05-27T17:18:54.738789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load the data","metadata":{}},{"cell_type":"code","source":"datapath = Path(\"../input/siim-covid19-detection\")","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:18:54.741651Z","iopub.execute_input":"2021-05-27T17:18:54.741916Z","iopub.status.idle":"2021-05-27T17:18:54.753507Z","shell.execute_reply.started":"2021-05-27T17:18:54.741891Z","shell.execute_reply":"2021-05-27T17:18:54.752611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datapath.ls()","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:18:54.756769Z","iopub.execute_input":"2021-05-27T17:18:54.757202Z","iopub.status.idle":"2021-05-27T17:18:54.772387Z","shell.execute_reply.started":"2021-05-27T17:18:54.757159Z","shell.execute_reply":"2021-05-27T17:18:54.770988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that we have:\n\n - train_study_level.csv - the train study-level metadata, with one row for each study, including correct labels.\n - train_image_level.csv - the train image-level metadata, with one row for each image, including both correct labels and any bounding boxes in a dictionary format. Some images in both test and train have multiple bounding boxes.\n - sample_submission.csv - a sample submission file containing all image- and study-level IDs.\n - train folder - comprises 6,334 chest scans in DICOM format, stored in paths with the form study/series/image\n - test folder - The hidden test dataset is of roughly the same scale as the training dataset.","metadata":{}},{"cell_type":"markdown","source":"# Lets look at the CSVs file\ncheck train_study_level.csv file","metadata":{}},{"cell_type":"code","source":"train_study_df = pd.read_csv(datapath/'train_study_level.csv')","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:18:54.77509Z","iopub.execute_input":"2021-05-27T17:18:54.775771Z","iopub.status.idle":"2021-05-27T17:18:54.791987Z","shell.execute_reply.started":"2021-05-27T17:18:54.775721Z","shell.execute_reply":"2021-05-27T17:18:54.791101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:18:54.793921Z","iopub.execute_input":"2021-05-27T17:18:54.794331Z","iopub.status.idle":"2021-05-27T17:18:54.806013Z","shell.execute_reply.started":"2021-05-27T17:18:54.794287Z","shell.execute_reply":"2021-05-27T17:18:54.805085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at the unique labels:","metadata":{}},{"cell_type":"code","source":"study_classes = [\"Negative for Pneumonia\" , \"Typical Appearance\",\n                \"Indeterminate Appearance\", \"Atypical Appearance\"]\nnp.unique(train_study_df[study_classes].values , axis = 0)","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:18:54.808026Z","iopub.execute_input":"2021-05-27T17:18:54.808463Z","iopub.status.idle":"2021-05-27T17:18:54.835594Z","shell.execute_reply.started":"2021-05-27T17:18:54.80842Z","shell.execute_reply":"2021-05-27T17:18:54.834662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, at the study-level, we are predicting the following classes:\n\n - Negative for Pneumonia\n - Typical Appearance\n - Indeterminate Appearance\n - Atypical Appearance\n - This here is a standard multi-label classification problem. In the training set, interestingly they are not multi-label, but it is mentioned that:\n\nStudies in the test set may contain more than one label.\n\nLet's look at the distribution:","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize = (10,5))\nplt.bar([1,2,3,4] , train_study_df[study_classes].values.sum(axis = 0))\nplt.xticks([1,2,3,4] , study_classes)\nplt.ylabel(\"Frequency\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:18:54.83776Z","iopub.execute_input":"2021-05-27T17:18:54.838042Z","iopub.status.idle":"2021-05-27T17:18:54.975287Z","shell.execute_reply.started":"2021-05-27T17:18:54.838015Z","shell.execute_reply":"2021-05-27T17:18:54.97462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's now look at train_image_level.csv:","metadata":{}},{"cell_type":"code","source":"train_image_df = pd.read_csv(datapath/\"train_image_level.csv\")\ntrain_image_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:18:54.976343Z","iopub.execute_input":"2021-05-27T17:18:54.97669Z","iopub.status.idle":"2021-05-27T17:18:55.013708Z","shell.execute_reply.started":"2021-05-27T17:18:54.976664Z","shell.execute_reply":"2021-05-27T17:18:55.012678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have our bounding box labels provided in the label column. The format is as follows:\n\n[class ID] [confidence score] [bounding box]\n\n - class ID - either opacity or none\n - confidence score - confidence from your neural network model. If none, the confidence is 1.\n - bounding box - typical xmin ymin xmax ymax format. If class ID is none, the bounding box is 1 0 0 1 1.\n \nThe bounding boxes are also provided in easily readable dictionary format in column boxes, and the study that each image is a part of is provided inStudyInstanceUID.\n\nLet's quick look at the distribution of opacity vs none:","metadata":{}},{"cell_type":"code","source":"train_image_df['split_label'] = train_image_df.label.apply(lambda x: [x.split()[offs:offs+6] for offs in range(0, len(x.split()), 6)])","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:18:55.015146Z","iopub.execute_input":"2021-05-27T17:18:55.015548Z","iopub.status.idle":"2021-05-27T17:18:55.044211Z","shell.execute_reply.started":"2021-05-27T17:18:55.015507Z","shell.execute_reply":"2021-05-27T17:18:55.043317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"classes_freq = []\nfor i in range(len(train_image_df)):\n    for j in train_image_df.iloc[i].split_label: classes_freq.append(j[0])\n\nplt.hist(classes_freq)\nplt.ylabel(\"Frequency\")","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:18:55.046009Z","iopub.execute_input":"2021-05-27T17:18:55.046429Z","iopub.status.idle":"2021-05-27T17:18:55.931901Z","shell.execute_reply.started":"2021-05-27T17:18:55.046399Z","shell.execute_reply":"2021-05-27T17:18:55.931163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's also look at the distribution of the bounding box areas:","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:18:55.933204Z","iopub.execute_input":"2021-05-27T17:18:55.933592Z","iopub.status.idle":"2021-05-27T17:18:55.938902Z","shell.execute_reply.started":"2021-05-27T17:18:55.933564Z","shell.execute_reply":"2021-05-27T17:18:55.937685Z"}}},{"cell_type":"code","source":"bbox_areas=[]\nfor i in range(len(train_image_df)):\n    for j in train_image_df.iloc[i].split_label:\n        bbox_areas.append((float(j[4])-float(j[2]))*(float(j[5])*float(j[3])))\n        \nplt.hist(bbox_areas)\nplt.ylabel(\"Frequency\")","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:20:48.786939Z","iopub.execute_input":"2021-05-27T17:20:48.787292Z","iopub.status.idle":"2021-05-27T17:20:49.830863Z","shell.execute_reply.started":"2021-05-27T17:20:48.78725Z","shell.execute_reply":"2021-05-27T17:20:49.830073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## A look at the images\nOkay, let's now look at some example images:","metadata":{}},{"cell_type":"code","source":"def dicom2array(path, voi_lut=True, fix_monochrome=True):\n    dicom = pydicom.read_file(path)\n    # VOI LUT (if available by DICOM device) is used to\n    # transform raw DICOM data to \"human-friendly\" view\n    if voi_lut:\n        data = apply_voi_lut(dicom.pixel_array, dicom)\n    else:\n        data = dicom.pixel_array\n    # depending on this value, X-ray may look inverted - fix that:\n    if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n        data = np.amax(data) - data\n    data = data - np.min(data)\n    data = data / np.max(data)\n    data = (data * 255).astype(np.uint8)\n    return data\n\n\ndef plot_img(img, size=(7, 7), is_rgb=True, title=\"\", cmap='gray'):\n    plt.figure(figsize=size)\n    plt.imshow(img, cmap=cmap)\n    plt.suptitle(title)\n    plt.show()\n\n    \ndef plot_imgs(imgs, cols=4, size=7, is_rgb=True, title=\"\", cmap='gray', img_size=(500,500)):\n    rows = len(imgs)//cols + 1\n    fig = plt.figure(figsize=(cols*size, rows*size))\n    for i, img in enumerate(imgs):\n        if img_size is not None:\n            img = cv2.resize(img, img_size)\n        fig.add_subplot(rows, cols, i+1)\n        plt.imshow(img, cmap=cmap)\n    plt.suptitle(title)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:40:01.790421Z","iopub.execute_input":"2021-05-27T17:40:01.790739Z","iopub.status.idle":"2021-05-27T17:40:01.802702Z","shell.execute_reply.started":"2021-05-27T17:40:01.790712Z","shell.execute_reply":"2021-05-27T17:40:01.801431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dicom_paths = get_dicom_files(datapath/'train')\nimgs = [dicom2array(path) for path in dicom_paths[:4]]\nplot_imgs(imgs)","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:40:03.300132Z","iopub.execute_input":"2021-05-27T17:40:03.300556Z","iopub.status.idle":"2021-05-27T17:40:10.362985Z","shell.execute_reply.started":"2021-05-27T17:40:03.300505Z","shell.execute_reply":"2021-05-27T17:40:10.362154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's actually look at how many images are available per study:","metadata":{}},{"cell_type":"code","source":"num_images_per_study = []\nfor i in (datapath/'train').ls():\n    num_images_per_study.append(len(get_dicom_files(i)))\n    if len(get_dicom_files(i)) > 5:\n        print(f\"Study {i} had {len(get_dicom_files(i))} images\")","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:45:52.340677Z","iopub.execute_input":"2021-05-27T17:45:52.34102Z","iopub.status.idle":"2021-05-27T17:46:01.214421Z","shell.execute_reply.started":"2021-05-27T17:45:52.340994Z","shell.execute_reply":"2021-05-27T17:46:01.2136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist(num_images_per_study)","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:47:12.830689Z","iopub.execute_input":"2021-05-27T17:47:12.83103Z","iopub.status.idle":"2021-05-27T17:47:13.043223Z","shell.execute_reply.started":"2021-05-27T17:47:12.831002Z","shell.execute_reply":"2021-05-27T17:47:13.042407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def image_path(row):\n    study_path = datapath/\"train\"/row.StudyInstanceUID\n    for i in get_dicom_files(study_path):\n        if row.id.split('_')[0] == i.stem: return i\n        \ntrain_image_df['image_path'] = train_image_df.apply(image_path , axis = 1)\n","metadata":{"execution":{"iopub.status.busy":"2021-05-27T18:32:21.072668Z","iopub.execute_input":"2021-05-27T18:32:21.073066Z","iopub.status.idle":"2021-05-27T18:32:27.640048Z","shell.execute_reply.started":"2021-05-27T18:32:21.073031Z","shell.execute_reply":"2021-05-27T18:32:27.639149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-05-27T18:33:14.572455Z","iopub.execute_input":"2021-05-27T18:33:14.572825Z","iopub.status.idle":"2021-05-27T18:33:14.590209Z","shell.execute_reply.started":"2021-05-27T18:33:14.572796Z","shell.execute_reply":"2021-05-27T18:33:14.589557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"imgs = []\nimage_paths = train_image_df['image_path'].values\n\n# map label_id to specify color\nthickness = 10\nscale = 5\n\n\nfor i in range(8):\n    image_path = random.choice(image_paths)\n    print(image_path)\n    img = dicom2array(path=image_path)\n    img = cv2.resize(img, None, fx=1/scale, fy=1/scale)\n    img = np.stack([img, img, img], axis=-1)\n    for i in train_image_df.loc[train_image_df['image_path'] == image_path].split_label.values[0]:\n        if i[0] == 'opacity':\n            img = cv2.rectangle(img,\n                                (int(float(i[2])/scale), int(float(i[3])/scale)),\n                                (int(float(i[4])/scale), int(float(i[5])/scale)),\n                                [255,0,0], thickness)\n    \n    img = cv2.resize(img, (500,500))\n    imgs.append(img)\n\nplot_imgs(imgs, cmap=None)","metadata":{"execution":{"iopub.status.busy":"2021-05-27T18:46:12.445743Z","iopub.execute_input":"2021-05-27T18:46:12.446343Z","iopub.status.idle":"2021-05-27T18:46:18.995848Z","shell.execute_reply.started":"2021-05-27T18:46:12.446295Z","shell.execute_reply":"2021-05-27T18:46:18.994699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## How to submit\nLet's now go over the sample_submission.csv file so we know how to submit our predictions.\n\nBefore we do so, it's worth reminding ourselves that this is a code-only competition, meaning that your submission file has to be generated in a script/notebook. The sample_submission.csv file demonstrated what kind of file needs to be produced:\n\n","metadata":{}},{"cell_type":"code","source":"submission_df = pd.read_csv(datapath/\"sample_submission.csv\")\nsubmission_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-05-27T18:47:57.901978Z","iopub.execute_input":"2021-05-27T18:47:57.902346Z","iopub.status.idle":"2021-05-27T18:47:57.926204Z","shell.execute_reply.started":"2021-05-27T18:47:57.902314Z","shell.execute_reply":"2021-05-27T18:47:57.92523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see we have to provide the study-level class label. These will be of the format [class] 1 0 0 1 1","metadata":{}},{"cell_type":"code","source":"submission_df.iloc[2000:2010]","metadata":{"execution":{"iopub.status.busy":"2021-05-27T18:48:45.491434Z","iopub.execute_input":"2021-05-27T18:48:45.491982Z","iopub.status.idle":"2021-05-27T18:48:45.504415Z","shell.execute_reply.started":"2021-05-27T18:48:45.491936Z","shell.execute_reply":"2021-05-27T18:48:45.503436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We also have to provide the image-level bounding box. These will be of the format [class ID] [confidence score] [bounding box] as described earlier.\n\nOf course, in both cases, you can have multi-label scenarios.","metadata":{}},{"cell_type":"code","source":"submission_df.to_csv(\"submission.csv\" , index = False)","metadata":{"execution":{"iopub.status.busy":"2021-05-27T18:49:56.941487Z","iopub.execute_input":"2021-05-27T18:49:56.941845Z","iopub.status.idle":"2021-05-27T18:49:56.95824Z","shell.execute_reply.started":"2021-05-27T18:49:56.941816Z","shell.execute_reply":"2021-05-27T18:49:56.957112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}