{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# COVID-19 Detection on Chest Radiographs\n**Please upvote if you found the notebook helpful :)**","metadata":{}},{"cell_type":"markdown","source":"In this competition, you’ll identify COVID-19 abnormalities on chest radiographs. In particular, you'll categorize the radiographs as **negative** for pneumonia or **typical**, **indeterminate**, or **atypical** for COVID-19. \n\nIf successful, you'll help radiologists diagnose the millions of COVID-19 patients more confidently and quickly. This will also enable doctors to see the extent of the disease and help them make decisions regarding treatment. ","metadata":{}},{"cell_type":"markdown","source":"dataset files used: \n* **rain_study_level.csv**:  the train study-level metadata, with one row for each study, including correct labels.\n\n* **train_image_level.csv**: the train image-level metadata, with one row for each image, including both correct labels and any bounding boxes in a dictionary format. Some images in both test and train have multiple bounding boxes.\n\n* **sample_submission.csv**: a sample submission file containing all image- and study-level IDs.","metadata":{}},{"cell_type":"markdown","source":"**Please upvote if you found it useful to you**","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nfrom glob import glob\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nfrom skimage import exposure\nimport ast\nimport cv2\nimport warnings\nimport os\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading and exploring data (EDA)","metadata":{}},{"cell_type":"code","source":"files_path = '../input/siim-covid19-detection'\ntrain_img_df = pd.read_csv(\"../input/siim-covid19-detection/train_image_level.csv\")\ntrain_study_data = pd.read_csv(\"../input/siim-covid19-detection/train_study_level.csv\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"| id    | unique study identifier                                      |\n|-------|--------------------------------------------------------------|\n| boxes | bounding boxes in easily-readable dictionary format          |\n| label | the correct prediction label for the provided bounding boxes |","metadata":{}},{"cell_type":"markdown","source":"# train_image_level.csv file","metadata":{}},{"cell_type":"code","source":"train_img_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Total (rows, cols)**","metadata":{}},{"cell_type":"code","source":"train_img_df.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_img_df.info()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Count missing values\ntrain_img_df.isnull().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# missing values display\n\ntrain_img_df[train_img_df.isnull().any(axis=1)].head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"2,040 null values on boxes column  ","metadata":{}},{"cell_type":"markdown","source":"# **train_study_level.csv file**","metadata":{}},{"cell_type":"code","source":"train_study_data = pd.read_csv(\"../input/siim-covid19-detection/train_study_level.csv\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study_data.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Total (rows, cols)**","metadata":{}},{"cell_type":"code","source":"train_study_data.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study_data.info()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_result = ['Negative for Pneumonia',\n                'Typical Appearance',\n                'Indeterminate Appearance',\n                 'Atypical Appearance']\nnp.unique(train_study_data[study_result].values, axis=0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n**train_study_level.csv:**\n\n\n| id                       | unique study identifier                                  | value                     |\n|--------------------------|----------------------------------------------------------|---------------------------|\n| Negative for Pneumonia   | 1: if the study is negative for pneumonia, 0: otherwise  | 1\t0\t0\t0 |\n| Typical Appearance       | 1: if the study has this appearance, 0: otherwise        | 0\t1\t0\t0 |\n| Indeterminate Appearance | 1: if the study has this appearance, 0: otherwise        | 0\t0\t1\t0 |\n| Atypical Appearance      | 1: if the study has this appearance, 0: otherwise        | 0\t0\t0\t1 |","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize = (10,5))\nplt.bar([1,2,3,4], train_study_data[study_result].values.sum(axis=0))\nplt.xticks([1,2,3,4],study_result)\nplt.ylabel('Frequency')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ndef img_list(path, voi_lut=True, fix_monochrome=True):\n    dicom = pydicom.read_file(path)\n    if voi_lut:\n        data = apply_voi_lut(dicom.pixel_array, dicom)\n    else:\n        data = dicom.pixel_array\n    if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n        data = np.amax(data) - data\n    data = data - np.min(data)\n    data = data / np.max(data)\n    data = (data * 255).astype(np.uint8)\n    return data\n        \n    \ndef plot_img(img, size=(7, 7), is_rgb=True,\n             title=\"\", cmap='gray'):\n    plt.figure(figsize=size)\n    plt.imshow(img, cmap=cmap)\n    plt.suptitle(title)\n    plt.show()\n\ndef plot_imgs(imgs, cols=5, size=7, is_rgb=True, title=\"\",\n              cmap='gray', img_size=(500,500)):\n    rows = len(imgs)//cols + 1\n    fig = plt.figure(figsize=(cols*size, rows*size))\n    for i, img in enumerate(imgs):\n        if img_size is not None:\n            img = cv2.resize(img, img_size)\n        fig.add_subplot(rows, cols, i+1)\n        plt.imshow(img, cmap=cmap)\n    plt.suptitle(title)\n    plt.show()\ndicom_paths = glob(f'{files_path}/train/*/*/*.dcm')\nimgs = [img_list(path) for path in dicom_paths[:10]]\nplot_imgs(imgs)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Apply histogram equalization on images (contrast adjustment).**","metadata":{}},{"cell_type":"code","source":"imgs = [exposure.equalize_hist(img) for img in imgs]\nplot_imgs(imgs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"boxes = ast.literal_eval(train_img_df.loc[0, 'boxes'])\nboxes","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study_data['StudyInstanceUID'] = train_study_data['id'].apply(lambda x: x.replace('_study', ''))\ndel train_study_data['id']\ntrain_img_df = train_img_df.merge(train_study_data, on='StudyInstanceUID')\ntrain_img_df.head()\ndef bar_plot(train_img_df, variable):\n    var = train_img_df[variable]\n    varValue = var.value_counts()\n    \n    # visualize\n    plt.figure(figsize = (9,3))\n    plt.bar(varValue.index, varValue)\n    plt.xticks(varValue.index, varValue.index.values)\n    plt.ylabel(\"Frequency\")\n    plt.title(variable)\n    plt.show()\n    print(\"{}: \\n {}\".format(variable,varValue))\n    \ntrain_img_df['target'] = 'Negative for Pneumonia'\ntrain_img_df.loc[train_img_df['Typical Appearance']==1, 'target'] = 'Typical Appearance'\ntrain_img_df.loc[train_img_df['Indeterminate Appearance']==1, 'target'] = 'Indeterminate Appearance'\ntrain_img_df.loc[train_img_df['Atypical Appearance']==1, 'target'] = 'Atypical Appearance'\nbar_plot(train_img_df, 'target') ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#train_img_df[\"target\"].value_counts().plot(kind = 'pie', autopct='%1.1f%%', figsize=(6, 6)).legend()\ntrain_img_df[\"target\"].value_counts().plot(kind = 'pie', autopct='%1.1f%%', figsize=(6, 6))\nprint(train_img_df.target.value_counts())\n#pie_plot(train_img_df, 'target')   ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot Images with bounding box","metadata":{}},{"cell_type":"code","source":"appearance_types = ['Typical Appearance','Negative for Pneumonia', 'Indeterminate Appearance', 'Atypical Appearance']\nunique_classes = np.unique(train_img_df[appearance_types].values, axis=0)\nimgs = []\nlabel2color = {\n    '[1, 0, 0, 0]': [0,255,0], # Typical Appearance\n    '[0, 1, 0, 0]': [255,0,0], # Indeterminate Appearance\n    '[0, 0, 1, 0]': [0,0,255], # Atypical Appearance\n    '[0, 0, 0, 1]': [255,255,0], # Negative for Pneumonia\n}\nprint('Typical Appearance: green')\nprint('Indeterminate Appearance: red')\nprint('Atypical Appearance: blue')\nprint('Negative for Pneumonia: yellow')\nprint(' \\n  \\n ')\nthickness = 5\nscale = 5\n\nfor _, row in train_img_df[train_img_df['Negative for Pneumonia']==0].iloc[:10].iterrows():\n    study_id = row['StudyInstanceUID']\n    img_path = glob(f'{files_path}/train/{study_id}/*/*')[0]\n    img = img_list(path=img_path)\n    img = cv2.resize(img, None, fx=1/scale, fy=1/scale)\n    img = np.stack([img, img, img], axis=-1)\n    \n    claz = row[appearance_types].values\n    color = label2color[str(claz.tolist())]\n\n    bboxes = []\n    bbox = []\n    for i, l in enumerate(row['label'].split(' ')):\n        if (i % 6 == 0) | (i % 6 == 1):\n            continue\n        bbox.append(float(l)/scale)\n        if i % 6 == 5:\n            bboxes.append(bbox)\n            bbox = []    \n    \n    for box_frame in bboxes:\n        img = cv2.rectangle(\n            img,\n            (int(box_frame[0]), int(box_frame[1])),\n            (int(box_frame[2]), int(box_frame[3])),\n            color, thickness\n    )\n    img = cv2.resize(img, (500,500))\n    imgs.append(img)\n    \nplot_imgs(imgs, cmap=None)\n\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submit the result","metadata":{}},{"cell_type":"code","source":"submission_df = pd.read_csv('../input/siim-covid19-detection/sample_submission.csv')\nsubmission_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Please upvote if you found it useful to you !.**","metadata":{}}]}