{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.10","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":26680,"databundleVersionId":2283525,"sourceType":"competition"}],"dockerImageVersionId":30096,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n

👨‍💻 Getting Started with SIIM-FISABIO-RSNA COVID-19 Detection

\n \n
","metadata":{}},{"cell_type":"markdown","source":"
","metadata":{}},{"cell_type":"markdown","source":"
\n

Introduction

\n

At the end of 2019, the coronavirus disease aroused in “Wuhan”, the city of china. Since January 2020, It has been extensively increased all around the world. On 30 January 2020, the WHO announced public health emergency internationally because of COVID-19. The disease was named coronavirus disease (COVID-19) by WHO in February 2020. The epidemic COVID-19 disease is caused by a virus called \"Severe Acute Respiratory Syndrome\" coronavirus-2 (SARS-CoV-2). The coronavirus is a family of several viruses that causes disease like \"Severe Acute Respiratory Syndrome\" (SARS-CoV) and \"Middle East Respiratory Syndrome\". Nowadays, countries like the USA, India, Brazil, and Russia struggling with the COVID-19 as it is the cause of the increase in the death rate. It can be spread to humans as well as other species of animals such as cats, cattle, bats, and camels. It is can be deadly for people with a weakened immune system. Physicians and specialists of infectious diseases around the world are starving hard to discover a treatment for the disease. The diseases can be spreading by touching and sitting or standing close to infected person. So, the COVID-19 infected person should to be identified and isolated before he might infect others. The earlier stage symptoms of COVID-19 include “cough”, “fatigue”, “fever”, and “myalgia”. The more intense situation of disease includes the heart damage, the respiratory problem, and internal infections. Some other abnormal states are observed in Covid-19 patients Chest CT and X-ray images. In most cases, the doctors use the patient’s X-ray and CT scan images of the patient’s chest for earlier stage diagnose of COVID-19. Chest radiography is the most widely used method of pneumonia diagnosis worldwide. It is a faster and cheaper clinical method of diagnosis. It is preferred over computed tomography (CT) and MRI because of lower radiation dose, easy availability, and cost. So, X-ray is the first choice of the radiologist for chest pathology detection and it is also applied for confirmation of COVID-19 patients. Therefore, this study focus on the use of X-ray images for COVID-19 detection in patients.

\n
\n
\n Problem description/definition: \n
\n\n\n\n
\nEvolution measures: \n
\n\n
\n\n \n
\nSource of Data: \n
\n https://www.kaggle.com/c/siim-covid19-detection\n \n
","metadata":{}},{"cell_type":"markdown","source":"
\n

Importing Python Libraries 📕 📗 📘 📙

\n \n
","metadata":{}},{"cell_type":"code","source":"import os\nimport seaborn as sns\nfrom pandas_profiling import ProfileReport\nimport pydicom\nfrom glob import glob\nimport numpy as np\nimport pandas as pd\nfrom tqdm.notebook import tqdm\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nimport matplotlib.pyplot as plt\nfrom skimage import exposure\nimport cv2\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:13:19.553350Z","iopub.execute_input":"2023-11-30T19:13:19.553910Z","iopub.status.idle":"2023-11-30T19:13:22.182884Z","shell.execute_reply.started":"2023-11-30T19:13:19.553812Z","shell.execute_reply":"2023-11-30T19:13:22.182020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

Sample Images

\n \n
","metadata":{}},{"cell_type":"code","source":"dataset_dir = '../input/siim-covid19-detection'\ndef dicom2array(path, voi_lut=True, fix_monochrome=True):\n dicom = pydicom.read_file(path)\n if voi_lut:\n data = apply_voi_lut(dicom.pixel_array, dicom)\n else:\n data = dicom.pixel_array\n if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\n data = data - np.min(data)\n data = data / np.max(data)\n data = (data * 255).astype(np.uint8)\n return data\n \n \ndef plot_img(img, size=(7, 7), is_rgb=True, title=\"\", cmap='gray'):\n plt.figure(figsize=size)\n plt.imshow(img, cmap=cmap)\n plt.suptitle(title)\n plt.show()\n\ndef plot_imgs(imgs, cols=5, size=7, is_rgb=True, title=\"\", cmap='gray', img_size=(500,500)):\n rows = len(imgs)//cols + 1\n fig = plt.figure(figsize=(cols*size, rows*size))\n for i, img in enumerate(imgs):\n if img_size is not None:\n img = cv2.resize(img, img_size)\n fig.add_subplot(rows, cols, i+1)\n plt.imshow(img, cmap=cmap)\n plt.suptitle(title)\n plt.show()\ndicom_paths = glob(f'{dataset_dir}/train/*/*/*.dcm')\nimgs = [dicom2array(path) for path in dicom_paths[:20]]\nplot_imgs(imgs)","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:13:22.184341Z","iopub.execute_input":"2023-11-30T19:13:22.184842Z","iopub.status.idle":"2023-11-30T19:14:48.898072Z","shell.execute_reply.started":"2023-11-30T19:13:22.184807Z","shell.execute_reply":"2023-11-30T19:14:48.897016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

The above sample Images after applying preprocessing Technique which is equalize histogram

\n \n
","metadata":{}},{"cell_type":"code","source":"imgs = [exposure.equalize_hist(img) for img in imgs]\nplot_imgs(imgs)","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:14:48.900084Z","iopub.execute_input":"2023-11-30T19:14:48.900665Z","iopub.status.idle":"2023-11-30T19:15:00.704694Z","shell.execute_reply.started":"2023-11-30T19:14:48.900622Z","shell.execute_reply":"2023-11-30T19:15:00.703403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If we foucs deeply on the last image, it was not clear before applying the equalize histogram but after that it is clear and give the meaningful information.","metadata":{}},{"cell_type":"markdown","source":"
\n

Loading data

\n \n
","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(f'{dataset_dir}/train_image_level.csv')\ntrain_study = pd.read_csv(f'{dataset_dir}/train_study_level.csv')","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:00.706560Z","iopub.execute_input":"2023-11-30T19:15:00.706915Z","iopub.status.idle":"2023-11-30T19:15:00.777258Z","shell.execute_reply.started":"2023-11-30T19:15:00.706880Z","shell.execute_reply":"2023-11-30T19:15:00.775969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

Exploratory data analysis

\n \n
","metadata":{}},{"cell_type":"markdown","source":"
\n

Train

\n \n
","metadata":{}},{"cell_type":"code","source":"# Five top records of data\n\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:00.778902Z","iopub.execute_input":"2023-11-30T19:15:00.779366Z","iopub.status.idle":"2023-11-30T19:15:00.808028Z","shell.execute_reply.started":"2023-11-30T19:15:00.779319Z","shell.execute_reply":"2023-11-30T19:15:00.806840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Five last records of data\n\ntrain.tail()","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:00.809790Z","iopub.execute_input":"2023-11-30T19:15:00.810171Z","iopub.status.idle":"2023-11-30T19:15:00.823484Z","shell.execute_reply.started":"2023-11-30T19:15:00.810136Z","shell.execute_reply":"2023-11-30T19:15:00.822284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Coloumns/features in data\n\ntrain.columns","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:00.824782Z","iopub.execute_input":"2023-11-30T19:15:00.825091Z","iopub.status.idle":"2023-11-30T19:15:00.831293Z","shell.execute_reply.started":"2023-11-30T19:15:00.825062Z","shell.execute_reply":"2023-11-30T19:15:00.830322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Length of data\n\nprint('lenght of data is', len(train))","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:00.834334Z","iopub.execute_input":"2023-11-30T19:15:00.834808Z","iopub.status.idle":"2023-11-30T19:15:00.842283Z","shell.execute_reply.started":"2023-11-30T19:15:00.834775Z","shell.execute_reply":"2023-11-30T19:15:00.840893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Shape of data\n\ntrain.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:00.844539Z","iopub.execute_input":"2023-11-30T19:15:00.844865Z","iopub.status.idle":"2023-11-30T19:15:00.853188Z","shell.execute_reply.started":"2023-11-30T19:15:00.844835Z","shell.execute_reply":"2023-11-30T19:15:00.852210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Data information\n\ntrain.info()","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:00.854635Z","iopub.execute_input":"2023-11-30T19:15:00.855263Z","iopub.status.idle":"2023-11-30T19:15:00.881860Z","shell.execute_reply.started":"2023-11-30T19:15:00.855194Z","shell.execute_reply":"2023-11-30T19:15:00.880352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Data types of all coloumns\n\ntrain.dtypes","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:00.883478Z","iopub.execute_input":"2023-11-30T19:15:00.883809Z","iopub.status.idle":"2023-11-30T19:15:00.892076Z","shell.execute_reply.started":"2023-11-30T19:15:00.883772Z","shell.execute_reply":"2023-11-30T19:15:00.890958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checking missing Values\n\ntrain[train.isnull().any(axis=1)].head()","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:00.893564Z","iopub.execute_input":"2023-11-30T19:15:00.893916Z","iopub.status.idle":"2023-11-30T19:15:00.913439Z","shell.execute_reply.started":"2023-11-30T19:15:00.893866Z","shell.execute_reply":"2023-11-30T19:15:00.912252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Count of missing values\n\nnp.sum(train.isnull().any(axis=1))","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:00.914908Z","iopub.execute_input":"2023-11-30T19:15:00.915564Z","iopub.status.idle":"2023-11-30T19:15:00.925385Z","shell.execute_reply.started":"2023-11-30T19:15:00.915524Z","shell.execute_reply":"2023-11-30T19:15:00.924200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Counts of missing values in each column\n\ntrain.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:00.926961Z","iopub.execute_input":"2023-11-30T19:15:00.927437Z","iopub.status.idle":"2023-11-30T19:15:00.941630Z","shell.execute_reply.started":"2023-11-30T19:15:00.927393Z","shell.execute_reply":"2023-11-30T19:15:00.940227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

Data Visualization of train

\n \n
","metadata":{}},{"cell_type":"code","source":"train_vis = ProfileReport(train, title=\"train_image_level_report\")\ntrain_vis","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:00.943109Z","iopub.execute_input":"2023-11-30T19:15:00.943505Z","iopub.status.idle":"2023-11-30T19:15:06.090067Z","shell.execute_reply.started":"2023-11-30T19:15:00.943471Z","shell.execute_reply":"2023-11-30T19:15:06.089157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

train_study

\n \n
","metadata":{}},{"cell_type":"code","source":"# Five top records of data\n\ntrain_study.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:06.091381Z","iopub.execute_input":"2023-11-30T19:15:06.091893Z","iopub.status.idle":"2023-11-30T19:15:06.104027Z","shell.execute_reply.started":"2023-11-30T19:15:06.091842Z","shell.execute_reply":"2023-11-30T19:15:06.103154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Five last records of data\n\ntrain_study.tail()","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:06.105431Z","iopub.execute_input":"2023-11-30T19:15:06.105955Z","iopub.status.idle":"2023-11-30T19:15:06.125681Z","shell.execute_reply.started":"2023-11-30T19:15:06.105900Z","shell.execute_reply":"2023-11-30T19:15:06.124346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

Data Visualization of train study

\n \n
","metadata":{}},{"cell_type":"code","source":"train_st = ProfileReport(train_study, title=\"train study of images\")\ntrain_st","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:06.127226Z","iopub.execute_input":"2023-11-30T19:15:06.127567Z","iopub.status.idle":"2023-11-30T19:15:15.718945Z","shell.execute_reply.started":"2023-11-30T19:15:06.127536Z","shell.execute_reply":"2023-11-30T19:15:15.718027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Merging train study in train","metadata":{}},{"cell_type":"code","source":"train_study['StudyInstanceUID'] = train_study['id'].apply(lambda x: x.replace('_study', ''))\ndel train_study['id']\ntrain = train.merge(train_study, on='StudyInstanceUID')\ntrain.head()\ndef bar_plot(train_df, variable):\n var = train_df[variable]\n varValue = var.value_counts()\n \n # visualize\n plt.figure(figsize = (9,3))\n plt.bar(varValue.index, varValue)\n plt.xticks(varValue.index, varValue.index.values)\n plt.ylabel(\"Frequency\")\n plt.title(variable)\n plt.show()\n print(\"{}: \\n {}\".format(variable,varValue))\n \ntrain['target'] = 'Negative for Pneumonia'\ntrain.loc[train['Typical Appearance']==1, 'target'] = 'Typical Appearance'\ntrain.loc[train['Indeterminate Appearance']==1, 'target'] = 'Indeterminate Appearance'\ntrain.loc[train['Atypical Appearance']==1, 'target'] = 'Atypical Appearance'\nbar_plot(train, 'target') ","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:15.720391Z","iopub.execute_input":"2023-11-30T19:15:15.720932Z","iopub.status.idle":"2023-11-30T19:15:15.910063Z","shell.execute_reply.started":"2023-11-30T19:15:15.720875Z","shell.execute_reply":"2023-11-30T19:15:15.909299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"target\"].value_counts().head(7).plot(kind = 'pie', autopct='%1.1f%%', figsize=(8, 8)).legend()","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:15.911377Z","iopub.execute_input":"2023-11-30T19:15:15.911879Z","iopub.status.idle":"2023-11-30T19:15:16.177587Z","shell.execute_reply.started":"2023-11-30T19:15:15.911837Z","shell.execute_reply":"2023-11-30T19:15:16.176318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(2,2,figsize=(20,16))\nsns.kdeplot(train[\"Negative for Pneumonia\"], shade=True,ax=ax[0,0],color=\"#ffb4a2\")\nax[0,0].set_title(\"Negative for Pneumonia Distribution\",font=\"Serif\")\nsns.kdeplot(train[\"Typical Appearance\"], shade=True,ax=ax[0,1],color=\"#e5989b\")\nax[0,1].set_title(\"Typical Appearance Distribution\",font=\"Serif\")\nsns.kdeplot(train[\"Indeterminate Appearance\"], shade=True,ax=ax[1,0],color=\"#b5838d\")\nax[1,0].set_title(\"Indeterminate Appearance Distribution\",font=\"Serif\")\nsns.kdeplot(train[\"Atypical Appearance\"], shade=True,ax=ax[1,1],color=\"#6d6875\")\nax[1,1].set_title(\"Atypical Appearance Distribution\",font=\"Serif\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:16.179436Z","iopub.execute_input":"2023-11-30T19:15:16.180205Z","iopub.status.idle":"2023-11-30T19:15:17.086284Z","shell.execute_reply.started":"2023-11-30T19:15:16.180156Z","shell.execute_reply":"2023-11-30T19:15:17.085039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

Localization of Labels with Bounding Boxes

\n \n
","metadata":{}},{"cell_type":"code","source":"#train = train[~train.boxes.isnull()] \nclass_names = ['Typical Appearance','Negative for Pneumonia', 'Indeterminate Appearance', 'Atypical Appearance']\nunique_classes = np.unique(train[class_names].values, axis=0)\nimgs = []\nlabel2color = {\n '[1, 0, 0, 0]': [255,0,0], # Typical Appearance\n '[0, 1, 0, 0]': [0,255,0], # Indeterminate Appearance\n '[0, 0, 1, 0]': [0,0,255], # Atypical Appearance\n '[0, 0, 0, 1]': [255,255,0], # Negative for Pneumonia\n}\nprint('Typical Appearance: red')\nprint('Indeterminate Appearance: green')\nprint('Atypical Appearance: blue')\nprint('Negative for Pneumonia: yellow')\nprint(' \\n \\n ')\nthickness = 4\nscale = 5\nfor _, row in train[train['Negative for Pneumonia']==0].iloc[:20].iterrows():\n study_id = row['StudyInstanceUID']\n img_path = glob(f'{dataset_dir}/train/{study_id}/*/*')[0]\n img = dicom2array(path=img_path)\n img = cv2.resize(img, None, fx=1/scale, fy=1/scale)\n img = np.stack([img, img, img], axis=-1)\n \n claz = row[class_names].values\n color = label2color[str(claz.tolist())]\n\n bboxes = []\n bbox = []\n for i, l in enumerate(row['label'].split(' ')):\n if (i % 6 == 0) | (i % 6 == 1):\n continue\n bbox.append(float(l)/scale)\n if i % 6 == 5:\n bboxes.append(bbox)\n bbox = [] \n \n for box in bboxes:\n img = cv2.rectangle(\n img,\n (int(box[0]), int(box[1])),\n (int(box[2]), int(box[3])),\n color, thickness\n )\n img = cv2.resize(img, (500,500))\n imgs.append(img)\n \nplot_imgs(imgs, cmap=None)","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:17.088008Z","iopub.execute_input":"2023-11-30T19:15:17.088484Z","iopub.status.idle":"2023-11-30T19:15:28.028798Z","shell.execute_reply.started":"2023-11-30T19:15:17.088437Z","shell.execute_reply":"2023-11-30T19:15:28.027531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

Guideline to submit the results

\n \n
","metadata":{}},{"cell_type":"markdown","source":"> **Check the sample_submission.csv file so we will know the format that how can submit the predictions. Before we do so, it's worth reminding ourselves that this is a code-only competition, meaning that your submission file has to be generated in a script/notebook. The sample_submission.csv file demonstrated what kind of file needs to be produced:**","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('../input/siim-covid19-detection/sample_submission.csv')\nid_laststr_list = []\nfor i in range(df.shape[0]):\n id_laststr_list.append(df.loc[i,'id'][-1])\ndf['id_last_str'] = id_laststr_list\nstudy_len = df[df['id_last_str'] == 'y'].shape[0]\nstudy_str = 'negative 1 0 0 1 1 atypical 1 0 0 1 1 typical 1 0 0 1 1 indeterminate 1 0 0 1 1'\nfor i in range(study_len):\n df.loc[i, 'PredictionString'] = study_str\nsubmission_file = df[['id', 'PredictionString']]","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:28.032355Z","iopub.execute_input":"2023-11-30T19:15:28.032733Z","iopub.status.idle":"2023-11-30T19:15:28.238849Z","shell.execute_reply.started":"2023-11-30T19:15:28.032700Z","shell.execute_reply":"2023-11-30T19:15:28.237573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_file.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:28.240495Z","iopub.execute_input":"2023-11-30T19:15:28.240843Z","iopub.status.idle":"2023-11-30T19:15:28.253987Z","shell.execute_reply.started":"2023-11-30T19:15:28.240804Z","shell.execute_reply":"2023-11-30T19:15:28.252906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> **The submission file is look like above. We also have to provide the image-level bounding box. These will be of the format [class ID] [confidence score] [bounding box] as described earlier.**","metadata":{}},{"cell_type":"code","source":"submission_file.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-11-30T19:15:28.255553Z","iopub.execute_input":"2023-11-30T19:15:28.255995Z","iopub.status.idle":"2023-11-30T19:15:28.280446Z","shell.execute_reply.started":"2023-11-30T19:15:28.255956Z","shell.execute_reply":"2023-11-30T19:15:28.279362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> **We submitted the predictions but we will come back soon with model training and a lot more.**","metadata":{}}]}