{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Overview","metadata":{}},{"cell_type":"markdown","source":"\n\"Xray.png\"\n","metadata":{}},{"cell_type":"markdown","source":" Covid-19 is a contagious disease. Many people lost their lives because of covid-19. Only way to stop this \n \n disease is maintaining social distancing. This disease had spread worldwide and the world is in financial \n \n crisis, many lost their jobs.\n \n Symptoms of covid-19 are variable, often cases include\n \n * Fever\n \n * Cough\n \n * Fatigue\n \n * Breathing difficulties\n \n * Loss of smell, taste\n \n One method of detecting covid-19 is from chest X-rays scans of a patients.\n \n In this competition, we’ll identify and localize COVID-19 abnormalities on chest radiographs. In particular, \n \n we'll categorize the radiographs as negative for pneumonia or typical, indeterminate, or atypical for \n \n COVID - 19. This is an object detection and classification problem.","metadata":{}},{"cell_type":"markdown","source":"## Data description","metadata":{}},{"cell_type":"code","source":"import warnings # importing warnings module\nwarnings.filterwarnings('ignore') # we will ignore all the warnings generated","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\npath = '../input/siim-covid19-detection' # specifying the path\nprint(os.listdir(path)) # printing all the files in the given path","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" The dataset has 5 files namely sample_submission.csv, train_image_level.csv, test,train,train_study_level.csv.","metadata":{}},{"cell_type":"markdown","source":"## Loading data","metadata":{}},{"cell_type":"code","source":"import pandas as pd\ntrain_image_level = pd.read_csv(path+'/train_image_level.csv') #read_csv is used to read the .csv file\ntrain_image_level.head() # head() is used to print the first five rows","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" train_image_level.csv - \n \n the train image-level metadata, with one row for each image, including both correct labels and \n \n any bounding boxes in a dictionary format. Some images in both test and train have multiple bounding boxes.\n \n columns :\n \n id - unique image identifier\n\n boxes - bounding boxes in easily-readable dictionary format\n\n label - the correct prediction label for the provided bounding boxes","metadata":{}},{"cell_type":"code","source":"train_study_level = pd.read_csv(path+'/train_study_level.csv') #read_csv is used to read the .csv file\ntrain_study_level.head() # head() is used to print the first five rows","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" train_study_level.csv - \n \n the train study-level metadata, with one row for each study, including correct labels.\n \n columns :\n \n id - unique study identifier\n \n Negative for Pneumonia - 1 if the study is negative for pneumonia, 0 otherwise\n\n Typical Appearance - 1 if the study has this appearance, 0 otherwise\n\n Indeterminate Appearance - 1 if the study has this appearance, 0 otherwise\n\n Atypical Appearance - 1 if the study has this appearance, 0 otherwise\n","metadata":{}},{"cell_type":"markdown","source":"## Distribution of classes","metadata":{}},{"cell_type":"code","source":"import plotly.express as pe # plotly.express is used for beautiful visualization\ncount_of_classes = train_study_level.loc[ : , train_study_level.columns != 'id'].sum() #get the sum of the classes\ncolumn_names = train_study_level.columns[1:] # creating a list with the column names\ndata = {'Count_of_classes':list(count_of_classes),\n 'class_names':list(column_names)}\ndf = pd.DataFrame(data) # creating a dataframe with count of classes and column names\npie_chart = pe.pie(df, values='Count_of_classes', names='class_names', title='Distribution of classes in data')\npie_chart.show() # displaying the pie chart","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" ","metadata":{}},{"cell_type":"markdown","source":" Typical appearances are most with 47.2 percentile and atypical appearances are least with 7.83 percentile.\n \n key take away :\n \n Data imbalance is there in the given data","metadata":{}},{"cell_type":"markdown","source":"## Plotting raw images","metadata":{}},{"cell_type":"code","source":"# reference : https://www.kaggle.com/trungthanhnguyen0502/eda-vinbigdata-chest-x-ray-abnormalities\nfrom glob import glob\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport cv2\ndef dicom_images(path, voi_lut=True, fix_monochrome=True):\n '''This function is used to convert the dcom images '''\n dicom = pydicom.read_file(path) # readingg the dcom files\n # VOI LUT (if available by DICOM device) is used to\n # transform raw DICOM data to \"human-friendly\" view\n if voi_lut:\n data = apply_voi_lut(dicom.pixel_array, dicom)\n else:\n data = dicom.pixel_array\n # depending on this value, X-ray may look inverted - fix that:\n if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\n data = data - np.min(data)\n data = data / np.max(data)\n data = (data * 255).astype(np.uint8)\n return data\n \n \ndef plot_img(img, size=(7, 7), is_rgb=True, title=\"\", cmap='gray'):\n '''This function is used to plot the images'''\n plt.figure(figsize=size)\n plt.imshow(img, cmap=cmap)\n plt.suptitle(title)\n plt.show()\n\n\ndef plot_imgs(imgs, cols=4, size=7, is_rgb=True, title=\"\", cmap='gray', img_size=(500,500)):\n rows = len(imgs)//cols + 1\n fig = plt.figure(figsize=(cols*size, rows*size))\n for i, img in enumerate(imgs):\n if img_size is not None:\n img = cv2.resize(img, img_size)\n fig.add_subplot(rows, cols, i+1)\n plt.imshow(img, cmap=cmap)\n plt.suptitle(title)\n plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_paths = glob(f'{path}/train/*/*/*.dcm') # specifying the image paths\nimgs = [dicom_images(path) for path in image_paths[:4]]\nplot_imgs(imgs) # plotting images","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preprocessing raw images - [Convert dicom to np.array - the correct way]","metadata":{}},{"cell_type":"markdown","source":" Thanks to this great idea : https://www.kaggle.com/raddar/convert-dicom-to-np-array-the-correct-way\n\n raw dicom data is not actually linearly convertable to \"human-friendly\" png/jpg. In fact, most of DICOM's store pixel \n \n values in exponential scale, which is resolved by standard standard DICOM viewers.So in order to get jpg/png as \n \n radiologists would initially see in their workspace, you need to apply some transformations. DICOM metadata stores \n \n information how to make such \"human-friendly\" transformations.","metadata":{}},{"cell_type":"code","source":"image_paths = glob(f'{path}/train/*/*/*.dcm') # path\nimgs = [dicom_images(path,fix_monochrome=False) for path in image_paths[:4]]\nplot_imgs(imgs) # plotting images","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" with fix_monochrome = False, images are clearer than the raw images","metadata":{}},{"cell_type":"markdown","source":"## Preprocessing raw images - trick (2) [Histogram equalization]","metadata":{}},{"cell_type":"code","source":"from skimage import exposure\nimgs = [exposure.equalize_hist(img) for img in imgs] # equalizing histograms\nplot_imgs(imgs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" With histogram equalization, images are more clearer than the raw images.","metadata":{}},{"cell_type":"markdown","source":"## Plotting bounding boxes","metadata":{}},{"cell_type":"code","source":"train_study_level['id'][:5] # displaying 1st five rows of id column","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image_level['StudyInstanceUID'][:5] # displaying 1st five rows of id column","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" If we observe clearly , in both the dataframe in id , we have suffixes different, we will remove those suffix and \n \n merge the dataframes","metadata":{}},{"cell_type":"code","source":"train_study_level['StudyInstanceUID'] = train_study_level['id'].apply(lambda x: x.replace('_study', '')) # replacing the suffix with ''\ndel train_study_level['id'] # deleting the id column in one dataframe\ntrain = train_study_level.merge(train_image_level, on='StudyInstanceUID') # merging the dataframes\ntrain.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# reference : https://www.kaggle.com/yujiariyasu/plot-3positive-classes by YujiAriyasu\nclass_names = ['Typical Appearance', 'Indeterminate Appearance', 'Atypical Appearance']\nimgs = []\n\nimgs = []\nlabel2color = {\n '[1, 0, 0]': [0,255,0], # Typical Appearance\n}\n\nthickness = 3\nscale = 5\n\nfor _, row in train[train['Typical Appearance'] == 1].iloc[:16].iterrows():\n study_id = row['StudyInstanceUID']\n img_path = glob(f'{path}/train/{study_id}/*/*')[0]\n img = dicom_images(path=img_path)\n img = cv2.resize(img, None, fx=1/scale, fy=1/scale)\n img = np.stack([img, img, img], axis=-1)\n \n claz = row[class_names].values\n color = label2color[str(claz.tolist())]\n\n bboxes = []\n bbox = []\n for i, l in enumerate(row['label'].split(' ')):\n if (i % 6 == 0) | (i % 6 == 1):\n continue\n bbox.append(float(l)/scale)\n if i % 6 == 5:\n bboxes.append(bbox)\n bbox = [] \n \n for box in bboxes:\n img = cv2.rectangle(\n img,\n (int(box[0]), int(box[1])),\n (int(box[2]), int(box[3])),\n color, thickness\n )\n img = cv2.resize(img, (500,500))\n imgs.append(img)\n \nplot_imgs(imgs, cmap=None)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# More work in progress !.... \n\n# If you like my work, please upvote it !..\n\n# Thanks for reading ....\n","metadata":{}}]}