{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# SIIM-FISABIO-RSNA COVID-19 Detection","metadata":{}},{"cell_type":"markdown","source":"### Business Objective: \n   #### Currently, COVID-19 can be diagnosed via polymerase chain reaction to detect genetic material from the virus\n   #### or chest radiograph. However, it can take a few hours and sometimes days before the molecular test results are back.\n   #### In this competition, we identify and localize COVID-19 abnormalities on chest radiographs. \n   #### In particular, we categorize the radiographs as negative for pneumonia or typical, \n   #### indeterminate, or atypical for COVID-19.THis will help radiologists diagnose the millions \n   #### of COVID-19 patients more confidently and quickly\n","metadata":{}},{"cell_type":"markdown","source":"##### Below is the approach taken in this Case Study analysis.\n","metadata":{}},{"cell_type":"markdown","source":"###### Step1 : Importing Data\n\nStep 2: Inspecting the Dataframe Train_Study_level dataset.\n\nStep3: Checking for Missing values and Duplicates in Train_Study_level dataset.\n\nStep4 : Adding new column Study_Result in Train_Study_level dataset and Performing EDA.\n\nStep5 : Inspecting dataframe Train_Image_level dataset.\n\nStep6 : Checking for Missing values and Duplicates in Train_Image_level dataset.\n\nStep7 : Merging both the datasets Train_Study_level and Train_Image_level\n\nStep8 : Feature Engineering on Train_Image_level dataset.Creating new columns by splitting data in Label column.\n\nStep9 : Performing EDA (both UniVariate and BiVariate analysis) on Merged dataset.","metadata":{}},{"cell_type":"markdown","source":"# DataSet Information\ntrain_study_level.csv - the train study-level metadata, with one row for each study, including correct labels.","metadata":{}},{"cell_type":"code","source":"# Suppressing Warnings\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.111127Z","iopub.execute_input":"2021-06-02T02:07:17.111398Z","iopub.status.idle":"2021-06-02T02:07:17.115786Z","shell.execute_reply.started":"2021-06-02T02:07:17.111371Z","shell.execute_reply":"2021-06-02T02:07:17.114846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Importing Pandas and NumPy\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n# Importing more modules\nimport seaborn as sns\n\n\n","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.121329Z","iopub.execute_input":"2021-06-02T02:07:17.121578Z","iopub.status.idle":"2021-06-02T02:07:17.131325Z","shell.execute_reply.started":"2021-06-02T02:07:17.121552Z","shell.execute_reply":"2021-06-02T02:07:17.130577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study_level_data = pd.read_csv(\"../input/siim-covid19-detection/train_study_level.csv\")","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.132498Z","iopub.execute_input":"2021-06-02T02:07:17.132928Z","iopub.status.idle":"2021-06-02T02:07:17.155710Z","shell.execute_reply.started":"2021-06-02T02:07:17.132892Z","shell.execute_reply":"2021-06-02T02:07:17.155063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"###### Step2: Inspecting Dataframe","metadata":{}},{"cell_type":"code","source":"\n\n#id:unique study identifier\n#Negative for Pneumonia:1 if the study is negative for pneumonia, 0 otherwise\n#Typical Appearance:1 if the study has this appearance, 0 otherwise\n#Indeterminate Appearance:1 if the study has this appearance, 0 otherwise\n#Atypical Appearance:1 if the study has this appearance, 0 otherwise\n","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.157173Z","iopub.execute_input":"2021-06-02T02:07:17.157639Z","iopub.status.idle":"2021-06-02T02:07:17.163220Z","shell.execute_reply.started":"2021-06-02T02:07:17.157601Z","shell.execute_reply":"2021-06-02T02:07:17.162207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study_level_data.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.164724Z","iopub.execute_input":"2021-06-02T02:07:17.165041Z","iopub.status.idle":"2021-06-02T02:07:17.188603Z","shell.execute_reply.started":"2021-06-02T02:07:17.165013Z","shell.execute_reply":"2021-06-02T02:07:17.187616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study_level_data.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.190176Z","iopub.execute_input":"2021-06-02T02:07:17.190390Z","iopub.status.idle":"2021-06-02T02:07:17.202443Z","shell.execute_reply.started":"2021-06-02T02:07:17.190366Z","shell.execute_reply":"2021-06-02T02:07:17.201885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study_level_data.info()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.204201Z","iopub.execute_input":"2021-06-02T02:07:17.204499Z","iopub.status.idle":"2021-06-02T02:07:17.222335Z","shell.execute_reply.started":"2021-06-02T02:07:17.204467Z","shell.execute_reply":"2021-06-02T02:07:17.220874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Above info shows that there are no null columns in the dataset.","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.223626Z","iopub.execute_input":"2021-06-02T02:07:17.223882Z","iopub.status.idle":"2021-06-02T02:07:17.235123Z","shell.execute_reply.started":"2021-06-02T02:07:17.223856Z","shell.execute_reply":"2021-06-02T02:07:17.234372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Check for duplicates\nsum(train_study_level_data.duplicated(subset = 'id')) == 0\n# No duplicate values","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.240595Z","iopub.execute_input":"2021-06-02T02:07:17.240901Z","iopub.status.idle":"2021-06-02T02:07:17.251728Z","shell.execute_reply.started":"2021-06-02T02:07:17.240820Z","shell.execute_reply":"2021-06-02T02:07:17.250324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are no duplicates in this dataset","metadata":{}},{"cell_type":"markdown","source":"##### Check for Missing Values","metadata":{}},{"cell_type":"code","source":"train_study_level_data.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.255202Z","iopub.execute_input":"2021-06-02T02:07:17.255438Z","iopub.status.idle":"2021-06-02T02:07:17.264590Z","shell.execute_reply.started":"2021-06-02T02:07:17.255415Z","shell.execute_reply":"2021-06-02T02:07:17.263349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are no missing values in the Train Study dataset.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Perform EDA","metadata":{}},{"cell_type":"code","source":"# Add a new feature \"Study Result\" which shows the result name of that row.\n\ntrain_study_level_data[\"Study_Result\"]='Negative'\ntrain_study_level_data.loc[train_study_level_data['Typical Appearance']==1, 'Study_Result'] = 'Typical'\ntrain_study_level_data.loc[train_study_level_data['Indeterminate Appearance']==1, 'Study_Result'] = 'Indeterminate'\ntrain_study_level_data.loc[train_study_level_data['Atypical Appearance']==1, 'Study_Result'] = 'Atypical'\n\ntrain_study_level_data.head(5)","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.364524Z","iopub.execute_input":"2021-06-02T02:07:17.364813Z","iopub.status.idle":"2021-06-02T02:07:17.381582Z","shell.execute_reply.started":"2021-06-02T02:07:17.364776Z","shell.execute_reply":"2021-06-02T02:07:17.380813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study_level_data[\"Study_Result\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.382869Z","iopub.execute_input":"2021-06-02T02:07:17.383141Z","iopub.status.idle":"2021-06-02T02:07:17.392251Z","shell.execute_reply.started":"2021-06-02T02:07:17.383108Z","shell.execute_reply":"2021-06-02T02:07:17.391259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Plot a bar plot to see the Study results.","metadata":{}},{"cell_type":"code","source":"train_study_level_data[\"Study_Result\"].value_counts().plot(kind=\"bar\")\nplt.xlabel('Study Results')\nplt.ylabel('Count')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.394087Z","iopub.execute_input":"2021-06-02T02:07:17.394463Z","iopub.status.idle":"2021-06-02T02:07:17.532582Z","shell.execute_reply.started":"2021-06-02T02:07:17.394427Z","shell.execute_reply":"2021-06-02T02:07:17.531575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above BarPlot shows that Typical Appearence is the mostly got result.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Analyse Train image dataset","metadata":{}},{"cell_type":"code","source":"train_image_level_data = pd.read_csv(\"../input/siim-covid19-detection/train_image_level.csv\")","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.534039Z","iopub.execute_input":"2021-06-02T02:07:17.534287Z","iopub.status.idle":"2021-06-02T02:07:17.572701Z","shell.execute_reply.started":"2021-06-02T02:07:17.534263Z","shell.execute_reply":"2021-06-02T02:07:17.571753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image_level_data.head(10)","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.573850Z","iopub.execute_input":"2021-06-02T02:07:17.574153Z","iopub.status.idle":"2021-06-02T02:07:17.587297Z","shell.execute_reply.started":"2021-06-02T02:07:17.574118Z","shell.execute_reply":"2021-06-02T02:07:17.585672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Check for missing values in Train Image dataset","metadata":{}},{"cell_type":"code","source":"train_image_level_data.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.589107Z","iopub.execute_input":"2021-06-02T02:07:17.589538Z","iopub.status.idle":"2021-06-02T02:07:17.604253Z","shell.execute_reply.started":"2021-06-02T02:07:17.589506Z","shell.execute_reply":"2021-06-02T02:07:17.602954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Only boxes column has 2040 missing values. Lets analyse further to check when the boxes column doesnt have value.","metadata":{}},{"cell_type":"code","source":"#Check for duplicates\nsum(train_image_level_data.duplicated(subset = 'id')) == 0\n# No duplicate values","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.605419Z","iopub.execute_input":"2021-06-02T02:07:17.605631Z","iopub.status.idle":"2021-06-02T02:07:17.619166Z","shell.execute_reply.started":"2021-06-02T02:07:17.605609Z","shell.execute_reply":"2021-06-02T02:07:17.618198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Feature Engineering on Train Image DataSet","metadata":{}},{"cell_type":"markdown","source":"In Label Column, the format is as follows: [class ID] [confidence score] [bounding box]\n\nClass ID: Either opacity or none \n\nConfidence score: confidence from your neural network model. If none, the confidence is 1.\n\nBounding box:typical xmin ymin xmax ymax format. If class ID is none, the bounding box is 1 0 0 1 1.","metadata":{}},{"cell_type":"markdown","source":"### Based on the above interpretation, lets split the Label column and start analysing the images ","metadata":{}},{"cell_type":"code","source":"# 1.Class ID\n#Class ID is Either Opacity or None.\n#So lets check how many have opacity by splitting the ClassID as new column\n","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.620260Z","iopub.execute_input":"2021-06-02T02:07:17.620632Z","iopub.status.idle":"2021-06-02T02:07:17.630733Z","shell.execute_reply.started":"2021-06-02T02:07:17.620608Z","shell.execute_reply":"2021-06-02T02:07:17.629962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image_level_data['Class_ID'] = train_image_level_data.label.apply(lambda x: x.split()[0])\ntrain_image_level_data['Class_ID'].value_counts().plot(kind=\"bar\")\nplt.xlabel('Class_ID')\nplt.ylabel('Count')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.633776Z","iopub.execute_input":"2021-06-02T02:07:17.634128Z","iopub.status.idle":"2021-06-02T02:07:17.768061Z","shell.execute_reply.started":"2021-06-02T02:07:17.634099Z","shell.execute_reply":"2021-06-02T02:07:17.767037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Column 2 is Confidence score: confidence from your neural network model. If none, the confidence is 1","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.771540Z","iopub.execute_input":"2021-06-02T02:07:17.771827Z","iopub.status.idle":"2021-06-02T02:07:17.775181Z","shell.execute_reply.started":"2021-06-02T02:07:17.771783Z","shell.execute_reply":"2021-06-02T02:07:17.774227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image_level_data['Confidence_Score'] = train_image_level_data.label.apply(lambda x: x.split()[1])\ntrain_image_level_data['Confidence_Score'].value_counts()\n","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.776443Z","iopub.execute_input":"2021-06-02T02:07:17.776716Z","iopub.status.idle":"2021-06-02T02:07:17.799032Z","shell.execute_reply.started":"2021-06-02T02:07:17.776686Z","shell.execute_reply":"2021-06-02T02:07:17.798098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Confidence Score is 1 for all the data.","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.800320Z","iopub.execute_input":"2021-06-02T02:07:17.800670Z","iopub.status.idle":"2021-06-02T02:07:17.809960Z","shell.execute_reply.started":"2021-06-02T02:07:17.800633Z","shell.execute_reply":"2021-06-02T02:07:17.809405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Column 3 is bounding box parameters. From Boxes column we either get xmin ,ymin,width and height.Or we can use\n#label column to get xmin,ymin,xmax,ymax.Both will give same values","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.810740Z","iopub.execute_input":"2021-06-02T02:07:17.810941Z","iopub.status.idle":"2021-06-02T02:07:17.821833Z","shell.execute_reply.started":"2021-06-02T02:07:17.810920Z","shell.execute_reply":"2021-06-02T02:07:17.820874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image_level_data['boxes'][0]","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.822825Z","iopub.execute_input":"2021-06-02T02:07:17.823042Z","iopub.status.idle":"2021-06-02T02:07:17.833009Z","shell.execute_reply.started":"2021-06-02T02:07:17.823021Z","shell.execute_reply":"2021-06-02T02:07:17.832134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image_level_data['label'][0]","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.833655Z","iopub.execute_input":"2021-06-02T02:07:17.833853Z","iopub.status.idle":"2021-06-02T02:07:17.845704Z","shell.execute_reply.started":"2021-06-02T02:07:17.833832Z","shell.execute_reply":"2021-06-02T02:07:17.844539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#So lets add new columns for xmin,ymin,xmax and ymax by retrieving values from Label column","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.847031Z","iopub.execute_input":"2021-06-02T02:07:17.847331Z","iopub.status.idle":"2021-06-02T02:07:17.854684Z","shell.execute_reply.started":"2021-06-02T02:07:17.847301Z","shell.execute_reply":"2021-06-02T02:07:17.854105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image_level_data['x_min'] = train_image_level_data.label.apply(lambda x: float(x.split()[2]))\ntrain_image_level_data['y_min'] = train_image_level_data.label.apply(lambda x: float(x.split()[3]))\ntrain_image_level_data['x_max'] = train_image_level_data.label.apply(lambda x: float(x.split()[4]))\ntrain_image_level_data['y_max'] = train_image_level_data.label.apply(lambda x: float(x.split()[5]))\n\n\n","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.855468Z","iopub.execute_input":"2021-06-02T02:07:17.855769Z","iopub.status.idle":"2021-06-02T02:07:17.885849Z","shell.execute_reply.started":"2021-06-02T02:07:17.855747Z","shell.execute_reply":"2021-06-02T02:07:17.885289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Check the parameters now for 1 record.\n\ntrain_image_level_data.head(1).T","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.886587Z","iopub.execute_input":"2021-06-02T02:07:17.886888Z","iopub.status.idle":"2021-06-02T02:07:17.894965Z","shell.execute_reply.started":"2021-06-02T02:07:17.886866Z","shell.execute_reply":"2021-06-02T02:07:17.894044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#train_image_level_data[train_image_level_data['label']=='none'].value_counts().plot(kind='bar')\ntrain_image_level_data['OpacityCount']=train_image_level_data['label'].str.count('opacity')\ntrain_image_level_data['OpacityCount']","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.895785Z","iopub.execute_input":"2021-06-02T02:07:17.896041Z","iopub.status.idle":"2021-06-02T02:07:17.916875Z","shell.execute_reply.started":"2021-06-02T02:07:17.896018Z","shell.execute_reply":"2021-06-02T02:07:17.916002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#  Merge both datasets train Study and Train image to understand report result and report parameters link\n","metadata":{}},{"cell_type":"code","source":"\n##Column ID value in train_study_level_data is same as StudyInstanceID column value in train_Image_level_data\n\n\ntrain_study_level_data['StudyInstanceUID'] = train_study_level_data['id'].apply(lambda x: x.replace('_study', ''))\n##del train_study_level_data['id']\ntrain_Merged_df = train_study_level_data.merge(train_image_level_data, on='StudyInstanceUID')\ntrain_Merged_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.917996Z","iopub.execute_input":"2021-06-02T02:07:17.918209Z","iopub.status.idle":"2021-06-02T02:07:17.959121Z","shell.execute_reply.started":"2021-06-02T02:07:17.918187Z","shell.execute_reply":"2021-06-02T02:07:17.958343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Perform UniVaraite and ByVariate analysis on Merged DataSet.","metadata":{}},{"cell_type":"markdown","source":"### Check UniVariate Analysis of few columns in Merged Data ","metadata":{}},{"cell_type":"markdown","source":"#### UniVaraite analysis for Negative for Pneumonia ","metadata":{}},{"cell_type":"code","source":"\nsns.distplot(train_Merged_df[\"Negative for Pneumonia\"])\n","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:17.960046Z","iopub.execute_input":"2021-06-02T02:07:17.960253Z","iopub.status.idle":"2021-06-02T02:07:18.136070Z","shell.execute_reply.started":"2021-06-02T02:07:17.960230Z","shell.execute_reply":"2021-06-02T02:07:18.135340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_Merged_df[\"Negative for Pneumonia\"].value_counts().plot(kind='bar')\nplt.show()\n\n","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:18.138465Z","iopub.execute_input":"2021-06-02T02:07:18.138725Z","iopub.status.idle":"2021-06-02T02:07:18.262976Z","shell.execute_reply.started":"2021-06-02T02:07:18.138697Z","shell.execute_reply":"2021-06-02T02:07:18.262013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### UniVaraite distribution for Typical Appearance","metadata":{}},{"cell_type":"code","source":"sns.distplot(train_Merged_df[\"Typical Appearance\"])","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:18.264193Z","iopub.execute_input":"2021-06-02T02:07:18.264541Z","iopub.status.idle":"2021-06-02T02:07:18.432051Z","shell.execute_reply.started":"2021-06-02T02:07:18.264504Z","shell.execute_reply":"2021-06-02T02:07:18.431232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_Merged_df[\"Typical Appearance\"].value_counts().plot(kind='bar')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:18.432945Z","iopub.execute_input":"2021-06-02T02:07:18.433151Z","iopub.status.idle":"2021-06-02T02:07:18.609094Z","shell.execute_reply.started":"2021-06-02T02:07:18.433129Z","shell.execute_reply":"2021-06-02T02:07:18.608245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### UniVariate analysis for Indeterminate Appearance","metadata":{}},{"cell_type":"code","source":"sns.distplot(train_Merged_df[\"Indeterminate Appearance\"])","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:18.610319Z","iopub.execute_input":"2021-06-02T02:07:18.610602Z","iopub.status.idle":"2021-06-02T02:07:18.919589Z","shell.execute_reply.started":"2021-06-02T02:07:18.610571Z","shell.execute_reply":"2021-06-02T02:07:18.918702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_Merged_df[\"Indeterminate Appearance\"].value_counts().plot(kind='bar')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:18.920662Z","iopub.execute_input":"2021-06-02T02:07:18.920954Z","iopub.status.idle":"2021-06-02T02:07:19.006854Z","shell.execute_reply.started":"2021-06-02T02:07:18.920922Z","shell.execute_reply":"2021-06-02T02:07:19.006039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### UniVariate analysis for Atypical Appearance","metadata":{}},{"cell_type":"code","source":"sns.distplot(train_Merged_df[\"Atypical Appearance\"])","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:19.009465Z","iopub.execute_input":"2021-06-02T02:07:19.009672Z","iopub.status.idle":"2021-06-02T02:07:19.212583Z","shell.execute_reply.started":"2021-06-02T02:07:19.009651Z","shell.execute_reply":"2021-06-02T02:07:19.212127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_Merged_df[\"Atypical Appearance\"].value_counts().plot(kind='bar')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:19.213968Z","iopub.execute_input":"2021-06-02T02:07:19.214362Z","iopub.status.idle":"2021-06-02T02:07:19.306264Z","shell.execute_reply.started":"2021-06-02T02:07:19.214330Z","shell.execute_reply":"2021-06-02T02:07:19.305387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Check UniVariate analysis for Opacity Count","metadata":{}},{"cell_type":"code","source":"train_Merged_df[\"OpacityCount\"].value_counts().plot(kind='bar')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:19.307167Z","iopub.execute_input":"2021-06-02T02:07:19.307357Z","iopub.status.idle":"2021-06-02T02:07:19.419160Z","shell.execute_reply.started":"2021-06-02T02:07:19.307336Z","shell.execute_reply":"2021-06-02T02:07:19.417462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By looking at the above plot, we can analyse that most of the data have 2 opacity values.","metadata":{}},{"cell_type":"markdown","source":"#### Lets check how the results are when opacity is available.","metadata":{}},{"cell_type":"code","source":"grouped_df = train_Merged_df[['Study_Result','Class_ID']]\ngrouped_df[grouped_df['Class_ID']=='opacity'].value_counts()\n           ","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:19.420974Z","iopub.execute_input":"2021-06-02T02:07:19.421330Z","iopub.status.idle":"2021-06-02T02:07:19.437579Z","shell.execute_reply.started":"2021-06-02T02:07:19.421297Z","shell.execute_reply":"2021-06-02T02:07:19.436531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"WHen Opacity is available, most of the results were Typical .Also includes Indeterminate and Atypical.","metadata":{}},{"cell_type":"code","source":"grouped_df[grouped_df['Class_ID']=='opacity'].value_counts().plot(kind='bar')","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:19.440398Z","iopub.execute_input":"2021-06-02T02:07:19.441212Z","iopub.status.idle":"2021-06-02T02:07:19.569771Z","shell.execute_reply.started":"2021-06-02T02:07:19.441159Z","shell.execute_reply":"2021-06-02T02:07:19.568924Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Check for None Class values\n\ngrouped_df[grouped_df['Class_ID']=='none'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:19.570639Z","iopub.execute_input":"2021-06-02T02:07:19.570859Z","iopub.status.idle":"2021-06-02T02:07:19.581392Z","shell.execute_reply.started":"2021-06-02T02:07:19.570831Z","shell.execute_reply":"2021-06-02T02:07:19.580537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grouped_df[grouped_df['Class_ID']=='none'].value_counts().plot(kind='bar')","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:19.582166Z","iopub.execute_input":"2021-06-02T02:07:19.582360Z","iopub.status.idle":"2021-06-02T02:07:19.708975Z","shell.execute_reply.started":"2021-06-02T02:07:19.582339Z","shell.execute_reply":"2021-06-02T02:07:19.708371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When there is no Opacity values available in Label column, most of the Study Results were Negative(1736 records) which indicates there was no lung infection in most cases.\nBut surprisingly there were records with Typical (153 records) ,Atypical(92) and Indeterminate (59) study results as well with None opacity values.","metadata":{}},{"cell_type":"markdown","source":"## Lets check whether Data is imbalanced or not","metadata":{}},{"cell_type":"code","source":"#RATIO OF IMBALANCE \n\nfrequency_values=train_study_level_data['Study_Result'].value_counts()\n\nfrequency_values.plot(kind='pie',autopct='%1.2f%%',fontsize=14,figsize=(6,6))","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:19.709936Z","iopub.execute_input":"2021-06-02T02:07:19.710310Z","iopub.status.idle":"2021-06-02T02:07:19.815558Z","shell.execute_reply.started":"2021-06-02T02:07:19.710278Z","shell.execute_reply":"2021-06-02T02:07:19.814657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From above Pie chart, it is clear that near to 50% of Study results are Typical.Rest 50% include Negative,Atypical and Indeterminate","metadata":{}},{"cell_type":"markdown","source":"#### Lets analyse further how the Study result is based on opacity count.","metadata":{}},{"cell_type":"code","source":"train_image_level_data['OpacityCount'].value_counts().plot(kind='bar')","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:19.816671Z","iopub.execute_input":"2021-06-02T02:07:19.817010Z","iopub.status.idle":"2021-06-02T02:07:19.961060Z","shell.execute_reply.started":"2021-06-02T02:07:19.816979Z","shell.execute_reply":"2021-06-02T02:07:19.960358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_Merged_df.head(1)","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:19.962110Z","iopub.execute_input":"2021-06-02T02:07:19.962473Z","iopub.status.idle":"2021-06-02T02:07:19.979739Z","shell.execute_reply.started":"2021-06-02T02:07:19.962439Z","shell.execute_reply":"2021-06-02T02:07:19.979141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_Merged_df['OpacityCount'].unique()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:19.980961Z","iopub.execute_input":"2021-06-02T02:07:19.981296Z","iopub.status.idle":"2021-06-02T02:07:19.996563Z","shell.execute_reply.started":"2021-06-02T02:07:19.981264Z","shell.execute_reply":"2021-06-02T02:07:19.994892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### When OpacityCount is 8, lets analyse the Study Result.","metadata":{}},{"cell_type":"code","source":"OpacityCount_8_df=train_Merged_df[train_Merged_df['OpacityCount']==8][['Study_Result','OpacityCount']]#.plot(kind='bar')\n\nOpacityCount_8_df","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:19.997589Z","iopub.execute_input":"2021-06-02T02:07:19.997843Z","iopub.status.idle":"2021-06-02T02:07:20.013448Z","shell.execute_reply.started":"2021-06-02T02:07:19.997820Z","shell.execute_reply":"2021-06-02T02:07:20.012524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n#grouped_df[grouped_df['Class_ID']=='opacity'].value_counts().plot(kind='bar')\nOpacityCount_8_df.value_counts().plot(kind='pie',autopct='%1.2f%%',fontsize=14,figsize=(4,4))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:20.014590Z","iopub.execute_input":"2021-06-02T02:07:20.014809Z","iopub.status.idle":"2021-06-02T02:07:20.094966Z","shell.execute_reply.started":"2021-06-02T02:07:20.014771Z","shell.execute_reply":"2021-06-02T02:07:20.093572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"THere is only one record with opacity count 8 and the study result is Typical.","metadata":{}},{"cell_type":"markdown","source":"### When Opacity Count is 5, Lets analyse the Study Result","metadata":{}},{"cell_type":"code","source":"OpacityCount_5_df=train_Merged_df[train_Merged_df['OpacityCount']==5][['Study_Result','OpacityCount']]\nOpacityCount_5_df","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:20.096377Z","iopub.execute_input":"2021-06-02T02:07:20.096636Z","iopub.status.idle":"2021-06-02T02:07:20.108020Z","shell.execute_reply.started":"2021-06-02T02:07:20.096609Z","shell.execute_reply":"2021-06-02T02:07:20.107194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"OpacityCount_5_df.value_counts().plot(kind='pie',autopct='%1.2f%%',fontsize=14,figsize=(4,4))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:20.108908Z","iopub.execute_input":"2021-06-02T02:07:20.109122Z","iopub.status.idle":"2021-06-02T02:07:20.187578Z","shell.execute_reply.started":"2021-06-02T02:07:20.109092Z","shell.execute_reply":"2021-06-02T02:07:20.186915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"THere is only one record with opacity count 5 and the study result is Indeterminate.","metadata":{}},{"cell_type":"code","source":"\nOpacityCount_4_df=train_Merged_df[train_Merged_df['OpacityCount']==4][['Study_Result','OpacityCount',]]\nOpacityCount_4_df","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:20.188443Z","iopub.execute_input":"2021-06-02T02:07:20.188663Z","iopub.status.idle":"2021-06-02T02:07:20.199083Z","shell.execute_reply.started":"2021-06-02T02:07:20.188638Z","shell.execute_reply":"2021-06-02T02:07:20.198405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"OpacityCount_4_df.value_counts().plot(kind='pie',autopct='%1.2f%%',fontsize=14,figsize=(4,4))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:20.199995Z","iopub.execute_input":"2021-06-02T02:07:20.200198Z","iopub.status.idle":"2021-06-02T02:07:20.299823Z","shell.execute_reply.started":"2021-06-02T02:07:20.200177Z","shell.execute_reply":"2021-06-02T02:07:20.299159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When OpacityCount is 4, there are 91.3% of data which has Typical Study Result,4.35% of ATypical and 4.35% of Indeterminate Study Results.","metadata":{}},{"cell_type":"code","source":"\nOpacityCount_3_df=train_Merged_df[train_Merged_df['OpacityCount']==3][['Study_Result','OpacityCount',]]\n","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:20.300674Z","iopub.execute_input":"2021-06-02T02:07:20.300909Z","iopub.status.idle":"2021-06-02T02:07:20.306445Z","shell.execute_reply.started":"2021-06-02T02:07:20.300883Z","shell.execute_reply":"2021-06-02T02:07:20.305591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"OpacityCount_3_df.value_counts().plot(kind='pie',autopct='%1.2f%%',fontsize=14,figsize=(4,4))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:20.307154Z","iopub.execute_input":"2021-06-02T02:07:20.307344Z","iopub.status.idle":"2021-06-02T02:07:20.403727Z","shell.execute_reply.started":"2021-06-02T02:07:20.307323Z","shell.execute_reply":"2021-06-02T02:07:20.402769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When OpacityCount is 3, there are 83.06 of data which has Typical Study Result,7.65% of ATypical and \n9.29% of Indeterminate Study Results.\n","metadata":{}},{"cell_type":"code","source":"OpacityCount_2_df=train_Merged_df[train_Merged_df['OpacityCount']==2][['Study_Result','OpacityCount',]]\nOpacityCount_2_df.value_counts().plot(kind='pie',autopct='%1.2f%%',fontsize=14,figsize=(4,4))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:20.405089Z","iopub.execute_input":"2021-06-02T02:07:20.405402Z","iopub.status.idle":"2021-06-02T02:07:20.494488Z","shell.execute_reply.started":"2021-06-02T02:07:20.405371Z","shell.execute_reply":"2021-06-02T02:07:20.493509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When OpacityCount is 2, there are 85.22 of data which has Typical Study Result,3.69% of ATypical and \n11.08% of Indeterminate Study Results.","metadata":{}},{"cell_type":"code","source":"OpacityCount_1_df=train_Merged_df[train_Merged_df['OpacityCount']==1][['Study_Result','OpacityCount',]]\nOpacityCount_1_df.value_counts().plot(kind='pie',autopct='%1.2f%%',fontsize=14,figsize=(4,4))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:20.495565Z","iopub.execute_input":"2021-06-02T02:07:20.495769Z","iopub.status.idle":"2021-06-02T02:07:20.580235Z","shell.execute_reply.started":"2021-06-02T02:07:20.495748Z","shell.execute_reply":"2021-06-02T02:07:20.579344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When OpacityCount is 1, there are 70.40 of data which has Indeterminate Study Result,2.77% of Typical and \n26.82% of ATypical Study Results.","metadata":{}},{"cell_type":"code","source":"OpacityCount_0_df=train_Merged_df[train_Merged_df['OpacityCount']==0][['Study_Result','OpacityCount','label']]\nOpacityCount_0_df.value_counts().plot(kind='pie',autopct='%1.2f%%',fontsize=14,figsize=(4,4))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:20.581199Z","iopub.execute_input":"2021-06-02T02:07:20.581404Z","iopub.status.idle":"2021-06-02T02:07:20.687506Z","shell.execute_reply.started":"2021-06-02T02:07:20.581382Z","shell.execute_reply":"2021-06-02T02:07:20.686964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When OpacityCount is 0, Mostly Study Results were Negative(85.1%).But also includes\nIndeterminate(2.89%),ATypical(4.5%) and Typical(7.5%).","metadata":{}},{"cell_type":"markdown","source":"### Analysing OpacityCount 0 data with Typical Study Result","metadata":{}},{"cell_type":"code","source":"OpacityCount_0_df[OpacityCount_0_df['Study_Result']=='Typical']","metadata":{"execution":{"iopub.status.busy":"2021-06-02T02:07:20.688346Z","iopub.execute_input":"2021-06-02T02:07:20.688534Z","iopub.status.idle":"2021-06-02T02:07:20.701282Z","shell.execute_reply.started":"2021-06-02T02:07:20.688513Z","shell.execute_reply":"2021-06-02T02:07:20.700387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## OBSERVATIONS FROM Basic Data Analysis and EDA:","metadata":{}},{"cell_type":"markdown","source":"1. There were no missing values in train_study_level dataset.\n\n\n2. When EDA is Performed on train_study_level dataset, we observed that Data is Highly Imbalanced \n   and near to 50% of Study results are Typical.Rest 50% include Negative,Atypical and Indeterminate.\n   \n\n3. There were 2040 missing values in Box Column in Train_Image_level dataset.\n\n\n4. There were no duplicate data.\n\n\nAfter Merging the both the datasets train_study_level and train_Image_level, we observed below points.\n\n5. When Opacity values are available in Label column, study results were mostly Typical(2854 records),\n   followed by Indeterminate(1049 records) and Atypical(391 records).\n   \n    \n6. When there is no Opacity values available in Label column, most of the Study Results were Negative(1736 records) \n    which indicates there was no lung infection in most cases.\n    But surprisingly there were records with Typical (153 records) ,Atypical(92) and Indeterminate (59) \n    study results as well with None opacity values.\n    \n\n7. There were multiple Opacity values available for each study result which indicates either multiple images available or \n   multiple boxes in single image.\n    Multiple values of Opacity Count is 2, 0, 1, 3, 4, 8, 5.\n    \n    \n8. There is only one record with opacity count 8 and the study result is Typical.\n\n\n9. There is only one record with opacity count 5 and the study result is Indeterminate.\n\n\n10. When OpacityCount is 4, there are 91.3% of data which has Typical Study Result,4.35% of ATypical and \n    4.35% of Indeterminate Study Results.\n    \n    \n11. When OpacityCount is 3, there are 83.06 of data which has Typical Study Result,7.65% of ATypical and \n    9.29% of Indeterminate Study Results.\n\n\n12. When OpacityCount is 2, there are 85.22 of data which has Typical Study Result,3.69% of ATypical and \n    11.08% of Indeterminate Study Results.\n\n\n13. When OpacityCount is 1, there are 70.40 of data which has Indeterminate Study Result,2.77% of Typical and \n    26.82% of ATypical Study Results.\n\n\n14. When OpacityCount is 0, Mostly Study Results were Negative(85.1%).But also includes\nIndeterminate(2.89%),ATypical(4.5%) and Typical(7.5%).\n\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}