{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# SIIM: Step-by-Step Image Detection for Beginners [Part 1 - EDA to Preprocessing]","metadata":{}},{"cell_type":"markdown","source":"Thanks for nice reference : \n\n`handling dcm file`\n- [SIIM-FISABIO-RSNA_COVID-19_Detection_Starter, DrCapa](https://www.kaggle.com/drcapa/siim-fisabio-rsna-covid-19-detection-starter)\n\n`Image Visualization`\n- [2. ONE STOP:Understanding+InDepth EDA+Model](https://www.kaggle.com/harshsharma511/one-stop-understanding-indepth-eda-model-progress)\n\n`using gdcm without internet access`\n- [pydicom_conda_helper](https://www.kaggle.com/awsaf49/pydicom-conda-helper)\n\n`Get the box informations`\n- [catch up on positive samples / plot submission.csv](https://www.kaggle.com/yujiariyasu/catch-up-on-positive-samples-plot-submission-csv?scriptVersionId=63394385)","metadata":{}},{"cell_type":"markdown","source":"\n\n```\nStep 1. Import Libraries\nStep 2. Load Data\nStep 3. Read DCM File\n     3-a. explore path with python code\n     3-b. make image extractor(function)\nStep 4. Show Sample Image\n     4-a. explore image data with python code\n     4-b. check position to draw box\nStep 5. Show Multiple Images\nStep 6. Feature Engineering I\n     6-a. count opacity\n     6-b. simplify 'id'\n     6-c. rename colume 'id' to 'StudyInstanceUID for merge on 'StudyInstanceUID'\n     6-d. check the relation between 'OpacityCount' and other columes in train_study\n     6-e. visualize the relation between 'OpacityCount' and other columes in train_study\n     6-f. check duplicate values(One row and Two Appearances)\nStep 7. Feature Engineering II\n     7-a. explore data analysis\n     7-b. check duplicates in dataset\n     7-c. modify some of the code in function that extract image(.dcm)\nStep 8. Visualize X-ray with bbox\n     8-a. negative for pneumonia\n     8-b. typical appearance\n     8-c. indeterminate appearance\n     8-d. atypical Appearance\nStep 9. Featrue Engineering III\n     9-a. anomaly detection\n     9-b. show outliers in `Typical Appearance`\n     9-c. show outliers in `Intermiate Appearance`\n     9-d. show outliers in `Atypical Appearance`\nStep 10. Image Data Preprocessing\n     10-a. add image path to a separate column\n     10-b. Resize the image (uniform to 150x150) and Scale each pixel values (uniform range 1~255)\n     10-c. Calculate the resize ratio(x, y) and Apply the same to the bounding box\n```  ","metadata":{}},{"cell_type":"markdown","source":"## Step 1. Import Libraries","metadata":{}},{"cell_type":"markdown","source":"We need to load the gdcm package before import `pydicom`. because some of the dcm files are `jpeg lossless` type.","metadata":{}},{"cell_type":"code","source":"!wget 'https://anaconda.org/conda-forge/gdcm/2.8.9/download/linux-64/gdcm-2.8.9-py37h500ead1_1.tar.bz2' -q\n!conda install 'gdcm-2.8.9-py37h500ead1_1.tar.bz2' -c conda-forge -y","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:25:53.254472Z","iopub.execute_input":"2021-05-24T06:25:53.254899Z","iopub.status.idle":"2021-05-24T06:26:18.44411Z","shell.execute_reply.started":"2021-05-24T06:25:53.254866Z","shell.execute_reply":"2021-05-24T06:26:18.443118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n\nimport os\n#for dirname, _, filenames in os.walk('/kaggle/input'):\n#    for filename in filenames:\n#        print(os.path.join(dirname, filename))\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-05-26T17:35:04.211440Z","iopub.execute_input":"2021-05-26T17:35:04.211794Z","iopub.status.idle":"2021-05-26T17:35:04.219718Z","shell.execute_reply.started":"2021-05-26T17:35:04.211715Z","shell.execute_reply":"2021-05-26T17:35:04.219070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport matplotlib\nimport pydicom as dicom\nimport cv2\nimport ast\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:35:12.610586Z","iopub.execute_input":"2021-05-26T17:35:12.611051Z","iopub.status.idle":"2021-05-26T17:35:12.993004Z","shell.execute_reply.started":"2021-05-26T17:35:12.611005Z","shell.execute_reply":"2021-05-26T17:35:12.992118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 2. Load Data","metadata":{}},{"cell_type":"code","source":"path = '/kaggle/input/siim-covid19-detection/'","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:35:17.693083Z","iopub.execute_input":"2021-05-26T17:35:17.693424Z","iopub.status.idle":"2021-05-26T17:35:17.696642Z","shell.execute_reply.started":"2021-05-26T17:35:17.693394Z","shell.execute_reply":"2021-05-26T17:35:17.696062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"os.listdir(path)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:35:21.069480Z","iopub.execute_input":"2021-05-26T17:35:21.069818Z","iopub.status.idle":"2021-05-26T17:35:21.077808Z","shell.execute_reply.started":"2021-05-26T17:35:21.069783Z","shell.execute_reply":"2021-05-26T17:35:21.076954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image = pd.read_csv(path+'train_image_level.csv')\ntrain_study = pd.read_csv(path+'train_study_level.csv')\nsample_submission = pd.read_csv(path+'sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:35:24.576494Z","iopub.execute_input":"2021-05-26T17:35:24.576828Z","iopub.status.idle":"2021-05-26T17:35:24.652352Z","shell.execute_reply.started":"2021-05-26T17:35:24.576794Z","shell.execute_reply":"2021-05-26T17:35:24.651455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(sample_submission)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:35:29.440775Z","iopub.execute_input":"2021-05-26T17:35:29.441138Z","iopub.status.idle":"2021-05-26T17:35:29.445920Z","shell.execute_reply.started":"2021-05-26T17:35:29.441105Z","shell.execute_reply":"2021-05-26T17:35:29.445105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:35:33.282633Z","iopub.execute_input":"2021-05-26T17:35:33.282945Z","iopub.status.idle":"2021-05-26T17:35:33.308766Z","shell.execute_reply.started":"2021-05-26T17:35:33.282914Z","shell.execute_reply":"2021-05-26T17:35:33.308039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:35:37.901706Z","iopub.execute_input":"2021-05-26T17:35:37.902045Z","iopub.status.idle":"2021-05-26T17:35:37.916057Z","shell.execute_reply.started":"2021-05-26T17:35:37.902001Z","shell.execute_reply":"2021-05-26T17:35:37.915123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 3. Read DCM File","metadata":{}},{"cell_type":"markdown","source":"### 3-a. explore path with python code","metadata":{}},{"cell_type":"code","source":"temp = train_image.loc[0, 'StudyInstanceUID']\ntemp","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:35:43.461550Z","iopub.execute_input":"2021-05-26T17:35:43.462003Z","iopub.status.idle":"2021-05-26T17:35:43.469663Z","shell.execute_reply.started":"2021-05-26T17:35:43.461971Z","shell.execute_reply":"2021-05-26T17:35:43.469081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp_depth2 = os.listdir(path+'train/'+temp)\ntemp_depth2[0]","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:35:47.255218Z","iopub.execute_input":"2021-05-26T17:35:47.255725Z","iopub.status.idle":"2021-05-26T17:35:47.266537Z","shell.execute_reply.started":"2021-05-26T17:35:47.255682Z","shell.execute_reply":"2021-05-26T17:35:47.265980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp_train_path = path+'train/'+temp+'/'+temp_depth2[0]\ntemp_train_path","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:35:50.715624Z","iopub.execute_input":"2021-05-26T17:35:50.716116Z","iopub.status.idle":"2021-05-26T17:35:50.721687Z","shell.execute_reply.started":"2021-05-26T17:35:50.716071Z","shell.execute_reply":"2021-05-26T17:35:50.721087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"os.listdir('/kaggle/input/siim-covid19-detection/train/5776db0cec75/81456c9c5423')  ","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:35:54.672413Z","iopub.execute_input":"2021-05-26T17:35:54.673340Z","iopub.status.idle":"2021-05-26T17:35:54.686092Z","shell.execute_reply.started":"2021-05-26T17:35:54.673289Z","shell.execute_reply":"2021-05-26T17:35:54.685460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image.loc[0, 'id']","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:35:58.187392Z","iopub.execute_input":"2021-05-26T17:35:58.187879Z","iopub.status.idle":"2021-05-26T17:35:58.192928Z","shell.execute_reply.started":"2021-05-26T17:35:58.187824Z","shell.execute_reply":"2021-05-26T17:35:58.192133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3-b. make image extractor(function)","metadata":{}},{"cell_type":"code","source":"def extraction(i):\n    path_train = path + 'train/' + train_image.loc[i, 'StudyInstanceUID']\n    last_folder_in_path = os.listdir(path_train)[0]\n    path_train = path_train + '/{}/'.format(last_folder_in_path)\n    img_id = train_image.loc[i, 'id'].replace('_image','.dcm')\n    print(img_id)\n    data_file = dicom.dcmread(path_train+img_id)\n    img = data_file.pixel_array\n    return img","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:36:02.741948Z","iopub.execute_input":"2021-05-26T17:36:02.742412Z","iopub.status.idle":"2021-05-26T17:36:02.746564Z","shell.execute_reply.started":"2021-05-26T17:36:02.742373Z","shell.execute_reply":"2021-05-26T17:36:02.745830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 4. Show Sample Image","metadata":{}},{"cell_type":"markdown","source":"### 4-a. Explore Image Data with python code","metadata":{}},{"cell_type":"code","source":"sample_img = extraction(0)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:36:09.254566Z","iopub.execute_input":"2021-05-26T17:36:09.254889Z","iopub.status.idle":"2021-05-26T17:36:10.265635Z","shell.execute_reply.started":"2021-05-26T17:36:09.254855Z","shell.execute_reply":"2021-05-26T17:36:10.264633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_img","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:36:14.682017Z","iopub.execute_input":"2021-05-26T17:36:14.682363Z","iopub.status.idle":"2021-05-26T17:36:14.689552Z","shell.execute_reply.started":"2021-05-26T17:36:14.682334Z","shell.execute_reply":"2021-05-26T17:36:14.688513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_img.shape","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:36:17.846894Z","iopub.execute_input":"2021-05-26T17:36:17.847347Z","iopub.status.idle":"2021-05-26T17:36:17.851733Z","shell.execute_reply.started":"2021-05-26T17:36:17.847315Z","shell.execute_reply":"2021-05-26T17:36:17.851171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4-b. check position to draw box","metadata":{}},{"cell_type":"code","source":"train_image.loc[0, 'boxes']","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:19.889694Z","iopub.execute_input":"2021-05-24T06:26:19.890009Z","iopub.status.idle":"2021-05-24T06:26:19.897652Z","shell.execute_reply.started":"2021-05-24T06:26:19.889985Z","shell.execute_reply":"2021-05-24T06:26:19.896816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"boxes = ast.literal_eval(train_image.loc[0, 'boxes'])\nboxes","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:37:18.921215Z","iopub.execute_input":"2021-05-26T17:37:18.921704Z","iopub.status.idle":"2021-05-26T17:37:18.926579Z","shell.execute_reply.started":"2021-05-26T17:37:18.921673Z","shell.execute_reply":"2021-05-26T17:37:18.926068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1,1, figsize=(8,4))\nfor box in boxes:\n    p = matplotlib.patches.Rectangle((box['x'], box['y']),\n                                      box['width'], box['height'],\n                                      ec='r', fc='none', lw=1.5)\n    ax.add_patch(p)\nax.imshow(sample_img, cmap='gray')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:39:29.788182Z","iopub.execute_input":"2021-05-26T17:39:29.788691Z","iopub.status.idle":"2021-05-26T17:39:31.056971Z","shell.execute_reply.started":"2021-05-26T17:39:29.788647Z","shell.execute_reply":"2021-05-26T17:39:31.056126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 5. Show Multiple Images","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(3,3, figsize=(20,16))\nfig.subplots_adjust(hspace=.1, wspace=.1)\naxes = axes.ravel()\n\nfor row in range(9):\n    img = extraction(row)\n    # if (nan == nan)\n    # False\n    if (train_image.loc[row,'boxes'] == train_image.loc[row,'boxes']):\n        boxes = ast.literal_eval(train_image.loc[row,'boxes'])\n        for box in boxes:\n            p = matplotlib.patches.Rectangle((box['x'], box['y']),\n                                              box['width'], box['height'],\n                                              ec='r', fc='none', lw=2.\n                                            )\n            axes[row].add_patch(p)\n    \n    axes[row].imshow(img, cmap='gray')\n    axes[row].set_title(train_image.loc[row, 'label'].split(' ')[0])\n    axes[row].set_xticklabels([])\n    axes[row].set_yticklabels([])","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:37:46.246070Z","iopub.execute_input":"2021-05-26T17:37:46.246396Z","iopub.status.idle":"2021-05-26T17:37:57.698290Z","shell.execute_reply.started":"2021-05-26T17:37:46.246367Z","shell.execute_reply":"2021-05-26T17:37:57.697310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 6. Feature Engineering I","metadata":{}},{"cell_type":"code","source":"train_image","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:39:51.710046Z","iopub.execute_input":"2021-05-26T17:39:51.710379Z","iopub.status.idle":"2021-05-26T17:39:51.722974Z","shell.execute_reply.started":"2021-05-26T17:39:51.710347Z","shell.execute_reply":"2021-05-26T17:39:51.722285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 6-a. Count Opacity in Image","metadata":{}},{"cell_type":"code","source":"OpacityCount = train_image['label'].str.count('opacity')\nOpacityCount","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:39:56.296477Z","iopub.execute_input":"2021-05-26T17:39:56.296996Z","iopub.status.idle":"2021-05-26T17:39:56.309681Z","shell.execute_reply.started":"2021-05-26T17:39:56.296950Z","shell.execute_reply":"2021-05-26T17:39:56.308766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image['OpacityCount'] = OpacityCount.values","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:49:05.470818Z","iopub.execute_input":"2021-05-26T17:49:05.471221Z","iopub.status.idle":"2021-05-26T17:49:05.476609Z","shell.execute_reply.started":"2021-05-26T17:49:05.471191Z","shell.execute_reply":"2021-05-26T17:49:05.475692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:49:28.549562Z","iopub.execute_input":"2021-05-26T17:49:28.549885Z","iopub.status.idle":"2021-05-26T17:49:28.566013Z","shell.execute_reply.started":"2021-05-26T17:49:28.549856Z","shell.execute_reply":"2021-05-26T17:49:28.565142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image['id'].isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:49:31.879558Z","iopub.execute_input":"2021-05-26T17:49:31.879856Z","iopub.status.idle":"2021-05-26T17:49:31.886540Z","shell.execute_reply.started":"2021-05-26T17:49:31.879830Z","shell.execute_reply":"2021-05-26T17:49:31.885605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 6-b. Simplify 'id' (study)","metadata":{}},{"cell_type":"code","source":"id_extract = lambda x : x[0]","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:49:34.680334Z","iopub.execute_input":"2021-05-26T17:49:34.680783Z","iopub.status.idle":"2021-05-26T17:49:34.683966Z","shell.execute_reply.started":"2021-05-26T17:49:34.680705Z","shell.execute_reply":"2021-05-26T17:49:34.683346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:49:39.594921Z","iopub.execute_input":"2021-05-26T17:49:39.595621Z","iopub.status.idle":"2021-05-26T17:49:39.611442Z","shell.execute_reply.started":"2021-05-26T17:49:39.595569Z","shell.execute_reply":"2021-05-26T17:49:39.610429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study['id'].isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:49:42.571403Z","iopub.execute_input":"2021-05-26T17:49:42.571700Z","iopub.status.idle":"2021-05-26T17:49:42.577988Z","shell.execute_reply.started":"2021-05-26T17:49:42.571673Z","shell.execute_reply":"2021-05-26T17:49:42.577376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study['id'].str.split('_')","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:49:45.258699Z","iopub.execute_input":"2021-05-26T17:49:45.259181Z","iopub.status.idle":"2021-05-26T17:49:45.271690Z","shell.execute_reply.started":"2021-05-26T17:49:45.259149Z","shell.execute_reply":"2021-05-26T17:49:45.271022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study['id'].str.split('_').apply(id_extract)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:49:48.062756Z","iopub.execute_input":"2021-05-26T17:49:48.064628Z","iopub.status.idle":"2021-05-26T17:49:48.076810Z","shell.execute_reply.started":"2021-05-26T17:49:48.064589Z","shell.execute_reply":"2021-05-26T17:49:48.075938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study['id'] = train_study['id'].str.split('_').apply(id_extract)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:49:58.799244Z","iopub.execute_input":"2021-05-26T17:49:58.799541Z","iopub.status.idle":"2021-05-26T17:49:58.809479Z","shell.execute_reply.started":"2021-05-26T17:49:58.799512Z","shell.execute_reply":"2021-05-26T17:49:58.808535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sum(train_study['id'].str.contains(train_image['StudyInstanceUID'][0]))","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:50:01.840785Z","iopub.execute_input":"2021-05-26T17:50:01.841154Z","iopub.status.idle":"2021-05-26T17:50:01.850788Z","shell.execute_reply.started":"2021-05-26T17:50:01.841121Z","shell.execute_reply":"2021-05-26T17:50:01.849977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 6-c. rename colume 'id' to 'StudyInstanceUID for merge on 'StudyInstanceUID'","metadata":{}},{"cell_type":"code","source":"train_study = train_study.rename({'id':'StudyInstanceUID'}, axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:50:16.493615Z","iopub.execute_input":"2021-05-26T17:50:16.493929Z","iopub.status.idle":"2021-05-26T17:50:16.499186Z","shell.execute_reply.started":"2021-05-26T17:50:16.493899Z","shell.execute_reply":"2021-05-26T17:50:16.498320Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:50:19.039567Z","iopub.execute_input":"2021-05-26T17:50:19.039907Z","iopub.status.idle":"2021-05-26T17:50:19.052822Z","shell.execute_reply.started":"2021-05-26T17:50:19.039877Z","shell.execute_reply":"2021-05-26T17:50:19.051926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.merge(train_image, train_study, on='StudyInstanceUID')\ntrain_df","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:50:22.874430Z","iopub.execute_input":"2021-05-26T17:50:22.874743Z","iopub.status.idle":"2021-05-26T17:50:22.906798Z","shell.execute_reply.started":"2021-05-26T17:50:22.874714Z","shell.execute_reply":"2021-05-26T17:50:22.905871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 6-d. Check the Relation between 'OpacityCount' and other Columes in train_study","metadata":{}},{"cell_type":"code","source":"train_df['OpacityCount'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:50:26.033242Z","iopub.execute_input":"2021-05-26T17:50:26.033674Z","iopub.status.idle":"2021-05-26T17:50:26.041227Z","shell.execute_reply.started":"2021-05-26T17:50:26.033644Z","shell.execute_reply":"2021-05-26T17:50:26.040654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.iloc[:,5:].columns","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:50:28.846101Z","iopub.execute_input":"2021-05-26T17:50:28.846609Z","iopub.status.idle":"2021-05-26T17:50:28.852276Z","shell.execute_reply.started":"2021-05-26T17:50:28.846576Z","shell.execute_reply":"2021-05-26T17:50:28.851428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"i = 5\nfor col in train_df.iloc[:,5:].columns:\n    print('The Count of {} : '.format(col), sum(train_df.iloc[:,i]))\n    i += 1","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:50:34.340972Z","iopub.execute_input":"2021-05-26T17:50:34.341443Z","iopub.status.idle":"2021-05-26T17:50:34.350637Z","shell.execute_reply.started":"2021-05-26T17:50:34.341413Z","shell.execute_reply":"2021-05-26T17:50:34.349905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[train_df['OpacityCount'] == 0]","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:50:40.223238Z","iopub.execute_input":"2021-05-26T17:50:40.223719Z","iopub.status.idle":"2021-05-26T17:50:40.256295Z","shell.execute_reply.started":"2021-05-26T17:50:40.223673Z","shell.execute_reply":"2021-05-26T17:50:40.255629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"OCount = sorted(list(train_df['OpacityCount'].value_counts().index))\nprint(OCount)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:50:44.288080Z","iopub.execute_input":"2021-05-26T17:50:44.288422Z","iopub.status.idle":"2021-05-26T17:50:44.294426Z","shell.execute_reply.started":"2021-05-26T17:50:44.288387Z","shell.execute_reply":"2021-05-26T17:50:44.293482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for count in OCount:\n    print('Opacity Count = {}\\n------------------------------'.format(count))\n    print(train_df[train_df['OpacityCount'] == count].iloc[:,5:].sum())\n    print(' ')","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:50:50.180338Z","iopub.execute_input":"2021-05-26T17:50:50.180666Z","iopub.status.idle":"2021-05-26T17:50:50.203154Z","shell.execute_reply.started":"2021-05-26T17:50:50.180637Z","shell.execute_reply":"2021-05-26T17:50:50.202163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 6-e. Visualize the Relation between 'OpacityCount' and other Columes in train_study","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:51:02.417826Z","iopub.execute_input":"2021-05-26T17:51:02.418160Z","iopub.status.idle":"2021-05-26T17:51:03.127642Z","shell.execute_reply.started":"2021-05-26T17:51:02.418126Z","shell.execute_reply":"2021-05-26T17:51:03.126828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for count in OCount:\n    Count_Series = train_df[train_df['OpacityCount'] == count].iloc[:,5:].sum()\n    fig = plt.figure(figsize=(12,3))\n    sns.barplot(x=Count_Series.index, y=Count_Series.values/sum(train_df['OpacityCount']==count))\n    plt.title('OpacityCount : {} '.format(count))\n    plt.plot();","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:51:06.483678Z","iopub.execute_input":"2021-05-26T17:51:06.483995Z","iopub.status.idle":"2021-05-26T17:51:07.374758Z","shell.execute_reply.started":"2021-05-26T17:51:06.483968Z","shell.execute_reply":"2021-05-26T17:51:07.373898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 6-f. Check Duplicate Values(One row and Two Appearances)","metadata":{}},{"cell_type":"code","source":"sum(train_df['OpacityCount']==1)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:51:15.136508Z","iopub.execute_input":"2021-05-26T17:51:15.136840Z","iopub.status.idle":"2021-05-26T17:51:15.143046Z","shell.execute_reply.started":"2021-05-26T17:51:15.136807Z","shell.execute_reply":"2021-05-26T17:51:15.141895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[(train_df['OpacityCount']==1)&(train_df['Indeterminate Appearance'] == 1)]","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:51:21.619746Z","iopub.execute_input":"2021-05-26T17:51:21.620090Z","iopub.status.idle":"2021-05-26T17:51:21.637089Z","shell.execute_reply.started":"2021-05-26T17:51:21.620059Z","shell.execute_reply":"2021-05-26T17:51:21.636109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[(train_df['OpacityCount']==1)&(train_df['Atypical Appearance'] == 1)]","metadata":{"execution":{"iopub.status.busy":"2021-05-26T17:51:26.084121Z","iopub.execute_input":"2021-05-26T17:51:26.084454Z","iopub.status.idle":"2021-05-26T17:51:26.099750Z","shell.execute_reply.started":"2021-05-26T17:51:26.084424Z","shell.execute_reply":"2021-05-26T17:51:26.099146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[(train_df['OpacityCount']==1)&(train_df['Typical Appearance'] == 1)]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.464425Z","iopub.execute_input":"2021-05-24T06:26:33.464663Z","iopub.status.idle":"2021-05-24T06:26:33.485659Z","shell.execute_reply.started":"2021-05-24T06:26:33.464641Z","shell.execute_reply":"2021-05-24T06:26:33.484646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(train_df[(train_df['OpacityCount']==1)&(train_df['Indeterminate Appearance'] == 1)]) + len(train_df[(train_df['OpacityCount']==1)&(train_df['Atypical Appearance'] == 1)]) + len(train_df[(train_df['OpacityCount']==1)&(train_df['Typical Appearance'] == 1)])","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.486743Z","iopub.execute_input":"2021-05-24T06:26:33.487197Z","iopub.status.idle":"2021-05-24T06:26:33.497511Z","shell.execute_reply.started":"2021-05-24T06:26:33.487166Z","shell.execute_reply":"2021-05-24T06:26:33.496886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sum(train_df['OpacityCount']==1)","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.498303Z","iopub.execute_input":"2021-05-24T06:26:33.498611Z","iopub.status.idle":"2021-05-24T06:26:33.507272Z","shell.execute_reply.started":"2021-05-24T06:26:33.498576Z","shell.execute_reply":"2021-05-24T06:26:33.506685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.508044Z","iopub.execute_input":"2021-05-24T06:26:33.50835Z","iopub.status.idle":"2021-05-24T06:26:33.521835Z","shell.execute_reply.started":"2021-05-24T06:26:33.508328Z","shell.execute_reply":"2021-05-24T06:26:33.521311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 7. Feature Engineering II","metadata":{}},{"cell_type":"markdown","source":"### 7-a. explore data analysis","metadata":{}},{"cell_type":"markdown","source":"The number of `StudyInstanceUID` in train_study(original id) is different from the number of `StudyInstanceUID` in train_df(==train_image)\n\nLet's check them","metadata":{}},{"cell_type":"code","source":"train_study","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.522566Z","iopub.execute_input":"2021-05-24T06:26:33.522871Z","iopub.status.idle":"2021-05-24T06:26:33.539953Z","shell.execute_reply.started":"2021-05-24T06:26:33.522849Z","shell.execute_reply":"2021-05-24T06:26:33.539203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.542546Z","iopub.execute_input":"2021-05-24T06:26:33.542782Z","iopub.status.idle":"2021-05-24T06:26:33.556384Z","shell.execute_reply.started":"2021-05-24T06:26:33.542759Z","shell.execute_reply":"2021-05-24T06:26:33.555339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.559061Z","iopub.execute_input":"2021-05-24T06:26:33.559318Z","iopub.status.idle":"2021-05-24T06:26:33.574727Z","shell.execute_reply.started":"2021-05-24T06:26:33.559294Z","shell.execute_reply":"2021-05-24T06:26:33.573804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(train_df['StudyInstanceUID'])","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.576145Z","iopub.execute_input":"2021-05-24T06:26:33.57656Z","iopub.status.idle":"2021-05-24T06:26:33.584019Z","shell.execute_reply.started":"2021-05-24T06:26:33.576522Z","shell.execute_reply":"2021-05-24T06:26:33.583422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(train_df['StudyInstanceUID'].unique())","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.584861Z","iopub.execute_input":"2021-05-24T06:26:33.585259Z","iopub.status.idle":"2021-05-24T06:26:33.594079Z","shell.execute_reply.started":"2021-05-24T06:26:33.585234Z","shell.execute_reply":"2021-05-24T06:26:33.593336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 7-b. Check duplicates in dataset","metadata":{}},{"cell_type":"markdown","source":"We can find that No duplicates in train_study(original ID) because the length of unique `StudyInstanceUID` in train_df and the length of train_study's rows are the same","metadata":{}},{"cell_type":"code","source":"train_image['StudyInstanceUID'].unique().sort() == train_study['StudyInstanceUID'].unique().sort()","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.594967Z","iopub.execute_input":"2021-05-24T06:26:33.595314Z","iopub.status.idle":"2021-05-24T06:26:33.614014Z","shell.execute_reply.started":"2021-05-24T06:26:33.595289Z","shell.execute_reply":"2021-05-24T06:26:33.613351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(train_image['StudyInstanceUID'].unique())","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.614812Z","iopub.execute_input":"2021-05-24T06:26:33.615166Z","iopub.status.idle":"2021-05-24T06:26:33.622202Z","shell.execute_reply.started":"2021-05-24T06:26:33.615131Z","shell.execute_reply":"2021-05-24T06:26:33.621104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Naturally, because `train_df` is a merged data frame based on `train_image`, `train_image` has also the same result.","metadata":{}},{"cell_type":"markdown","source":"Now, Let's check duplicated images(id)","metadata":{}},{"cell_type":"code","source":"train_image[train_image.duplicated(['StudyInstanceUID'])==True]['StudyInstanceUID']","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.623055Z","iopub.execute_input":"2021-05-24T06:26:33.623382Z","iopub.status.idle":"2021-05-24T06:26:33.633053Z","shell.execute_reply.started":"2021-05-24T06:26:33.62336Z","shell.execute_reply":"2021-05-24T06:26:33.632277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"du_StudyId = train_image[train_image.duplicated(['StudyInstanceUID'])==True]['StudyInstanceUID'].values","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.634484Z","iopub.execute_input":"2021-05-24T06:26:33.634936Z","iopub.status.idle":"2021-05-24T06:26:33.64246Z","shell.execute_reply.started":"2021-05-24T06:26:33.6349Z","shell.execute_reply":"2021-05-24T06:26:33.641713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"du_images = train_image[train_image['StudyInstanceUID'].isin(du_StudyId)].sort_values(by=['StudyInstanceUID'])\ndu_images","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.64366Z","iopub.execute_input":"2021-05-24T06:26:33.643934Z","iopub.status.idle":"2021-05-24T06:26:33.661479Z","shell.execute_reply.started":"2021-05-24T06:26:33.643911Z","shell.execute_reply":"2021-05-24T06:26:33.660627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"```\n232 original ID\n280 duplicate ID\n```\n\n- one or more duplicate Image at the same ID","metadata":{"execution":{"iopub.status.busy":"2021-05-21T13:24:56.705889Z","iopub.execute_input":"2021-05-21T13:24:56.706513Z","iopub.status.idle":"2021-05-21T13:24:56.713431Z","shell.execute_reply.started":"2021-05-21T13:24:56.706475Z","shell.execute_reply":"2021-05-21T13:24:56.711993Z"}}},{"cell_type":"markdown","source":"### 7-c. modify some of the code in function that extract image(.dcm)","metadata":{}},{"cell_type":"markdown","source":"So, Some of the code(function extraction) needs to be modified to accurately target the image to be extracted.","metadata":{"execution":{"iopub.status.busy":"2021-05-21T14:42:12.225005Z","iopub.execute_input":"2021-05-21T14:42:12.225436Z","iopub.status.idle":"2021-05-21T14:42:12.231859Z","shell.execute_reply.started":"2021-05-21T14:42:12.225402Z","shell.execute_reply":"2021-05-21T14:42:12.230716Z"}}},{"cell_type":"markdown","source":"Duplicate ID - ex. 1 ID(74ba8f2badcb) - 4 Path(in each path, All 4 Images are the same)","metadata":{}},{"cell_type":"code","source":"train_df[train_df['StudyInstanceUID'].str.contains('74ba8f2')]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.662472Z","iopub.execute_input":"2021-05-24T06:26:33.6627Z","iopub.status.idle":"2021-05-24T06:26:33.67758Z","shell.execute_reply.started":"2021-05-24T06:26:33.662679Z","shell.execute_reply":"2021-05-24T06:26:33.676681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"os.listdir(path + 'train/' + '74ba8f2badcb')","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.678298Z","iopub.execute_input":"2021-05-24T06:26:33.678717Z","iopub.status.idle":"2021-05-24T06:26:33.691984Z","shell.execute_reply.started":"2021-05-24T06:26:33.678692Z","shell.execute_reply":"2021-05-24T06:26:33.691417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"long_path = path + 'train/' + '74ba8f2badcb/'\nfor i in os.listdir(long_path):\n    print(os.listdir(long_path+i))","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.692744Z","iopub.execute_input":"2021-05-24T06:26:33.69312Z","iopub.status.idle":"2021-05-24T06:26:33.711366Z","shell.execute_reply.started":"2021-05-24T06:26:33.693095Z","shell.execute_reply":"2021-05-24T06:26:33.710524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"os.listdir('/kaggle/input/siim-covid19-detection/train/ff0879eb20ed/d8a644cc4f93')","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.712691Z","iopub.execute_input":"2021-05-24T06:26:33.713061Z","iopub.status.idle":"2021-05-24T06:26:33.720512Z","shell.execute_reply.started":"2021-05-24T06:26:33.713025Z","shell.execute_reply":"2021-05-24T06:26:33.719571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\nSearch all paths through a loop and check whether it matches the id value.","metadata":{}},{"cell_type":"code","source":"def error_processed_extraction(i):\n    long_path = path + 'train/' + train_df.loc[i, 'StudyInstanceUID'] + '/'\n    img_id = train_df.loc[i, 'id'].replace('_image','.dcm')\n    for dcm in os.listdir(long_path):\n        dcm_path = long_path+dcm+'/'\n        if img_id == os.listdir(dcm_path)[0]:\n            data_file = dicom.dcmread(dcm_path+img_id)\n            print('index : {} - DCM File Path :{}'.format(i, dcm_path+img_id))\n        else:\n            continue\n            \n    img = data_file.pixel_array\n    return img","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.723544Z","iopub.execute_input":"2021-05-24T06:26:33.723911Z","iopub.status.idle":"2021-05-24T06:26:33.729729Z","shell.execute_reply.started":"2021-05-24T06:26:33.723866Z","shell.execute_reply":"2021-05-24T06:26:33.728793Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 8. Visualize X-ray with bbox","metadata":{}},{"cell_type":"code","source":"OpacityType = list(train_df.iloc[:,5:].columns)\nOpacityType","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.738262Z","iopub.execute_input":"2021-05-24T06:26:33.73856Z","iopub.status.idle":"2021-05-24T06:26:33.744355Z","shell.execute_reply.started":"2021-05-24T06:26:33.738531Z","shell.execute_reply":"2021-05-24T06:26:33.743382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[train_df[OpacityType[0]]==1]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.748092Z","iopub.execute_input":"2021-05-24T06:26:33.748387Z","iopub.status.idle":"2021-05-24T06:26:33.765764Z","shell.execute_reply.started":"2021-05-24T06:26:33.748362Z","shell.execute_reply":"2021-05-24T06:26:33.764911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 8-a. Negative for Pneumonia","metadata":{}},{"cell_type":"code","source":"Negative_Idx = list(train_df[train_df[OpacityType[0]]==1].index)\nNegative_Idx[:9]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.766726Z","iopub.execute_input":"2021-05-24T06:26:33.766973Z","iopub.status.idle":"2021-05-24T06:26:33.77282Z","shell.execute_reply.started":"2021-05-24T06:26:33.76695Z","shell.execute_reply":"2021-05-24T06:26:33.772129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.iloc[Negative_Idx, :]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.773731Z","iopub.execute_input":"2021-05-24T06:26:33.773983Z","iopub.status.idle":"2021-05-24T06:26:33.794311Z","shell.execute_reply.started":"2021-05-24T06:26:33.77396Z","shell.execute_reply":"2021-05-24T06:26:33.793743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(3,3, figsize=(20,16))\nfig.subplots_adjust(hspace=.1, wspace=.1)\naxes = axes.ravel()\nrow = 0\nfor idx in Negative_Idx[:9]:\n    img = error_processed_extraction(idx)\n    # if (nan == nan)\n    # False\n    if (train_df.loc[idx,'boxes'] == train_df.loc[idx,'boxes']):\n        boxes = ast.literal_eval(train_df.loc[idx,'boxes'])\n        for box in boxes:\n            p = matplotlib.patches.Rectangle((box['x'], box['y']),\n                                              box['width'], box['height'],\n                                              ec='r', fc='none', lw=2.\n                                            )\n            axes[row].add_patch(p)\n    \n    axes[row].imshow(img, cmap='gray')\n    axes[row].set_title(str(train_df.loc[idx, 'label'].split(' ')[0])+ str(idx))\n    axes[row].set_xticklabels([])\n    axes[row].set_yticklabels([])\n    row += 1","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:33.795056Z","iopub.execute_input":"2021-05-24T06:26:33.795386Z","iopub.status.idle":"2021-05-24T06:26:42.482512Z","shell.execute_reply.started":"2021-05-24T06:26:33.795363Z","shell.execute_reply":"2021-05-24T06:26:42.481536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 8-b. Typical Appearance","metadata":{}},{"cell_type":"code","source":"Typical_Idx = list(train_df[train_df[OpacityType[1]]==1].index)\nTypical_Idx[:9]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:42.483771Z","iopub.execute_input":"2021-05-24T06:26:42.484107Z","iopub.status.idle":"2021-05-24T06:26:42.493055Z","shell.execute_reply.started":"2021-05-24T06:26:42.484075Z","shell.execute_reply":"2021-05-24T06:26:42.492173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.iloc[Typical_Idx, :]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:42.494437Z","iopub.execute_input":"2021-05-24T06:26:42.494773Z","iopub.status.idle":"2021-05-24T06:26:42.516938Z","shell.execute_reply.started":"2021-05-24T06:26:42.494739Z","shell.execute_reply":"2021-05-24T06:26:42.516311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(3,3, figsize=(20,16))\nfig.subplots_adjust(hspace=.1, wspace=.1)\naxes = axes.ravel()\nrow = 0\nfor idx in Typical_Idx[:9]:\n    img = error_processed_extraction(idx)\n    # if (nan == nan)\n    # False\n    if (train_df.loc[idx,'boxes'] == train_df.loc[idx,'boxes']):\n        boxes = ast.literal_eval(train_df.loc[idx,'boxes'])\n        for box in boxes:\n            p = matplotlib.patches.Rectangle((box['x'], box['y']),\n                                              box['width'], box['height'],\n                                              ec='r', fc='none', lw=2.\n                                            )\n            axes[row].add_patch(p)\n    \n    axes[row].imshow(img, cmap='gray')\n    axes[row].set_title(train_df.loc[idx, 'label'].split(' ')[0])\n    axes[row].set_xticklabels([])\n    axes[row].set_yticklabels([])\n    row += 1","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:42.517882Z","iopub.execute_input":"2021-05-24T06:26:42.518114Z","iopub.status.idle":"2021-05-24T06:26:53.537637Z","shell.execute_reply.started":"2021-05-24T06:26:42.518091Z","shell.execute_reply":"2021-05-24T06:26:53.536772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 8-c. Indeterminate Appearance","metadata":{}},{"cell_type":"code","source":"Indeterminate_Idx = list(train_df[train_df[OpacityType[2]]==1].index)\nIndeterminate_Idx[:9]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:53.538883Z","iopub.execute_input":"2021-05-24T06:26:53.539152Z","iopub.status.idle":"2021-05-24T06:26:53.546618Z","shell.execute_reply.started":"2021-05-24T06:26:53.539124Z","shell.execute_reply":"2021-05-24T06:26:53.545744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.iloc[Indeterminate_Idx, :]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:53.547892Z","iopub.execute_input":"2021-05-24T06:26:53.548154Z","iopub.status.idle":"2021-05-24T06:26:53.568046Z","shell.execute_reply.started":"2021-05-24T06:26:53.54813Z","shell.execute_reply":"2021-05-24T06:26:53.567108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(3,3, figsize=(20,16))\nfig.subplots_adjust(hspace=.1, wspace=.1)\naxes = axes.ravel()\nrow = 0\nfor idx in Indeterminate_Idx[:9]:\n    img = error_processed_extraction(idx)\n    # if (nan == nan)\n    # False\n    if (train_df.loc[idx,'boxes'] == train_df.loc[idx,'boxes']):\n        boxes = ast.literal_eval(train_df.loc[idx,'boxes'])\n        for box in boxes:\n            p = matplotlib.patches.Rectangle((box['x'], box['y']),\n                                              box['width'], box['height'],\n                                              ec='b', fc='none', lw=2.\n                                            )\n            axes[row].add_patch(p)\n    \n    axes[row].imshow(img, cmap='gray')\n    axes[row].set_title(train_df.loc[idx, 'label'].split(' ')[0])\n    axes[row].set_xticklabels([])\n    axes[row].set_yticklabels([])\n    row += 1","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:26:53.56915Z","iopub.execute_input":"2021-05-24T06:26:53.569416Z","iopub.status.idle":"2021-05-24T06:27:05.404963Z","shell.execute_reply.started":"2021-05-24T06:26:53.569389Z","shell.execute_reply":"2021-05-24T06:27:05.404062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 8-d. Atypical Appearance","metadata":{"execution":{"iopub.status.busy":"2021-05-21T14:35:54.16389Z","iopub.execute_input":"2021-05-21T14:35:54.164393Z","iopub.status.idle":"2021-05-21T14:35:54.171615Z","shell.execute_reply.started":"2021-05-21T14:35:54.164347Z","shell.execute_reply":"2021-05-21T14:35:54.170341Z"}}},{"cell_type":"code","source":"Atypical_Idx = list(train_df[train_df[OpacityType[3]]==1].index)\nAtypical_Idx[:9]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:05.405973Z","iopub.execute_input":"2021-05-24T06:27:05.406201Z","iopub.status.idle":"2021-05-24T06:27:05.413812Z","shell.execute_reply.started":"2021-05-24T06:27:05.406178Z","shell.execute_reply":"2021-05-24T06:27:05.412943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.iloc[Atypical_Idx, :]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:05.414969Z","iopub.execute_input":"2021-05-24T06:27:05.415223Z","iopub.status.idle":"2021-05-24T06:27:05.440155Z","shell.execute_reply.started":"2021-05-24T06:27:05.415198Z","shell.execute_reply":"2021-05-24T06:27:05.439294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(3,3, figsize=(20,16))\nfig.subplots_adjust(hspace=.1, wspace=.1)\naxes = axes.ravel()\nrow = 0\nfor idx in Atypical_Idx[:9]:\n    img = error_processed_extraction(idx)\n    # if (nan == nan)\n    # False\n    if (train_df.loc[idx,'boxes'] == train_df.loc[idx,'boxes']):\n        boxes = ast.literal_eval(train_df.loc[idx,'boxes'])\n        for box in boxes:\n            p = matplotlib.patches.Rectangle((box['x'], box['y']),\n                                              box['width'], box['height'],\n                                              ec='g', fc='none', lw=2.\n                                            )\n            axes[row].add_patch(p)\n    \n    axes[row].imshow(img, cmap='gray')\n    axes[row].set_title(train_df.loc[idx, 'label'].split(' ')[0])\n    axes[row].set_xticklabels([])\n    axes[row].set_yticklabels([])\n    row += 1","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:05.44131Z","iopub.execute_input":"2021-05-24T06:27:05.44158Z","iopub.status.idle":"2021-05-24T06:27:18.73735Z","shell.execute_reply.started":"2021-05-24T06:27:05.441553Z","shell.execute_reply":"2021-05-24T06:27:18.736364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 9. Feature Engineering III","metadata":{}},{"cell_type":"markdown","source":"\nOutliers detected through above visualizations. let's check them","metadata":{}},{"cell_type":"code","source":"train_df[(train_df['OpacityCount']==0) & (train_df['Negative for Pneumonia']!=1)]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:18.738731Z","iopub.execute_input":"2021-05-24T06:27:18.739142Z","iopub.status.idle":"2021-05-24T06:27:18.760062Z","shell.execute_reply.started":"2021-05-24T06:27:18.739104Z","shell.execute_reply":"2021-05-24T06:27:18.759267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Anomaly 304 rows : label is 'none' but Non Negative for Pneumonia**","metadata":{}},{"cell_type":"markdown","source":"### 9-a. anomaly detection\n\n- Cases with no opacity detected but classified as symptomatic","metadata":{}},{"cell_type":"code","source":"i=0\nfig, axes = plt.subplots(nrows=1,ncols=3, figsize=(12,4))\nfor type in OpacityType[1:]:\n    sr = train_df[(train_df['OpacityCount']==0) & (train_df['Negative for Pneumonia']!=1)].loc[:,type].value_counts()\n    sns.barplot(x=sr.index, y=sr.values, ax=axes[i])\n    axes[i].set_title(type)\n    i += 1","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:18.761099Z","iopub.execute_input":"2021-05-24T06:27:18.761546Z","iopub.status.idle":"2021-05-24T06:27:19.049074Z","shell.execute_reply.started":"2021-05-24T06:27:18.761514Z","shell.execute_reply":"2021-05-24T06:27:19.04799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Anom_Count = train_df[(train_df['OpacityCount']==0) & (train_df['Negative for Pneumonia']!=1)][OpacityType[1:]].sum()\nAnom_Count","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:19.050667Z","iopub.execute_input":"2021-05-24T06:27:19.051103Z","iopub.status.idle":"2021-05-24T06:27:19.062472Z","shell.execute_reply.started":"2021-05-24T06:27:19.051063Z","shell.execute_reply":"2021-05-24T06:27:19.06153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(8,4))\nsns.barplot(x=Anom_Count.index, y=Anom_Count.values)\nplt.title('Count of \"label==none\"')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:19.064072Z","iopub.execute_input":"2021-05-24T06:27:19.06451Z","iopub.status.idle":"2021-05-24T06:27:19.187025Z","shell.execute_reply.started":"2021-05-24T06:27:19.064469Z","shell.execute_reply":"2021-05-24T06:27:19.186423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 9-b. Show Outliers in `Typical Appearance`","metadata":{}},{"cell_type":"code","source":"train_df[(train_df['OpacityCount']==0) & (train_df['Negative for Pneumonia']!=1) & (train_df['Typical Appearance']==1)].head()","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:19.188025Z","iopub.execute_input":"2021-05-24T06:27:19.188487Z","iopub.status.idle":"2021-05-24T06:27:19.20251Z","shell.execute_reply.started":"2021-05-24T06:27:19.18845Z","shell.execute_reply":"2021-05-24T06:27:19.20189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Outlier_Typical_Idx = list(train_df[(train_df['OpacityCount']==0) & (train_df['Negative for Pneumonia']!=1) & (train_df['Typical Appearance']==1)].index)\nOutlier_Typical_Idx[:6]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:19.203434Z","iopub.execute_input":"2021-05-24T06:27:19.203794Z","iopub.status.idle":"2021-05-24T06:27:19.215872Z","shell.execute_reply.started":"2021-05-24T06:27:19.203768Z","shell.execute_reply":"2021-05-24T06:27:19.214967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(2,3, figsize=(20,10))\nfig.subplots_adjust(hspace=.1, wspace=.1)\naxes = axes.ravel()\nrow = 0\nfor idx in Outlier_Typical_Idx[:6]:\n    img = error_processed_extraction(idx)\n\n    axes[row].imshow(img, cmap='gray')\n    axes[row].set_title(train_df.loc[idx, 'label'].split(' ')[0])\n    axes[row].set_xticklabels([])\n    axes[row].set_yticklabels([])\n    row += 1","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:19.217199Z","iopub.execute_input":"2021-05-24T06:27:19.217503Z","iopub.status.idle":"2021-05-24T06:27:27.264185Z","shell.execute_reply.started":"2021-05-24T06:27:19.217477Z","shell.execute_reply":"2021-05-24T06:27:27.263283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 9-c. Show Outliers in `Indeterminate Appearance`","metadata":{"execution":{"iopub.status.busy":"2021-05-22T12:38:32.299964Z","iopub.execute_input":"2021-05-22T12:38:32.300377Z","iopub.status.idle":"2021-05-22T12:38:32.305434Z","shell.execute_reply.started":"2021-05-22T12:38:32.300345Z","shell.execute_reply":"2021-05-22T12:38:32.304023Z"}}},{"cell_type":"code","source":"train_df[(train_df['OpacityCount']==0) & (train_df['Negative for Pneumonia']!=1) & (train_df['Indeterminate Appearance']==1)].head()","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:27.265458Z","iopub.execute_input":"2021-05-24T06:27:27.265995Z","iopub.status.idle":"2021-05-24T06:27:27.28357Z","shell.execute_reply.started":"2021-05-24T06:27:27.265952Z","shell.execute_reply":"2021-05-24T06:27:27.282898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Outlier_Indeterminate_Idx = list(train_df[(train_df['OpacityCount']==0) & (train_df['Negative for Pneumonia']!=1) & (train_df['Indeterminate Appearance']==1)].index)\nOutlier_Indeterminate_Idx[:6]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:27.28472Z","iopub.execute_input":"2021-05-24T06:27:27.285142Z","iopub.status.idle":"2021-05-24T06:27:27.295552Z","shell.execute_reply.started":"2021-05-24T06:27:27.285101Z","shell.execute_reply":"2021-05-24T06:27:27.294763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(2,3, figsize=(20,10))\nfig.subplots_adjust(hspace=.1, wspace=.1)\naxes = axes.ravel()\nrow = 0\nfor idx in Outlier_Indeterminate_Idx[:6]:\n    img = error_processed_extraction(idx)\n    \n    axes[row].imshow(img, cmap='gray')\n    axes[row].set_title(train_df.loc[idx, 'label'].split(' ')[0])\n    axes[row].set_xticklabels([])\n    axes[row].set_yticklabels([])\n    row += 1","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:27.296508Z","iopub.execute_input":"2021-05-24T06:27:27.296739Z","iopub.status.idle":"2021-05-24T06:27:34.751345Z","shell.execute_reply.started":"2021-05-24T06:27:27.296717Z","shell.execute_reply":"2021-05-24T06:27:34.750795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 9-d. Show Outliers in `Atypical Appearance`","metadata":{}},{"cell_type":"code","source":"train_df[(train_df['OpacityCount']==0) & (train_df['Negative for Pneumonia']!=1) & (train_df['Atypical Appearance']==1)].head()","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:34.752267Z","iopub.execute_input":"2021-05-24T06:27:34.752653Z","iopub.status.idle":"2021-05-24T06:27:34.766304Z","shell.execute_reply.started":"2021-05-24T06:27:34.752626Z","shell.execute_reply":"2021-05-24T06:27:34.76545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Outlier_Atypical_Idx = list(train_df[(train_df['OpacityCount']==0) & (train_df['Negative for Pneumonia']!=1) & (train_df['Atypical Appearance']==1)].index)\nOutlier_Atypical_Idx[:6]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:34.767402Z","iopub.execute_input":"2021-05-24T06:27:34.767875Z","iopub.status.idle":"2021-05-24T06:27:34.78424Z","shell.execute_reply.started":"2021-05-24T06:27:34.767801Z","shell.execute_reply":"2021-05-24T06:27:34.783312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(2,3, figsize=(20,10))\nfig.subplots_adjust(hspace=.1, wspace=.1)\naxes = axes.ravel()\nrow = 0\nfor idx in Outlier_Atypical_Idx[:6]:\n    img = error_processed_extraction(idx)\n    \n    axes[row].imshow(img, cmap='gray')\n    axes[row].set_title(train_df.loc[idx, 'label'].split(' ')[0])\n    axes[row].set_xticklabels([])\n    axes[row].set_yticklabels([])\n    row += 1","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:34.785505Z","iopub.execute_input":"2021-05-24T06:27:34.785877Z","iopub.status.idle":"2021-05-24T06:27:42.132406Z","shell.execute_reply.started":"2021-05-24T06:27:34.78582Z","shell.execute_reply":"2021-05-24T06:27:42.131511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 10. Image Data Preprocessing","metadata":{}},{"cell_type":"markdown","source":"### 10-a. Add image path to a separate column","metadata":{}},{"cell_type":"code","source":"for _, row in train_df.iloc[:5].iterrows():\n    print(row)","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:42.133431Z","iopub.execute_input":"2021-05-24T06:27:42.133673Z","iopub.status.idle":"2021-05-24T06:27:42.142469Z","shell.execute_reply.started":"2021-05-24T06:27:42.133649Z","shell.execute_reply":"2021-05-24T06:27:42.141593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from glob import glob","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:42.143761Z","iopub.execute_input":"2021-05-24T06:27:42.144121Z","iopub.status.idle":"2021-05-24T06:27:42.152475Z","shell.execute_reply.started":"2021-05-24T06:27:42.144086Z","shell.execute_reply":"2021-05-24T06:27:42.151695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for _, row in train_df.iloc[:5].iterrows():\n    image_id = row['id'].split('_')[0]\n    study_id = row['StudyInstanceUID']\n    img_path = glob(f'{path}/train/{study_id}/*/{image_id}.dcm')\n    print(img_path)","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:42.153644Z","iopub.execute_input":"2021-05-24T06:27:42.15414Z","iopub.status.idle":"2021-05-24T06:27:42.171679Z","shell.execute_reply.started":"2021-05-24T06:27:42.154038Z","shell.execute_reply":"2021-05-24T06:27:42.170896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path_list = []\nfor _, row in train_df.iterrows():\n    image_id = row['id'].split('_')[0]\n    study_id = row['StudyInstanceUID']\n    img_path = glob(f'{path}/train/{study_id}/*/{image_id}.dcm')\n    if len(img_path)==1:\n        path_list.append(img_path[0])\n    else:\n        print(img_path)","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:27:42.172568Z","iopub.execute_input":"2021-05-24T06:27:42.172781Z","iopub.status.idle":"2021-05-24T06:28:09.729276Z","shell.execute_reply.started":"2021-05-24T06:27:42.172761Z","shell.execute_reply":"2021-05-24T06:28:09.72846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(path_list)","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:09.730235Z","iopub.execute_input":"2021-05-24T06:28:09.730479Z","iopub.status.idle":"2021-05-24T06:28:09.735434Z","shell.execute_reply.started":"2021-05-24T06:28:09.730456Z","shell.execute_reply":"2021-05-24T06:28:09.734824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path_list[:10]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:09.736469Z","iopub.execute_input":"2021-05-24T06:28:09.736709Z","iopub.status.idle":"2021-05-24T06:28:09.74574Z","shell.execute_reply.started":"2021-05-24T06:28:09.736685Z","shell.execute_reply":"2021-05-24T06:28:09.744828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df['Path'] = path_list","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:09.746569Z","iopub.execute_input":"2021-05-24T06:28:09.746866Z","iopub.status.idle":"2021-05-24T06:28:09.754992Z","shell.execute_reply.started":"2021-05-24T06:28:09.746842Z","shell.execute_reply":"2021-05-24T06:28:09.754217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All Image path have been saved (6334)","metadata":{}},{"cell_type":"code","source":"train_df","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:09.755866Z","iopub.execute_input":"2021-05-24T06:28:09.756142Z","iopub.status.idle":"2021-05-24T06:28:09.77729Z","shell.execute_reply.started":"2021-05-24T06:28:09.756118Z","shell.execute_reply":"2021-05-24T06:28:09.776608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We use this dataframe at Part 2. Let's save this to csv file.","metadata":{}},{"cell_type":"code","source":"train_df.to_csv('train_df.csv')","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:09.778353Z","iopub.execute_input":"2021-05-24T06:28:09.778644Z","iopub.status.idle":"2021-05-24T06:28:09.858643Z","shell.execute_reply.started":"2021-05-24T06:28:09.778621Z","shell.execute_reply":"2021-05-24T06:28:09.858004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 10-b. Resize the image (uniform to 150x150) and Scale each pixel values (uniform range 1~255)","metadata":{}},{"cell_type":"code","source":"data_file = dicom.read_file(path_list[0])","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:09.859658Z","iopub.execute_input":"2021-05-24T06:28:09.859895Z","iopub.status.idle":"2021-05-24T06:28:09.87221Z","shell.execute_reply.started":"2021-05-24T06:28:09.859872Z","shell.execute_reply":"2021-05-24T06:28:09.871589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_file.pixel_array","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:09.873143Z","iopub.execute_input":"2021-05-24T06:28:09.873358Z","iopub.status.idle":"2021-05-24T06:28:09.889407Z","shell.execute_reply.started":"2021-05-24T06:28:09.873337Z","shell.execute_reply":"2021-05-24T06:28:09.888711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_file","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:09.890289Z","iopub.execute_input":"2021-05-24T06:28:09.890532Z","iopub.status.idle":"2021-05-24T06:28:09.897352Z","shell.execute_reply.started":"2021-05-24T06:28:09.890509Z","shell.execute_reply":"2021-05-24T06:28:09.896604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_file.Rows","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:09.898277Z","iopub.execute_input":"2021-05-24T06:28:09.898697Z","iopub.status.idle":"2021-05-24T06:28:09.908094Z","shell.execute_reply.started":"2021-05-24T06:28:09.898661Z","shell.execute_reply":"2021-05-24T06:28:09.907226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_file.Columns","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:09.909181Z","iopub.execute_input":"2021-05-24T06:28:09.909661Z","iopub.status.idle":"2021-05-24T06:28:09.916288Z","shell.execute_reply.started":"2021-05-24T06:28:09.909624Z","shell.execute_reply":"2021-05-24T06:28:09.915636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Test function, We don't use this\ndef extract_img_size(path_list):\n    origin_img_heights = []\n    origin_img_widths = []\n    i = 0\n    for path in path_list:\n        data_file = dicom.read_file(path)\n        origin_img_heights.append(data_file.Rows)\n        origin_img_widths.append(data_file.Columns)\n        i += 1\n        if i % 100 == 0:\n            print('{}/{}'.format(i,len(path_list)))\n            \n    return origin_img_heights, origin_img_widths","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:09.917748Z","iopub.execute_input":"2021-05-24T06:28:09.918295Z","iopub.status.idle":"2021-05-24T06:28:09.924428Z","shell.execute_reply.started":"2021-05-24T06:28:09.918259Z","shell.execute_reply":"2021-05-24T06:28:09.923794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"origin_img_heights, origin_img_widths = extract_img_size(path_list[:10])","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:09.925353Z","iopub.execute_input":"2021-05-24T06:28:09.925804Z","iopub.status.idle":"2021-05-24T06:28:10.007492Z","shell.execute_reply.started":"2021-05-24T06:28:09.925776Z","shell.execute_reply":"2021-05-24T06:28:10.006804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"origin_img_heights","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.008456Z","iopub.execute_input":"2021-05-24T06:28:10.008819Z","iopub.status.idle":"2021-05-24T06:28:10.013692Z","shell.execute_reply.started":"2021-05-24T06:28:10.008778Z","shell.execute_reply":"2021-05-24T06:28:10.012818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"origin_img_widths","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.014958Z","iopub.execute_input":"2021-05-24T06:28:10.015278Z","iopub.status.idle":"2021-05-24T06:28:10.024385Z","shell.execute_reply.started":"2021-05-24T06:28:10.015248Z","shell.execute_reply":"2021-05-24T06:28:10.023619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We use this function\nimport cv2\ndef extract_resized_and_origin_img_info(path_list):\n    img_list = []\n    origin_img_heights = []\n    origin_img_widths = []\n    i = 0\n    for path in path_list:\n        data_file = dicom.read_file(path)\n        img = data_file.pixel_array\n\n            \n        origin_img_heights.append(img.shape[0])\n        origin_img_widths.append(img.shape[1])\n\n        \n        # scailing to 0~255\n        img = (img - np.min(img)) / np.max(img)\n        img = (img * 255).astype(np.uint8)\n        \n        # resizing to 4000+ to 150 default\n        img = cv2.resize(img, (150,150))\n        img_list.append(img)\n        img_array = np.array(img_list)\n        i += 1\n        if i % 100 == 0:\n            print('{} / {}'.format(len(img_array),len(path_list)))\n    return img_array, origin_img_heights, origin_img_widths","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.02606Z","iopub.execute_input":"2021-05-24T06:28:10.026282Z","iopub.status.idle":"2021-05-24T06:28:10.03289Z","shell.execute_reply.started":"2021-05-24T06:28:10.02626Z","shell.execute_reply":"2021-05-24T06:28:10.032119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_imgs, origin_img_heights2, origin_img_widths2 = extract_resized_and_origin_img_info(path_list[:10])","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.033613Z","iopub.execute_input":"2021-05-24T06:28:10.03392Z","iopub.status.idle":"2021-05-24T06:28:10.870324Z","shell.execute_reply.started":"2021-05-24T06:28:10.033897Z","shell.execute_reply":"2021-05-24T06:28:10.869585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_imgs.shape","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.871382Z","iopub.execute_input":"2021-05-24T06:28:10.87162Z","iopub.status.idle":"2021-05-24T06:28:10.879005Z","shell.execute_reply.started":"2021-05-24T06:28:10.871596Z","shell.execute_reply":"2021-05-24T06:28:10.878222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_imgs[0].shape","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.880192Z","iopub.execute_input":"2021-05-24T06:28:10.880676Z","iopub.status.idle":"2021-05-24T06:28:10.887197Z","shell.execute_reply.started":"2021-05-24T06:28:10.880638Z","shell.execute_reply":"2021-05-24T06:28:10.88644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_imgs[0]","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.888129Z","iopub.execute_input":"2021-05-24T06:28:10.888363Z","iopub.status.idle":"2021-05-24T06:28:10.896429Z","shell.execute_reply.started":"2021-05-24T06:28:10.888341Z","shell.execute_reply":"2021-05-24T06:28:10.895694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('pixel range : ', '{} ~ {}'.format(min(test_imgs[0].reshape(-1)),max(test_imgs[0].reshape(-1))))","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.897538Z","iopub.execute_input":"2021-05-24T06:28:10.897802Z","iopub.status.idle":"2021-05-24T06:28:10.911864Z","shell.execute_reply.started":"2021-05-24T06:28:10.897779Z","shell.execute_reply":"2021-05-24T06:28:10.911186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"origin_img_heights2","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.912853Z","iopub.execute_input":"2021-05-24T06:28:10.913249Z","iopub.status.idle":"2021-05-24T06:28:10.920021Z","shell.execute_reply.started":"2021-05-24T06:28:10.91322Z","shell.execute_reply":"2021-05-24T06:28:10.919391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"origin_img_widths2","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.921248Z","iopub.execute_input":"2021-05-24T06:28:10.921799Z","iopub.status.idle":"2021-05-24T06:28:10.929074Z","shell.execute_reply.started":"2021-05-24T06:28:10.921764Z","shell.execute_reply":"2021-05-24T06:28:10.928349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print((origin_img_heights == origin_img_heights2), (origin_img_widths == origin_img_widths2))","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.930389Z","iopub.execute_input":"2021-05-24T06:28:10.931079Z","iopub.status.idle":"2021-05-24T06:28:10.937603Z","shell.execute_reply.started":"2021-05-24T06:28:10.931039Z","shell.execute_reply":"2021-05-24T06:28:10.936635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"exactly same result on two function","metadata":{}},{"cell_type":"markdown","source":"### 10-c. Calculate the resize ratio(x, y) and Apply the same to the bounding box","metadata":{}},{"cell_type":"code","source":"x_scale_list=[]\ny_scale_list=[]\nif len(origin_img_heights) == len(origin_img_widths):\n    for i in range(len(origin_img_heights)):\n        x_scale = 150 / origin_img_widths[i]\n        x_scale_list.append(x_scale)\n        print(i)\n        y_scale = 150 / origin_img_heights[i]\n        y_scale_list.append(y_scale)","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.938528Z","iopub.execute_input":"2021-05-24T06:28:10.938924Z","iopub.status.idle":"2021-05-24T06:28:10.946784Z","shell.execute_reply.started":"2021-05-24T06:28:10.938896Z","shell.execute_reply":"2021-05-24T06:28:10.945852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_scale_list","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.947943Z","iopub.execute_input":"2021-05-24T06:28:10.948246Z","iopub.status.idle":"2021-05-24T06:28:10.954707Z","shell.execute_reply.started":"2021-05-24T06:28:10.948221Z","shell.execute_reply":"2021-05-24T06:28:10.95395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_scale_list","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.956029Z","iopub.execute_input":"2021-05-24T06:28:10.9566Z","iopub.status.idle":"2021-05-24T06:28:10.964125Z","shell.execute_reply.started":"2021-05-24T06:28:10.956556Z","shell.execute_reply":"2021-05-24T06:28:10.963128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before vs after resizing","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(3,3, figsize=(20,16))\nfig.subplots_adjust(hspace=.1, wspace=.1)\naxes = axes.ravel()\nrow = 0\nfor idx in range(9):\n    img = error_processed_extraction(idx)\n    # if (nan == nan)\n    # False\n    if (train_df.loc[idx,'boxes'] == train_df.loc[idx,'boxes']):\n        boxes = ast.literal_eval(train_df.loc[idx,'boxes'])\n        for box in boxes:\n            p = matplotlib.patches.Rectangle((box['x'], box['y']),\n                                              box['width'], box['height'],\n                                              ec='b', fc='none', lw=2.)\n            axes[row].add_patch(p)\n    \n    axes[row].imshow(img, cmap='gray')\n    axes[row].set_title(train_df.loc[idx, 'label'].split(' ')[0])\n    axes[row].set_xticklabels([])\n    axes[row].set_yticklabels([])\n    row += 1","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:10.965414Z","iopub.execute_input":"2021-05-24T06:28:10.965764Z","iopub.status.idle":"2021-05-24T06:28:19.143541Z","shell.execute_reply.started":"2021-05-24T06:28:10.965734Z","shell.execute_reply":"2021-05-24T06:28:19.142901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(3,3, figsize=(16,16))\nfig.subplots_adjust(hspace=.1, wspace=.05)\naxes = axes.ravel()\nrow = 0\nfor idx in range(9):\n    img = test_imgs[idx]\n    # if (nan == nan)\n    # False\n    if (train_df.loc[idx,'boxes'] == train_df.loc[idx,'boxes']):\n        boxes = ast.literal_eval(train_df.loc[idx,'boxes'])\n        for box in boxes:\n            p = matplotlib.patches.Rectangle((box['x']*x_scale_list[idx], box['y']*y_scale_list[idx]),\n                                              box['width']*x_scale_list[idx], box['height']*y_scale_list[idx],\n                                              ec='b', fc='none', lw=2.)\n            axes[row].add_patch(p)\n    \n    axes[row].imshow(img, cmap='gray')\n    axes[row].set_title(train_df.loc[idx, 'label'].split(' ')[0])\n    axes[row].set_xticklabels([])\n    axes[row].set_yticklabels([])\n    row += 1","metadata":{"execution":{"iopub.status.busy":"2021-05-24T06:28:19.144393Z","iopub.execute_input":"2021-05-24T06:28:19.14493Z","iopub.status.idle":"2021-05-24T06:28:20.401364Z","shell.execute_reply.started":"2021-05-24T06:28:19.144897Z","shell.execute_reply":"2021-05-24T06:28:20.400802Z"},"trusted":true},"execution_count":null,"outputs":[]}]}