{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# EfficientDet\n\nARXIV [https://arxiv.org/pdf/1911.09070.pdf](https://arxiv.org/pdf/1911.09070.pdf) \nGithub [https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch](https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch) \n\nEfficientDet employs EfficientNet as the backbone network, BiFPN as the feature network, and shared class/box prediction network. Both BiFPN layers and class/box net layers are repeated multiple times based on different resource constraints.\n\n[Object Detection SOTA model](https://paperswithcode.com/sota/object-detection-on-coco) \nThis page shows object detection models' score on COCO test-dev. I focus AP50 score, because this competitions' metric is AP50. \n1. DyHead (Based Swin-L) : 78.5\n2. DetectoRS (Based ResNeXt) : 74.2\n3. YOLOv4-P7 (Based Scaled-YOLO) : 73.3\n4. EfficientDet-D7 (Based EfficientNet) : 72.4\n5. YOLOv4-608 (Based YOLO) : 65.7\n\n\n\n\n EfficientDet is better model than YOLOv5 on AP.\n\n\n\nEveryone used model based YOLOv4 or YOLOv5, but this model isn't SOTA model. I'll try EfficientDet first and then Scaled-YOLOv4, DyHead. \n \nThree notebooks summarize how to use this model.\n1. [Preprocessing](https://www.kaggle.com/adldotori/efficientdet-preprocessing-better-than-yolov5/)\n2. [Training](https://www.kaggle.com/adldotori/efficientdet-training-better-than-yolov5/)\n3. [Inference](https://www.kaggle.com/adldotori/efficientdet-inference-better-than-yolov5/) - 2days later open!\n\nThis notebook is first notebook which includes how to preprocess the data. \nLet's start! \n\nThis picture shows the rough structure of efficientdet.\n","metadata":{"execution":{"iopub.status.busy":"2021-07-03T10:35:03.427345Z","iopub.execute_input":"2021-07-03T10:35:03.427684Z","iopub.status.idle":"2021-07-03T10:35:03.431325Z","shell.execute_reply.started":"2021-07-03T10:35:03.427651Z","shell.execute_reply":"2021-07-03T10:35:03.430412Z"}}},{"cell_type":"markdown","source":"# Environment","metadata":{}},{"cell_type":"code","source":"!conda install gdcm -c conda-forge -y\n!pip install pycocotools numpy opencv-python tqdm tensorboard tensorboardX pyyaml webcolors matplotlib","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Base Setting","metadata":{}},{"cell_type":"markdown","source":"Clone EfficientDet repository. In this repo, we can training after preprocessing **pre-trained weights**, **yml file**, **image files(not dcm)**, **annotation files**.","metadata":{}},{"cell_type":"code","source":"!git clone https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch\n\nimport os\nos.chdir(\"Yet-Another-EfficientDet-Pytorch\")","metadata":{"execution":{"iopub.status.busy":"2021-07-03T11:52:24.505012Z","iopub.execute_input":"2021-07-03T11:52:24.505308Z","iopub.status.idle":"2021-07-03T11:52:27.143203Z","shell.execute_reply.started":"2021-07-03T11:52:24.505278Z","shell.execute_reply":"2021-07-03T11:52:27.141947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# load checkpoint\n! mkdir weights\n! wget https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/releases/download/1.0/efficientdet-d0.pth -O weights/efficientdet-d0.pth","metadata":{"execution":{"iopub.status.busy":"2021-07-03T11:52:27.145951Z","iopub.execute_input":"2021-07-03T11:52:27.146275Z","iopub.status.idle":"2021-07-03T11:52:30.405008Z","shell.execute_reply.started":"2021-07-03T11:52:27.146244Z","shell.execute_reply":"2021-07-03T11:52:30.403695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"siim_yml = '''\nproject_name: siim # also the folder name of the dataset that under data_path folder\ntrain_set: train\nval_set: val\nnum_gpus: 1\n\n# mean and std in RGB order, actually this part should remain unchanged as long as your dataset is similar to coco.\nmean: [ 0.485, 0.456, 0.406 ]\nstd: [ 0.229, 0.224, 0.225 ]\n\n# this anchor is adapted to the dataset\nanchors_scales: '[2 ** 0, 2 ** (1.0 / 3.0), 2 ** (2.0 / 3.0)]'\nanchors_ratios: '[(1.0, 1.0), (1.3, 0.8), (1.9, 0.5)]'\n\nobj_list: ['typical', 'indeterminate', 'atypical']\n'''\nwith open('projects/siim.yml', 'w') as f:\n f.write(siim_yml)","metadata":{"execution":{"iopub.status.busy":"2021-07-03T11:52:30.407011Z","iopub.execute_input":"2021-07-03T11:52:30.407337Z","iopub.status.idle":"2021-07-03T11:52:30.412382Z","shell.execute_reply.started":"2021-07-03T11:52:30.407304Z","shell.execute_reply":"2021-07-03T11:52:30.411649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Change to 256x256px Image","metadata":{}},{"cell_type":"code","source":"import os\nfrom PIL import Image\nimport pandas as pd\nfrom tqdm.auto import tqdm\nimport numpy as np\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\n\nimport torch\n\ndef read_xray(path, voi_lut=False, fix_monochrome=True):\n # Original from: https://www.kaggle.com/raddar/convert-dicom-to-np-array-the-correct-way\n dicom = pydicom.read_file(path)\n # VOI LUT (if available by DICOM device) is used to transform raw DICOM data to\n # \"human-friendly\" view\n if voi_lut:\n data = apply_voi_lut(dicom.pixel_array, dicom)\n else:\n data = dicom.pixel_array\n\n # depending on this value, X-ray may look inverted - fix that:\n if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\n\n data = data - np.min(data)\n data = data / np.max(data)\n data = (data * 255).astype(np.uint8)\n\n return data\n\n\ndef resize(array, size, keep_ratio=False, resample=Image.LANCZOS):\n # Original from: https://www.kaggle.com/xhlulu/vinbigdata-process-and-resize-to-image\n im = Image.fromarray(array)\n\n if keep_ratio:\n im.thumbnail((size, size), resample)\n else:\n im = im.resize((size, size), resample)\n\n return im","metadata":{"execution":{"iopub.status.busy":"2021-07-03T11:52:30.41364Z","iopub.execute_input":"2021-07-03T11:52:30.414137Z","iopub.status.idle":"2021-07-03T11:52:30.429003Z","shell.execute_reply.started":"2021-07-03T11:52:30.414105Z","shell.execute_reply":"2021-07-03T11:52:30.427896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from glob import glob\nINPUT_PATH = \"/kaggle/input/siim-covid19-detection/\"\n\nfor split in [\"test\", \"train\"]:\n save_dir = f\"datasets/siim/{split}/\"\n\n os.makedirs(save_dir, exist_ok=True)\n\n for path in tqdm(glob(INPUT_PATH + split + '/*/*/*')):\n # set keep_ratio=True to have original aspect ratio\n xray = read_xray(path)\n im = resize(xray, size=256)\n im.save(os.path.join(save_dir, path.split('/')[-1][:-3]+'jpg'))","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:44:55.580063Z","iopub.execute_input":"2021-07-08T11:44:55.581496Z","iopub.status.idle":"2021-07-08T11:44:55.713778Z","shell.execute_reply.started":"2021-07-08T11:44:55.581235Z","shell.execute_reply":"2021-07-08T11:44:55.711755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocessing","metadata":{}},{"cell_type":"markdown","source":"Let's check what the csv file looks like.","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nfrom glob import glob\nimport pydicom","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:44:58.329728Z","iopub.execute_input":"2021-07-08T11:44:58.33017Z","iopub.status.idle":"2021-07-08T11:44:58.673261Z","shell.execute_reply.started":"2021-07-08T11:44:58.330128Z","shell.execute_reply":"2021-07-08T11:44:58.671848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study = pd.read_csv(INPUT_PATH + 'train_study_level.csv')\ntrain_image = pd.read_csv(INPUT_PATH + 'train_image_level.csv')","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:45:41.517305Z","iopub.execute_input":"2021-07-08T11:45:41.517771Z","iopub.status.idle":"2021-07-08T11:45:41.561394Z","shell.execute_reply.started":"2021-07-08T11:45:41.517737Z","shell.execute_reply":"2021-07-08T11:45:41.560382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:45:51.082237Z","iopub.execute_input":"2021-07-08T11:45:51.082993Z","iopub.status.idle":"2021-07-08T11:45:51.095347Z","shell.execute_reply.started":"2021-07-08T11:45:51.082952Z","shell.execute_reply":"2021-07-08T11:45:51.094042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"id is too long, and column name too. Let's shorten the name.","metadata":{}},{"cell_type":"code","source":"train_study = train_study.rename(columns = {\n 'Negative for Pneumonia': 'Negative', 'Typical Appearance': 'Typical',\n 'Indeterminate Appearance': 'Indeterminate', 'Atypical Appearance': 'Atypical'},\n inplace = False)\ntrain_study['StudyInstanceUID'] = train_study['id'].str[:-6]\ntrain_study.drop(columns=['id'], inplace=True)\ntrain_study.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:44:59.925646Z","iopub.execute_input":"2021-07-08T11:44:59.926123Z","iopub.status.idle":"2021-07-08T11:45:00.071478Z","shell.execute_reply.started":"2021-07-08T11:44:59.926082Z","shell.execute_reply":"2021-07-08T11:45:00.070228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:46:44.663991Z","iopub.execute_input":"2021-07-08T11:46:44.664402Z","iopub.status.idle":"2021-07-08T11:46:44.678293Z","shell.execute_reply.started":"2021-07-08T11:46:44.664368Z","shell.execute_reply":"2021-07-08T11:46:44.677247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This dataframe's id is too long, and merge train_study dataframe.","metadata":{}},{"cell_type":"code","source":"train_image = train_image.merge(train_study, on='StudyInstanceUID')\ntrain_image['id'] = train_image['id'].str[:-6]\ntrain_image.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:45:01.017657Z","iopub.execute_input":"2021-07-08T11:45:01.018041Z","iopub.status.idle":"2021-07-08T11:45:01.064363Z","shell.execute_reply.started":"2021-07-08T11:45:01.018Z","shell.execute_reply":"2021-07-08T11:45:01.063169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Add path of image.","metadata":{}},{"cell_type":"code","source":"id_path = []\nfor i in glob('/kaggle/input/siim-covid19-detection/train/*/*/*'):\n id_path.append((i, i.split('/')[-1][:-4]))\nid_path = pd.DataFrame(id_path, columns=['path', 'id'])\ntrain_image = train_image.merge(id_path, on='id')\ntrain_image.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:45:01.512166Z","iopub.execute_input":"2021-07-08T11:45:01.512619Z","iopub.status.idle":"2021-07-08T11:45:30.075425Z","shell.execute_reply.started":"2021-07-08T11:45:01.512581Z","shell.execute_reply":"2021-07-08T11:45:30.074258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image.iloc[0]['boxes']","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:45:30.077378Z","iopub.execute_input":"2021-07-08T11:45:30.077737Z","iopub.status.idle":"2021-07-08T11:45:30.085827Z","shell.execute_reply.started":"2021-07-08T11:45:30.077704Z","shell.execute_reply":"2021-07-08T11:45:30.08474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image.iloc[0]['label']","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:45:30.088098Z","iopub.execute_input":"2021-07-08T11:45:30.088454Z","iopub.status.idle":"2021-07-08T11:45:30.104182Z","shell.execute_reply.started":"2021-07-08T11:45:30.088407Z","shell.execute_reply":"2021-07-08T11:45:30.103286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"boxes column includes x, y, width, height.\n\n","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:52:14.949131Z","iopub.execute_input":"2021-07-08T11:52:14.949554Z","iopub.status.idle":"2021-07-08T11:52:14.958366Z","shell.execute_reply.started":"2021-07-08T11:52:14.949518Z","shell.execute_reply":"2021-07-08T11:52:14.956958Z"}}},{"cell_type":"markdown","source":"label column includes x1, y1, x2, y2.\n\n","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:54:34.551847Z","iopub.execute_input":"2021-07-08T11:54:34.552313Z","iopub.status.idle":"2021-07-08T11:54:34.558844Z","shell.execute_reply.started":"2021-07-08T11:54:34.552273Z","shell.execute_reply":"2021-07-08T11:54:34.557529Z"}}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"pydicom.read_file(train_image.iloc[0]['path']).pixel_array.shape","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:45:30.105404Z","iopub.execute_input":"2021-07-08T11:45:30.106244Z","iopub.status.idle":"2021-07-08T11:45:30.979772Z","shell.execute_reply.started":"2021-07-08T11:45:30.106185Z","shell.execute_reply":"2021-07-08T11:45:30.978448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can know the image size in the same way as above.","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:55:13.744579Z","iopub.execute_input":"2021-07-08T11:55:13.745248Z","iopub.status.idle":"2021-07-08T11:55:13.751132Z","shell.execute_reply.started":"2021-07-08T11:55:13.745181Z","shell.execute_reply":"2021-07-08T11:55:13.749939Z"}}},{"cell_type":"code","source":"xy = []\nfor i, data in train_image.iterrows():\n xy.append(pydicom.read_file(data['path']).pixel_array.shape)\ntrain_image[['xcell','ycell']] = xy\ntrain_image.to_csv('datasets/train_image.csv', index=None)\ntrain_image.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T12:26:12.74229Z","iopub.execute_input":"2021-07-03T12:26:12.742803Z","iopub.status.idle":"2021-07-03T12:26:29.762888Z","shell.execute_reply.started":"2021-07-03T12:26:12.742764Z","shell.execute_reply":"2021-07-03T12:26:29.760479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Split train to train&val","metadata":{}},{"cell_type":"markdown","source":"Make the validation set.","metadata":{}},{"cell_type":"code","source":"import random\nimport os\nimport shutil\nimport pandas as pd\nimport json\n\nrandom.seed(481)\nSRC_PATH = 'datasets/siim/train/'\nTRG_PATH = 'datasets/siim/'\ntrain_list = os.listdir(SRC_PATH)\nrandom.shuffle(train_list)\n\nimport shutil\nos.makedirs(TRG_PATH+'val', exist_ok=True)\nfor path in train_list[int(len(train_list)*0.8):]:\n shutil.move(SRC_PATH + path, TRG_PATH + 'val/' + path)","metadata":{"execution":{"iopub.status.busy":"2021-07-03T12:30:38.02535Z","iopub.execute_input":"2021-07-03T12:30:38.026017Z","iopub.status.idle":"2021-07-03T12:30:38.042852Z","shell.execute_reply.started":"2021-07-03T12:30:38.025961Z","shell.execute_reply":"2021-07-03T12:30:38.041586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Annotation Files","metadata":{}},{"cell_type":"markdown","source":"We need categories, images, annotations(box). Let's make this file.","metadata":{}},{"cell_type":"code","source":"def anno(sets='train'):\n image_id = pd.DataFrame(os.listdir(TRG_PATH + sets))[0].str[:-4].values.tolist()\n annotation = {}\n annotation['type'] = 'instances'\n annotation['categories'] = []\n annotation['images'] = []\n annotation['annotations'] = []\n annotation['categories'].append({'supercategory': 'none', 'id': 1, 'name': 'typical'})\n annotation['categories'].append({'supercategory': 'none', 'id': 2, 'name': 'indeterminate'})\n annotation['categories'].append({'supercategory': 'none', 'id': 3, 'name': 'atypical'})\n for i, data in train_image[train_image.id.isin(image_id)].iterrows():\n dic = {}\n dic['file_name'] = data['id']+'.jpg'\n dic['height'] = 256\n dic['width'] = 256\n dic['id'] = data.name + 1\n annotation['images'].append(dic)\n cnt = 1\n\n for i, data in train_image.iterrows():\n if type(data['boxes']) == float: # nan\n continue\n # split box string\n boxes = json.loads(data['boxes'].replace('\\'', '\\\"'))\n \n # reverse x,y cell count\n ycell, xcell = data['xcell'], data['ycell']\n \n # category\n t, i, a = data['Typical'], data['Indeterminate'], data['Atypical']\n if t==1:\n category = 1\n elif i==1:\n category = 2\n elif a == 1:\n category = 3\n \n # add boxes\n for j in boxes:\n dic = {}\n dic['area'] = (j['width']*256)//xcell * (j['height']*256)//ycell\n dic['iscrowd'] = 0\n dic['image_id'] = data.name + 1\n dic['bbox'] = [(j['x']*256)//xcell, (j['y']*256)//ycell,\n (j['width']*256)//xcell, (j['height']*256)//ycell]\n dic['category_id'] = category\n dic['id'] = cnt\n dic['ignore'] = 0\n dic['segmentation'] = []\n cnt += 1\n annotation['annotations'].append(dic)\n \n # save annotation json files\n with open(f'{TRG_PATH}annotations/instances_{sets}.json', 'w') as f:\n json.dump(annotation, f)","metadata":{"execution":{"iopub.status.busy":"2021-07-03T12:32:09.437258Z","iopub.execute_input":"2021-07-03T12:32:09.437681Z","iopub.status.idle":"2021-07-03T12:32:09.465902Z","shell.execute_reply.started":"2021-07-03T12:32:09.437645Z","shell.execute_reply":"2021-07-03T12:32:09.464999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"os.makedirs(TRG_PATH + 'annotations', exist_ok=True)\nanno('train')\nanno('val')","metadata":{"execution":{"iopub.status.busy":"2021-07-03T12:32:13.814563Z","iopub.execute_input":"2021-07-03T12:32:13.815165Z","iopub.status.idle":"2021-07-03T12:32:13.97124Z","shell.execute_reply.started":"2021-07-03T12:32:13.815116Z","shell.execute_reply":"2021-07-03T12:32:13.969679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Finished!","metadata":{}},{"cell_type":"markdown","source":"The whole preprocessing process is complete. I will train on the next notebook with the files from here. Please wait for the next notebook. Thank you for read my notebook!","metadata":{}}]}
![](\"https://blog.roboflow.com/content/images/2020/06/yolov5-performance.png\")
\n ![](\"https://aihub-storage.s3.ap-northeast-2.amazonaws.com/file/%E1%84%89%E1%85%B3%E1%84%8F%E1%85%B3%E1%84%85%E1%85%B5%E1%86%AB%E1%84%89%E1%85%A3%E1%86%BA_2021-07-08_%E1%84%8B%E1%85%A9%E1%84%92%E1%85%AE_8.51.15.png\")
","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:52:14.949131Z","iopub.execute_input":"2021-07-08T11:52:14.949554Z","iopub.status.idle":"2021-07-08T11:52:14.958366Z","shell.execute_reply.started":"2021-07-08T11:52:14.949518Z","shell.execute_reply":"2021-07-08T11:52:14.956958Z"}}},{"cell_type":"markdown","source":"label column includes x1, y1, x2, y2.\n\n![](\"https://aihub-storage.s3.ap-northeast-2.amazonaws.com/file/%E1%84%89%E1%85%B3%E1%84%8F%E1%85%B3%E1%84%85%E1%85%B5%E1%86%AB%E1%84%89%E1%85%A3%E1%86%BA_2021-07-08_%E1%84%8B%E1%85%A9%E1%84%92%E1%85%AE_8.54.13.png\")
","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:54:34.551847Z","iopub.execute_input":"2021-07-08T11:54:34.552313Z","iopub.status.idle":"2021-07-08T11:54:34.558844Z","shell.execute_reply.started":"2021-07-08T11:54:34.552273Z","shell.execute_reply":"2021-07-08T11:54:34.557529Z"}}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"pydicom.read_file(train_image.iloc[0]['path']).pixel_array.shape","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:45:30.105404Z","iopub.execute_input":"2021-07-08T11:45:30.106244Z","iopub.status.idle":"2021-07-08T11:45:30.979772Z","shell.execute_reply.started":"2021-07-08T11:45:30.106185Z","shell.execute_reply":"2021-07-08T11:45:30.978448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can know the image size in the same way as above.","metadata":{"execution":{"iopub.status.busy":"2021-07-08T11:55:13.744579Z","iopub.execute_input":"2021-07-08T11:55:13.745248Z","iopub.status.idle":"2021-07-08T11:55:13.751132Z","shell.execute_reply.started":"2021-07-08T11:55:13.745181Z","shell.execute_reply":"2021-07-08T11:55:13.749939Z"}}},{"cell_type":"code","source":"xy = []\nfor i, data in train_image.iterrows():\n xy.append(pydicom.read_file(data['path']).pixel_array.shape)\ntrain_image[['xcell','ycell']] = xy\ntrain_image.to_csv('datasets/train_image.csv', index=None)\ntrain_image.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T12:26:12.74229Z","iopub.execute_input":"2021-07-03T12:26:12.742803Z","iopub.status.idle":"2021-07-03T12:26:29.762888Z","shell.execute_reply.started":"2021-07-03T12:26:12.742764Z","shell.execute_reply":"2021-07-03T12:26:29.760479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Split train to train&val","metadata":{}},{"cell_type":"markdown","source":"Make the validation set.","metadata":{}},{"cell_type":"code","source":"import random\nimport os\nimport shutil\nimport pandas as pd\nimport json\n\nrandom.seed(481)\nSRC_PATH = 'datasets/siim/train/'\nTRG_PATH = 'datasets/siim/'\ntrain_list = os.listdir(SRC_PATH)\nrandom.shuffle(train_list)\n\nimport shutil\nos.makedirs(TRG_PATH+'val', exist_ok=True)\nfor path in train_list[int(len(train_list)*0.8):]:\n shutil.move(SRC_PATH + path, TRG_PATH + 'val/' + path)","metadata":{"execution":{"iopub.status.busy":"2021-07-03T12:30:38.02535Z","iopub.execute_input":"2021-07-03T12:30:38.026017Z","iopub.status.idle":"2021-07-03T12:30:38.042852Z","shell.execute_reply.started":"2021-07-03T12:30:38.025961Z","shell.execute_reply":"2021-07-03T12:30:38.041586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Annotation Files","metadata":{}},{"cell_type":"markdown","source":"We need categories, images, annotations(box). Let's make this file.","metadata":{}},{"cell_type":"code","source":"def anno(sets='train'):\n image_id = pd.DataFrame(os.listdir(TRG_PATH + sets))[0].str[:-4].values.tolist()\n annotation = {}\n annotation['type'] = 'instances'\n annotation['categories'] = []\n annotation['images'] = []\n annotation['annotations'] = []\n annotation['categories'].append({'supercategory': 'none', 'id': 1, 'name': 'typical'})\n annotation['categories'].append({'supercategory': 'none', 'id': 2, 'name': 'indeterminate'})\n annotation['categories'].append({'supercategory': 'none', 'id': 3, 'name': 'atypical'})\n for i, data in train_image[train_image.id.isin(image_id)].iterrows():\n dic = {}\n dic['file_name'] = data['id']+'.jpg'\n dic['height'] = 256\n dic['width'] = 256\n dic['id'] = data.name + 1\n annotation['images'].append(dic)\n cnt = 1\n\n for i, data in train_image.iterrows():\n if type(data['boxes']) == float: # nan\n continue\n # split box string\n boxes = json.loads(data['boxes'].replace('\\'', '\\\"'))\n \n # reverse x,y cell count\n ycell, xcell = data['xcell'], data['ycell']\n \n # category\n t, i, a = data['Typical'], data['Indeterminate'], data['Atypical']\n if t==1:\n category = 1\n elif i==1:\n category = 2\n elif a == 1:\n category = 3\n \n # add boxes\n for j in boxes:\n dic = {}\n dic['area'] = (j['width']*256)//xcell * (j['height']*256)//ycell\n dic['iscrowd'] = 0\n dic['image_id'] = data.name + 1\n dic['bbox'] = [(j['x']*256)//xcell, (j['y']*256)//ycell,\n (j['width']*256)//xcell, (j['height']*256)//ycell]\n dic['category_id'] = category\n dic['id'] = cnt\n dic['ignore'] = 0\n dic['segmentation'] = []\n cnt += 1\n annotation['annotations'].append(dic)\n \n # save annotation json files\n with open(f'{TRG_PATH}annotations/instances_{sets}.json', 'w') as f:\n json.dump(annotation, f)","metadata":{"execution":{"iopub.status.busy":"2021-07-03T12:32:09.437258Z","iopub.execute_input":"2021-07-03T12:32:09.437681Z","iopub.status.idle":"2021-07-03T12:32:09.465902Z","shell.execute_reply.started":"2021-07-03T12:32:09.437645Z","shell.execute_reply":"2021-07-03T12:32:09.464999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"os.makedirs(TRG_PATH + 'annotations', exist_ok=True)\nanno('train')\nanno('val')","metadata":{"execution":{"iopub.status.busy":"2021-07-03T12:32:13.814563Z","iopub.execute_input":"2021-07-03T12:32:13.815165Z","iopub.status.idle":"2021-07-03T12:32:13.97124Z","shell.execute_reply.started":"2021-07-03T12:32:13.815116Z","shell.execute_reply":"2021-07-03T12:32:13.969679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Finished!","metadata":{}},{"cell_type":"markdown","source":"The whole preprocessing process is complete. I will train on the next notebook with the files from here. Please wait for the next notebook. Thank you for read my notebook!","metadata":{}}]}