{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# EfficientDet\n\nARXIV [https://arxiv.org/pdf/1911.09070.pdf](https://arxiv.org/pdf/1911.09070.pdf)  \nGithub [https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch](https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch)  \n\nEfficientDet employs EfficientNet as the backbone network, BiFPN as the feature network, and shared class/box prediction network. Both BiFPN layers and class/box net layers are repeated multiple times based on different resource constraints.\n\n[Object Detection SOTA model](https://paperswithcode.com/sota/object-detection-on-coco)  \nThis page shows object detection models' score on COCO test-dev. I focus AP50 score, because this competitions' metric is AP50.    \n1. DyHead (Based Swin-L) : 78.5\n2. DetectoRS (Based ResNeXt) : 74.2\n3. YOLOv4-P7 (Based Scaled-YOLO) : 73.3\n4. EfficientDet-D7 (Based EfficientNet) : 72.4\n5. YOLOv4-608 (Based YOLO) : 65.7\n\n<figure>\n<img src=\"https://blog.roboflow.com/content/images/2020/06/yolov5-performance.png\" style=\"width:700px\">\n    <figcaption>EfficientDet is better model than YOLOv5 on AP.</figcaption>\n</figure>\n\n\nEveryone used model based YOLOv4 or YOLOv5, but this model isn't SOTA model. I'll try EfficientDet first and then Scaled-YOLOv4, DyHead.  \n  \n## Three notebooks summarize how to use this model.\n#### Preprocessing\nhttps://www.kaggle.com/adldotori/efficientdet-preprocessing-better-than-yolov5/\n#### Training\nhttps://www.kaggle.com/adldotori/efficientdet-training-better-than-yolov5/\n#### Inference\nhttps://www.kaggle.com/adldotori/efficientdet-inference-better-than-yolov5/\n  \n  \n  \nThis notebook is second notebook which includes how to training EfficientDet.  \nLet's start!  \n\nThis picture shows the rough structure of efficientdet.\n![image](https://aihub-storage.s3.ap-northeast-2.amazonaws.com/file/efficientdet.png)","metadata":{}},{"cell_type":"markdown","source":"# Environment","metadata":{}},{"cell_type":"code","source":"!cp -r ../input/efficientdet-preprocessing-better-than-yolov5/Yet-Another-EfficientDet-Pytorch/ ./","metadata":{"execution":{"iopub.status.busy":"2021-07-18T13:16:33.101382Z","iopub.execute_input":"2021-07-18T13:16:33.101771Z","iopub.status.idle":"2021-07-18T13:17:27.268133Z","shell.execute_reply.started":"2021-07-18T13:16:33.101681Z","shell.execute_reply":"2021-07-18T13:17:27.267082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nos.chdir('Yet-Another-EfficientDet-Pytorch/')","metadata":{"execution":{"iopub.status.busy":"2021-07-18T13:17:27.269754Z","iopub.execute_input":"2021-07-18T13:17:27.270081Z","iopub.status.idle":"2021-07-18T13:17:27.276217Z","shell.execute_reply.started":"2021-07-18T13:17:27.270043Z","shell.execute_reply":"2021-07-18T13:17:27.275125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install pycocotools webcolors","metadata":{"execution":{"iopub.status.busy":"2021-07-18T13:17:27.278565Z","iopub.execute_input":"2021-07-18T13:17:27.278921Z","iopub.status.idle":"2021-07-18T13:17:41.881344Z","shell.execute_reply.started":"2021-07-18T13:17:27.278886Z","shell.execute_reply":"2021-07-18T13:17:41.880255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{}},{"cell_type":"markdown","source":"## Train a custom dataset with pretrained weights (Highly Recommended)","metadata":{}},{"cell_type":"code","source":"!python train.py -c 0 -p siim --head_only True --lr 5e-3 --batch_size 32 --load_weights weights/efficientdet-d0.pth --num_epochs 10 --save_interval 100","metadata":{"execution":{"iopub.status.busy":"2021-07-18T13:19:29.667308Z","iopub.execute_input":"2021-07-18T13:19:29.667674Z","iopub.status.idle":"2021-07-18T13:21:30.651445Z","shell.execute_reply.started":"2021-07-18T13:19:29.667639Z","shell.execute_reply":"2021-07-18T13:21:30.650483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Resume training","metadata":{}},{"cell_type":"code","source":"! python train.py -c 0 -p siim --head_only False --lr 1e-3 --batch_size 16 --load_weights last  --num_epochs 40 --save_interval 100","metadata":{"execution":{"iopub.status.busy":"2021-07-18T13:21:30.66452Z","iopub.execute_input":"2021-07-18T13:21:30.664762Z","iopub.status.idle":"2021-07-18T13:24:07.851598Z","shell.execute_reply.started":"2021-07-18T13:21:30.664737Z","shell.execute_reply":"2021-07-18T13:24:07.850529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluation","metadata":{}},{"cell_type":"markdown","source":"## mAP","metadata":{}},{"cell_type":"code","source":"weight_file = !ls logs/siim/\n! python coco_eval.py -c 0 -p siim -w \"logs/siim/{weight_file[-2]}\"","metadata":{"execution":{"iopub.status.busy":"2021-07-18T13:24:07.875725Z","iopub.execute_input":"2021-07-18T13:24:07.876111Z","iopub.status.idle":"2021-07-18T13:25:52.675544Z","shell.execute_reply.started":"2021-07-18T13:24:07.876069Z","shell.execute_reply":"2021-07-18T13:25:52.674592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We focus Average Precision (AP) @[IoU=0.50] score.","metadata":{}},{"cell_type":"markdown","source":"## test image result","metadata":{}},{"cell_type":"code","source":"import torch\nimport random\nfrom torch.backends import cudnn\n\nfrom backbone import EfficientDetBackbone\nimport cv2\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nfrom efficientdet.utils import BBoxTransform, ClipBoxes\nfrom utils.utils import preprocess, invert_affine, postprocess\nfrom colorama import Fore","metadata":{"execution":{"iopub.status.busy":"2021-07-18T13:25:52.678819Z","iopub.execute_input":"2021-07-18T13:25:52.679087Z","iopub.status.idle":"2021-07-18T13:25:53.190815Z","shell.execute_reply.started":"2021-07-18T13:25:52.679057Z","shell.execute_reply":"2021-07-18T13:25:53.189916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"compound_coef = 0\nforce_input_size = None  # set None to use default size\nimg_path = 'datasets/siim/val/'\ntest_cnt = 30\n\nthreshold = 0.1\niou_threshold = 0.1\n\nuse_cuda = True\nuse_float16 = False\ncudnn.fastest = True\ncudnn.benchmark = True\n\nclass_names = ['Typical', 'Indeterminate', 'Atypical']\nlabel2color = {\n    '[0, 0, 0]': [0, 0, 0], # Typical Appearance\n    '[1, 0, 0]': [66,9,255], # Typical Appearance\n    '[0, 1, 0]': [255,186,8], # Indeterminate Appearance\n    '[0, 0, 1]': [247,37,69], # Atypical Appearance\n}\nname2color = {\n    'Negative' : [0, 0, 0], # Typical Appearance\n    'Typical': [66,9,255], # Typical Appearance\n    'Indeterminate': [255,186,8], # Indeterminate Appearance\n    'Atypical': [247,37,69], # Atypical Appearance\n}\n\nobj_list = ['Typical','Indeterminate', 'Atypical']\n\n# tf bilinear interpolation is different from any other's, just make do\ninput_sizes = [512, 640, 768, 896, 1024, 1280, 1280, 1536]\ninput_size = input_sizes[compound_coef] if force_input_size is None else force_input_size\n\nimagelist = os.listdir(img_path)\nrandom.shuffle(imagelist)\nimages = imagelist[:test_cnt]","metadata":{"execution":{"iopub.status.busy":"2021-07-18T13:52:23.997678Z","iopub.execute_input":"2021-07-18T13:52:23.99802Z","iopub.status.idle":"2021-07-18T13:52:24.009543Z","shell.execute_reply.started":"2021-07-18T13:52:23.997986Z","shell.execute_reply":"2021-07-18T13:52:24.008628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![](https://aihub-storage.s3.ap-northeast-2.amazonaws.com/file/%E1%84%89%E1%85%B3%E1%84%8F%E1%85%B3%E1%84%85%E1%85%B5%E1%86%AB%E1%84%89%E1%85%A3%E1%86%BA_2021-07-18_%E1%84%8B%E1%85%A9%E1%84%92%E1%85%AE_10.53.32.png)","metadata":{}},{"cell_type":"markdown","source":"## load etc files","metadata":{}},{"cell_type":"code","source":"import pandas as pd\ntrain_image = pd.read_csv('datasets/train_image.csv')\nimport json\nwith open('datasets/siim/annotations/instances_val.json', 'r') as f:\n    files = json.load(f)","metadata":{"execution":{"iopub.status.busy":"2021-07-18T13:52:26.76652Z","iopub.execute_input":"2021-07-18T13:52:26.766849Z","iopub.status.idle":"2021-07-18T13:52:27.045351Z","shell.execute_reply.started":"2021-07-18T13:52:26.766817Z","shell.execute_reply":"2021-07-18T13:52:27.044494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inference","metadata":{}},{"cell_type":"code","source":"imgs = []\nfor image in images:\n    ori_imgs, framed_imgs, framed_metas = preprocess(img_path + image, max_size=input_size)\n\n    if use_cuda:\n        x = torch.stack([torch.from_numpy(fi).cuda() for fi in framed_imgs], 0)\n    else:\n        x = torch.stack([torch.from_numpy(fi) for fi in framed_imgs], 0)\n\n    x = x.to(torch.float32 if not use_float16 else torch.float16).permute(0, 3, 1, 2)\n\n    model = EfficientDetBackbone(compound_coef=compound_coef, num_classes=len(obj_list),\n\n                                 # replace this part with your project's anchor config\n                                 ratios=[(1.0, 1.0), (1.3, 0.8), (1.9, 0.5)],\n                                 scales=[2 ** 0, 2 ** (1.0 / 3.0), 2 ** (2.0 / 3.0)])\n\n    model.load_state_dict(torch.load('logs/siim/'+weight_file[-2]))\n    model.requires_grad_(False)\n    model.eval()\n\n    if use_cuda:\n        model = model.cuda()\n    if use_float16:\n        model = model.half()\n\n    with torch.no_grad():\n        features, regression, classification, anchors = model(x)\n\n        regressBoxes = BBoxTransform()\n        clipBoxes = ClipBoxes()\n\n        out = postprocess(x,\n                          anchors, regression, classification,\n                          regressBoxes, clipBoxes,\n                          threshold, iou_threshold)\n\n    out = invert_affine(framed_metas, out)\n\n    for i in range(len(ori_imgs)):\n        if len(out[i]['rois']) == 0:\n            continue\n        for j in range(len(out[i]['rois'])):\n            (x1, y1, x2, y2) = out[i]['rois'][j].astype(np.int)\n            obj = obj_list[out[i]['class_ids'][j]]\n            cv2.rectangle(ori_imgs[i], (x1, y1), (x2, y2), name2color[obj], 2)\n            score = float(out[i]['scores'][j])\n\n            cv2.putText(ori_imgs[i], '{:.3f}'.format(score),\n                        (x1, y1 + 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5,\n                        (255,255,255), 1)\n            \n    tmp = train_image[train_image.id==image[:-4]]\n    label = tmp['label']\n    img = cv2.imread(img_path + image)\n    claz = tmp[class_names].values\n    color = label2color[str(claz.tolist()[0])]\n\n    for i in files['images']:\n        if i['file_name'] == image:\n            image_id = i['id']\n    bboxes = []\n    for i in files['annotations']:\n        if i['image_id'] == image_id:\n            bboxes.append(i['bbox'])\n            \n    for box in bboxes:\n        img = cv2.rectangle(\n            img,\n            (int(box[0]), int(box[1])),\n            (int(box[2]+box[0]), int(box[3]+box[1])),\n            color, 2\n        )\n    imgs += [ori_imgs[0], img]","metadata":{"execution":{"iopub.status.busy":"2021-07-18T13:52:39.541716Z","iopub.execute_input":"2021-07-18T13:52:39.542036Z","iopub.status.idle":"2021-07-18T13:52:50.358838Z","shell.execute_reply.started":"2021-07-18T13:52:39.542005Z","shell.execute_reply":"2021-07-18T13:52:50.35796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![](https://aihub-storage.s3.ap-northeast-2.amazonaws.com/file/%E1%84%89%E1%85%B3%E1%84%8F%E1%85%B3%E1%84%85%E1%85%B5%E1%86%AB%E1%84%89%E1%85%A3%E1%86%BA_2021-07-18_%E1%84%8B%E1%85%A9%E1%84%92%E1%85%AE_10.53.32.png)","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(2*5, len(imgs)//2*5))\nfor i, img in enumerate(imgs[:30]):\n    img = cv2.resize(img, (300,300))\n    ax = fig.add_subplot(len(imgs)//2, 2, i+1)\n    ax.get_xaxis().set_visible(False)\n    ax.get_yaxis().set_visible(False)\n    if i == 0:\n        ax.set_title('Prediction Image',fontsize=20)\n    elif i == 1:\n        ax.set_title('Ground Truth Image',fontsize=20)\n    plt.imshow(img)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-18T13:52:50.360387Z","iopub.execute_input":"2021-07-18T13:52:50.360743Z","iopub.status.idle":"2021-07-18T13:52:54.550395Z","shell.execute_reply.started":"2021-07-18T13:52:50.360705Z","shell.execute_reply":"2021-07-18T13:52:54.549346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Finished!","metadata":{"execution":{"iopub.status.busy":"2021-07-18T14:08:28.210724Z","iopub.execute_input":"2021-07-18T14:08:28.21109Z","iopub.status.idle":"2021-07-18T14:08:28.217119Z","shell.execute_reply.started":"2021-07-18T14:08:28.211052Z","shell.execute_reply":"2021-07-18T14:08:28.216238Z"}}},{"cell_type":"markdown","source":"The white letter means confidence. This result is pretty cool. The model is used to draw boxes by image and classify studies through labels on those boxes. This content will be shared on the next notebook. Please look forward to it.","metadata":{}}]}