{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🐠 Reef - DETR - Detection Transformer - Infer\n\n## DETR Baseline model for the [Great Barrier Reef Competition](https://www.kaggle.com/c/tensorflow-great-barrier-reef) with `LB=0.189`\n\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/31703/logos/header.png)\n\n## An adaption of [End to End Object Detection with Transformers:DETR](https://www.kaggle.com/tanulsingh077/end-to-end-object-detection-with-transformers-detr) to the [Great Barrier Reef Competition](https://www.kaggle.com/c/tensorflow-great-barrier-reef)\n\nI made various adaptations to it in order to work, based on the following code and documentation:\n* This awesome fork [End to End Object Detection with Transformers:DETR](https://www.kaggle.com/prokaj/end-to-end-object-detection-with-transformers-detr) by [prvi](https://www.kaggle.com/prokaj), correctly formatting the input, which is not coco and not pascal_voc, but something else.\n* Albumentation code for bbox normalize and denormalize functions: [here](https://github.com/albumentations-team/albumentations/blob/master/albumentations/augmentations/bbox_utils.py#L88)\n* [DETR's hands on Colab Notebook](https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/detr_attention.ipynb): Shows how to load a model from hub, generate predictions, then visualize the attention of the model (similar to the figures of the paper)\n* [Standalone Colab Notebook](https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/detr_demo.ipynb): In this notebook, we demonstrate how to implement a simplified version of DETR from the grounds up in 50 lines of Python, then visualize the predictions. It is a good starting point if you want to gain better understanding the architecture and poke around before diving in the codebase.\n* [Panoptic Colab Notebook](https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/DETR_panoptic.ipynb): Demonstrates how to use DETR for panoptic segmentation and plot the predictions.\n* [Hugging Face DETR Documentation](https://huggingface.co/docs/transformers/model_doc/detr)\n\nThe main changes to the original notebook I forked are:\n* Data format changed from `[x_min, y_min, w, h]` to `[x_center, y_center, w, h]`\n* Resnet-like normalization instead of `[0...1]`\n\n\n## This is the inference notebook. You can find the training one here: [🐠 Reef - DETR - Detection Transformer - Train](https://www.kaggle.com/julian3833/reef-detr-detection-transformer-train).\n\n\n\n# Please, _DO_ upvote if you find this useful!!\n\n\n&nbsp;\n&nbsp;\n&nbsp;\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"# About DETR (Detection Transformer)\n\nAttention is all you need,paper for Transformers,changed the state of NLP and has achieved great hieghts. Though mainly developed for NLP , the latest research around it focuses on how to leverage it across different verticals of deep learning. Transformer acrhitecture is very very powerful, and is something which is very close to my part,this is the reason I am motivated to explore anything that uses transformers , be it google's recently released Tabnet or OpenAI's ImageGPT .\n\nDetection Transformer leverages the transformer network(both encoder and the decoder) for Detecting Objects in Images . Facebook's researchers argue that for object detection one part of the image should be in contact with the other part of the image for greater result especially with ocluded objects and partially visible objects, and what's better than to use transformer for it.\n\n**The main motive behind DETR is effectively removing the need for many hand-designed components like a non-maximum suppression procedure or anchor generation that explicitly encode prior knowledge about the task and makes the process complex and computationally expensive**\n\nThe main ingredients of the new framework, called DEtection TRansformer or DETR, <font color='green'>are a set-based global loss that forces unique predictions via bipartite matching, and a transformer encoder-decoder architecture.</font>\n\n![](https://cdn.analyticsvidhya.com/wp-content/uploads/2020/05/Screenshot-from-2020-05-27-17-48-38.png)\n\n---\n","metadata":{}},{"cell_type":"code","source":"from IPython.display import IFrame, YouTubeVideo\nYouTubeVideo('T35ba_VXkMY',width=600, height=400)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-12T03:44:46.921576Z","iopub.execute_input":"2022-01-12T03:44:46.922381Z","iopub.status.idle":"2022-01-12T03:45:06.980618Z","shell.execute_reply.started":"2022-01-12T03:44:46.922282Z","shell.execute_reply":"2022-01-12T03:45:06.97969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# References:\n* The video [above](https://www.youtube.com/watch?v=T35ba_VXkMY) in youtube\n* [Other Video](https://www.youtube.com/watch?v=LfUsGv-ESbc)\n* The original notebook: [End to End Object Detection with Transformers:DETR](https://www.kaggle.com/tanulsingh077/end-to-end-object-detection-with-transformers-detr)\n* [Paper](https://scontent.flko3-1.fna.fbcdn.net/v/t39.8562-6/101177000_245125840263462_1160672288488554496_n.pdf?_nc_cat=104&_nc_sid=ae5e01&_nc_ohc=KwU3i7_izOgAX9bxMVv&_nc_ht=scontent.flko3-1.fna&oh=64dad6ce7a7b4807bb3941690beaee69&oe=5F1E8347) is the link to the paper\n* [Github repo](https://github.com/facebookresearch/detr)\n* [Blogpost](https://ai.facebook.com/blog/end-to-end-object-detection-with-transformers/)\n\n\nOk, enough chit chat, show me the code!!","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np \nimport pandas as pd \nfrom datetime import datetime\nimport time\nimport random\nfrom tqdm import tqdm\n\n\n#Torch\nimport torch\nimport torch.nn as nn\n\n#CV\nimport cv2\n\n#Albumenatations\nimport albumentations as A\nimport matplotlib.pyplot as plt\nfrom albumentations.pytorch.transforms import ToTensorV2","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-01-12T03:45:06.982385Z","iopub.execute_input":"2022-01-12T03:45:06.982629Z","iopub.status.idle":"2022-01-12T03:45:10.667024Z","shell.execute_reply.started":"2022-01-12T03:45:06.982597Z","shell.execute_reply":"2022-01-12T03:45:10.66617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Constants","metadata":{}},{"cell_type":"code","source":"num_classes = 2\nnum_queries = 18\nnull_class_coef = 0.1\nBATCH_SIZE = 8\n\nWIDTH = 1280\nHEIGHT = 720\n\nDETECTION_THRESHOLD = 0.5\n\nDEVICE = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:45:10.668425Z","iopub.execute_input":"2022-01-12T03:45:10.668742Z","iopub.status.idle":"2022-01-12T03:45:10.72307Z","shell.execute_reply.started":"2022-01-12T03:45:10.668671Z","shell.execute_reply":"2022-01-12T03:45:10.722349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Pretrained weights\n\nThese come from the training notebook here:  [🐠 Reef - DETR - Detection Transformer - Train](https://www.kaggle.com/julian3833/reef-detr-detection-transformer-train)\n\nTurned into a dataset here: [DETR - Weights and Supplies](https://www.kaggle.com/julian3833/detr-weights-and-supplies)\n ","metadata":{}},{"cell_type":"code","source":"WEIGHTS_FILE = \"../input/detr-weights-and-supplies/pytorch_model.bin\"","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:45:10.726459Z","iopub.execute_input":"2022-01-12T03:45:10.727035Z","iopub.status.idle":"2022-01-12T03:45:10.735525Z","shell.execute_reply.started":"2022-01-12T03:45:10.726993Z","shell.execute_reply":"2022-01-12T03:45:10.73462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Setup pre-trained backbone and torchhub dependecies \n\nWe are overriding the resources torch hub would download from the Internet in `/root/.cache/torch/hub/` with what we know it downloaded (from the training notebook).\nThis way we can submit with Internet disabled.\n","metadata":{}},{"cell_type":"code","source":"!mkdir -p /root/.cache/torch/hub/\n!cp -R ../input/detr-weights-and-supplies/torch_hub/* /root/.cache/torch/hub\n!ls -l /root/.cache/torch/hub/","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:45:10.737948Z","iopub.execute_input":"2022-01-12T03:45:10.738508Z","iopub.status.idle":"2022-01-12T03:45:17.694336Z","shell.execute_reply.started":"2022-01-12T03:45:10.738468Z","shell.execute_reply":"2022-01-12T03:45:17.693475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Get the model\nWe create the model and set the fine-tuned weights with `torch.load`","metadata":{}},{"cell_type":"code","source":"class DETRModel(nn.Module):\n    def __init__(self,num_classes, num_queries):\n        super(DETRModel,self).__init__()\n        self.num_classes = num_classes\n        self.num_queries = num_queries\n        \n        self.model = torch.hub.load('facebookresearch/detr', 'detr_resnet50', pretrained=True)\n        self.in_features = self.model.class_embed.in_features\n        \n        self.model.class_embed = nn.Linear(in_features=self.in_features,out_features=self.num_classes)\n        self.model.num_queries = self.num_queries\n        \n    def forward(self,images):\n        return self.model(images)\n\n\ndef get_model():\n    model = DETRModel(num_classes=num_classes,num_queries=num_queries)\n    model.load_state_dict(torch.load(WEIGHTS_FILE, map_location=DEVICE))\n    model.eval()\n    model = model.to(DEVICE)\n    return model\n\nmodel = get_model()","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:45:17.696366Z","iopub.execute_input":"2022-01-12T03:45:17.696669Z","iopub.status.idle":"2022-01-12T03:45:24.019063Z","shell.execute_reply.started":"2022-01-12T03:45:17.696629Z","shell.execute_reply":"2022-01-12T03:45:24.018279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predict functions","metadata":{}},{"cell_type":"code","source":"import torchvision.transforms as T","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:45:24.020341Z","iopub.execute_input":"2022-01-12T03:45:24.020584Z","iopub.status.idle":"2022-01-12T03:45:24.025274Z","shell.execute_reply.started":"2022-01-12T03:45:24.020553Z","shell.execute_reply":"2022-01-12T03:45:24.024394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def transform():\n    return A.Compose([\n        A.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n        ToTensorV2(p=1.0),\n        \n    ])\n\n\ndef format_prediction_string(boxes, scores):\n    # Format as specified in the evaluation page\n    pred_strings = []\n    for j in zip(scores, boxes):\n        pred_strings.append(\"{0:.10f} {1} {2} {3} {4}\".format(j[0], j[1][0], j[1][1], j[1][2], j[1][3]))\n    return \" \".join(pred_strings)\n\n\ndef predict(model, pixel_array):\n    # Predictions for a single image\n    \n    # Apply all the transformations that are required\n    pixel_array = pixel_array.astype(np.float32)# / 255.\n    tensor_img = transform()(image=pixel_array)['image'].unsqueeze(0)\n    \n    # TODO!!: un-scale boxes\n    \n    # Get predictions\n    with torch.no_grad():\n        outputs = model(tensor_img.to(DEVICE))\n    \n    #import pdb; pdb.set_trace()\n    # Move predictions to cpu and numpy\n    boxes = outputs['pred_boxes'][0].data.cpu().numpy()\n    boxes = np.array([np.array(box).astype(np.int32) for box in A.augmentations.bbox_utils.denormalize_bboxes(boxes, HEIGHT, WIDTH)])\n    \n    scores  = outputs['pred_logits'][0].softmax(1).detach().cpu().numpy()[:,0]\n    \n    #import pdb; pdb.set_trace()\n    # Filter predictions with low score\n    boxes = boxes[scores >= DETECTION_THRESHOLD].astype(np.int32)\n    \n    #[x_min, y_min, width, height]\n    boxes[:, 0] = boxes[:, 0] - (boxes[:, 2] / 2) # x_center --> x_min\n    boxes[:, 1] = boxes[:, 1] - (boxes[:, 3] / 2) # y_center --> y_min\n    \n    scores = scores[scores >= DETECTION_THRESHOLD]\n    \n    scored_boxes = list(zip(boxes, scores))\n    sorted_boxes = list(sorted(scored_boxes, key=lambda y: -y[1]))\n    top_n_boxes = sorted_boxes[:18]\n    boxes = [box for box, score in top_n_boxes]\n    scores = [score for box, score in top_n_boxes]\n  \n    # Format results as requested in the Evaluation tab\n    return format_prediction_string(boxes, scores)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:45:24.026479Z","iopub.execute_input":"2022-01-12T03:45:24.026897Z","iopub.status.idle":"2022-01-12T03:45:24.099382Z","shell.execute_reply.started":"2022-01-12T03:45:24.026863Z","shell.execute_reply":"2022-01-12T03:45:24.098471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submit\n\n(See: [Great Barrier Reef API Tutorial](https://www.kaggle.com/sohier/great-barrier-reef-api-tutorial))","metadata":{}},{"cell_type":"code","source":"import greatbarrierreef\nenv = greatbarrierreef.make_env()\niter_test = env.iter_test() \n#pixel_arrays = []\nfor (pixel_array, df_pred) in iter_test:\n    # Predictions\n    #pixel_arrays.append(pixel_array)\n    df_pred['annotations'] = predict(model, pixel_array)\n    #display(df_pred)\n    \n    env.predict(df_pred)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:45:24.101877Z","iopub.execute_input":"2022-01-12T03:45:24.102351Z","iopub.status.idle":"2022-01-12T03:45:30.187913Z","shell.execute_reply.started":"2022-01-12T03:45:24.102312Z","shell.execute_reply":"2022-01-12T03:45:30.187004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Please, _DO_ upvote if you find it useful or interesting!!","metadata":{}}]}