{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🐠 Reef - DETR - Detection Transformer - Train\n\n## DETR Baseline model for the [Great Barrier Reef Competition](https://www.kaggle.com/c/tensorflow-great-barrier-reef)\n\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/31703/logos/header.png)\n\n## An adaption of [End to End Object Detection with Transformers:DETR](https://www.kaggle.com/tanulsingh077/end-to-end-object-detection-with-transformers-detr) to the [Great Barrier Reef Competition](https://www.kaggle.com/c/tensorflow-great-barrier-reef)\n\nI made various adaptations to it in order to work, based on the following code and documentation:\n* This awesome fork [End to End Object Detection with Transformers:DETR](https://www.kaggle.com/prokaj/end-to-end-object-detection-with-transformers-detr) by [prvi](https://www.kaggle.com/prokaj), correctly formatting the input, which is not coco and not pascal_voc, but something else.\n* Albumentation code for bbox normalize and denormalize functions: [here](https://github.com/albumentations-team/albumentations/blob/master/albumentations/augmentations/bbox_utils.py#L88)\n* [DETR's hands on Colab Notebook](https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/detr_attention.ipynb): Shows how to load a model from hub, generate predictions, then visualize the attention of the model (similar to the figures of the paper)\n* [Standalone Colab Notebook](https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/detr_demo.ipynb): In this notebook, we demonstrate how to implement a simplified version of DETR from the grounds up in 50 lines of Python, then visualize the predictions. It is a good starting point if you want to gain better understanding the architecture and poke around before diving in the codebase.\n* [Panoptic Colab Notebook](https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/DETR_panoptic.ipynb): Demonstrates how to use DETR for panoptic segmentation and plot the predictions.\n* [Hugging Face DETR Documentation](https://huggingface.co/docs/transformers/model_doc/detr)\n\nThe main changes to the original notebook I forked are:\n* Data format changed from `[x_min, y_min, w, h]` to `[x_center, y_center, w, h]`\n* Resnet-like normalization instead of `[0...1]`\n\n\n## This is the training notebook. You can find the inference one here: [🐠 Reef - DETR - Detection Transformer - Infer](https://www.kaggle.com/julian3833/reef-detr-detection-transformer-infer).\n\n\n\n# Please, _DO_ upvote if you find this useful!!\n\n\n&nbsp;\n&nbsp;\n&nbsp;\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"# About DETR (Detection Transformer)\n\nAttention is all you need,paper for Transformers,changed the state of NLP and has achieved great hieghts. Though mainly developed for NLP , the latest research around it focuses on how to leverage it across different verticals of deep learning. Transformer acrhitecture is very very powerful, and is something which is very close to my part,this is the reason I am motivated to explore anything that uses transformers , be it google's recently released Tabnet or OpenAI's ImageGPT .\n\nDetection Transformer leverages the transformer network(both encoder and the decoder) for Detecting Objects in Images . Facebook's researchers argue that for object detection one part of the image should be in contact with the other part of the image for greater result especially with ocluded objects and partially visible objects, and what's better than to use transformer for it.\n\n**The main motive behind DETR is effectively removing the need for many hand-designed components like a non-maximum suppression procedure or anchor generation that explicitly encode prior knowledge about the task and makes the process complex and computationally expensive**\n\nThe main ingredients of the new framework, called DEtection TRansformer or DETR, <font color='green'>are a set-based global loss that forces unique predictions via bipartite matching, and a transformer encoder-decoder architecture.</font>\n\n![](https://cdn.analyticsvidhya.com/wp-content/uploads/2020/05/Screenshot-from-2020-05-27-17-48-38.png)\n\n---\n","metadata":{}},{"cell_type":"code","source":"from IPython.display import IFrame, YouTubeVideo\nYouTubeVideo('T35ba_VXkMY',width=600, height=400)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-12T03:03:13.093441Z","iopub.execute_input":"2022-01-12T03:03:13.093738Z","iopub.status.idle":"2022-01-12T03:03:13.212022Z","shell.execute_reply.started":"2022-01-12T03:03:13.093703Z","shell.execute_reply":"2022-01-12T03:03:13.211257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# References:\n* The video [above](https://www.youtube.com/watch?v=T35ba_VXkMY) in youtube\n* [Other Video](https://www.youtube.com/watch?v=LfUsGv-ESbc)\n* The original notebook: [End to End Object Detection with Transformers:DETR](https://www.kaggle.com/tanulsingh077/end-to-end-object-detection-with-transformers-detr)\n* [Paper](https://scontent.flko3-1.fna.fbcdn.net/v/t39.8562-6/101177000_245125840263462_1160672288488554496_n.pdf?_nc_cat=104&_nc_sid=ae5e01&_nc_ohc=KwU3i7_izOgAX9bxMVv&_nc_ht=scontent.flko3-1.fna&oh=64dad6ce7a7b4807bb3941690beaee69&oe=5F1E8347) is the link to the paper\n* [Github repo](https://github.com/facebookresearch/detr)\n* [Blogpost](https://ai.facebook.com/blog/end-to-end-object-detection-with-transformers/)","metadata":{}},{"cell_type":"markdown","source":"Ok, enough chit chat, show me the code!!\n\n&nbsp;\n&nbsp;\n&nbsp;\n\n# Clone github repo of detr\n\nWe will not use it for the inference notebook, but we need it for training. The model is a pytorch base model that relies on the torch hub for the weights.","metadata":{}},{"cell_type":"code","source":"!git clone https://github.com/facebookresearch/detr.git   ","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-12T03:03:13.881401Z","iopub.execute_input":"2022-01-12T03:03:13.881704Z","iopub.status.idle":"2022-01-12T03:03:16.495048Z","shell.execute_reply.started":"2022-01-12T03:03:13.881644Z","shell.execute_reply":"2022-01-12T03:03:16.494166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Now if you have seen the video , you know that DETR uses a special loss called Bipartite Matching loss where it assigns one ground truth bbox to a predicted box using a matcher , thus when fine tuning we need the matcher (hungarian matcher as used in paper) and also the fucntion SetCriterion which gives Bipartite matching loss for backpropogation. This is the reason for forking the github repo","metadata":{}},{"cell_type":"markdown","source":"* So I did not know that we can add the path to environment variables using sys , hence I was changine directories , but now I have made changes so I do not have to change directories and import detr easily. A big Thanks to @prvi for his help","metadata":{}},{"cell_type":"code","source":"import os\nimport time\nimport random\nimport numpy as np \nimport pandas as pd \n\nfrom tqdm import tqdm\n\n\n#Torch\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset,DataLoader\n\n\n#CV\nimport cv2\n\n################# DETR FUCNTIONS FOR LOSS######################## \nimport sys\nsys.path.append('./detr/')\n\nfrom detr.models.matcher import HungarianMatcher\nfrom detr.models.detr import SetCriterion\n#################################################################\n\n#Albumenatations\nimport albumentations as A\nimport matplotlib.pyplot as plt\nfrom albumentations.pytorch.transforms import ToTensorV2\n\n\n# Fix randomness\n\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True\n    \nseed_everything(seed=42)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-01-12T03:03:16.500423Z","iopub.execute_input":"2022-01-12T03:03:16.501026Z","iopub.status.idle":"2022-01-12T03:03:20.671066Z","shell.execute_reply.started":"2022-01-12T03:03:16.500982Z","shell.execute_reply":"2022-01-12T03:03:20.670254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configuration\n\nBasic configuration for this model","metadata":{}},{"cell_type":"code","source":"NUM_CLASSES = 2\nNUM_QUERIES = 18\nNULL_CLASS_COEF = 0.1\nBATCH_SIZE = 8\nLR = 1e-5\nEPOCHS = 20\n\nWIDTH = 1280\nHEIGHT = 720\n\nBASE_DIR = \"../input/tensorflow-great-barrier-reef/train_images/\"","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:20.672357Z","iopub.execute_input":"2022-01-12T03:03:20.672619Z","iopub.status.idle":"2022-01-12T03:03:20.680216Z","shell.execute_reply.started":"2022-01-12T03:03:20.672583Z","shell.execute_reply":"2022-01-12T03:03:20.679369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train-validation split\n\nWe are splitting using subsequences. I have tried other strategies and this is the one that works the best for now. The dataset has just 3 videos, each of them split into sequences, but in total there are only 20 sequences. A **subsequences**, as we defined them, are parts of a sequences where objects are continually present or are continually not present. \n\n&nbsp;\n\nLet's see an **example**. Consider the sequence `A` with the following frames:\n* `1-20` - No annotations present\n* `21-30` - Annotations present\n* `31-60` - No annotations\n* `61-80` - Annotations present\n\nIn this case, we say that the sequence `A` has `4` subsequences (`1-20`, `21-30`, `31-60`, `61-80`).\n\n\nSee: [🐠 Reef - CV strategy: subsequences!](https://www.kaggle.com/julian3833/reef-cv-strategy-subsequences) for more details about this","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"../input/reef-cv-strategy-subsequences-dataframes/train-validation-split/train-0.1.csv\")\n\n# Turn annotations from strings into lists of dictionaries\ndf['annotations'] = df['annotations'].apply(eval)\n\n# Create the image path for the row\ndf['image_path'] = \"video_\" + df['video_id'].astype(str) + \"/\" + df['video_frame'].astype(str) + \".jpg\"\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:20.682919Z","iopub.execute_input":"2022-01-12T03:03:20.683654Z","iopub.status.idle":"2022-01-12T03:03:21.095236Z","shell.execute_reply.started":"2022-01-12T03:03:20.683611Z","shell.execute_reply":"2022-01-12T03:03:21.09451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def clean_annotations(annotations):\n    new_annotations = []\n    for a in annotations:\n        x_max = a['x'] + a['width']\n        y_max = a['y'] + a['height']\n        if x_max > WIDTH or y_max > HEIGHT:\n            print(\"Found broken annotation:\", a, x_max, y_max)\n        else:\n            new_annotations.append(a)\n    return new_annotations","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:21.096554Z","iopub.execute_input":"2022-01-12T03:03:21.097092Z","iopub.status.idle":"2022-01-12T03:03:21.103791Z","shell.execute_reply.started":"2022-01-12T03:03:21.097052Z","shell.execute_reply":"2022-01-12T03:03:21.103036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Drop annotation exceeding the image frame\n# We could clip them also. Smarter and possibly better?\ndf['annotations'] = df['annotations'].apply(clean_annotations)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:21.105038Z","iopub.execute_input":"2022-01-12T03:03:21.105482Z","iopub.status.idle":"2022-01-12T03:03:21.144044Z","shell.execute_reply.started":"2022-01-12T03:03:21.105443Z","shell.execute_reply":"2022-01-12T03:03:21.143318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Drop images with no annotations. The background works as negative samples anyway\ndf = df[df.annotations.str.len() > 0 ].reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:21.145487Z","iopub.execute_input":"2022-01-12T03:03:21.145788Z","iopub.status.idle":"2022-01-12T03:03:21.172236Z","shell.execute_reply.started":"2022-01-12T03:03:21.145742Z","shell.execute_reply":"2022-01-12T03:03:21.171156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train-validation split\ndf_train, df_val = df[df['is_train']], df[~df['is_train']]","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:21.174101Z","iopub.execute_input":"2022-01-12T03:03:21.174512Z","iopub.status.idle":"2022-01-12T03:03:21.18199Z","shell.execute_reply.started":"2022-01-12T03:03:21.174469Z","shell.execute_reply":"2022-01-12T03:03:21.181134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Augmentations\n","metadata":{}},{"cell_type":"code","source":"import torchvision.transforms as T","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:21.18485Z","iopub.execute_input":"2022-01-12T03:03:21.185523Z","iopub.status.idle":"2022-01-12T03:03:21.190499Z","shell.execute_reply.started":"2022-01-12T03:03:21.185486Z","shell.execute_reply":"2022-01-12T03:03:21.189421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_train_transform():\n    return A.Compose([\n        #A.Flip(0.5),\n        A.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n        ToTensorV2(p=1.0),\n        \n    ])#, bbox_params={'format': 'coco', 'label_fields': ['labels']})\n\n\ndef get_valid_transform():\n    return A.Compose([\n        A.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n        ToTensorV2(p=1.0),\n        \n    ])#, bbox_params={'format': 'coco', 'label_fields': ['labels']})","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:21.193344Z","iopub.execute_input":"2022-01-12T03:03:21.19408Z","iopub.status.idle":"2022-01-12T03:03:21.202349Z","shell.execute_reply.started":"2022-01-12T03:03:21.194035Z","shell.execute_reply":"2022-01-12T03:03:21.201209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Creating Dataset\n\n* I hope you have the video by now , DETR accepts data in coco format which is (x,y,w,h)(for those who do not know there are two formats coco and pascal(smin,ymin,xmax,ymax) which are widely used) . So now we need to prepare data in that format","metadata":{}},{"cell_type":"code","source":"class ReefDataset(Dataset):\n\n    def __init__(self, df, transforms):\n        self.df = df\n        self.transforms = transforms\n\n    def get_boxes(self, row):\n        \"\"\"Returns the bboxes for a given row as a 3D matrix with format [x_center, y_center, w, h]\"\"\"\n        boxes = pd.DataFrame(row['annotations'], columns=['x', 'y', 'width', 'height'])\n        boxes['x'] = boxes['x'] + (boxes['width'] / 2)\n        boxes['y'] = boxes['y'] + (boxes['height'] / 2)\n        \n        return boxes.astype(float).values\n    \n    def get_image(self, row):\n        \"\"\"Gets the image for a given row\"\"\"\n        \n        image = cv2.imread(f'{BASE_DIR}/{row[\"image_path\"]}', cv2.IMREAD_COLOR)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n        #image /= 255.0\n        return image\n    \n    \n    \n    def plot_img(self, i):\n        img, _, _ = self[i]\n        plt.imshow(img.permute(1, 2, 0))\n        plt.show()\n    \n    def __getitem__(self, i):\n\n        row = self.df.iloc[i]\n        image = self.get_image(row)\n        boxes = self.get_boxes(row)\n        \n        n_boxes = boxes.shape[0]\n        \n        # Calculate the area\n        #Area of bb\n        #area = boxes[:,2] * boxes[:,3]\n        \n        \n        target = {\n            'boxes': torch.as_tensor(boxes, dtype=torch.float32),\n            #'area': torch.as_tensor(area, dtype=torch.float32),\n            \n             'image_id': torch.tensor([i]),\n            \n            # There is only one class\n            'labels': torch.zeros((n_boxes,), dtype=torch.int64),\n        }\n\n        image_id = self.df.iloc[i]['image_path']\n        \n        \n        sample = {\n            'image': image,\n            'bboxes': target['boxes'],\n            'labels': target['labels']\n        }\n        image = self.transforms(image=sample['image'])['image']\n        #image = sample['image']\n\n        #import pdb; pdb.set_trace()\n        #target['boxes'] = torch.stack(tuple(map(torch.tensor, zip(*sample['bboxes'])))).permute(1, 0)\n        \n        target['boxes'] = A.augmentations.bbox_utils.normalize_bboxes(target['boxes'],rows=HEIGHT,cols=WIDTH)\n        target['boxes'] = torch.as_tensor(target['boxes'], dtype=torch.float32)\n        \n        return image, target, image_id\n\n    def __len__(self):\n        return len(self.df)\n","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:22.196276Z","iopub.execute_input":"2022-01-12T03:03:22.1969Z","iopub.status.idle":"2022-01-12T03:03:22.212899Z","shell.execute_reply.started":"2022-01-12T03:03:22.196862Z","shell.execute_reply":"2022-01-12T03:03:22.211988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Datasets and DataLoaders","metadata":{}},{"cell_type":"code","source":"def collate_fn(batch):\n    return tuple(zip(*batch))\n\nds_train = ReefDataset(df_train, get_train_transform())\nds_val = ReefDataset(df_val, get_valid_transform())\n\ndl_train = DataLoader(ds_train, batch_size=BATCH_SIZE,\n                      shuffle=True, num_workers=4, collate_fn=collate_fn)\n\n\ndl_val = DataLoader(ds_val, batch_size=BATCH_SIZE, shuffle=False,\n                    num_workers=4, collate_fn=collate_fn)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:22.934051Z","iopub.execute_input":"2022-01-12T03:03:22.934601Z","iopub.status.idle":"2022-01-12T03:03:22.944934Z","shell.execute_reply.started":"2022-01-12T03:03:22.934563Z","shell.execute_reply":"2022-01-12T03:03:22.943849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds_train.plot_img(3)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:23.272853Z","iopub.execute_input":"2022-01-12T03:03:23.273596Z","iopub.status.idle":"2022-01-12T03:03:23.98012Z","shell.execute_reply.started":"2022-01-12T03:03:23.273556Z","shell.execute_reply":"2022-01-12T03:03:23.977065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model\n\n* Initial DETR model is trained on coco dataset , which has 91 classes + 1 background class , hence we need to modify it to take our own number of classes\n* Also DETR model takes in 100 queries ie, it outputs total of 100 bboxes for every image, we can very well change that too","metadata":{}},{"cell_type":"code","source":"class DETRModel(nn.Module):\n    def __init__(self, num_classes, num_queries):\n        super(DETRModel,self).__init__()\n        self.num_classes = num_classes\n        self.num_queries = num_queries\n        \n        self.model = torch.hub.load('facebookresearch/detr', 'detr_resnet50', pretrained=True)\n        self.in_features = self.model.class_embed.in_features\n        \n        self.model.class_embed = nn.Linear(in_features=self.in_features,out_features=self.num_classes)\n        self.model.num_queries = self.num_queries\n        \n    def forward(self,images):\n        return self.model(images)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:26.662134Z","iopub.execute_input":"2022-01-12T03:03:26.66388Z","iopub.status.idle":"2022-01-12T03:03:26.670309Z","shell.execute_reply.started":"2022-01-12T03:03:26.663831Z","shell.execute_reply":"2022-01-12T03:03:26.669578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Utils\n* AverageMeter - class for averaging loss,metric,etc over epochs","metadata":{}},{"cell_type":"code","source":"class AverageMeter(object):\n    \"\"\"Computes and stores the average and current value\"\"\"\n    def __init__(self):\n        self.reset()\n\n    def reset(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n\n    def update(self, val, n=1):\n        self.val = val\n        self.sum += val * n\n        self.count += n\n        self.avg = self.sum / self.count","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:27.782558Z","iopub.execute_input":"2022-01-12T03:03:27.783337Z","iopub.status.idle":"2022-01-12T03:03:27.789642Z","shell.execute_reply.started":"2022-01-12T03:03:27.783296Z","shell.execute_reply":"2022-01-12T03:03:27.788606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Matcher and Bipartite Matching Loss\n\nNow we make use of the unique loss that the model uses and for that we need to define the matcher. DETR calcuates three individual losses :\n* Classification Loss for labels(its weight can be set by loss_ce)\n* Bbox Loss (its weight can be set by loss_bbox)\n* Loss for Background class","metadata":{}},{"cell_type":"code","source":"'''\ncode taken from github repo detr , 'code present in engine.py'\n'''\n\nmatcher = HungarianMatcher()\n\nweight_dict = weight_dict = {'loss_ce': 1, 'loss_bbox': 1 , 'loss_giou': 1}\n\nlosses = ['labels', 'boxes', 'cardinality']","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:28.753685Z","iopub.execute_input":"2022-01-12T03:03:28.754294Z","iopub.status.idle":"2022-01-12T03:03:28.759719Z","shell.execute_reply.started":"2022-01-12T03:03:28.754255Z","shell.execute_reply":"2022-01-12T03:03:28.758941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training Function\n\nTraining of DETR is unique and different from FasteRRcnn and EfficientDET, as we train the criterion as well , the training function can be viewed here : https://github.com/facebookresearch/detr/blob/master/engine.py","metadata":{}},{"cell_type":"code","source":"def train_fn(data_loader, model, criterion, optimizer, device, scheduler, epoch):\n    model.train()\n    criterion.train()\n    \n    summary_loss = AverageMeter()\n    \n    tk0 = tqdm(data_loader, total=len(data_loader))\n    \n    for step, (images, targets, image_ids) in enumerate(tk0):\n        \n        images = list(image.to(device) for image in images)\n        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n        \n\n        output = model(images)\n        loss_dict = criterion(output, targets)\n        weight_dict = criterion.weight_dict\n        \n        losses = sum(loss_dict[k] * weight_dict[k] for k in loss_dict.keys() if k in weight_dict)\n        \n        optimizer.zero_grad()\n\n        losses.backward()\n        optimizer.step()\n        if scheduler is not None:\n            scheduler.step()\n        \n        summary_loss.update(losses.item(),BATCH_SIZE)\n        tk0.set_postfix(loss=summary_loss.avg)\n        \n    return summary_loss","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:31.040184Z","iopub.execute_input":"2022-01-12T03:03:31.040476Z","iopub.status.idle":"2022-01-12T03:03:31.05282Z","shell.execute_reply.started":"2022-01-12T03:03:31.040445Z","shell.execute_reply":"2022-01-12T03:03:31.051846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Eval Function","metadata":{}},{"cell_type":"code","source":"def eval_fn(data_loader, model,criterion, device):\n    model.eval()\n    criterion.eval()\n    summary_loss = AverageMeter()\n    \n    with torch.no_grad():\n        \n        tk0 = tqdm(data_loader, total=len(data_loader))\n        for step, (images, targets, image_ids) in enumerate(tk0):\n            \n            images = list(image.to(device) for image in images)\n            targets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n\n            output = model(images)\n\n            loss_dict = criterion(output, targets)\n            weight_dict = criterion.weight_dict\n        \n            losses = sum(loss_dict[k] * weight_dict[k] for k in loss_dict.keys() if k in weight_dict)\n            \n            summary_loss.update(losses.item(),BATCH_SIZE)\n            tk0.set_postfix(loss=summary_loss.avg)\n    \n    return summary_loss","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:32.168529Z","iopub.execute_input":"2022-01-12T03:03:32.16954Z","iopub.status.idle":"2022-01-12T03:03:32.179476Z","shell.execute_reply.started":"2022-01-12T03:03:32.169485Z","shell.execute_reply":"2022-01-12T03:03:32.178501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training/validation loop","metadata":{}},{"cell_type":"code","source":"device = torch.device('cuda')\nmodel = DETRModel(num_classes=NUM_CLASSES,num_queries=NUM_QUERIES)\nmodel = model.to(device)\ncriterion = SetCriterion(NUM_CLASSES-1, matcher, weight_dict, eos_coef = NULL_CLASS_COEF, losses=losses)\ncriterion = criterion.to(device)\n\noptimizer = torch.optim.AdamW(model.parameters(), lr=LR)\n\nbest_loss = 10**5\nfor epoch in range(EPOCHS):\n    time_start = time.time()\n    train_loss = train_fn(dl_train, model,criterion, optimizer,device,scheduler=None,epoch=epoch)\n    valid_loss = eval_fn(dl_val, model,criterion, device)\n\n    elapsed = time.time() - time_start\n    chk_name = f'pytorch_model_e{epoch}.bin'\n    torch.save(model.state_dict(), chk_name)\n    print(f\"[Epoch {epoch+1:2d} / {EPOCHS:2d}] Train loss: {train_loss.avg:.3f}. Val loss: {valid_loss.avg:.3f} --> {chk_name}  [{elapsed/60:.0f} mins]\")   \n\n    if valid_loss.avg < best_loss:\n        best_loss = valid_loss.avg\n        print(f'Best model found in epoch {epoch+1}........Saving Model')\n        torch.save(model.state_dict(), 'pytorch_model.bin')","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:03:33.059805Z","iopub.execute_input":"2022-01-12T03:03:33.060473Z","iopub.status.idle":"2022-01-12T03:39:08.785188Z","shell.execute_reply.started":"2022-01-12T03:03:33.060436Z","shell.execute_reply":"2022-01-12T03:39:08.783894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Save the torch hub cache path \n\nSave the torch hub path to the working directory so it's stored as the output of the notebook and we can use everything without Internet in the submission notebook.","metadata":{}},{"cell_type":"code","source":"cp -R /root/.cache/torch/hub/ torch_hub/","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:39:11.472794Z","iopub.execute_input":"2022-01-12T03:39:11.473632Z","iopub.status.idle":"2022-01-12T03:39:12.550761Z","shell.execute_reply.started":"2022-01-12T03:39:11.47359Z","shell.execute_reply":"2022-01-12T03:39:12.549417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Sample\n\n* I know we might be naive to visualize the model ouput just after one epoch but lets do that and see what are the results like","metadata":{}},{"cell_type":"code","source":"def view_sample(df_sample, model, device):\n    '''\n    Code taken from Peter's Kernel \n    https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-train\n    '''\n    ds_val = ReefDataset(df_sample, get_valid_transform())\n    dl_val = DataLoader(ds_val, batch_size=BATCH_SIZE, shuffle=False,\n                        num_workers=4, collate_fn=collate_fn)\n    \n    images, targets, image_ids = next(iter(dl_val))\n    \n    images = list(img.to(device) for img in images)\n    targets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n    \n    # [x_center, y_center, width, height] # All scaled\n    boxes = targets[0]['boxes'].cpu().numpy()\n    # De-scaled\n    boxes = np.array([np.array(box).astype(np.int32) for box in A.augmentations.bbox_utils.denormalize_bboxes(boxes, HEIGHT, WIDTH)])\n    #boxes['x'] = boxes['x'] + (boxes['width'] / 2)\n    #boxes['y'] = boxes['y'] + (boxes['height'] / 2)\n\n    \n    #[x_min, y_min, width, height]\n    boxes[:, 0] = boxes[:, 0] - (boxes[:, 2] / 2) # x_center --> x_min\n    boxes[:, 1] = boxes[:, 1] - (boxes[:, 3] / 2) # y_center --> y_min\n    \n    sample = images[0].permute(1,2,0).cpu().numpy()\n    \n    model.eval()\n    model.to(device)\n    cpu_device = torch.device(\"cpu\")\n    \n    with torch.no_grad():\n        outputs = model(images)\n        \n    outputs = [{k: v.to(cpu_device) for k, v in outputs.items()}]\n    \n    fig, ax = plt.subplots(1, 1, figsize=(16, 8))\n    for box in boxes:\n        x, y, w, h = box\n        cv2.rectangle(sample, (x, y), (x+w, y+h), (0, 220, 0), 3)\n        \n        \n\n    oboxes = outputs[0]['pred_boxes'][0].detach().cpu().numpy()\n    oboxes = np.array([np.array(box).astype(np.int32) for box in A.augmentations.bbox_utils.denormalize_bboxes(oboxes, HEIGHT, WIDTH)])\n    \n    # [x_min, y_min, width, height]\n    oboxes[:, 0] = oboxes[:, 0] - (oboxes[:, 2] / 2) # x_center --> x_min\n    oboxes[:, 1] = oboxes[:, 1] - (oboxes[:, 3] / 2) # y_center --> y_min\n    \n    prob   = outputs[0]['pred_logits'][0].softmax(1).detach().cpu().numpy()[:,0]\n    #print(f\"Probabilities: {prob}\")\n    scored_boxes = list(zip(oboxes, prob))\n    sorted_boxes = list(sorted(scored_boxes, key=lambda y: -y[1]))\n    print([score for _, score in sorted_boxes][:10])\n    for i, (box, p) in enumerate(sorted_boxes):\n        x, y, w, h = box\n        \n        if p > 0.5:\n            cv2.rectangle(sample, (x, y), (x+w, y+h), (220, 0, 0), 3)\n        \n        if i > 18:\n            break\n\n            \n    ax.set_axis_off()\n    ax.imshow(sample)\n    \n    \nmodel = DETRModel(num_classes=NUM_CLASSES, num_queries=NUM_QUERIES)\nmodel.load_state_dict(torch.load(\"./pytorch_model.bin\"))\nview_sample(df_sample=df_val[df_val['n_annotations'] > 3].iloc[[30]], model=model,device=torch.device('cuda'))","metadata":{"execution":{"iopub.status.busy":"2022-01-12T03:39:41.368101Z","iopub.execute_input":"2022-01-12T03:39:41.36842Z","iopub.status.idle":"2022-01-12T03:39:43.55858Z","shell.execute_reply.started":"2022-01-12T03:39:41.368386Z","shell.execute_reply":"2022-01-12T03:39:43.55759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Please, _DO_ upvote if you find it useful or interesting!!","metadata":{}}]}