{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🐠 Reef - Pytorch Starter - FasterRCNN Train\n\n## A self-contained, simple, pure pytorch 🔥 Faster R-CNN implementation with `LB=0.416`\n\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/31703/logos/header.png)\n\n#### FasterR-CNN is one of the SOTA models for Object detection.\n\n### In this notebook we will cover a full pipeline from zero to a submission using a simple, pure pytorch implementation of a Faster R-CNN with pretrained weights, and fine-tuning it to this specific dataset. \n### The inference notebook is separated for organization purposes, as is it a common practice in Kaggle nowadays.\n\nI hope it helps to get started in this amazing competition!\n\n## You can find the [inference notebook here](https://www.kaggle.com/julian3833/coral-reef-pytorch-fasterrcnn-infer-0-xxx).\n\n## Details: \n\n- It is an adapted version of [this notebook](https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-train) mentioned in [this comment](https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/290016).\n- FasterRCNN from torchvision\n- Use Resnet50 backbone\n\n\n# Please, _DO_ upvote if you find this useful!!\n\n\n \n \n \n\n#### Changelog\n\n| Version | Description| Dataset| Best LB |\n| --- | ----| --- | --- |\n| [**V8**](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-train-lb-0-293?scriptVersionId=80517118)  | 2 epochs - Save last epoch | [coral-reef-pytorch-starter-fasterrcnn-weights](https://www.kaggle.com/julian3833/coral-reef-pytorch-starter-fasterrcnn-weights)| `0.293`|\n| [**V16**](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-train-lb-0-293?scriptVersionId=80601095) | 4 epochs - Save all epochs | [reef-starter-torch-fasterrcnn-4e](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-4e)| `0.361` |\n| [**V17**](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-train-lb-0-369?scriptVersionId=80604402) | **Add validation (on subsequences)**. 95-5 split. 8 epochs, keeping track of validation loss. | [reef-starter-torch-fasterrcnn-8e](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-8e)| `0.369` |\n| [**V19**](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-train-lb-0-369?scriptVersionId=80610403) | 12 epochs, lower LR | [reef-starter-torch-fasterrcnn-12e](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-12e)| `0.413` |\n| [**V24**](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-train-lb-0-416?scriptVersionId=806809369) | V19 with 90-10 train-validation split. Tidy up code. Add Flip. Correct problem with augmentations. | [reef-starter-torch-fasterrcnn-12e-v2](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-12e-v2) | `0.416` |\n| [**V30**](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-train-lb-0-416) | Split in 90-10 subsequences. Train 16 epochs. Rollback validation to standard metrics. Log bbox validation score. Remove VerticalFlip (Only Horizontal). Add clip of masks as suggested [here](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-train-lb-0-416/comments#1597371)|  | `??` |\n\n---\n\n### References:\n\n* [TorchVision Object Detection Finetuning Tutorial - PyTorch](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html)\n* [Finetuning Torchvision Models - PyTorch](https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html)\n\n\nOk, enough chit chat, show me the code!!","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"# Very few imports. This is a pure torch solution!\nimport os\nimport cv2\nimport time\nimport random\n\nimport numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt\n\nimport albumentations as A\nfrom albumentations.pytorch.transforms import ToTensorV2\n\nimport torch\nimport torchvision\nfrom torch.utils.data import DataLoader\nfrom torchvision.models.detection.faster_rcnn import FastRCNNPredictor\nfrom torchvision.models.detection import FasterRCNN\n\n\n# Fix randomness\n\ndef fix_all_seeds(seed):\n    np.random.seed(seed)\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\n    \nfix_all_seeds(2021)","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2021-11-28T03:24:33.227354Z","iopub.execute_input":"2021-11-28T03:24:33.227727Z","iopub.status.idle":"2021-11-28T03:24:35.864645Z","shell.execute_reply.started":"2021-11-28T03:24:33.227675Z","shell.execute_reply":"2021-11-28T03:24:35.863511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configuration","metadata":{}},{"cell_type":"code","source":"DEVICE = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n\nBASE_DIR = \"../input/tensorflow-great-barrier-reef/train_images/\"\n\n\n# Configuration for the Optimizer\nLEARNING_RATE = 0.002\nMOMENTUM = 0.9\nWEIGHT_DECAY = 0.0005\n\n# Number of epochs\nNUM_EPOCHS = 16\n\nBATCH_SIZE = 8","metadata":{"execution":{"iopub.status.busy":"2021-11-28T03:24:35.869425Z","iopub.execute_input":"2021-11-28T03:24:35.869715Z","iopub.status.idle":"2021-11-28T03:24:35.906586Z","shell.execute_reply.started":"2021-11-28T03:24:35.869669Z","shell.execute_reply":"2021-11-28T03:24:35.905313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load `df`","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"../input/reef-cv-strategy-subsequences-dataframes/train-validation-split/train-0.1.csv\")\n\n# Turn annotations from strings into lists of dictionaries\ndf['annotations'] = df['annotations'].apply(eval)\n\n# Create the image path for the row\ndf['image_path'] = \"video_\" + df['video_id'].astype(str) + \"/\" + df['video_frame'].astype(str) + \".jpg\"\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-28T03:24:35.908292Z","iopub.execute_input":"2021-11-28T03:24:35.909079Z","iopub.status.idle":"2021-11-28T03:24:36.576022Z","shell.execute_reply.started":"2021-11-28T03:24:35.909026Z","shell.execute_reply":"2021-11-28T03:24:36.575066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are a lot of images with no annotations, about `80%` or `18k`.","metadata":{}},{"cell_type":"code","source":"(df['annotations'].str.len() > 0).value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-11-28T03:24:36.577236Z","iopub.execute_input":"2021-11-28T03:24:36.577735Z","iopub.status.idle":"2021-11-28T03:24:36.620647Z","shell.execute_reply.started":"2021-11-28T03:24:36.577686Z","shell.execute_reply":"2021-11-28T03:24:36.619759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(df['annotations'].str.len() > 0).value_counts(normalize=True).round(2)","metadata":{"execution":{"iopub.status.busy":"2021-11-28T03:24:36.626292Z","iopub.execute_input":"2021-11-28T03:24:36.628356Z","iopub.status.idle":"2021-11-28T03:24:36.661927Z","shell.execute_reply.started":"2021-11-28T03:24:36.628301Z","shell.execute_reply":"2021-11-28T03:24:36.660915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" We drop them to make it easier and faster to train.","metadata":{}},{"cell_type":"code","source":"# Drop images with no annotations. The background works as negative samples anyway\ndf = df[df['annotations'].str.len() > 0].reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2021-11-28T03:24:36.666324Z","iopub.execute_input":"2021-11-28T03:24:36.668596Z","iopub.status.idle":"2021-11-28T03:24:36.698618Z","shell.execute_reply.started":"2021-11-28T03:24:36.668542Z","shell.execute_reply":"2021-11-28T03:24:36.697840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train-validation split\n\nWe are splitting using subsequences. I have tried other strategies and this is the one that works the best for now. The dataset has just 3 videos, each of them split into sequences, but in total there are only 20 sequences. A **subsequences**, as we defined them, are parts of a sequences where objects are continually present or are continually not present. \n\n \n\nLet's see an **example**. Consider the sequence `A` with the following frames:\n* `1-20` - No annotations present\n* `21-30` - Annotations present\n* `31-60` - No annotations\n* `61-80` - Annotations present\n\nIn this case, we say that the sequence `A` has `4` subsequences (`1-20`, `21-30`, `31-60`, `61-80`).\n\n\nSee: [🐠 Reef - CV strategy: subsequences!](https://www.kaggle.com/julian3833/reef-cv-strategy-subsequences) for more details about this","metadata":{}},{"cell_type":"code","source":"df_train = df[df['is_train']].reset_index(drop=True)\ndf_val = df[~df['is_train']].reset_index(drop=True)\n\ndf_train.shape[0], df_val.shape[0]","metadata":{"execution":{"iopub.status.busy":"2021-11-28T03:24:36.702787Z","iopub.execute_input":"2021-11-28T03:24:36.704850Z","iopub.status.idle":"2021-11-28T03:24:36.718967Z","shell.execute_reply.started":"2021-11-28T03:24:36.704798Z","shell.execute_reply":"2021-11-28T03:24:36.718014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset class","metadata":{}},{"cell_type":"code","source":"class ReefDataset:\n\n    def __init__(self, df, transforms=None):\n        self.df = df\n        self.transforms = transforms\n\n    def get_boxes(self, row):\n        \"\"\"Returns the bboxes for a given row as a 3D matrix with format [x_min, y_min, x_max, y_max]\"\"\"\n        \n        boxes = pd.DataFrame(row['annotations'], columns=['x', 'y', 'width', 'height']).astype(float).values\n        \n        # Change from [x_min, y_min, w, h] to [x_min, y_min, x_max, y_max]\n        boxes[:, 2] = boxes[:, 0] + boxes[:, 2]\n        boxes[:, 3] = boxes[:, 1] + boxes[:, 3]\n        \n        # Clip boxes as suggested by Lukazs in the comment below:\n        # https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-train-lb-0-416/comments#1597371\n        boxes[:, 2] = np.clip(boxes[:, 2], 0, 1280)\n        boxes[:, 3] = np.clip(boxes[:, 3], 0, 720)\n        \n        return boxes\n    \n    def get_image(self, row):\n        \"\"\"Gets the image for a given row\"\"\"\n        \n        image = cv2.imread(f'{BASE_DIR}/{row[\"image_path\"]}', cv2.IMREAD_COLOR)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n        image /= 255.0\n        return image\n    \n    def __getitem__(self, i):\n\n        row = self.df.iloc[i]\n        image = self.get_image(row)\n        boxes = self.get_boxes(row)\n        \n        n_boxes = boxes.shape[0]\n        \n        # Calculate the area\n        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])\n        \n        \n        target = {\n            'boxes': torch.as_tensor(boxes, dtype=torch.float32),\n            'area': torch.as_tensor(area, dtype=torch.float32),\n            \n            'image_id': torch.tensor([i]),\n            \n            # There is only one class\n            'labels': torch.ones((n_boxes,), dtype=torch.int64),\n            \n            # Suppose all instances are not crowd\n            'iscrowd': torch.zeros((n_boxes,), dtype=torch.int64)            \n        }\n\n        \n        sample = {\n            'image': image,\n            'bboxes': target['boxes'],\n            'labels': target['labels']\n        }\n        sample = self.transforms(**sample)\n        image = sample['image']\n\n        if n_boxes > 0:\n            target['boxes'] = torch.stack(tuple(map(torch.tensor, zip(*sample['bboxes'])))).permute(1, 0)\n\n        return image, target\n\n    def __len__(self):\n        return len(self.df)","metadata":{"execution":{"iopub.status.busy":"2021-11-28T03:24:36.723230Z","iopub.execute_input":"2021-11-28T03:24:36.725393Z","iopub.status.idle":"2021-11-28T03:24:36.753744Z","shell.execute_reply.started":"2021-11-28T03:24:36.725330Z","shell.execute_reply":"2021-11-28T03:24:36.752761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Augmentations\n\nA very simple set of augmentations. There should be a lot of low-hanging fruits here to explore!","metadata":{}},{"cell_type":"code","source":"def get_train_transform():\n    return A.Compose([\n        A.HorizontalFlip(0.5),\n        ToTensorV2(p=1.0)\n    ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})\n\n\ndef get_valid_transform():\n    return A.Compose([\n        ToTensorV2(p=1.0)\n    ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})","metadata":{"execution":{"iopub.status.busy":"2021-11-28T03:24:36.758331Z","iopub.execute_input":"2021-11-28T03:24:36.760663Z","iopub.status.idle":"2021-11-28T03:24:36.769082Z","shell.execute_reply.started":"2021-11-28T03:24:36.760600Z","shell.execute_reply":"2021-11-28T03:24:36.768168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define datasets\nds_train = ReefDataset(df_train, get_train_transform())\nds_val = ReefDataset(df_val, get_valid_transform())","metadata":{"execution":{"iopub.status.busy":"2021-11-28T03:24:36.773141Z","iopub.execute_input":"2021-11-28T03:24:36.773928Z","iopub.status.idle":"2021-11-28T03:24:36.782125Z","shell.execute_reply.started":"2021-11-28T03:24:36.773877Z","shell.execute_reply":"2021-11-28T03:24:36.781140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Check one sample","metadata":{}},{"cell_type":"code","source":"try:\n    # Let's get an interesting one ;)\n    idx = df_train[df_train.annotations.str.len() > 12].iloc[0].name\nexcept:\n    idx = 0\n    \n\nimage, targets = ds_train[idx]\n\n\nboxes = targets['boxes'].cpu().numpy().astype(np.int32)\nimg = image.permute(1,2,0).cpu().numpy()\nfig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\nfor box in boxes:\n    cv2.rectangle(img,\n                  (box[0], box[1]),\n                  (box[2], box[3]),\n                  (220, 0, 0), 3)\n    \nax.set_axis_off()\nax.imshow(img);","metadata":{"execution":{"iopub.status.busy":"2021-11-28T03:24:36.783486Z","iopub.execute_input":"2021-11-28T03:24:36.784035Z","iopub.status.idle":"2021-11-28T03:24:37.398231Z","shell.execute_reply.started":"2021-11-28T03:24:36.783990Z","shell.execute_reply":"2021-11-28T03:24:37.397196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## DataLoaders","metadata":{}},{"cell_type":"code","source":"# Create dataloaders\n\ndef collate_fn(batch):\n    return tuple(zip(*batch))\n\ndl_train = DataLoader(ds_train, batch_size=BATCH_SIZE, shuffle=True, num_workers=4, collate_fn=collate_fn)\ndl_val = DataLoader(ds_val, batch_size=BATCH_SIZE, shuffle=False, num_workers=4, collate_fn=collate_fn)","metadata":{"execution":{"iopub.status.busy":"2021-11-28T03:24:37.399515Z","iopub.execute_input":"2021-11-28T03:24:37.399841Z","iopub.status.idle":"2021-11-28T03:24:37.409134Z","shell.execute_reply.started":"2021-11-28T03:24:37.399799Z","shell.execute_reply":"2021-11-28T03:24:37.407725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create the model","metadata":{}},{"cell_type":"code","source":"def get_model():\n    # load a model; pre-trained on COCO\n    model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)\n\n    num_classes = 2  # 1 class (starfish) + background\n\n    # get number of input features for the classifier\n    in_features = model.roi_heads.box_predictor.cls_score.in_features\n\n    # replace the pre-trained head with a new one\n    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)\n\n    model.to(DEVICE)\n    return model\n\nmodel = get_model()","metadata":{"execution":{"iopub.status.busy":"2021-11-28T03:24:37.410996Z","iopub.execute_input":"2021-11-28T03:24:37.411877Z","iopub.status.idle":"2021-11-28T03:24:42.485611Z","shell.execute_reply.started":"2021-11-28T03:24:37.411662Z","shell.execute_reply":"2021-11-28T03:24:42.484729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training  loop!!","metadata":{}},{"cell_type":"code","source":"# Create the optimizer\nparams = [p for p in model.parameters() if p.requires_grad]\noptimizer = torch.optim.SGD(params, lr=LEARNING_RATE, momentum=MOMENTUM, weight_decay=WEIGHT_DECAY)\n\nn_batches, n_batches_val = len(dl_train), len(dl_val)\nval_losses = []\nval_box_losses = []\n\nfor epoch in range(NUM_EPOCHS):\n    \n    model.train()\n    \n    time_start = time.time()\n    loss_accum = 0\n    loss_box_accum = 0\n    \n    # Go over training batches\n    for batch_idx, (images, targets) in enumerate(dl_train, 1):\n        images = list(image.to(DEVICE) for image in images)\n        targets = [{k: v.to(DEVICE) for k, v in t.items()} for t in targets]\n\n        # Predict!\n        # This dict has the following keys:\n        #    loss_classifier, loss_box_reg, loss_objectness, loss_rpn_box_reg\n        loss_dict = model(images, targets)\n        \n        # We optimize the full set of losses\n        losses = sum(loss for loss in loss_dict.values())\n        loss_value = losses.item()\n\n        loss_accum += loss_value\n        loss_box_accum += loss_dict['loss_box_reg'].item()\n        \n\n        # Back-prop\n        optimizer.zero_grad()\n        losses.backward()\n        optimizer.step()\n\n        \n    # Validation step!\n    val_loss_accum = 0\n    val_loss_box_accum = 0\n    \n    with torch.no_grad():\n        for batch_idx, (images, targets) in enumerate(dl_val, 1):\n            images = list(image.to(DEVICE) for image in images)\n            targets = [{k: v.to(DEVICE) for k, v in t.items()} for t in targets]\n            \n            val_loss_dict = model(images, targets)\n            val_batch_loss = sum(loss for loss in val_loss_dict.values())\n            \n            val_loss_accum += val_batch_loss.item()\n            val_loss_box_accum += val_loss_dict['loss_box_reg'].item()\n\n    \n    # Calculate epoch losses\n    val_loss = val_loss_accum / n_batches_val\n    val_loss_box = val_loss_box_accum / n_batches_val\n    \n    train_loss = loss_accum / n_batches\n    train_loss_box = loss_box_accum / n_batches\n    \n    val_losses.append(val_loss)\n    val_box_losses.append(val_loss_box)\n    \n    # Save model\n    chk_name = f'pytorch_model-e{epoch}.bin'\n    torch.save(model.state_dict(), chk_name)\n    \n    \n    # Logging\n    elapsed = time.time() - time_start\n    \n    prefix = f\"[Epoch {epoch+1:2d} / {NUM_EPOCHS:2d}]\"\n    print(f\"{prefix} Train loss: {train_loss:.3f}.  Train loss (bbox only): {train_loss_box:.3f}.  Val loss (bbox only): {val_loss_box:.3f}\")   \n    print(prefix)\n    print(f\"{prefix} Val loss  : {val_loss:.3f}\")\n    print(f\"{prefix} Saved to  : {chk_name}  [{elapsed:.0f} secs]\")\n    print()","metadata":{"execution":{"iopub.status.busy":"2021-11-28T03:24:42.486969Z","iopub.execute_input":"2021-11-28T03:24:42.487335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Best model based on lowest validation loss\nnp.argmin(val_losses)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Best model based on lowest bbox loss\nnp.argmin(val_box_losses)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Check result","metadata":{}},{"cell_type":"code","source":"idx = 0\n\nimages, targets = next(iter(dl_val))\nimages = list(img.to(DEVICE) for img in images)\ntargets = [{k: v.to(DEVICE) for k, v in t.items()} for t in targets]\n\nboxes = targets[idx]['boxes'].cpu().numpy().astype(np.int32)\nsample = images[idx].permute(1,2,0).cpu().numpy()\n\nmodel.eval()\n\noutputs = model(images)\noutputs = [{k: v.detach().cpu().numpy() for k, v in t.items()} for t in outputs]\n\nfig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\n# Red for ground truth\nfor box in boxes:\n    cv2.rectangle(sample,\n                  (box[0], box[1]),\n                  (box[2], box[3]),\n                  (220, 0, 0), 3)\n\n    \n# Green for predictions\n# Print the first 5\nfor box in outputs[idx]['boxes'][:5]:\n    cv2.rectangle(sample,\n                  (box[0], box[1]),\n                  (box[2], box[3]),\n                  (0, 220, 0), 3)\n\nax.set_axis_off()\nax.imshow(sample);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Please, _DO_ upvote if you found it useful!","metadata":{}}]}