{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🐠 Reef - Pytorch Starter - FasterRCNN Train\n\n## A self-contained, simple, pure pytorch 🔥 Faster R-CNN implementation with `LB=0.413`\n\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/31703/logos/header.png)\n\n#### FasterR-CNN is one of the SOTA models for Object detection.\n\n### In this notebook we present a simple solution using a pure pytorch Faster R-CNN with pretrained weights, and finetuning it for few epochs.\n\nIt is an adapted version of [this notebook](https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-train) mentioned in [this comment](https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/290016).\n\n## You can find the [inference notebook here](https://www.kaggle.com/julian3833/coral-reef-pytorch-fasterrcnn-infer-0-xxx).\n\n## Details: \n- FasterRCNN from torchvision\n- Use Resnet50 backbone\n\n**Update**: Added simple train/validation split in this version, using the \"subsequence\" split of this notebook: [🐠 Reef - CV strategy: subsequences!](https://www.kaggle.com/julian3833/reef-cv-strategy-subsequences)\n\nStill dropping all the images with no objects, as the model doesn't support them out-of-the-box. The other starters are removing the empty images as well, so it might be a general condition of Object Detection. I'm quite noob in the field to be honest.\n\n# Please, _DO_ upvote if you find this useful!!\n\n\n&nbsp;\n&nbsp;\n&nbsp;\n\n#### Changelog\n\n| Version | Description| Dataset| Best LB |\n| --- | ----| --- | --- |\n| [**V8**](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-train-lb-0-293?scriptVersionId=80517118)  | 2 epochs - Save last epoch | [coral-reef-pytorch-starter-fasterrcnn-weights](https://www.kaggle.com/julian3833/coral-reef-pytorch-starter-fasterrcnn-weights)| `0.293`|\n| [**V16**](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-train-lb-0-293?scriptVersionId=80601095) | 4 epochs - Save all epochs | [reef-starter-torch-fasterrcnn-4e](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-4e)| `0.361` |\n| [**V17**](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-train-lb-0-369?scriptVersionId=80604402) | Add **validation**. 95-5 split. 8 epochs, keeping track of validation loss. | [reef-starter-torch-fasterrcnn-8e](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-8e)| `0.369` |\n| [**V19**](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-train-lb-0-369?scriptVersionId=80610403) | 12 epochs, lower LR | [reef-starter-torch-fasterrcnn-12e](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-12e)| `0.413` |\n| [**V24**](https://www.kaggle.com/julian3833/reef-starter-torch-fasterrcnn-train-lb-0-369) | V19 with 90-10 train-validation split. Tidy up code. Add Flip. Correct problem with augmentations. | -- | `??` |\n\n\n","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"# Very few imports. This is a pure torch solution!\nimport os\nimport shutil\nimport cv2\nimport json\nimport torch\nimport random\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom PIL import Image\nfrom albumentations.pytorch.transforms import ToTensorV2\nfrom sklearn.model_selection import train_test_split\nfrom tqdm.notebook import tqdm\nimport albumentations \nfrom torch.utils.data import DataLoader","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2022-11-14T13:34:02.257596Z","iopub.execute_input":"2022-11-14T13:34:02.257936Z","iopub.status.idle":"2022-11-14T13:34:02.264347Z","shell.execute_reply.started":"2022-11-14T13:34:02.257885Z","shell.execute_reply":"2022-11-14T13:34:02.262972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# reletive path (src/tensorflow-great-barrier-reef)\nDATASET_PATH = \"tensorflow-great-barrier-reef\"\n\n# update dataset\nUPDATE_DATASET = False\n\n# remove no label images\nREMOVE_NON_LABELED = True\n\n# training data percentage\nTRAINING_SIZE = 0.8\n\n# turn on the debug\nDEBUGGING_PREPROCESS = False\n\n# image size\nIMAGE_SIZE = (1280, 720)\n\n# if train preprocess\nTRAIN_PREPROCESS = False","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:02.271162Z","iopub.execute_input":"2022-11-14T13:34:02.271471Z","iopub.status.idle":"2022-11-14T13:34:02.276965Z","shell.execute_reply.started":"2022-11-14T13:34:02.271408Z","shell.execute_reply":"2022-11-14T13:34:02.276031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### W&B Setup","metadata":{}},{"cell_type":"code","source":"#!pip install wandb --upgrade","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:02.290616Z","iopub.execute_input":"2022-11-14T13:34:02.290885Z","iopub.status.idle":"2022-11-14T13:34:02.296569Z","shell.execute_reply.started":"2022-11-14T13:34:02.290839Z","shell.execute_reply":"2022-11-14T13:34:02.295747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#!wandb login ce5676f856caf561584c75f8175f6967876f1c77","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:02.299378Z","iopub.execute_input":"2022-11-14T13:34:02.299906Z","iopub.status.idle":"2022-11-14T13:34:02.30542Z","shell.execute_reply.started":"2022-11-14T13:34:02.299834Z","shell.execute_reply":"2022-11-14T13:34:02.304547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import wandb\n\n# wandb.init(project=\"new_COTS\", entity=\"pppetrr\")","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:02.30826Z","iopub.execute_input":"2022-11-14T13:34:02.308969Z","iopub.status.idle":"2022-11-14T13:34:02.316073Z","shell.execute_reply.started":"2022-11-14T13:34:02.308902Z","shell.execute_reply":"2022-11-14T13:34:02.315138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# config = {\n#     \"epochs\": 10,\n#     \"classes\": 2,\n#     \"batch_size\" : 4,\n#     \"learning_rate\" : 0.005\n# }","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:02.325612Z","iopub.execute_input":"2022-11-14T13:34:02.325866Z","iopub.status.idle":"2022-11-14T13:34:02.329258Z","shell.execute_reply.started":"2022-11-14T13:34:02.325821Z","shell.execute_reply":"2022-11-14T13:34:02.328234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Constants","metadata":{}},{"cell_type":"code","source":"!ls ../input/tensorflow-great-barrier-reef","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:02.335831Z","iopub.execute_input":"2022-11-14T13:34:02.336145Z","iopub.status.idle":"2022-11-14T13:34:03.348433Z","shell.execute_reply.started":"2022-11-14T13:34:02.336093Z","shell.execute_reply":"2022-11-14T13:34:03.347542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DEVICE = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\nDATASET_PATH = \"../input/tensorflow-great-barrier-reef/\"\n\nNUM_EPOCHS = 1\n","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:03.35092Z","iopub.execute_input":"2022-11-14T13:34:03.351352Z","iopub.status.idle":"2022-11-14T13:34:03.356741Z","shell.execute_reply.started":"2022-11-14T13:34:03.351293Z","shell.execute_reply":"2022-11-14T13:34:03.355729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(\"../input/reef-cv-strategy-subsequences-dataframes/train-validation-split/train-0.1.csv\")\n\n# Turn annotations from strings into lists of dictionaries\ndf['annotations'] = df['annotations'].apply(eval)\n\n# Create the image path for the row\ndf['image_path'] = \"video_\" + df['video_id'].astype(str) + \"/\" + df['video_frame'].astype(str) + \".jpg\"\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:03.358268Z","iopub.execute_input":"2022-11-14T13:34:03.358911Z","iopub.status.idle":"2022-11-14T13:34:03.846155Z","shell.execute_reply.started":"2022-11-14T13:34:03.358857Z","shell.execute_reply":"2022-11-14T13:34:03.845185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train, df_val = df[df['is_train']], df[~df['is_train']]","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:03.847658Z","iopub.execute_input":"2022-11-14T13:34:03.848017Z","iopub.status.idle":"2022-11-14T13:34:03.857351Z","shell.execute_reply.started":"2022-11-14T13:34:03.847964Z","shell.execute_reply":"2022-11-14T13:34:03.856622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.shape\ndf_train","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:03.861689Z","iopub.execute_input":"2022-11-14T13:34:03.862032Z","iopub.status.idle":"2022-11-14T13:34:03.887141Z","shell.execute_reply.started":"2022-11-14T13:34:03.86198Z","shell.execute_reply":"2022-11-14T13:34:03.886008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load `df`\n\n### See: [🐠 Reef - CV strategy: subsequences!](https://www.kaggle.com/julian3833/reef-cv-strategy-subsequences)","metadata":{}},{"cell_type":"code","source":"# list of String(image path)\nTOTAL_images = []\n\n# list of dict(e.g. [{'x': 559, 'y': 213, 'width': 50, 'height': 32}])\nTOTAL_labels = []\n\nimage_dict = {\n    \"0\":[],\n    \"1\":[],\n    \"2\":[]\n}\nlabel_dict = {\n    \"0\":[],\n    \"1\":[],\n    \"2\":[]\n}\n\ntrain_CSV = pd.read_csv(f\"../input/tensorflow-great-barrier-reef/train.csv\")\nsize = len(train_CSV[\"video_id\"])\nfor row in range(size):\n    video_id = train_CSV[\"video_id\"][row]\n    img_id = train_CSV[\"video_frame\"][row]\n    img = f\"{DATASET_PATH}/train_images/video_{video_id}/{img_id}.jpg\"\n\n    labels = list(eval(train_CSV[\"annotations\"][row]))\n    if REMOVE_NON_LABELED and len(labels) == 0:\n        continue\n\n#     TOTAL_images.append(img)\n#     TOTAL_labels.append(labels)\n    image_dict[str(video_id)].append(img)\n    label_dict[str(video_id)].append(labels)\n\nprint(f\"Total number of images collected: {len(TOTAL_images)}\")\n\n# split training, test\n# _train_set, test_set, _train_label, test_label = train_test_split(\n#     TOTAL_images, TOTAL_labels,\n#     train_size=TRAINING_SIZE,\n#     shuffle=True)\n\n# # get valid\n# train_set, valid_set, train_label, valid_label = train_test_split(\n#     _train_set, _train_label,\n#     train_size=TRAINING_SIZE,\n#     shuffle=True)\ntrain_set = image_dict[\"0\"]\nvalid_set = image_dict[\"1\"]\ntest_set = image_dict[\"2\"]\n\ntrain_label = label_dict[\"0\"]\nvalid_label = label_dict[\"1\"]\ntest_label = label_dict[\"2\"]\n\n\n\nprint(f\"Traning size: {len(train_set)}\")\nprint(f\"Test size: {len(test_set)}\")\n\n# show first image and its labels\nimg_0 = cv2.cvtColor(cv2.imread(train_set[0]), cv2.COLOR_BGR2RGB)\nprint(f\"Image size: {img_0.size}\")\nprint(str(train_label[0]))\n\n# drow labels on the image\nfor dirct in train_label[0]:\n    x = dirct[\"x\"]\n    y = dirct[\"y\"]\n    width = dirct[\"width\"]\n    height = dirct[\"height\"]\n    start_point = (x, y)\n    end_point = (x+width, y+height)\n    img_0 = cv2.rectangle(img_0, start_point, end_point, (255,255,0), 2)\nplt.imshow(img_0)\nplt.axis(\"off\")\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:03.89024Z","iopub.execute_input":"2022-11-14T13:34:03.890847Z","iopub.status.idle":"2022-11-14T13:34:05.697099Z","shell.execute_reply.started":"2022-11-14T13:34:03.890794Z","shell.execute_reply":"2022-11-14T13:34:05.695848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset class","metadata":{}},{"cell_type":"code","source":"def preProcess(img_path):\n    \"\"\"Preprocess the image\n    Args:\n        img (string): image path\n\n    Returns:\n        cv2.Mat: image\n    \"\"\"\n    # read image\n    img = cv2.imread(img_path)\n\n    # image enhancement\n    hsv_img = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)\n\n    h, s, v = hsv_img[:,:,0], hsv_img[:,:,1], hsv_img[:,:,2]\n    clahe = cv2.createCLAHE(clipLimit = 15.0, tileGridSize = (20,20))\n    v = clahe.apply(v)\n\n    hsv_img = np.dstack((h,s,v))\n\n    img = cv2.cvtColor(hsv_img, cv2.COLOR_HSV2RGB)\n\n    # gamma\n    R = 255.0\n    img = (R * np.power(img.astype(np.uint32)/R, 1/0.6)).astype(np.uint8)\n\n    # sharpening\n    kernel = np.array([[0, -1, 0],\n                    [-1, 5,-1],\n                    [0, -1, 0]])\n    img = cv2.filter2D(src=img, ddepth=-1, kernel=kernel)\n\n    # improve contrast\n    alpha = 1.1 # Contrast control (1.0-3.0)\n    beta = 0 # Brightness control (0-100)\n    img = cv2.convertScaleAbs(img, alpha=alpha, beta=beta)\n\n    # reduce size\n    scale = 1\n    x = int(img.shape[0] * scale)\n    y = int(img.shape[1] * scale)\n    img = cv2.resize(img, dsize=(y, x), interpolation=cv2.INTER_AREA)\n\n    return img","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:05.698675Z","iopub.execute_input":"2022-11-14T13:34:05.700099Z","iopub.status.idle":"2022-11-14T13:34:05.712731Z","shell.execute_reply.started":"2022-11-14T13:34:05.698957Z","shell.execute_reply":"2022-11-14T13:34:05.711781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if DEBUGGING_PREPROCESS:\n    # random 5 images for preProcess eval\n    random_5_img = []\n    random_5_label = []\n\n    for _ in range(5):\n        idx = random.choice(range(0, len(train_set)))\n        random_5_img.append(TOTAL_images[idx])\n        random_5_label.append(TOTAL_labels[idx])\n\n    for i in range(len(random_5_img)):\n        img = preProcess(random_5_img[i])\n\n        print(str(random_5_label[i]))\n\n        # draw rectangle\n        org_img = cv2.imread(random_5_img[i])\n        org_img = cv2.cvtColor(org_img, cv2.COLOR_BGR2RGB)\n        for lable in random_5_label[i]:\n            x = lable[\"x\"]\n            y = lable[\"y\"]\n            width = lable[\"width\"]\n            height = lable[\"height\"]\n            start_point = (x-width, y-height)\n            end_point = (x+width, y+height)\n            org_img = cv2.rectangle(org_img, start_point, end_point, (255,255,0), 2)\n            img = cv2.rectangle(img, start_point, end_point, (255,255,0), 2)\n\n        plt.imshow(org_img)\n        plt.axis(\"off\")\n        plt.show()\n\n        plt.imshow(img)\n        plt.axis(\"off\")\n        plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:05.714637Z","iopub.execute_input":"2022-11-14T13:34:05.715072Z","iopub.status.idle":"2022-11-14T13:34:05.729766Z","shell.execute_reply.started":"2022-11-14T13:34:05.714998Z","shell.execute_reply":"2022-11-14T13:34:05.728401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"org_img = cv2.imread(train_set[3])\norg_img = cv2.cvtColor(org_img, cv2.COLOR_BGR2RGB)\nplt.imshow(org_img)\nplt.axis(\"off\")\nplt.show()\nprint(train_label[3])\na = np.array([list(idx.values()) for idx in train_label[3] ])\nprint(a)\nprint(a[:,2])","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:05.731861Z","iopub.execute_input":"2022-11-14T13:34:05.732552Z","iopub.status.idle":"2022-11-14T13:34:05.885596Z","shell.execute_reply.started":"2022-11-14T13:34:05.732496Z","shell.execute_reply":"2022-11-14T13:34:05.884727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Dataset:\n\n    def __init__(self, images,boxes,transforms=None):\n        self.images = images \n        self.boxes = boxes\n        self.transforms = transforms\n    def can_augment(self, boxes):\n        box_outside_image = ((boxes[:, 0] < 0).any() or (boxes[:, 1] < 0).any() \n                             or (boxes[:, 2] > 1280).any() or (boxes[:, 3] > 720).any())\n        return not box_outside_image\n\n    def get_boxes(self, i):\n        \"\"\"Returns boxes in the form of [x_min, y_min, x_max, y_max], requires conversion\"\"\"\n        \n        boxes = np.array([list(idx.values()) for idx in self.boxes[i] ]).astype(float) # Convert into list now\n        # Change from [x_min, y_min, w, h] to [x_min, y_min, x_max, y_max]\n        boxes[:, 2] = boxes[:, 0] + boxes[:, 2]\n        boxes[:, 3] = boxes[:, 1] + boxes[:, 3]\n        return boxes\n    \n    def get_image(self, i):\n        \"\"\"Gets the image\"\"\"\n        if TRAIN_PREPROCESS:\n            image = preProcess(self.images[i])\n        else:  \n            image = cv2.imread(self.images[i])\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n        image /= 255.0\n        return image\n    \n    def __getitem__(self, i):\n        '''\n        Return image and item\n        '''\n        image = self.get_image(i)\n        boxes = self.get_boxes(i)\n        \n        n_boxes = boxes.shape[0]\n        \n        # Calculate the area\n        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])\n        \n        \n        target = {\n            'boxes': torch.as_tensor(boxes, dtype=torch.float32),\n            # There is only one class\n            'labels': torch.ones((n_boxes,), dtype=torch.int64),           \n        }\n\n\n        sample = {\n            'image':image,\n            'bboxes': target['boxes'],\n            'labels': target['labels']\n        }\n#         if self.can_augment(boxes):\n#         sample = self.transforms(**sample)\n#         image = sample['image']\n\n        if n_boxes > 0:\n            target['boxes'] = torch.stack(tuple(map(torch.tensor, zip(*sample['bboxes'])))).permute(1, 0)\n#         image = ToTensorV2(p=1.0)(image=image)['image']\n        # Convert image to the size required in RCNN\n#         else:\n#             image = ToTensorV2(p=1.0)(image=image)['image']\n        return torch.Tensor(image).permute(2,0,1), target\n\n    def __len__(self):\n        return len(self.images)","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:05.887074Z","iopub.execute_input":"2022-11-14T13:34:05.887667Z","iopub.status.idle":"2022-11-14T13:34:05.905944Z","shell.execute_reply.started":"2022-11-14T13:34:05.887614Z","shell.execute_reply":"2022-11-14T13:34:05.905052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_train_transform():\n    return albumentations.Compose([\n        albumentations.Flip(0.5),\n        ToTensorV2(p=1.0)\n    ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})\n\n\ndef get_normal_transform():\n    return albumentations.Compose([\n        ToTensorV2(p=1.0)\n    ], bbox_params={'format': 'pascal_voc', 'label_fields': ['labels']})","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:05.90871Z","iopub.execute_input":"2022-11-14T13:34:05.909504Z","iopub.status.idle":"2022-11-14T13:34:05.920453Z","shell.execute_reply.started":"2022-11-14T13:34:05.908929Z","shell.execute_reply":"2022-11-14T13:34:05.919195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds_train = Dataset(train_set,train_label)\nds_val = Dataset(valid_set,valid_label)\nds_test = Dataset(test_set,test_label)","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:05.923672Z","iopub.execute_input":"2022-11-14T13:34:05.923935Z","iopub.status.idle":"2022-11-14T13:34:05.934972Z","shell.execute_reply.started":"2022-11-14T13:34:05.92389Z","shell.execute_reply":"2022-11-14T13:34:05.933961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Example plot","metadata":{}},{"cell_type":"code","source":"image,target = ds_train[1000]\nlen(ds_train)","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:05.937428Z","iopub.execute_input":"2022-11-14T13:34:05.93779Z","iopub.status.idle":"2022-11-14T13:34:05.974956Z","shell.execute_reply.started":"2022-11-14T13:34:05.937742Z","shell.execute_reply":"2022-11-14T13:34:05.97411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"boxes = target['boxes'].cpu().numpy().astype(np.int32)\nprint(image.shape)\nimg = image.permute(1,2,0).cpu().numpy() # conver tot normal dimension to plot it\nfig, ax = plt.subplots(1, 1, figsize=(16, 8))\nprint(img.shape)\ncolour = (0,20,0)\nfor box in boxes:\n    print(box[0],box[1],box[2],box[3])\n    print(img.shape)\n    cv2.rectangle(img,\n                  (box[0], box[1]),\n                  (box[2], box[3]),\n                  (220, 0, 0), 3)\n    \nax.set_axis_off()\nax.imshow(img);","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:05.978012Z","iopub.execute_input":"2022-11-14T13:34:05.978289Z","iopub.status.idle":"2022-11-14T13:34:06.412484Z","shell.execute_reply.started":"2022-11-14T13:34:05.978244Z","shell.execute_reply":"2022-11-14T13:34:06.408483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Faster R-CNN","metadata":{}},{"cell_type":"code","source":"\nLEARNING_RATE = 0.001\nBATCH_SIZE = 2\nIOU_TARGET = 0.45","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:06.413628Z","iopub.execute_input":"2022-11-14T13:34:06.414089Z","iopub.status.idle":"2022-11-14T13:34:06.419425Z","shell.execute_reply.started":"2022-11-14T13:34:06.414008Z","shell.execute_reply":"2022-11-14T13:34:06.418419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from torchvision import models\nfrom torchvision.models.detection.faster_rcnn import FastRCNNPredictor\nmodel = models.detection.fasterrcnn_resnet50_fpn()\nnum_class = 2  # object and background\n\n\nin_features = model.roi_heads.box_predictor.cls_score.in_features\n\nmodel.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_class)","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:06.420737Z","iopub.execute_input":"2022-11-14T13:34:06.421294Z","iopub.status.idle":"2022-11-14T13:34:07.621467Z","shell.execute_reply.started":"2022-11-14T13:34:06.421248Z","shell.execute_reply":"2022-11-14T13:34:07.62049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## DataLoaders","metadata":{}},{"cell_type":"code","source":"# prac_train_set = convert_all_image(train_set)\n# prac_val_set = convert_all_image(valid_set)\n# prac_test_set = convert_all_image(test_set)\ndef collate_fn(batch):\n    return tuple(zip(*batch))\n\n    \n# prac_train_label = convert_all_labels(train_label)\n# prac_val_label = convert_all_labels(valid_label)\n# prac_test_label = convert_all_labels(test_label)\ntrain_loader = DataLoader(ds_train, batch_size=4, shuffle=True, drop_last=False,pin_memory= True,num_workers = 2,collate_fn=collate_fn)\nval_loader = DataLoader(ds_val, batch_size=4, shuffle=True, drop_last=False,pin_memory= True,num_workers = 2,collate_fn=collate_fn)\ntest_loader = DataLoader(ds_test,batch_size = 4,shuffle = True,drop_last=False,pin_memory= True,num_workers = 2,collate_fn=collate_fn)","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:07.622814Z","iopub.execute_input":"2022-11-14T13:34:07.623166Z","iopub.status.idle":"2022-11-14T13:34:07.631748Z","shell.execute_reply.started":"2022-11-14T13:34:07.623114Z","shell.execute_reply":"2022-11-14T13:34:07.630772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_loader = DataLoader(dataset=prac_train_set, batch_size=BATCH_SIZE, shuffle=True, num_workers=0, drop_last=False,pin_memory= True)\n# val_loader = DataLoader(dataset=prac_val_set, batch_size=BATCH_SIZE, shuffle=True, num_workers=0, drop_last=False,pin_memory= True)\n# test_loader = DataLoader(dataset=prac_test_set, batch_size=BATCH_SIZE, shuffle=True, num_workers=0, drop_last=False,pin_memory= True)\n","metadata":{"execution":{"iopub.status.busy":"2022-11-14T13:34:07.633271Z","iopub.execute_input":"2022-11-14T13:34:07.633951Z","iopub.status.idle":"2022-11-14T13:34:07.643848Z","shell.execute_reply.started":"2022-11-14T13:34:07.633842Z","shell.execute_reply":"2022-11-14T13:34:07.643058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create the model","metadata":{}},{"cell_type":"code","source":"LOAD_MODEL = False","metadata":{"execution":{"iopub.status.busy":"2022-11-14T14:26:54.716566Z","iopub.execute_input":"2022-11-14T14:26:54.716909Z","iopub.status.idle":"2022-11-14T14:26:54.720561Z","shell.execute_reply.started":"2022-11-14T14:26:54.716855Z","shell.execute_reply":"2022-11-14T14:26:54.719689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def save_checkpoint(state,epoch):\n    # .../abc.pth.tar\n    base_filename = \"../working/models/\"\n    file_name = base_filename+\"model_epoch\"+str(epoch)+\".pth.tar\"\n    print(\"> Saving the State of model now\")\n    torch.save(state,file_name)","metadata":{"execution":{"iopub.status.busy":"2022-11-14T14:26:54.735444Z","iopub.execute_input":"2022-11-14T14:26:54.735718Z","iopub.status.idle":"2022-11-14T14:26:54.740799Z","shell.execute_reply.started":"2022-11-14T14:26:54.735671Z","shell.execute_reply":"2022-11-14T14:26:54.739549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_checkpoint(model,optimizer,checkpoint):\n    print(\"Loading State of Model now\")\n    model.load_state_dict(checkpoint['state'])\n    optimizer.load_state_dict(checkpoint['optimizer'])\n    epoch = checkpoint['epoch']\n    losses_list = checkpoint['losses_list']\n    return model,optimizer, epoch, losses_list","metadata":{"execution":{"iopub.status.busy":"2022-11-14T14:26:54.750562Z","iopub.execute_input":"2022-11-14T14:26:54.750818Z","iopub.status.idle":"2022-11-14T14:26:54.755803Z","shell.execute_reply.started":"2022-11-14T14:26:54.750773Z","shell.execute_reply":"2022-11-14T14:26:54.754773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_model(model,optimizer,file_name):\n    file_name = file_name # Need to change this\n    base_filename = \"../working/models/\"\n    checkout,optimizer,epoch,losses_list = load_checkpoint(model,optimizer,torch.load(base_filename+file_name))\n    return model, optimizer , epoch, losses_list\n    ","metadata":{"execution":{"iopub.status.busy":"2022-11-14T14:26:54.765559Z","iopub.execute_input":"2022-11-14T14:26:54.765815Z","iopub.status.idle":"2022-11-14T14:26:54.770689Z","shell.execute_reply.started":"2022-11-14T14:26:54.765771Z","shell.execute_reply":"2022-11-14T14:26:54.769494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"NUM_EPOCHS = 10\nBASE_EPOCHS = 9","metadata":{"execution":{"iopub.status.busy":"2022-11-14T14:26:54.776229Z","iopub.execute_input":"2022-11-14T14:26:54.776651Z","iopub.status.idle":"2022-11-14T14:26:54.781386Z","shell.execute_reply.started":"2022-11-14T14:26:54.7766Z","shell.execute_reply":"2022-11-14T14:26:54.780295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tqdm import tqdm\ndevice = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n\nmodel.to(device)\n\noptimizer = torch.optim.Adam(model.parameters(),LEARNING_RATE)\n\nval_list = []\nbase_epoch = 0\nif LOAD_MODEL:\n    model,optimzer,base_epoch,val_list = load_model(model,optimizer,\"model_epoch6.pth.tar\")\n#     model, optimizer = load_checkpoint(model, optimizer, losslogger)\n    model = model.to(device)\n    # now individually transfer the optimizer parts...\n    for state in optimizer.state.values():\n        for k, v in state.items():\n            if isinstance(v, torch.Tensor):\n                state[k] = v.to(device)\n\n                             \nt_batches = len(train_loader)\nv_batches = len(val_loader)    \n\ndef F_RCNN_val(valid_loader):\n    val_loss_sum = 0\n    loss = 0\n    with torch.no_grad():\n        for b_id ,(images,targets) in enumerate(valid_loader, 1):\n            images1 = list(image.to(device) for image in images)\n            targets1 = [{key: vaule.to(device) for key, vaule in t.items()} for t in targets] \n            val_loss_dict = model(images1, targets1)\n            val_loss = sum(loss for loss in val_loss_dict.values())\n            val_loss_value = val_loss.item()\n            val_loss_sum += val_loss_value\n\n        val_loss = val_loss_sum / v_batches\n#         wandb.log({\"validation_loss\": val_loss,\"epoch\":epoch})\n        print(f\"Validation loss is {val_loss}\")\n    return val_loss\n\n\ndef F_RCNN_train(train_loader,valid_loader,base_epoch,val_list):\n#     wandb.watch(model,log = \"all\")\n#     if val_list is None:\n#         val_list = []\n    for epoch in tqdm(range(NUM_EPOCHS)):\n        # Optional\n        \n        t_loss_sum = 0\n        for b_id ,(images,targets) in enumerate(train_loader, 1):\n            images1 = list(image.to(device) for image in images)\n            targets1 = [{key: vaule.to(device) for key, vaule in t.items()} for t in targets] \n\n            t_loss_dict = model(images1, targets1)\n            t_loss = sum(loss for loss in t_loss_dict.values())\n            t_loss_value = t_loss.item()\n            t_loss_sum += t_loss_value\n\n            optimizer.zero_grad()\n            t_loss.backward()\n            optimizer.step()\n        train_loss = t_loss_sum / t_batches\n        print(f\"[Epoch {epoch+base_epoch+1:2d} / {(NUM_EPOCHS+base_epoch):2d}] Train loss: {train_loss:.3f}\")\n        val_loss = F_RCNN_val(valid_loader)\n        val_list.append(val_loss)\n        if (epoch+base_epoch) % 3 ==0:\n            checkpoint = {'state':model.state_dict(),'optimizer':optimizer.state_dict(),\"epoch\":base_epoch+epoch+1,\"losses_list\":val_list}\n            save_checkpoint(checkpoint,epoch)\n#     wandb.save(\"../working/models/trained_model.onnx\")\n    return val_list\n    \n    \n","metadata":{"execution":{"iopub.status.busy":"2022-11-14T14:26:54.803888Z","iopub.execute_input":"2022-11-14T14:26:54.804507Z","iopub.status.idle":"2022-11-14T14:26:54.826298Z","shell.execute_reply.started":"2022-11-14T14:26:54.804458Z","shell.execute_reply":"2022-11-14T14:26:54.825443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train","metadata":{}},{"cell_type":"code","source":"\nF_RCNN_train(train_loader,val_loader,base_epoch,val_list)\n","metadata":{"execution":{"iopub.status.busy":"2022-11-14T14:26:54.829273Z","iopub.execute_input":"2022-11-14T14:26:54.829604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Check result","metadata":{}},{"cell_type":"code","source":"'''\nPass in the two box and compute intersection over union\n'''\ndef intersection_over_union(box1,box2):\n        \n    # in the form x,y,w,h\n    box1_x1 = box1[0]\n    box1_y1 = box1[1]\n    box1_x2 = box1[2]\n    box1_y2 = box1[3]\n    box2_x1 = box2[0]\n    box2_y1 = box2[1]\n    box2_x2 = box2[2]\n    box2_y2 = box2[3]\n\n    # Intersection key point\n    int_x1 = torch.max(box1[0],box2[0])\n    int_y1 = torch.max(box1[1],box2[1])\n    int_x2 = torch.min(box1[2],box2[2])\n    int_y2 = torch.min(box2[3],box2[3])\n    \n   \n    \n    # Intersection area\n    intersection = (int_x2-int_x1).clamp(0)*(int_y2-int_y1).clamp(0)\n    # Seperate area\n    box1_area = abs(box1_x2-box1_x1)*abs(box1_y2-box1_y1)\n    box2_area = abs(box2_x2-box2_x1)*abs(box2_y2-box2_y1)\n\n    \n    # Union\n    union = box1_area+box2_area-intersection\n    return intersection/(union+1e-6) #\n    ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%ls","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#!mkdir ../working/models","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nreal has shape(M,4)\npred has shape(N,4)\n'''\ndef calculate_iou_pair(box1,box2,used):\n    if box2 in used:\n        return 0\n    else:\n        return intersection_over_union(box1,box2)\n\n'''\nReturn  true positive, false positive and false negative in the image's boxes\n'''\ndef group_iou(real,pred,iou_target):\n    tp = 0\n    fp = 0\n    fn = 0\n    used = np.zeros(len(real))\n    for i,box1 in enumerate(real):\n        all_iou = [calculate_iou_pair(box1,box2,used)for box2 in pred]\n        if len(all_iou) == 0:\n            continue\n\n        max_iou = max(all_iou)\n        if max_iou > iou_target:\n            used[i] = all_iou[all_iou.index(max_iou)]\n            tp+=1\n    fp = len(pred)-tp\n    fn = len(real)-tp\n#     prediction = [1 for i in used if i!=0]\n    return tp,fp,fn\n    ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_metrics(tp,fp,fn):\n    if tp > 0:\n        precision = tp / (tp+fp)\n        recall = tp / (tp+fn)\n        f1_score = 2 * ((precision * recall) / (precision + recall))\n        f2_score =  (5 * precision * recall) / (4 * precision + recall)\n        iou = tp / (tp + fp + fn)\n    else:\n        f2_score = precision = recall = f1_score = iou = float('NaN')\n    return {\n        \"FP\": fp,\n        \"FN\": fn,\n        \"TP\": tp,\n        \"precision\": precision,\n        \"recall\": recall,\n        \"f1_score\": f1_score,\n        \"f2_score\":f2_score,\n        \"iou\": iou\n    }","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # model.eval()\n# for b_id,(images,targets) in enumerate(test_loader,1):\n#     a = images\n#     a = list(img.to(device) for img in a)\n#     print(a[0].shape)\n#     b = model([a[0]])\n#     print(b)\n#     print(targets[0]['boxes'].shape)\n#     print(type(b[0]['boxes']))\n#     print(b[0]['boxes'].shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\nIOU_TARGET = 0.4\nmodel.eval()\nwith torch.no_grad():\n    tp = fp = fn = 0\n    pred_label = []\n    true_label = []\n    for b_id,(images,targets) in enumerate(test_loader,1):\n        images = list(img.to(device) for img in images)\n        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n#         print(len(images))\n#         print(images[0].shape)\n        outputs = model(images)\n        \n        #outputs = [{k: v.detach().cpu() for k, v in t.items()} for t in outputs]\n        #print(f\"output type is {type(outputs)} and output is {outputs}\")\n        \n        for i, image in enumerate(images):\n            boxes = outputs[i]['boxes'].data\n            labels = outputs[i]['labels'].data\n            scores = outputs[i]['scores'].data\n            gt_boxes = targets[i]['boxes'].data\n            tp_temp,fp_temp,fn_temp = group_iou(gt_boxes,boxes,IOU_TARGET)\n            tp += tp_temp\n            fp += fp_temp\n            fn += fn_temp\n    print(f\"tp is {tp} fp is {fp} and fn is {fn}\")\n\n        \n        ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(get_metrics(tp,fp,fn))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"targets = [{k: v.to(DEVICE) for k, v in t.items()} for t in targets]\ntargets[0]['labels'].tolist()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"idx = 0\nimages = list(img.to(DEVICE) for img in images)\ntargets = [{k: v.to(DEVICE) for k, v in t.items()} for t in targets]\n\nboxes = targets[idx]['boxes'].cpu().numpy().astype(np.int32)\nsample = images[idx].permute(1,2,0).cpu().numpy()\n\nmodel.eval()\nprint(type(images[0]))\nprint(images[0].shape)\noutputs = model(images)\n# outputs = [{k: v.detach().cpu().numpy() for k, v in t.items()} for t in outputs]\nprint(outputs[0]['boxes'].data)\nprint(targets[0])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def RCNN_inference(images):\n#   \n\n\n    info = pd.DataFrame(columns = [\"xmin\",\"ymin\",\"xmax\",\"ymax\",\"confidence\",\"class\",\"name\"])\n    if type(images) is not list:\n        outputs = model([images])\n    else:\n        outputs= model(images)\n    print(outputs)\n    boxes = outputs[0]['boxes'].data\n    scores = outputs[0]['scores'].data\n\n\n    for index,value in enumerate(boxes):\n        info.loc[index] = [int(value[0].item()),int(value[1].item()),int(value[2].item()),int(value[3].item()),scores[index].item(),1,\"starfish\"]\n    return info\n#     x1,y1,x2,y2 = boxes[0].item(),boxes[1].item(),boxes[2].item(),boxes[3].item()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nprint(RCNN_inference(images))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#!ls ../input/tensorflow-great-barrier-reef/train_images/video_0/64.jpg","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"imgg = None\nfor b_id,(images,targets) in enumerate(test_loader,1):\n    imgg = images[0]\n    break\nprint(imgg)\nprint(type(imgg))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image = cv2.imread(\"../input/tensorflow-great-barrier-reef/train_images/video_0/1000.jpg\")\nimage = cv2.cvtColor(img, cv2.COLOR_BGR2RGB).astype(np.float32)\nimage /= 255.0\n# image = image.transpose(2,0,1)\n# image = torch.Tensor(image)\nimage = torch.Tensor(image).permute(2,0,1)\n# image = torch.from_numpy(image.transpose(2,0,1))\nimage = image.to(device)\n# print(image)\n# print(type(image))\nprint(RCNN_inference(image))\n\n\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nmode = 1\na = images[0]\n\n# img = a\ndef Plot_Inference_CNN(img_dir,num_items=-1, mode=1, out_dir=\"./inference\"):\n    \"\"\"Inference\n    Args:\n        img_dir: directory where images to infer\n        num_items (default = -1): random pick the num of images to infer if specified. -1 for all images.\n        mode (default = 1): 0 will output image to the \"inference\" directory. 1 will display inline.\n\n    Returns:\n        None\n    \"\"\"\n    if num_items == -1:\n        img_list = os.listdir(img_dir)\n    elif num_items >= 0:\n        img_list = random.sample(os.listdir(img_dir), num_items)\n    if mode == 0 and not os.path.exists(out_dir):\n        os.mkdir(out_dir)\n\n\n    fig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\n    for i in range(len(img_list)):\n        full_path = img_dir + '/' + img_list[i]\n        img = plt.imread(full_path)\n        results = model(full_path)\n        \n        #     for i in range(len(img_list)):\n        boxes = RCNN_inference(a)\n        img = a.permute(1,2,0).cpu().numpy() # conver tot normal dimension to plot it\n        for box in boxes.itertuples():\n            print(box)\n            color = np.random.choice(range(256), size=3).tolist()\n            # color = [0,0,245]\n            print(color)\n            label = f\"starfish {box.confidence:.2f}\"\n            # For bounding box\n            print(f\"type of image is {type(img)} shape is {img.shape}\")\n            img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)\n            print(f\"starfish xmin {box.xmin} ymin {box.ymin} xmax {box.xmax} and ymax {box.ymax}\")\n            img = cv2.rectangle(img, (int(box.xmin),int(box.ymin)),(int(box.xmax),int(box.ymax)), color =\"white\",thickness = 3)\n\n            # For the text background\n            # Finds space required by the text so that we can put a background with that amount of width.\n            (w, h), _ = cv2.getTextSize(\n                    label, cv2.FONT_HERSHEY_SIMPLEX, 0.6, 1)\n            # Prints the text.\n            img = cv2.rectangle(img, (box.xmin, box.ymin - 20), (box.xmin + w, box.ymin), color, -1)\n\n            # For printing text\n            img = cv2.putText(img, label, (box.xmin, box.ymin),\n                                cv2.FONT_HERSHEY_SIMPLEX, 0.6, (255,255,255), 2)\n        plt.figure(figsize=(18,10))\n\n        #         plt.subplot(1,2,i%2 + 1)\n        plt.axis(\"off\")\n        plt.text(50, 50, f\"starfishes count: {boxes.shape[0]}\", color=\"white\", fontsize=15)\n        if mode:\n            plt.imshow(img)\n        else:\n            plt.savefig(f\"{out_dir}/{img_list[i]}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nPlot the outcome\n'''\ndef plot_and_compare_outcome(image,real,pred):\n    fig, ax = plt.subplots(1, 1, figsize=(16, 8))\n    count_actual = len(real)\n    count_predict = len(pred)\n    # Red for ground truth\n    for box in boxes:\n        cv2.rectangle(sample,(box[0], box[1]),\n                      (box[2], box[3]),\n                      (255, 0, 0), 3)\n\n\n    # Green for predictions\n    # Print the first 5\n    #    for box in outputs[idx]['boxes'][:5]:\n\n    for box in pred:\n        cv2.rectangle(sample,(box[0], box[1]),\n                      (box[2], box[3]),\n                      (0, 255, 0), 3)\n    add_string = f\"Actual {count_actual}, Predicted(count_predict)\"\n#     plt.text( add_string, bbox=dict(fill=False, edgecolor='red', linewidth=2))\n    ax.set_axis_off()\n    ax.imshow(sample);\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_and_compute_outcome(image[0],targets[])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Please, _DO_ upvote if you found it useful!","metadata":{}}]}