{
  "id": 427834,
  "title": "Newbie - Help Debugging COCO Evaluation",
  "url": "/competitions/hubmap-hacking-the-human-vasculature/discussion/427834",
  "author_name": "Duck-Bongos",
  "post_date": "2023-07-29T21:55:55.491000",
  "votes": 0,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hi All, </p>\n<p><em>TL;DR</em>: My evaluation step doesn't work. I think it's because of how I'm loading the data. I think at this point, my options for evaluation are:</p>\n<ol>\n<li>Debug the issue with the data loading (preferred)</li>\n<li>Change the evaluation function</li>\n<li><em>Open to suggestions</em></li>\n</ol>\n<p>I'm excited that after years of lurking, I'm finally participating in a competition! I'm new to Machine Learning and PyTorch and wanted to get more experience doing both! My goal is to submit a model that can make predictions, making mistakes and learning a lot along the way! So far so good 😅 I decided to try and follow the <a href=\"https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html\" target=\"_blank\">PyTorch MaskRCNN tutorial</a> because it seemed close to what I want to do with this. It's been fun so far, but I've run into an issue with the evaluation step and COCO. Frankly I'm not even sure I'm allowed to ask this question and if I'm not, sorry! Here goes:</p>\n<h2>What I see</h2>\n<p>The error I'm getting is below, which is <a href=\"https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/coco.py#L324\" target=\"_blank\">in line 324</a> of the <code>pycocotools/coco.py</code> file, which I import for evaluation.</p>\n<pre><code> (annsImgIds)  ((annsImgIds)  (self.getImgIds())), \\\n                  correspond  current coco '\n</code></pre>\n<p>When I debugged the error, I found something odd - not only are the set sizes different, the types are different:</p>\n<pre><code> \n{tensor([344])}\n \n{0, 1, 2, 3, 4, 5, 6, ... }  # len &gt; 500, depending\n</code></pre>\n<h2>Dataset and DataLoader</h2>\n<p>Here's my Dataset and DataLoader code. I include only one import because everything else is boiler plate, the import I included comes from a <a href=\"https://github.com/pytorch/vision/blob/main/references/detection/engine.py\" target=\"_blank\">torchvision reference file</a> used in the aforementioned tutorial.</p>\n<h2>DataLoader setup and Training Loop</h2>\n<pre><code> engine import evaluate, train_one_epoch\n\ndef my_collate(batch):\n    data = [item[0]  item  batch]\n    target = [item[1]  item  batch]\n    data = torch.stack(data)\n    # target = torch.LongTensor(target)\n\n    return [data, target]\n\nhd_train = HubmapDataset(\n        =,\n        =,\n        =get_transform(train=True),\n    )\n    hd_test = HubmapDataset(\n        =,\n        =,\n        =get_transform(train=False),\n    )\n\n    dataset_size = len(hd_train)\n    indices = torch.randperm(dataset_size).tolist()\n    dataset = torch.utils.data.Subset(hd_train, indices[: -int(dataset_size * 0.99)])\n    dataset_test = torch.utils.data.Subset(\n        hd_test, indices[-int(dataset_size * 0.99) :]\n    )\n\n    data_loader = DataLoader(\n        dataset,\n        =2,\n        =,\n        =1,\n        =my_collate,\n        =,\n    )\n    data_loader_test = DataLoader(\n        dataset_test,\n        =1,\n        =,\n        =1,\n        =my_collate,\n        =,\n    )\n\n\n    running_loss = 0\n    num_epochs = 10\n\n     epoch  range(num_epochs):\n        # train  one epoch, printing every 10 iterations\n        train_one_epoch(model, optimizer, data_loader, device, epoch, =10)\n        # update the learning rate\n        lr_scheduler.()\n        # evaluate on the test dataset\n        evaluate(model, data_loader_test, =device)\n\n    ()\n</code></pre>\n<h3>Dataset setup</h3>\n<pre><code> :\n    def  -&gt; None:\n        super.\n        self.data_dir = data_dir\n        self.annotations = self.\n        self.transforms = transforms\n        self._labels = {\n            : ,\n            : ,\n            : ,\n        }\n        self.image_list =   self.annotations\n        ]  # might need  filter this down\n\n    def :\n        return (box - box)(box - box)\n\n    def  -&gt; Dict:\n        l = \n         (fp)  polygon:\n            j = polygon.read\n            l = j.split()\n        z = {}\n         i, row  enumerate(l):\n            :\n                r = json.loads(row)\n                z] = r\n            except json.JSONDecodeError  jde:\n                print(i, jde, row)\n        return z\n\n    def :\n        xmin = (np.min(coords))\n        ymin = (np.min(coords))\n        xmax = (np.max(coords))\n        ymax = (np.max(coords))\n        return \n\n    def :\n        return len(self.image_list)\n\n    def :\n        :\n            torch.where(labels &gt; )\n            return True\n        except Exception:\n            return False\n\n    def :\n        \n        labels_ = labels.copy\n\n         len(np.unique(labels)):\n            labels_ = np.zeros(labels.shape)\n\n        elif np.min(labels):\n            labels_ = labels - \n\n        elif np.min(labels):\n            labels_ = labels - \n\n        elif np.min(labels)  len(np.where(labels)):\n            labels] = \n            labels_ = labels\n\n         len(np.unique(labels_)) - np.max(labels_)\n\n        return labels_\n\n    def  -&gt; Any:\n        img_name = self.image_list\n        image_path = os.path.join(self.data_dir, img_name)\n        image = (image_path)\n\n        annotations = self.annotations]\n        num_objs = len(annotations)\n\n        # create the masks  size (,,num_objs)\n        masks = np.zeros((num_objs, , ), dtype=np.uint8)\n        boxes = num_objs\n        areas = num_objs\n        labels = \n\n        #  each mask, add labels, boxes\n         i  range(num_objs):\n            l_type = annotations\n            label_color = self._labels\n            coords = np.(annotations)\n\n            # set the mask coordinates equal  the label color\n            m = np.zeros((, ))\n            m, coords] = label_color\n            cv2.fill\n            masks = m\n\n            # update the label\n            labels.append(label_color)\n\n            # create the bounding boxes\n            bbox = self.\n            areas = self.\n            boxes = bbox\n\n        labels = self.)\n\n        # labels = np.((np.unique(labels)))\n\n        target = {}\n        target = torch.\n        target = torch.\n        target = torch.\n        target = torch.\n         target.shapenum_objs\n        target = torch.tensor()\n\n         self.transforms is not None:\n            image, target = self.transforms(image, target)\n        :\n            image = (image)\n\n        return image, target\n</code></pre>\n<p>Any helpful pointers would be greatly appreciated! I'd love to reciprocate in the future.</p>\n<p>Happy Coding!</p>",
  "messages": [
    {
      "id": 2365041,
      "postDate": "2023-07-29T21:55:55.490Z",
      "content": "<p>Hi All, </p>\n<p><em>TL;DR</em>: My evaluation step doesn't work. I think it's because of how I'm loading the data. I think at this point, my options for evaluation are:</p>\n<ol>\n<li>Debug the issue with the data loading (preferred)</li>\n<li>Change the evaluation function</li>\n<li><em>Open to suggestions</em></li>\n</ol>\n<p>I'm excited that after years of lurking, I'm finally participating in a competition! I'm new to Machine Learning and PyTorch and wanted to get more experience doing both! My goal is to submit a model that can make predictions, making mistakes and learning a lot along the way! So far so good 😅 I decided to try and follow the <a href=\"https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html\" target=\"_blank\">PyTorch MaskRCNN tutorial</a> because it seemed close to what I want to do with this. It's been fun so far, but I've run into an issue with the evaluation step and COCO. Frankly I'm not even sure I'm allowed to ask this question and if I'm not, sorry! Here goes:</p>\n<h2>What I see</h2>\n<p>The error I'm getting is below, which is <a href=\"https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/coco.py#L324\" target=\"_blank\">in line 324</a> of the <code>pycocotools/coco.py</code> file, which I import for evaluation.</p>\n<pre><code> (annsImgIds)  ((annsImgIds)  (self.getImgIds())), \\\n                  correspond  current coco '\n</code></pre>\n<p>When I debugged the error, I found something odd - not only are the set sizes different, the types are different:</p>\n<pre><code> \n{tensor([344])}\n \n{0, 1, 2, 3, 4, 5, 6, ... }  # len &gt; 500, depending\n</code></pre>\n<h2>Dataset and DataLoader</h2>\n<p>Here's my Dataset and DataLoader code. I include only one import because everything else is boiler plate, the import I included comes from a <a href=\"https://github.com/pytorch/vision/blob/main/references/detection/engine.py\" target=\"_blank\">torchvision reference file</a> used in the aforementioned tutorial.</p>\n<h2>DataLoader setup and Training Loop</h2>\n<pre><code> engine import evaluate, train_one_epoch\n\ndef my_collate(batch):\n    data = [item[0]  item  batch]\n    target = [item[1]  item  batch]\n    data = torch.stack(data)\n    # target = torch.LongTensor(target)\n\n    return [data, target]\n\nhd_train = HubmapDataset(\n        =,\n        =,\n        =get_transform(train=True),\n    )\n    hd_test = HubmapDataset(\n        =,\n        =,\n        =get_transform(train=False),\n    )\n\n    dataset_size = len(hd_train)\n    indices = torch.randperm(dataset_size).tolist()\n    dataset = torch.utils.data.Subset(hd_train, indices[: -int(dataset_size * 0.99)])\n    dataset_test = torch.utils.data.Subset(\n        hd_test, indices[-int(dataset_size * 0.99) :]\n    )\n\n    data_loader = DataLoader(\n        dataset,\n        =2,\n        =,\n        =1,\n        =my_collate,\n        =,\n    )\n    data_loader_test = DataLoader(\n        dataset_test,\n        =1,\n        =,\n        =1,\n        =my_collate,\n        =,\n    )\n\n\n    running_loss = 0\n    num_epochs = 10\n\n     epoch  range(num_epochs):\n        # train  one epoch, printing every 10 iterations\n        train_one_epoch(model, optimizer, data_loader, device, epoch, =10)\n        # update the learning rate\n        lr_scheduler.()\n        # evaluate on the test dataset\n        evaluate(model, data_loader_test, =device)\n\n    ()\n</code></pre>\n<h3>Dataset setup</h3>\n<pre><code> :\n    def  -&gt; None:\n        super.\n        self.data_dir = data_dir\n        self.annotations = self.\n        self.transforms = transforms\n        self._labels = {\n            : ,\n            : ,\n            : ,\n        }\n        self.image_list =   self.annotations\n        ]  # might need  filter this down\n\n    def :\n        return (box - box)(box - box)\n\n    def  -&gt; Dict:\n        l = \n         (fp)  polygon:\n            j = polygon.read\n            l = j.split()\n        z = {}\n         i, row  enumerate(l):\n            :\n                r = json.loads(row)\n                z] = r\n            except json.JSONDecodeError  jde:\n                print(i, jde, row)\n        return z\n\n    def :\n        xmin = (np.min(coords))\n        ymin = (np.min(coords))\n        xmax = (np.max(coords))\n        ymax = (np.max(coords))\n        return \n\n    def :\n        return len(self.image_list)\n\n    def :\n        :\n            torch.where(labels &gt; )\n            return True\n        except Exception:\n            return False\n\n    def :\n        \n        labels_ = labels.copy\n\n         len(np.unique(labels)):\n            labels_ = np.zeros(labels.shape)\n\n        elif np.min(labels):\n            labels_ = labels - \n\n        elif np.min(labels):\n            labels_ = labels - \n\n        elif np.min(labels)  len(np.where(labels)):\n            labels] = \n            labels_ = labels\n\n         len(np.unique(labels_)) - np.max(labels_)\n\n        return labels_\n\n    def  -&gt; Any:\n        img_name = self.image_list\n        image_path = os.path.join(self.data_dir, img_name)\n        image = (image_path)\n\n        annotations = self.annotations]\n        num_objs = len(annotations)\n\n        # create the masks  size (,,num_objs)\n        masks = np.zeros((num_objs, , ), dtype=np.uint8)\n        boxes = num_objs\n        areas = num_objs\n        labels = \n\n        #  each mask, add labels, boxes\n         i  range(num_objs):\n            l_type = annotations\n            label_color = self._labels\n            coords = np.(annotations)\n\n            # set the mask coordinates equal  the label color\n            m = np.zeros((, ))\n            m, coords] = label_color\n            cv2.fill\n            masks = m\n\n            # update the label\n            labels.append(label_color)\n\n            # create the bounding boxes\n            bbox = self.\n            areas = self.\n            boxes = bbox\n\n        labels = self.)\n\n        # labels = np.((np.unique(labels)))\n\n        target = {}\n        target = torch.\n        target = torch.\n        target = torch.\n        target = torch.\n         target.shapenum_objs\n        target = torch.tensor()\n\n         self.transforms is not None:\n            image, target = self.transforms(image, target)\n        :\n            image = (image)\n\n        return image, target\n</code></pre>\n<p>Any helpful pointers would be greatly appreciated! I'd love to reciprocate in the future.</p>\n<p>Happy Coding!</p>",
      "rawMarkdown": "Hi All, \n\n_TL;DR_: My evaluation step doesn't work. I think it's because of how I'm loading the data. I think at this point, my options for evaluation are:\n1. Debug the issue with the data loading (preferred)\n2. Change the evaluation function\n3. *Open to suggestions*\n\n\nI'm excited that after years of lurking, I'm finally participating in a competition! I'm new to Machine Learning and PyTorch and wanted to get more experience doing both! My goal is to submit a model that can make predictions, making mistakes and learning a lot along the way! So far so good 😅 I decided to try and follow the [PyTorch MaskRCNN tutorial](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html) because it seemed close to what I want to do with this. It's been fun so far, but I've run into an issue with the evaluation step and COCO. Frankly I'm not even sure I'm allowed to ask this question and if I'm not, sorry! Here goes:\n\n## What I see\nThe error I'm getting is below, which is [in line 324](https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/coco.py#L324) of the `pycocotools/coco.py` file, which I import for evaluation.\n```\nassert set(annsImgIds) == (set(annsImgIds) & set(self.getImgIds())), \\\n               'Results do not correspond to current coco set'\n```\n\nWhen I debugged the error, I found something odd - not only are the set sizes different, the types are different:\n```\n>>> set(annsImgIds)\n{tensor([344])}\n>>> self.getImgIds()\n{0, 1, 2, 3, 4, 5, 6, ... }  # len > 500, depending\n```\n\n## Dataset and DataLoader\nHere's my Dataset and DataLoader code. I include only one import because everything else is boiler plate, the import I included comes from a [torchvision reference file](https://github.com/pytorch/vision/blob/main/references/detection/engine.py) used in the aforementioned tutorial.\n\n## DataLoader setup and Training Loop\n```\nfrom engine import evaluate, train_one_epoch\n\ndef my_collate(batch):\n    data = [item[0] for item in batch]\n    target = [item[1] for item in batch]\n    data = torch.stack(data)\n    # target = torch.LongTensor(target)\n\n    return [data, target]\n\nhd_train = HubmapDataset(\n        data_dir=\"train/\",\n        annotations_path=\"polygons.jsonl\",\n        transforms=get_transform(train=True),\n    )\n    hd_test = HubmapDataset(\n        data_dir=\"train/\",\n        annotations_path=\"polygons.jsonl\",\n        transforms=get_transform(train=False),\n    )\n\n    dataset_size = len(hd_train)\n    indices = torch.randperm(dataset_size).tolist()\n    dataset = torch.utils.data.Subset(hd_train, indices[: -int(dataset_size * 0.99)])\n    dataset_test = torch.utils.data.Subset(\n        hd_test, indices[-int(dataset_size * 0.99) :]\n    )\n\n    data_loader = DataLoader(\n        dataset,\n        batch_size=2,\n        shuffle=True,\n        num_workers=1,\n        collate_fn=my_collate,\n        pin_memory=True,\n    )\n    data_loader_test = DataLoader(\n        dataset_test,\n        batch_size=1,\n        shuffle=True,\n        num_workers=1,\n        collate_fn=my_collate,\n        pin_memory=True,\n    )\n\n# TRAININT SET UP\n    running_loss = 0\n    num_epochs = 10\n\n    for epoch in range(num_epochs):\n        # train for one epoch, printing every 10 iterations\n        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)\n        # update the learning rate\n        lr_scheduler.step()\n        # evaluate on the test dataset\n        evaluate(model, data_loader_test, device=device)\n\n    print(\"Complete!\")\n```\n\n### Dataset setup\n```\nclass HubmapDataset(Dataset):\n    def __init__(self, data_dir: str, annotations_path: str, transforms=None) -> None:\n        super().__init__()\n        self.data_dir = data_dir\n        self.annotations = self._extract_annotations(annotations_path)\n        self.transforms = transforms\n        self._labels = {\n            \"blood_vessel\": 0,\n            \"glomerulus\": 1,\n            \"unsure\": 2,\n        }\n        self.image_list = [\n            f for f in os.listdir(data_dir) if f[:-4] in self.annotations\n        ]  # might need to filter this down\n\n    def _calc_area(self, box):\n        return (box[2] - box[0]) * (box[3] - box[1])\n\n    def _extract_annotations(self, fp) -> Dict[str, Any]:\n        l = []\n        with open(fp) as polygon:\n            j = polygon.read()\n            l = j.split(\"\\n\")\n        z = {}\n        for i, row in enumerate(l):\n            try:\n                r = json.loads(row)\n                z[r[\"id\"]] = r[\"annotations\"]\n            except json.JSONDecodeError as jde:\n                print(i, jde, row)\n        return z\n\n    def _get_bbox(self, coords: np.ndarray):\n        xmin = int(np.min(coords[:, 1]))\n        ymin = int(np.min(coords[:, 0]))\n        xmax = int(np.max(coords[:, 1]))\n        ymax = int(np.max(coords[:, 0]))\n        return [xmin, ymin, xmax, ymax]\n\n    def __len__(self):\n        return len(self.image_list)\n\n    def _valid_labels(self, labels):\n        try:\n            torch.where(labels > 0)[0]\n            return True\n        except Exception:\n            return False\n\n    def _convert_labels(self, labels: np.ndarray):\n        \"\"\"Convert labels to be from 0-2\"\"\"\n        labels_ = labels.copy()\n\n        if len(np.unique(labels)) == 1:\n            labels_ = np.zeros(labels.shape)\n\n        elif np.min(labels) == 2:\n            labels_ = labels - 2\n\n        elif np.min(labels) == 1:\n            labels_ = labels - 1\n\n        elif np.min(labels) == 0 and len(np.where(labels == 1)[0]) == 0:\n            labels[np.where(labels == 2)[0]] = 1\n            labels_ = labels\n\n        assert len(np.unique(labels_)) - 1 == np.max(labels_)\n\n        return labels_\n\n    def __getitem__(self, index) -> Any:\n        img_name = self.image_list[index]\n        image_path = os.path.join(self.data_dir, img_name)\n        image = Image.open(image_path)\n\n        annotations = self.annotations[img_name[:-4]]\n        num_objs = len(annotations)\n\n        # create the masks of size (512,512,num_objs)\n        masks = np.zeros((num_objs, 512, 512), dtype=np.uint8)\n        boxes = [None] * num_objs\n        areas = [None] * num_objs\n        labels = []\n\n        # for each mask, add labels, boxes\n        for i in range(num_objs):\n            l_type = annotations[i][\"type\"]\n            label_color = self._labels[l_type]\n            coords = np.array(annotations[i][\"coordinates\"])[0]\n\n            # set the mask coordinates equal to the label color\n            m = np.zeros((512, 512))\n            m[coords[:, 1], coords[:, 0]] = label_color\n            cv2.fillPoly(m, pts=[coords], color=label_color)\n            masks[i, :, :] = m\n\n            # update the label\n            labels.append(label_color)\n\n            # create the bounding boxes\n            bbox = self._get_bbox(coords)\n            areas[i] = self._calc_area(bbox)\n            boxes[i] = bbox\n\n        labels = self._convert_labels(np.array(labels))\n\n        # labels = np.array(list(np.unique(labels)))\n\n        target = {}\n        target[\"boxes\"] = torch.as_tensor(boxes, dtype=torch.float32)\n        target[\"area\"] = torch.as_tensor(areas, dtype=torch.float32)\n        target[\"labels\"] = torch.as_tensor(labels, dtype=torch.int64)\n        target[\"masks\"] = torch.as_tensor(masks, dtype=torch.uint8)\n        assert target[\"masks\"].shape[0] == num_objs\n        target[\"image_id\"] = torch.tensor([index])\n\n        if self.transforms is not None:\n            image, target = self.transforms(image, target)\n        else:\n            image = PILToTensor()(image)\n\n        return image, target\n\n```\n\nAny helpful pointers would be greatly appreciated! I'd love to reciprocate in the future.\n\nHappy Coding!"
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2365041": "Hi All, \n\n_TL;DR_: My evaluation step doesn't work. I think it's because of how I'm loading the data. I think at this point, my options for evaluation are:\n1. Debug the issue with the data loading (preferred)\n2. Change the evaluation function\n3. *Open to suggestions*\n\n\nI'm excited that after years of lurking, I'm finally participating in a competition! I'm new to Machine Learning and PyTorch and wanted to get more experience doing both! My goal is to submit a model that can make predictions, making mistakes and learning a lot along the way! So far so good 😅 I decided to try and follow the [PyTorch MaskRCNN tutorial](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html) because it seemed close to what I want to do with this. It's been fun so far, but I've run into an issue with the evaluation step and COCO. Frankly I'm not even sure I'm allowed to ask this question and if I'm not, sorry! Here goes:\n\n## What I see\nThe error I'm getting is below, which is [in line 324](https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/coco.py#L324) of the `pycocotools/coco.py` file, which I import for evaluation.\n```\nassert set(annsImgIds) == (set(annsImgIds) & set(self.getImgIds())), \\\n               'Results do not correspond to current coco set'\n```\n\nWhen I debugged the error, I found something odd - not only are the set sizes different, the types are different:\n```\n>>> set(annsImgIds)\n{tensor([344])}\n>>> self.getImgIds()\n{0, 1, 2, 3, 4, 5, 6, ... }  # len > 500, depending\n```\n\n## Dataset and DataLoader\nHere's my Dataset and DataLoader code. I include only one import because everything else is boiler plate, the import I included comes from a [torchvision reference file](https://github.com/pytorch/vision/blob/main/references/detection/engine.py) used in the aforementioned tutorial.\n\n## DataLoader setup and Training Loop\n```\nfrom engine import evaluate, train_one_epoch\n\ndef my_collate(batch):\n    data = [item[0] for item in batch]\n    target = [item[1] for item in batch]\n    data = torch.stack(data)\n    # target = torch.LongTensor(target)\n\n    return [data, target]\n\nhd_train = HubmapDataset(\n        data_dir=\"train/\",\n        annotations_path=\"polygons.jsonl\",\n        transforms=get_transform(train=True),\n    )\n    hd_test = HubmapDataset(\n        data_dir=\"train/\",\n        annotations_path=\"polygons.jsonl\",\n        transforms=get_transform(train=False),\n    )\n\n    dataset_size = len(hd_train)\n    indices = torch.randperm(dataset_size).tolist()\n    dataset = torch.utils.data.Subset(hd_train, indices[: -int(dataset_size * 0.99)])\n    dataset_test = torch.utils.data.Subset(\n        hd_test, indices[-int(dataset_size * 0.99) :]\n    )\n\n    data_loader = DataLoader(\n        dataset,\n        batch_size=2,\n        shuffle=True,\n        num_workers=1,\n        collate_fn=my_collate,\n        pin_memory=True,\n    )\n    data_loader_test = DataLoader(\n        dataset_test,\n        batch_size=1,\n        shuffle=True,\n        num_workers=1,\n        collate_fn=my_collate,\n        pin_memory=True,\n    )\n\n# TRAININT SET UP\n    running_loss = 0\n    num_epochs = 10\n\n    for epoch in range(num_epochs):\n        # train for one epoch, printing every 10 iterations\n        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)\n        # update the learning rate\n        lr_scheduler.step()\n        # evaluate on the test dataset\n        evaluate(model, data_loader_test, device=device)\n\n    print(\"Complete!\")\n```\n\n### Dataset setup\n```\nclass HubmapDataset(Dataset):\n    def __init__(self, data_dir: str, annotations_path: str, transforms=None) -> None:\n        super().__init__()\n        self.data_dir = data_dir\n        self.annotations = self._extract_annotations(annotations_path)\n        self.transforms = transforms\n        self._labels = {\n            \"blood_vessel\": 0,\n            \"glomerulus\": 1,\n            \"unsure\": 2,\n        }\n        self.image_list = [\n            f for f in os.listdir(data_dir) if f[:-4] in self.annotations\n        ]  # might need to filter this down\n\n    def _calc_area(self, box):\n        return (box[2] - box[0]) * (box[3] - box[1])\n\n    def _extract_annotations(self, fp) -> Dict[str, Any]:\n        l = []\n        with open(fp) as polygon:\n            j = polygon.read()\n            l = j.split(\"\\n\")\n        z = {}\n        for i, row in enumerate(l):\n            try:\n                r = json.loads(row)\n                z[r[\"id\"]] = r[\"annotations\"]\n            except json.JSONDecodeError as jde:\n                print(i, jde, row)\n        return z\n\n    def _get_bbox(self, coords: np.ndarray):\n        xmin = int(np.min(coords[:, 1]))\n        ymin = int(np.min(coords[:, 0]))\n        xmax = int(np.max(coords[:, 1]))\n        ymax = int(np.max(coords[:, 0]))\n        return [xmin, ymin, xmax, ymax]\n\n    def __len__(self):\n        return len(self.image_list)\n\n    def _valid_labels(self, labels):\n        try:\n            torch.where(labels > 0)[0]\n            return True\n        except Exception:\n            return False\n\n    def _convert_labels(self, labels: np.ndarray):\n        \"\"\"Convert labels to be from 0-2\"\"\"\n        labels_ = labels.copy()\n\n        if len(np.unique(labels)) == 1:\n            labels_ = np.zeros(labels.shape)\n\n        elif np.min(labels) == 2:\n            labels_ = labels - 2\n\n        elif np.min(labels) == 1:\n            labels_ = labels - 1\n\n        elif np.min(labels) == 0 and len(np.where(labels == 1)[0]) == 0:\n            labels[np.where(labels == 2)[0]] = 1\n            labels_ = labels\n\n        assert len(np.unique(labels_)) - 1 == np.max(labels_)\n\n        return labels_\n\n    def __getitem__(self, index) -> Any:\n        img_name = self.image_list[index]\n        image_path = os.path.join(self.data_dir, img_name)\n        image = Image.open(image_path)\n\n        annotations = self.annotations[img_name[:-4]]\n        num_objs = len(annotations)\n\n        # create the masks of size (512,512,num_objs)\n        masks = np.zeros((num_objs, 512, 512), dtype=np.uint8)\n        boxes = [None] * num_objs\n        areas = [None] * num_objs\n        labels = []\n\n        # for each mask, add labels, boxes\n        for i in range(num_objs):\n            l_type = annotations[i][\"type\"]\n            label_color = self._labels[l_type]\n            coords = np.array(annotations[i][\"coordinates\"])[0]\n\n            # set the mask coordinates equal to the label color\n            m = np.zeros((512, 512))\n            m[coords[:, 1], coords[:, 0]] = label_color\n            cv2.fillPoly(m, pts=[coords], color=label_color)\n            masks[i, :, :] = m\n\n            # update the label\n            labels.append(label_color)\n\n            # create the bounding boxes\n            bbox = self._get_bbox(coords)\n            areas[i] = self._calc_area(bbox)\n            boxes[i] = bbox\n\n        labels = self._convert_labels(np.array(labels))\n\n        # labels = np.array(list(np.unique(labels)))\n\n        target = {}\n        target[\"boxes\"] = torch.as_tensor(boxes, dtype=torch.float32)\n        target[\"area\"] = torch.as_tensor(areas, dtype=torch.float32)\n        target[\"labels\"] = torch.as_tensor(labels, dtype=torch.int64)\n        target[\"masks\"] = torch.as_tensor(masks, dtype=torch.uint8)\n        assert target[\"masks\"].shape[0] == num_objs\n        target[\"image_id\"] = torch.tensor([index])\n\n        if self.transforms is not None:\n            image, target = self.transforms(image, target)\n        else:\n            image = PILToTensor()(image)\n\n        return image, target\n\n```\n\nAny helpful pointers would be greatly appreciated! I'd love to reciprocate in the future.\n\nHappy Coding!"
  }
}