I'm a big fan of @hengck23 's public solution threads, and always find them to be full of useful information, so in that spirit I will try a solution and keep track of it publicly, at least for a while.
\nSince Meta released their SegmentAnything Model (SAM), I've been looking for an excuse to play around with it, so that's what I'll do here.
\nSo the interesting thing about SAM, is that it's been designed to be prompted, i.e. given an image and a point/box/mask, it will return a mask. In the SAM paper , they point out this can be done by hand or programmatically.
\nNaturally the latter is more relevant here, so how about we train a lightweight object detection model (e.g. YOLO) to find objects, and then use SAM to refine the masks? The weakness here might be that we rely quite heavily on the object detector to do a good job first. Is it even necessary? As SAM can do automatic mask generation.
\nMy first attempt is here, and naively takes a mask and box from YOLOv7 as prompts, and then tries do predict a segmentation masks.
\nSo far, it does worse than just using using the YOLOv7 masks directly (LB 0.25).
Next, some fine-tuning of SAM?
", "messages": [ { "id": "2296060", "postDate": "06/11/2023 14:11:14", "content": "I'm a big fan of @hengck23 's public solution threads, and always find them to be full of useful information, so in that spirit I will try a solution and keep track of it publicly, at least for a while.
\nSince Meta released their SegmentAnything Model (SAM), I've been looking for an excuse to play around with it, so that's what I'll do here.
\nSo the interesting thing about SAM, is that it's been designed to be prompted, i.e. given an image and a point/box/mask, it will return a mask. In the SAM paper , they point out this can be done by hand or programmatically.
\nNaturally the latter is more relevant here, so how about we train a lightweight object detection model (e.g. YOLO) to find objects, and then use SAM to refine the masks? The weakness here might be that we rely quite heavily on the object detector to do a good job first. Is it even necessary? As SAM can do automatic mask generation.
\nMy first attempt is here, and naively takes a mask and box from YOLOv7 as prompts, and then tries do predict a segmentation masks.
\nSo far, it does worse than just using using the YOLOv7 masks directly (LB 0.25).
Next, some fine-tuning of SAM?
", "rawMarkdown": "I'm a big fan of @hengck23 's [public](https://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/332941) [solution](https://www.kaggle.com/competitions/champs-scalar-coupling/discussion/93972) [threads](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/370333), and always find them to be full of useful information, so in that spirit I will try a solution and keep track of it publicly, at least for a while. \n\nSince Meta released their [SegmentAnything Model](https://github.com/facebookresearch/segment-anything/tree/main) (SAM), I've been looking for an excuse to play around with it, so that's what I'll do here. \n\nSo the interesting thing about SAM, is that it's been designed to be prompted, i.e. given an image and a point/box/mask, it will return a mask. In the [SAM paper ](https://ai.facebook.com/research/publications/segment-anything/), they point out this can be done by hand or programmatically. \n\nNaturally the latter is more relevant here, so how about we train a lightweight object detection model (e.g. YOLO) to find objects, and then use SAM to refine the masks? The weakness here might be that we rely quite heavily on the object detector to do a good job first. Is it even necessary? As SAM can do [automatic mask generation](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/automatic_mask_generator.py). \n\nMy first attempt [is here](https://www.kaggle.com/code/fnands/yolov7-sam-inference-only), and naively takes a mask and box from YOLOv7 as prompts, and then tries do predict a segmentation masks. \nSo far, it does worse than just using using the YOLOv7 masks directly (LB 0.25). \n\nNext, some fine-tuning of SAM?", "votes": null }, { "id": "2305674", "postDate": "06/16/2023 20:46:58", "content": "Adding dilation 0.148 -> 0.195, without any other changes.
\nGot the tip from: https://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature/discussion/416901
Fine-tuning the mask decoder on dataset 1 takes me from 0.195 -> 0.364.
\nWas a bit of a pain in the ass to get training, but clipping the gradients helped a lot.
\nHaven't implemented CV, which is bad practice. Will do so now to get an idea of the difference between CV and LB.
Funnily, training on dataset 1 and 2 lowers the LB score compared to training on dataset 1 only. 0.364 -> 0.352.
\nThis does make sense, especially when considering we are trying to get the annotation quality to match dataset 1 as closely as possible, so fine tuning the segmentation part on only dataset 1 might be the way to go.
", "rawMarkdown": "Funnily, training on dataset 1 and 2 lowers the LB score compared to training on dataset 1 only. 0.364 -> 0.352. \n\nThis does make sense, especially when considering we are trying to get the annotation quality to match dataset 1 as closely as possible, so fine tuning the segmentation part on only dataset 1 might be the way to go.", "votes": null }, { "id": "2307532", "postDate": "06/18/2023 08:56:03", "content": "Training the prompt encoder as well seems to make basically no difference…
", "rawMarkdown": "Training the prompt encoder as well seems to make basically no difference...", "votes": null }, { "id": "2307741", "postDate": "06/18/2023 11:44:14", "content": "Going from the ViT-b to the ViT-l size of SAM only nets a 0.001 increase (0.364 -> 0.365), which might just be noise.
", "rawMarkdown": "Going from the ViT-b to the ViT-l size of SAM only nets a 0.001 increase (0.364 -> 0.365), which might just be noise.", "votes": null }, { "id": "2308670", "postDate": "06/19/2023 06:11:05", "content": "Fine tuning the encoder as well for the ViT-b size gets me to 0.365 -> 0.372 on the LB. Takes a bit of work to get it in the notebook GPU memory, so batch size=1 and 16-bit training.
\nIn related news, has someone found a reliable CV metric?
\nI'm using the torchmetrics implementation and get a mAP at IOU threshold 0.6 of 0.512, which is pretty far off from my LB score.
Adding dilation 0.148 -> 0.195, without any other changes.
\nGot the tip from: https://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature/discussion/416901
Fine-tuning the mask decoder on dataset 1 takes me from 0.195 -> 0.364.
\nWas a bit of a pain in the ass to get training, but clipping the gradients helped a lot.
\nHaven't implemented CV, which is bad practice. Will do so now to get an idea of the difference between CV and LB.
Funnily, training on dataset 1 and 2 lowers the LB score compared to training on dataset 1 only. 0.364 -> 0.352.
\nThis does make sense, especially when considering we are trying to get the annotation quality to match dataset 1 as closely as possible, so fine tuning the segmentation part on only dataset 1 might be the way to go.
", "votes": null, "replies": [] }, { "id": 2307532, "author_name": "fnands", "author_url": "", "post_date": "06/18/2023 08:56:03", "content": "Training the prompt encoder as well seems to make basically no difference…
", "votes": null, "replies": [] }, { "id": 2307741, "author_name": "fnands", "author_url": "", "post_date": "06/18/2023 11:44:14", "content": "Going from the ViT-b to the ViT-l size of SAM only nets a 0.001 increase (0.364 -> 0.365), which might just be noise.
", "votes": null, "replies": [] }, { "id": 2308670, "author_name": "fnands", "author_url": "", "post_date": "06/19/2023 06:11:05", "content": "Fine tuning the encoder as well for the ViT-b size gets me to 0.365 -> 0.372 on the LB. Takes a bit of work to get it in the notebook GPU memory, so batch size=1 and 16-bit training.
\nIn related news, has someone found a reliable CV metric?
\nI'm using the torchmetrics implementation and get a mAP at IOU threshold 0.6 of 0.512, which is pretty far off from my LB score.