{ "id": 302985, "title": "How to handle bounding boxes overlapping with edges when using tiling/cropping?", "url": "/competitions/tensorflow-great-barrier-reef/discussion/302985", "author_name": "Bilzard", "post_date": "2022-01-25T11:17:04.717000", "votes": 7, "comment_count": 13, "views": 0, "content": "

In Short

\n

When training a model using a tiled or cropped image, one question arises: \"how to properly define the bounding box that overlaps the edge of the cropped region?\" (see figure below).
\nThe simplest way is to use the original box as it is. This allows for a bbox whose center is outside the clipped area. The other way is to adjust the bounds of the bbox so that it fits into the cropped area.

\n

In this article, I will give you my thoughts on which option is better.
\nPlease do not take my word for it, as it is just my opinion.

\n

Detail

\n

We have two design options for bounding boxes overlapping edges:

\n

a) Use the overlapped box as is.
\nb) Cut off at the edge

\n

We want to lower the predicted IoU of the bbox that overlaps the edge.
\nOtherwise, when merging tiles, the bbox with the higher confidence and overlapping edges will be selected instead of the lower confidence and non-overlapping bbox.
\nAs a result, the trained model tends to produce more partially overlapping predictions, decreasing TP and increasing FN.

\n

From this point of view, I think option (A) is more suitable for learning with cropped images.
\nThis is because in option (A), the bboxes that overlap with the edges have only partial information about the cropped image, whereas in option (B), the bboxes of the overlapping edges have complete information about the cropped image.
\nThis makes option (B) more likely than option (A) to assign a higher confidence to the bboxes that overlap with the edges, and this would violate our design intent.

\n

How to handle with bbox overlapping with real edges?

\n

Someone want to say, \"No, I don't think option (A) is better. Because if trained with option (A), the trained model will predicts poor prediction for bbox which overlaps to real edges. So we should choose option (B)\".

\n

I agree with the first part, but I still think option (A) is better.
\nBecause we can handle this issue with post-processing the final prediction result: adjusting final predicted bboxes to get inside of the cropped region.
\nSo I think applying option (A) while training is better choice.

\n

Other Choice

\n\n

Reference

\n

[1] obss/sahi

\n
\n

\"Screen-Shot-2022-01-25-at-19-58-49\"

\n

\"Screen-Shot-2022-01-25-at-20-40-28\"

", "messages": [ { "id": 1663708, "postDate": "2022-01-25T11:17:04.717Z", "content": "

In Short

\n

When training a model using a tiled or cropped image, one question arises: \"how to properly define the bounding box that overlaps the edge of the cropped region?\" (see figure below).
\nThe simplest way is to use the original box as it is. This allows for a bbox whose center is outside the clipped area. The other way is to adjust the bounds of the bbox so that it fits into the cropped area.

\n

In this article, I will give you my thoughts on which option is better.
\nPlease do not take my word for it, as it is just my opinion.

\n

Detail

\n

We have two design options for bounding boxes overlapping edges:

\n

a) Use the overlapped box as is.
\nb) Cut off at the edge

\n

We want to lower the predicted IoU of the bbox that overlaps the edge.
\nOtherwise, when merging tiles, the bbox with the higher confidence and overlapping edges will be selected instead of the lower confidence and non-overlapping bbox.
\nAs a result, the trained model tends to produce more partially overlapping predictions, decreasing TP and increasing FN.

\n

From this point of view, I think option (A) is more suitable for learning with cropped images.
\nThis is because in option (A), the bboxes that overlap with the edges have only partial information about the cropped image, whereas in option (B), the bboxes of the overlapping edges have complete information about the cropped image.
\nThis makes option (B) more likely than option (A) to assign a higher confidence to the bboxes that overlap with the edges, and this would violate our design intent.

\n

How to handle with bbox overlapping with real edges?

\n

Someone want to say, \"No, I don't think option (A) is better. Because if trained with option (A), the trained model will predicts poor prediction for bbox which overlaps to real edges. So we should choose option (B)\".

\n

I agree with the first part, but I still think option (A) is better.
\nBecause we can handle this issue with post-processing the final prediction result: adjusting final predicted bboxes to get inside of the cropped region.
\nSo I think applying option (A) while training is better choice.

\n

Other Choice

\n\n

Reference

\n

[1] obss/sahi

\n
\n

\"Screen-Shot-2022-01-25-at-19-58-49\"

\n

\"Screen-Shot-2022-01-25-at-20-40-28\"

", "rawMarkdown": "# In Short\n\nWhen training a model using a tiled or cropped image, one question arises: \"how to properly define the bounding box that overlaps the edge of the cropped region?\" (see figure below).\nThe simplest way is to use the original box as it is. This allows for a bbox whose center is outside the clipped area. The other way is to adjust the bounds of the bbox so that it fits into the cropped area.\n\nIn this article, I will give you my thoughts on which option is better.\nPlease do not take my word for it, as it is just my opinion.\n\n# Detail\n\nWe have two design options for bounding boxes overlapping edges:\n\na) Use the overlapped box as is.\nb) Cut off at the edge\n\nWe want to lower the predicted IoU of the bbox that overlaps the edge.\nOtherwise, when merging tiles, the bbox with the higher confidence and overlapping edges will be selected instead of the lower confidence and non-overlapping bbox.\nAs a result, the trained model tends to produce more partially overlapping predictions, decreasing TP and increasing FN.\n\nFrom this point of view, I think option (A) is more suitable for learning with cropped images.\nThis is because in option (A), the bboxes that overlap with the edges have only partial information about the cropped image, whereas in option (B), the bboxes of the overlapping edges have complete information about the cropped image.\nThis makes option (B) more likely than option (A) to assign a higher confidence to the bboxes that overlap with the edges, and this would violate our design intent.\n\n## How to handle with bbox overlapping with *real* edges?\n\nSomeone want to say, \"No, I don't think option (A) is better. Because if trained with option (A), the trained model will predicts poor prediction for bbox which overlaps to *real* edges. So we should choose option (B)\".\n\nI agree with the first part, but I still think option (A) is better.\nBecause we can handle this issue with post-processing the final prediction result: adjusting final predicted bboxes to get inside of the cropped region.\nSo I think applying option (A) while training is better choice.\n\n# Other Choice\n\n* select boxes with wider intersection with cropped area by a threshold. ( @zzy990106 pointed out that SAHI[1] uses this strategy.)\n* only use the cropped image without overlapped boxes ( @atom1231 suggested)\n* pre-train with crop, fine-tune with full image size ( @hengck23 suggested)\n* train with crop, infer with full image size ( @tatamikenn suggested)\n\n# Reference\n\n[1] [obss/sahi](https://github.com/obss/sahi)\n\n---\n\n\"Screen-Shot-2022-01-25-at-19-58-49\"\n\n\"Screen-Shot-2022-01-25-at-20-40-28\"", "votes": 7 }, { "id": 1663766, "postDate": "2022-01-25T12:22:35.370Z", "content": "

You can set min_area_ratio(like >=0.5 to discard incomplete boxes) for training images in SAHI. It greatly reduces extreme(aspect ratio) predictions.

", "rawMarkdown": "You can set `min_area_ratio`(like >=0.5 to discard incomplete boxes) for training images in SAHI. It greatly reduces extreme(aspect ratio) predictions.", "votes": 2, "replies": [ { "id": 1664276, "postDate": "2022-01-25T20:34:23.040Z", "content": "

Yes, it is one remedy to mitigate the issue.
\nHowever, I think we can’t avoid the issue completely with it since the model will predict high-confidence, partly-covered boxes which have IoU greater than threshold.

", "rawMarkdown": "Yes, it is one remedy to mitigate the issue.\nHowever, I think we can’t avoid the issue completely with it since the model will predict high-confidence, partly-covered boxes which have IoU greater than threshold." } ] }, { "id": 1664492, "postDate": "2022-01-26T02:26:25.203Z", "content": "

I choose 3.
\n\"use the cropped image without overlapped box \"

", "rawMarkdown": "I choose 3.\n\"use the cropped image without overlapped box \"", "votes": 1 }, { "id": 1665722, "postDate": "2022-01-27T04:57:33.523Z", "content": "

pretrain with crop, finetune with full image size

", "rawMarkdown": "pretrain with crop, finetune with full image size", "votes": 2, "replies": [ { "id": 1667208, "postDate": "2022-01-28T11:56:43.940Z", "content": "

there is a different in using full images and crops for training.
\nassuming that we are either using full pos image or pos crop:

\n\n

whether using full image or crops lead to better results will depends on the data and the nature of the problem. more balance +ve and -ve class ratio and more variations usually geives better results

", "rawMarkdown": "there is a different in using full images and crops for training.\nassuming that we are either using full pos image or pos crop:\n\n- pos crop will give higher +ve object per pixel (since you discard neg crop)\n- pos crop limits context of the object. (more focus on objects and maybe less on context)\n- using crop from from various images gives more variation in data. this gives less bias per batch and better BN statistics\n\nwhether using full image or crops lead to better results will depends on the data and the nature of the problem. more balance +ve and -ve class ratio and more variations usually geives better results", "votes": 1 } ] }, { "id": 1666158, "postDate": "2022-01-27T13:53:37.483Z", "content": "

I haven't test but \"train with crop, infer with full image size\" might also be effective.

", "rawMarkdown": "I haven't test but \"train with crop, infer with full image size\" might also be effective." }, { "id": 1664317, "postDate": "2022-01-25T21:11:33.443Z", "content": "

I assumed that we are adopting box merging method as NMS, but more sophisticated merging method might mitigate the issue.

", "rawMarkdown": "I assumed that we are adopting box merging method as NMS, but more sophisticated merging method might mitigate the issue." }, { "id": 1664278, "postDate": "2022-01-25T20:40:19.040Z", "content": "

Note: since Yolov5 assumes the GT boxes should be within the input image, we have to fix the code to allow boxes out of the image if we want adopt option (A).

\n

We also consider the case when using mosaic and other geometric augmentation, because yolov5 must adopts option (B) for these algorithm. We might have to fix other code due to this design change.

", "rawMarkdown": "Note: since Yolov5 assumes the GT boxes should be within the input image, we have to fix the code to allow boxes out of the image if we want adopt option (A).\n\nWe also consider the case when using mosaic and other geometric augmentation, because yolov5 must adopts option (B) for these algorithm. We might have to fix other code due to this design change." }, { "id": 1664244, "postDate": "2022-01-25T20:11:59.180Z", "content": "

@tatamikenn we can mask the starfish completely with @hengck23 image blending

", "rawMarkdown": "@tatamikenn we can mask the starfish completely with @hengck23 image blending", "replies": [ { "id": 1664274, "postDate": "2022-01-25T20:30:12.323Z", "content": "

I haven’t read image blending (mixup?) part of his/her code.
\nI will check it. Thanks.

", "rawMarkdown": "I haven’t read image blending (mixup?) part of his/her code.\nI will check it. Thanks." }, { "id": 1664302, "postDate": "2022-01-25T20:57:49.780Z", "content": "

Oh, you are talking about this one[1].

\n

Sorry, I’m not sure what you are suggesting.
\nAre you suggesting masking labels that overlap with edges to avoid this issue?
\nIf so, isn’t this too rich as a remedy for the issue? I’d like more simple ones because I want to integrate the algorithm into the training pipeline.
\nBesides, erasing every GT boxes that overlap edges will cause low recall for boxes overlapped with real edges.
\n(Sorry if I misunderstood your point.)

\n

[1] https://www.kaggle.com/hengck23/augmentation-using-image-blending

", "rawMarkdown": "Oh, you are talking about this one[1].\n\nSorry, I’m not sure what you are suggesting.\nAre you suggesting masking labels that overlap with edges to avoid this issue?\nIf so, isn’t this too rich as a remedy for the issue? I’d like more simple ones because I want to integrate the algorithm into the training pipeline.\nBesides, erasing every GT boxes that overlap edges will cause low recall for boxes overlapped with *real* edges.\n(Sorry if I misunderstood your point.)\n\n[1] https://www.kaggle.com/hengck23/augmentation-using-image-blending" } ] }, { "id": 1663729, "postDate": "2022-01-25T11:46:48.897Z", "content": "

That is my thought, but I don't know if option (A) is really do better performance in the real situation.
\nDoes anyone know the proper way to handle this?

", "rawMarkdown": "That is my thought, but I don't know if option (A) is really do better performance in the real situation.\nDoes anyone know the proper way to handle this?" }, { "id": 1663723, "postDate": "2022-01-25T11:30:16.003Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1663766, "author_name": "Leon", "author_url": "", "post_date": "2022-01-25T12:22:35.370000", "content": "

You can set min_area_ratio(like >=0.5 to discard incomplete boxes) for training images in SAHI. It greatly reduces extreme(aspect ratio) predictions.

", "votes": 2, "replies": [ { "id": 1664276, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-25T20:34:23.040000", "content": "

Yes, it is one remedy to mitigate the issue.
\nHowever, I think we can’t avoid the issue completely with it since the model will predict high-confidence, partly-covered boxes which have IoU greater than threshold.

", "votes": 0, "replies": [] } ] }, { "id": 1664492, "author_name": "atom1231", "author_url": "", "post_date": "2022-01-26T02:26:25.203000", "content": "

I choose 3.
\n\"use the cropped image without overlapped box \"

", "votes": 1, "replies": [] }, { "id": 1665722, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-27T04:57:33.523000", "content": "

pretrain with crop, finetune with full image size

", "votes": 2, "replies": [ { "id": 1667208, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-28T11:56:43.940000", "content": "

there is a different in using full images and crops for training.
\nassuming that we are either using full pos image or pos crop:

\n\n

whether using full image or crops lead to better results will depends on the data and the nature of the problem. more balance +ve and -ve class ratio and more variations usually geives better results

", "votes": 1, "replies": [] } ] }, { "id": 1666158, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-27T13:53:37.483000", "content": "

I haven't test but \"train with crop, infer with full image size\" might also be effective.

", "votes": 0, "replies": [] }, { "id": 1664317, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-25T21:11:33.443000", "content": "

I assumed that we are adopting box merging method as NMS, but more sophisticated merging method might mitigate the issue.

", "votes": 0, "replies": [] }, { "id": 1664278, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-25T20:40:19.040000", "content": "

Note: since Yolov5 assumes the GT boxes should be within the input image, we have to fix the code to allow boxes out of the image if we want adopt option (A).

\n

We also consider the case when using mosaic and other geometric augmentation, because yolov5 must adopts option (B) for these algorithm. We might have to fix other code due to this design change.

", "votes": 0, "replies": [] }, { "id": 1664244, "author_name": "SeshuRaju 🧘‍♂️", "author_url": "", "post_date": "2022-01-25T20:11:59.180000", "content": "

@tatamikenn we can mask the starfish completely with @hengck23 image blending

", "votes": 0, "replies": [ { "id": 1664274, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-25T20:30:12.323000", "content": "

I haven’t read image blending (mixup?) part of his/her code.
\nI will check it. Thanks.

", "votes": 0, "replies": [] }, { "id": 1664302, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-25T20:57:49.780000", "content": "

Oh, you are talking about this one[1].

\n

Sorry, I’m not sure what you are suggesting.
\nAre you suggesting masking labels that overlap with edges to avoid this issue?
\nIf so, isn’t this too rich as a remedy for the issue? I’d like more simple ones because I want to integrate the algorithm into the training pipeline.
\nBesides, erasing every GT boxes that overlap edges will cause low recall for boxes overlapped with real edges.
\n(Sorry if I misunderstood your point.)

\n

[1] https://www.kaggle.com/hengck23/augmentation-using-image-blending

", "votes": 0, "replies": [] } ] }, { "id": 1663729, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-25T11:46:48.897000", "content": "

That is my thought, but I don't know if option (A) is really do better performance in the real situation.
\nDoes anyone know the proper way to handle this?

", "votes": 0, "replies": [] }, { "id": 1663723, "author_name": "", "author_url": "", "post_date": "2022-01-25T11:30:16.003000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1663708": "# In Short\n\nWhen training a model using a tiled or cropped image, one question arises: \"how to properly define the bounding box that overlaps the edge of the cropped region?\" (see figure below).\nThe simplest way is to use the original box as it is. This allows for a bbox whose center is outside the clipped area. The other way is to adjust the bounds of the bbox so that it fits into the cropped area.\n\nIn this article, I will give you my thoughts on which option is better.\nPlease do not take my word for it, as it is just my opinion.\n\n# Detail\n\nWe have two design options for bounding boxes overlapping edges:\n\na) Use the overlapped box as is.\nb) Cut off at the edge\n\nWe want to lower the predicted IoU of the bbox that overlaps the edge.\nOtherwise, when merging tiles, the bbox with the higher confidence and overlapping edges will be selected instead of the lower confidence and non-overlapping bbox.\nAs a result, the trained model tends to produce more partially overlapping predictions, decreasing TP and increasing FN.\n\nFrom this point of view, I think option (A) is more suitable for learning with cropped images.\nThis is because in option (A), the bboxes that overlap with the edges have only partial information about the cropped image, whereas in option (B), the bboxes of the overlapping edges have complete information about the cropped image.\nThis makes option (B) more likely than option (A) to assign a higher confidence to the bboxes that overlap with the edges, and this would violate our design intent.\n\n## How to handle with bbox overlapping with *real* edges?\n\nSomeone want to say, \"No, I don't think option (A) is better. Because if trained with option (A), the trained model will predicts poor prediction for bbox which overlaps to *real* edges. So we should choose option (B)\".\n\nI agree with the first part, but I still think option (A) is better.\nBecause we can handle this issue with post-processing the final prediction result: adjusting final predicted bboxes to get inside of the cropped region.\nSo I think applying option (A) while training is better choice.\n\n# Other Choice\n\n* select boxes with wider intersection with cropped area by a threshold. ( @zzy990106 pointed out that SAHI[1] uses this strategy.)\n* only use the cropped image without overlapped boxes ( @atom1231 suggested)\n* pre-train with crop, fine-tune with full image size ( @hengck23 suggested)\n* train with crop, infer with full image size ( @tatamikenn suggested)\n\n# Reference\n\n[1] [obss/sahi](https://github.com/obss/sahi)\n\n---\n\n\"Screen-Shot-2022-01-25-at-19-58-49\"\n\n\"Screen-Shot-2022-01-25-at-20-40-28\"", "1663766": "You can set `min_area_ratio`(like >=0.5 to discard incomplete boxes) for training images in SAHI. It greatly reduces extreme(aspect ratio) predictions.", "1664492": "I choose 3.\n\"use the cropped image without overlapped box \"", "1665722": "pretrain with crop, finetune with full image size", "1666158": "I haven't test but \"train with crop, infer with full image size\" might also be effective.", "1664317": "I assumed that we are adopting box merging method as NMS, but more sophisticated merging method might mitigate the issue.", "1664278": "Note: since Yolov5 assumes the GT boxes should be within the input image, we have to fix the code to allow boxes out of the image if we want adopt option (A).\n\nWe also consider the case when using mosaic and other geometric augmentation, because yolov5 must adopts option (B) for these algorithm. We might have to fix other code due to this design change.", "1664244": "@tatamikenn we can mask the starfish completely with @hengck23 image blending", "1663729": "That is my thought, but I don't know if option (A) is really do better performance in the real situation.\nDoes anyone know the proper way to handle this?", "1663723": "" } }