{ "id": 307605, "title": "simplest way to \"overfit\" LB (dangerous!!)", "url": "/competitions/tensorflow-great-barrier-reef/discussion/307605", "author_name": "", "post_date": "2022-02-15T00:34:42.671465Z", "votes": 31, "comment_count": 27, "views": 0, "content": "

Congratultion to al winners ~~

\n

Let me show how we overfit LB in simplest way :)

\n
        x_min = x_min+0.06*bbox_width\n        x_max = x_max-0.06*bbox_width\n        y_min = y_min+0.06*bbox_width\n        y_max = y_max-0.06*bbox_width  \n
\n

Before apply the trick
\npublic 702 private 699

\n

After
\npublic 788 private 641
\nlb rank 2 pb rank 376

", "messages": [ { "id": "1690449", "postDate": "02/15/2022 00:34:42", "content": "

Congratultion to al winners ~~

\n

Let me show how we overfit LB in simplest way :)

\n
        x_min = x_min+0.06*bbox_width\n        x_max = x_max-0.06*bbox_width\n        y_min = y_min+0.06*bbox_width\n        y_max = y_max-0.06*bbox_width  \n
\n

Before apply the trick
\npublic 702 private 699

\n

After
\npublic 788 private 641
\nlb rank 2 pb rank 376

", "rawMarkdown": "Congratultion to al winners ~~\n\nLet me show how we overfit LB in simplest way :)\n\n\n```\n x_min = x_min+0.06*bbox_width\n x_max = x_max-0.06*bbox_width\n y_min = y_min+0.06*bbox_width\n y_max = y_max-0.06*bbox_width \n```\n\nBefore apply the trick\npublic 702 private 699\n\nAfter\npublic 788 private 641\nlb rank **2** pb rank **376**", "votes": null }, { "id": "1690455", "postDate": "02/15/2022 00:37:47", "content": "

Reminds me of \"scaling\" that people did in jigsaw.

", "rawMarkdown": "Reminds me of \"scaling\" that people did in jigsaw.", "votes": null }, { "id": "1690456", "postDate": "02/15/2022 00:37:51", "content": "

Wait how exactly does this overfit public lb? You are just offsetting bbox values? Is it just because you are making the bbox smaller?

", "rawMarkdown": "Wait how exactly does this overfit public lb? You are just offsetting bbox values? Is it just because you are making the bbox smaller?", "votes": null }, { "id": "1690461", "postDate": "02/15/2022 00:39:11", "content": "

yes~XD

", "rawMarkdown": "yes~~~~~~XD", "votes": null }, { "id": "1690462", "postDate": "02/15/2022 00:39:26", "content": "

The real big issue is that it works for both videos in test, so it is not only a strange artefact of one subsequence or video. Why does it only work in public and not private? It does not make much sense to me… Everything hinted towards boxes in test being hand-labeled and tight, and labels in train being semi-automatic and messy. But suddenly private is completely different. Here I really hope they release the test set.

", "rawMarkdown": "The real big issue is that it works for both videos in test, so it is not only a strange artefact of one subsequence or video. Why does it only work in public and not private? It does not make much sense to me... Everything hinted towards boxes in test being hand-labeled and tight, and labels in train being semi-automatic and messy. But suddenly private is completely different. Here I really hope they release the test set.", "votes": null }, { "id": "1690470", "postDate": "02/15/2022 00:41:51", "content": "

hmm that is interesting. It doesnt add up if the annotation process was the same for all images.

", "rawMarkdown": "hmm that is interesting. It doesnt add up if the annotation process was the same for all images.", "votes": null }, { "id": "1690474", "postDate": "02/15/2022 00:43:59", "content": "

Yes, we checked that it holds for all public subsequences and both videos in test. So we were so sure it will also hold on private.

\n

Upscaling inference vs train resolution has a very similar effect, you can check it locally.

\n

Sub one month ago would be also third place for us :)

", "rawMarkdown": "Yes, we checked that it holds for all public subsequences and both videos in test. So we were so sure it will also hold on private. \n\nUpscaling inference vs train resolution has a very similar effect, you can check it locally.\n\nSub one month ago would be also third place for us :)", "votes": null }, { "id": "1690475", "postDate": "02/15/2022 00:44:34", "content": "

Wow. That's very interesting.

", "rawMarkdown": "Wow. That's very interesting.", "votes": null }, { "id": "1690518", "postDate": "02/15/2022 01:19:13", "content": "

Being able to do LB rank 2 is also a skill. Thanks for sharing.

", "rawMarkdown": "Being able to do LB rank 2 is also a skill. Thanks for sharing.", "votes": null }, { "id": "1690538", "postDate": "02/15/2022 01:53:17", "content": "

That's very interesting.
\nHow do you determine specific parameter 0.06? through experiments?

", "rawMarkdown": "That's very interesting. \nHow do you determine specific parameter 0.06? through experiments?", "votes": null }, { "id": "1690540", "postDate": "02/15/2022 01:56:04", "content": "

F1\\F2... are not stable metrics.

", "rawMarkdown": "F1\\F2\\... are not stable metrics.", "votes": null }, { "id": "1690550", "postDate": "02/15/2022 02:04:58", "content": "

yes~

", "rawMarkdown": "yes~~~~~~~~~~~", "votes": null }, { "id": "1690625", "postDate": "02/15/2022 03:17:24", "content": "

I guess they only finetune the annotation in public set. πŸ˜…

", "rawMarkdown": "I guess they only finetune the annotation in public set. πŸ˜…", "votes": null }, { "id": "1690631", "postDate": "02/15/2022 03:28:05", "content": "

@outrunner I am guessing they had different annotators? I am not sure, I did not read the dataset paper. I can't believe that they have a big annotation difference.

", "rawMarkdown": "outrunner I am guessing they had different annotators? I am not sure, I did not read the dataset paper. I can't believe that they have a big annotation difference.", "votes": null }, { "id": "1690714", "postDate": "02/15/2022 04:25:47", "content": "

haha, different annotators that one in public set, and others in private set. πŸ˜„

", "rawMarkdown": "haha, different annotators that one in public set, and others in private set. πŸ˜„", "votes": null }, { "id": "1690727", "postDate": "02/15/2022 04:31:10", "content": "

omg i was trying with ratio but not such gain, impressive! :) pity it didn't work on private :(

", "rawMarkdown": "omg i was trying with ratio but not such gain, impressive! :) pity it didn't work on private :(", "votes": null }, { "id": "1690737", "postDate": "02/15/2022 04:40:26", "content": "

@philippsinger How can we determine which subsequences and which videos are which frames in test data?

\n

For one of my final submissions I used one model with 2.5x image size infer for frame<2600 and another model with 1.0x image size infer for frame>5200. The idea was that (i thought) public LB (roughly frames 2600 thru 5200) was half video 4 and needed 2.5x and private was the other half video 4 and full unknown video 5. (This idea didn't work).

", "rawMarkdown": "philippsinger How can we determine which subsequences and which videos are which frames in test data? \n\nFor one of my final submissions I used one model with 2.5x image size infer for `frame<2600` and another model with 1.0x image size infer for `frame>5200`. The idea was that (i thought) public LB (roughly frames 2600 thru 5200) was half video 4 and needed 2.5x and private was the other half video 4 and full unknown video 5. (This idea didn't work).", "votes": null }, { "id": "1690741", "postDate": "02/15/2022 04:41:46", "content": "

Interesting @atom1231 . Nice discovery! Sorry about your shake down. Congrats on achieving 2nd place public LB. That was awesome!

", "rawMarkdown": "Interesting @atom1231 . Nice discovery! Sorry about your shake down. Congrats on achieving 2nd place public LB. That was awesome!", "votes": null }, { "id": "1690893", "postDate": "02/15/2022 06:32:50", "content": "

CV , ensemble , new model are the weapons, plus hardware to quick test ideas.

", "rawMarkdown": "CV , ensemble , new model are the weapons, plus hardware to quick test ideas.", "votes": null }, { "id": "1691056", "postDate": "02/15/2022 07:56:43", "content": "

@cdeotte just use test.csv, it has for each index information about sequence and video (see data tab)

\n

@outrunner maybe they indeed split public/private by annotator :P

", "rawMarkdown": "cdeotte just use `test.csv`, it has for each index information about sequence and video (see data tab)\n\n@outrunner maybe they indeed split public/private by annotator :P", "votes": null }, { "id": "1691062", "postDate": "02/15/2022 08:08:38", "content": "

We found it works both in yolovt/mmdet/TD OD api framework with similar ratio
\ntherefore we just go~~~~XD

", "rawMarkdown": "We found it works both in yolovt/mmdet/TD OD api framework with similar ratio\ntherefore we just go~~~~XD", "votes": null }, { "id": "1691068", "postDate": "02/15/2022 08:13:49", "content": "

You've solved the mystery in my mind, thanks! πŸ‘

", "rawMarkdown": "You've solved the mystery in my mind, thanks! πŸ‘", "votes": null }, { "id": "1691320", "postDate": "02/15/2022 10:49:11", "content": "

it is a bit string that the public bbox is different from the rest.
\ni don't think it is becuase of different annotator.

\n

could it be some other reasons?
\ne.g. the cots are occluded, so giving smaller box

\n

if the annotations are different in train public test and private test, then it is a mistake of the data collection in the whole application building pipleline (i.e. mistake of the organizer)

", "rawMarkdown": "it is a bit string that the public bbox is different from the rest.\ni don't think it is becuase of different annotator.\n\ncould it be some other reasons?\ne.g. the cots are occluded, so giving smaller box\n\nif the annotations are different in train public test and private test, then it is a mistake of the data collection in the whole application building pipleline (i.e. mistake of the organizer)", "votes": null }, { "id": "1692290", "postDate": "02/16/2022 00:28:53", "content": "

Could you share your cascade rcnn config file (mmdet)? I tuned the parameters for a week but only got a 0.15 LB, confusing now πŸ‘€

", "rawMarkdown": "Could you share your cascade rcnn config file (mmdet)? I tuned the parameters for a week but only got a 0.15 LB, confusing now πŸ‘€", "votes": null }, { "id": "1692370", "postDate": "02/16/2022 02:08:47", "content": "

In train set, there are same size bbox in sequence, and usually match the last(biggest) cots. I guess it is due to their sequence-based annotations and annotated in backward. Maybe some sequences are annotated in forward and happened in public test set. Even so, organizer should still avoid this.

", "rawMarkdown": "In train set, there are same size bbox in sequence, and usually match the last(biggest) cots. I guess it is due to their sequence-based annotations and annotated in backward. Maybe some sequences are annotated in forward and happened in public test set. Even so, organizer should still avoid this.", "votes": null }, { "id": "1694690", "postDate": "02/17/2022 16:27:03", "content": "

check the file ,
\nIt seems not the best one but actually works.
\nhttps://www.kaggle.com/atom1231/gbr-model?select=job31_cascade_rcnn_r50_fpn_1x_coco_f6_sc4.py

", "rawMarkdown": "check the file , \nIt seems not the best one but actually works.\nhttps://www.kaggle.com/atom1231/gbr-model?select=job31_cascade_rcnn_r50_fpn_1x_coco_f6_sc4.py", "votes": null }, { "id": "1695928", "postDate": "02/18/2022 13:19:58", "content": "

Wow , although it failed in Private Board, it has to be said that this is a very novel cut-in angle. Meaning that the annotator who labeled LB video prefers a tighter box …? How you guys find it , Probe LB ?

", "rawMarkdown": "Wow , although it failed in Private Board, it has to be said that this is a very novel cut-in angle. Meaning that the annotator who labeled LB video prefers a tighter box ...? How you guys find it , Probe LB ?", "votes": null }, { "id": "1696012", "postDate": "02/18/2022 14:40:44", "content": "

Actually it is a common issue happened in many previous competitons.

", "rawMarkdown": "Actually it is a common issue happened in many previous competitons.", "votes": null } ], "comments": [ { "id": 1690455, "author_name": "trushk", "author_url": "", "post_date": "02/15/2022 00:37:47", "content": "

Reminds me of \"scaling\" that people did in jigsaw.

", "votes": null, "replies": [] }, { "id": 1690456, "author_name": "vexxingbanana", "author_url": "", "post_date": "02/15/2022 00:37:51", "content": "

Wait how exactly does this overfit public lb? You are just offsetting bbox values? Is it just because you are making the bbox smaller?

", "votes": null, "replies": [ { "id": 1690461, "author_name": "atom1231", "author_url": "", "post_date": "02/15/2022 00:39:11", "content": "

yes~XD

", "votes": null, "replies": [] }, { "id": 1690475, "author_name": "vexxingbanana", "author_url": "", "post_date": "02/15/2022 00:44:34", "content": "

Wow. That's very interesting.

", "votes": null, "replies": [] } ] }, { "id": 1690462, "author_name": "philippsinger", "author_url": "", "post_date": "02/15/2022 00:39:26", "content": "

The real big issue is that it works for both videos in test, so it is not only a strange artefact of one subsequence or video. Why does it only work in public and not private? It does not make much sense to me… Everything hinted towards boxes in test being hand-labeled and tight, and labels in train being semi-automatic and messy. But suddenly private is completely different. Here I really hope they release the test set.

", "votes": null, "replies": [ { "id": 1690470, "author_name": "trushk", "author_url": "", "post_date": "02/15/2022 00:41:51", "content": "

hmm that is interesting. It doesnt add up if the annotation process was the same for all images.

", "votes": null, "replies": [] }, { "id": 1690474, "author_name": "philippsinger", "author_url": "", "post_date": "02/15/2022 00:43:59", "content": "

Yes, we checked that it holds for all public subsequences and both videos in test. So we were so sure it will also hold on private.

\n

Upscaling inference vs train resolution has a very similar effect, you can check it locally.

\n

Sub one month ago would be also third place for us :)

", "votes": null, "replies": [] }, { "id": 1690625, "author_name": "outrunner", "author_url": "", "post_date": "02/15/2022 03:17:24", "content": "

I guess they only finetune the annotation in public set. πŸ˜…

", "votes": null, "replies": [] }, { "id": 1690631, "author_name": "outwrest", "author_url": "", "post_date": "02/15/2022 03:28:05", "content": "

@outrunner I am guessing they had different annotators? I am not sure, I did not read the dataset paper. I can't believe that they have a big annotation difference.

", "votes": null, "replies": [] }, { "id": 1690714, "author_name": "outrunner", "author_url": "", "post_date": "02/15/2022 04:25:47", "content": "

haha, different annotators that one in public set, and others in private set. πŸ˜„

", "votes": null, "replies": [] }, { "id": 1690737, "author_name": "cdeotte", "author_url": "", "post_date": "02/15/2022 04:40:26", "content": "

@philippsinger How can we determine which subsequences and which videos are which frames in test data?

\n

For one of my final submissions I used one model with 2.5x image size infer for frame<2600 and another model with 1.0x image size infer for frame>5200. The idea was that (i thought) public LB (roughly frames 2600 thru 5200) was half video 4 and needed 2.5x and private was the other half video 4 and full unknown video 5. (This idea didn't work).

", "votes": null, "replies": [] }, { "id": 1691056, "author_name": "philippsinger", "author_url": "", "post_date": "02/15/2022 07:56:43", "content": "

@cdeotte just use test.csv, it has for each index information about sequence and video (see data tab)

\n

@outrunner maybe they indeed split public/private by annotator :P

", "votes": null, "replies": [] } ] }, { "id": 1690518, "author_name": "tomkerl", "author_url": "", "post_date": "02/15/2022 01:19:13", "content": "

Being able to do LB rank 2 is also a skill. Thanks for sharing.

", "votes": null, "replies": [] }, { "id": 1690538, "author_name": "yuanshug", "author_url": "", "post_date": "02/15/2022 01:53:17", "content": "

That's very interesting.
\nHow do you determine specific parameter 0.06? through experiments?

", "votes": null, "replies": [ { "id": 1690550, "author_name": "atom1231", "author_url": "", "post_date": "02/15/2022 02:04:58", "content": "

yes~

", "votes": null, "replies": [] } ] }, { "id": 1690540, "author_name": "yuanzhezhou", "author_url": "", "post_date": "02/15/2022 01:56:04", "content": "

F1\\F2... are not stable metrics.

", "votes": null, "replies": [] }, { "id": 1690727, "author_name": "lukaszborecki", "author_url": "", "post_date": "02/15/2022 04:31:10", "content": "

omg i was trying with ratio but not such gain, impressive! :) pity it didn't work on private :(

", "votes": null, "replies": [ { "id": 1691062, "author_name": "atom1231", "author_url": "", "post_date": "02/15/2022 08:08:38", "content": "

We found it works both in yolovt/mmdet/TD OD api framework with similar ratio
\ntherefore we just go~~~~XD

", "votes": null, "replies": [] } ] }, { "id": 1690741, "author_name": "cdeotte", "author_url": "", "post_date": "02/15/2022 04:41:46", "content": "

Interesting @atom1231 . Nice discovery! Sorry about your shake down. Congrats on achieving 2nd place public LB. That was awesome!

", "votes": null, "replies": [] }, { "id": 1690893, "author_name": "dragonzhang", "author_url": "", "post_date": "02/15/2022 06:32:50", "content": "

CV , ensemble , new model are the weapons, plus hardware to quick test ideas.

", "votes": null, "replies": [] }, { "id": 1691068, "author_name": "haqishen", "author_url": "", "post_date": "02/15/2022 08:13:49", "content": "

You've solved the mystery in my mind, thanks! πŸ‘

", "votes": null, "replies": [] }, { "id": 1691320, "author_name": "hengck23", "author_url": "", "post_date": "02/15/2022 10:49:11", "content": "

it is a bit string that the public bbox is different from the rest.
\ni don't think it is becuase of different annotator.

\n

could it be some other reasons?
\ne.g. the cots are occluded, so giving smaller box

\n

if the annotations are different in train public test and private test, then it is a mistake of the data collection in the whole application building pipleline (i.e. mistake of the organizer)

", "votes": null, "replies": [ { "id": 1692370, "author_name": "outrunner", "author_url": "", "post_date": "02/16/2022 02:08:47", "content": "

In train set, there are same size bbox in sequence, and usually match the last(biggest) cots. I guess it is due to their sequence-based annotations and annotated in backward. Maybe some sequences are annotated in forward and happened in public test set. Even so, organizer should still avoid this.

", "votes": null, "replies": [] } ] }, { "id": 1692290, "author_name": "lixxxxx", "author_url": "", "post_date": "02/16/2022 00:28:53", "content": "

Could you share your cascade rcnn config file (mmdet)? I tuned the parameters for a week but only got a 0.15 LB, confusing now πŸ‘€

", "votes": null, "replies": [ { "id": 1694690, "author_name": "atom1231", "author_url": "", "post_date": "02/17/2022 16:27:03", "content": "

check the file ,
\nIt seems not the best one but actually works.
\nhttps://www.kaggle.com/atom1231/gbr-model?select=job31_cascade_rcnn_r50_fpn_1x_coco_f6_sc4.py

", "votes": null, "replies": [] } ] }, { "id": 1695928, "author_name": "evilpsycho42", "author_url": "", "post_date": "02/18/2022 13:19:58", "content": "

Wow , although it failed in Private Board, it has to be said that this is a very novel cut-in angle. Meaning that the annotator who labeled LB video prefers a tighter box …? How you guys find it , Probe LB ?

", "votes": null, "replies": [ { "id": 1696012, "author_name": "atom1231", "author_url": "", "post_date": "02/18/2022 14:40:44", "content": "

Actually it is a common issue happened in many previous competitons.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1690449": "Congratultion to al winners ~~\n\nLet me show how we overfit LB in simplest way :)\n\n\n```\n x_min = x_min+0.06*bbox_width\n x_max = x_max-0.06*bbox_width\n y_min = y_min+0.06*bbox_width\n y_max = y_max-0.06*bbox_width \n```\n\nBefore apply the trick\npublic 702 private 699\n\nAfter\npublic 788 private 641\nlb rank **2** pb rank **376**", "1690455": "Reminds me of \"scaling\" that people did in jigsaw.", "1690456": "Wait how exactly does this overfit public lb? You are just offsetting bbox values? Is it just because you are making the bbox smaller?", "1690461": "yes~~~~~~XD", "1690462": "The real big issue is that it works for both videos in test, so it is not only a strange artefact of one subsequence or video. Why does it only work in public and not private? It does not make much sense to me... Everything hinted towards boxes in test being hand-labeled and tight, and labels in train being semi-automatic and messy. But suddenly private is completely different. Here I really hope they release the test set.", "1690470": "hmm that is interesting. It doesnt add up if the annotation process was the same for all images.", "1690474": "Yes, we checked that it holds for all public subsequences and both videos in test. So we were so sure it will also hold on private. \n\nUpscaling inference vs train resolution has a very similar effect, you can check it locally.\n\nSub one month ago would be also third place for us :)", "1690475": "Wow. That's very interesting.", "1690518": "Being able to do LB rank 2 is also a skill. Thanks for sharing.", "1690538": "That's very interesting. \nHow do you determine specific parameter 0.06? through experiments?", "1690540": "F1\\F2\\... are not stable metrics.", "1690550": "yes~~~~~~~~~~~", "1690625": "I guess they only finetune the annotation in public set. πŸ˜…", "1690631": "outrunner I am guessing they had different annotators? I am not sure, I did not read the dataset paper. I can't believe that they have a big annotation difference.", "1690714": "haha, different annotators that one in public set, and others in private set. πŸ˜„", "1690727": "omg i was trying with ratio but not such gain, impressive! :) pity it didn't work on private :(", "1690737": "philippsinger How can we determine which subsequences and which videos are which frames in test data? \n\nFor one of my final submissions I used one model with 2.5x image size infer for `frame<2600` and another model with 1.0x image size infer for `frame>5200`. The idea was that (i thought) public LB (roughly frames 2600 thru 5200) was half video 4 and needed 2.5x and private was the other half video 4 and full unknown video 5. (This idea didn't work).", "1690741": "Interesting @atom1231 . Nice discovery! Sorry about your shake down. Congrats on achieving 2nd place public LB. That was awesome!", "1690893": "CV , ensemble , new model are the weapons, plus hardware to quick test ideas.", "1691056": "cdeotte just use `test.csv`, it has for each index information about sequence and video (see data tab)\n\n@outrunner maybe they indeed split public/private by annotator :P", "1691062": "We found it works both in yolovt/mmdet/TD OD api framework with similar ratio\ntherefore we just go~~~~XD", "1691068": "You've solved the mystery in my mind, thanks! πŸ‘", "1691320": "it is a bit string that the public bbox is different from the rest.\ni don't think it is becuase of different annotator.\n\ncould it be some other reasons?\ne.g. the cots are occluded, so giving smaller box\n\nif the annotations are different in train public test and private test, then it is a mistake of the data collection in the whole application building pipleline (i.e. mistake of the organizer)", "1692290": "Could you share your cascade rcnn config file (mmdet)? I tuned the parameters for a week but only got a 0.15 LB, confusing now πŸ‘€", "1692370": "In train set, there are same size bbox in sequence, and usually match the last(biggest) cots. I guess it is due to their sequence-based annotations and annotated in backward. Maybe some sequences are annotated in forward and happened in public test set. Even so, organizer should still avoid this.", "1694690": "check the file , \nIt seems not the best one but actually works.\nhttps://www.kaggle.com/atom1231/gbr-model?select=job31_cascade_rcnn_r50_fpn_1x_coco_f6_sc4.py", "1695928": "Wow , although it failed in Private Board, it has to be said that this is a very novel cut-in angle. Meaning that the annotator who labeled LB video prefers a tighter box ...? How you guys find it , Probe LB ?", "1696012": "Actually it is a common issue happened in many previous competitons." }, "source": "meta" }