{ "id": 567295, "title": "Best Single Model CV LB thread", "url": "/competitions/byu-locating-bacterial-flagellar-motors-2025/discussion/567295", "author_name": "", "post_date": "2025-03-09T14:23:33.737309300Z", "votes": 32, "comment_count": 25, "views": 0, "content": "

\n

I haven't made a table yet, but I'm having a CV of over 0.9, while my LB is stll wondering around 0.5~0.6.
\nThe correlation is very very weak, and I'm begining to think that this competition is a \"trust LB\" type of competition, which means that build a CV/LB table is almost meaningless.
\nThere is still the possibility that my CV method is not good, so I'd be happy to hear from anyone who has good CV/LB correlation!

\n\n\n\n\n\n\n\n\n\n\n
modelcvlb
", "messages": [ { "id": "3145212", "postDate": "03/09/2025 14:23:33", "content": "

\n

I haven't made a table yet, but I'm having a CV of over 0.9, while my LB is stll wondering around 0.5~0.6.
\nThe correlation is very very weak, and I'm begining to think that this competition is a \"trust LB\" type of competition, which means that build a CV/LB table is almost meaningless.
\nThere is still the possibility that my CV method is not good, so I'd be happy to hear from anyone who has good CV/LB correlation!

\n\n\n\n\n\n\n\n\n\n\n
modelcvlb
", "rawMarkdown": "~~I'll start updating my results when I join this competition. (Currently running LuxAI)\n~~\n\nI haven't made a table yet, but I'm having a CV of over 0.9, while my LB is stll wondering around 0.5~0.6.\nThe correlation is very very weak, and I'm begining to think that this competition is a \"trust LB\" type of competition, which means that build a CV/LB table is almost meaningless.\nThere is still the possibility that my CV method is not good, so I'd be happy to hear from anyone who has good CV/LB correlation!\n\n| model | cv | lb |\n| --- | --- | --- |", "votes": null }, { "id": "3146073", "postDate": "03/10/2025 13:50:32", "content": "

haha, battle of giants

", "rawMarkdown": "haha, battle of giants", "votes": null }, { "id": "3146101", "postDate": "03/10/2025 14:13:24", "content": "

battle for ranking 1

", "rawMarkdown": "battle for ranking 1", "votes": null }, { "id": "3146113", "postDate": "03/10/2025 14:27:58", "content": "

Do you use 3D U-Net?

", "rawMarkdown": "Do you use 3D U-Net?", "votes": null }, { "id": "3146753", "postDate": "03/11/2025 08:07:16", "content": "

one fold, CV 0.84, but LB only scored 0.018, seems like data leakage, but I can't find any leakage in my training process till now. :(

", "rawMarkdown": "one fold, CV 0.84, but LB only scored 0.018, seems like data leakage, but I can't find any leakage in my training process till now. :(", "votes": null }, { "id": "3146758", "postDate": "03/11/2025 08:10:04", "content": "

It can't be the domain gap, can it?

", "rawMarkdown": "It can't be the domain gap, can it?", "votes": null }, { "id": "3147550", "postDate": "03/12/2025 05:54:20", "content": "

I have CV 0.87 and LB 0.55, there is a gap, but not that huge. Maybe you can check your inference pipeline or subission format.

", "rawMarkdown": "I have CV 0.87 and LB 0.55, there is a gap, but not that huge. Maybe you can check your inference pipeline or subission format.", "votes": null }, { "id": "3147558", "postDate": "03/12/2025 06:10:08", "content": "

I have checked for many times, now I suspect the reason is domain gap and my model overfit to trian set, but I am not sure about it. I wonder if others encounter this CV-LB not same problem. If so, the problem maybe caused by domain gap between train and hidden test set.

", "rawMarkdown": "I have checked for many times, now I suspect the reason is domain gap and my model overfit to trian set, but I am not sure about it. I wonder if others encounter this CV-LB not same problem. If so, the problem maybe caused by domain gap between train and hidden test set.", "votes": null }, { "id": "3147560", "postDate": "03/12/2025 06:18:13", "content": "

And host mention that there is population shift between train set and test set in this disscussion, if I understand correctly, the population shift means domain gap in train and hidden test set.

", "rawMarkdown": "And host mention that there is population shift between train set and test set in [this disscussion](https://www.kaggle.com/competitions/byu-locating-bacterial-flagellar-motors-2025/discussion/566137#3142976), if I understand correctly, the population shift means domain gap in train and hidden test set.", "votes": null }, { "id": "3147581", "postDate": "03/12/2025 06:53:19", "content": "

Thank you, that explain why i'm getting ~0.3 gap between CV、LB. You can try using yolov8n and train for 30 epochs like the starting notebook, I can get similar LB result with starting notebooks setup.

", "rawMarkdown": "Thank you, that explain why i'm getting ~0.3 gap between CV、LB. You can try using yolov8n and train for 30 epochs like the starting notebook, I can get similar LB result with starting notebooks setup.", "votes": null }, { "id": "3150217", "postDate": "03/15/2025 07:19:53", "content": "

I'm having the same problem right now.

", "rawMarkdown": "I'm having the same problem right now.", "votes": null }, { "id": "3150230", "postDate": "03/15/2025 07:47:57", "content": "

Do you guys have adjacent slices from same tomograms on different folds?

", "rawMarkdown": "Do you guys have adjacent slices from same tomograms on different folds?", "votes": null }, { "id": "3150240", "postDate": "03/15/2025 08:03:30", "content": "

No, I'm being careful with that.

", "rawMarkdown": "No, I'm being careful with that.", "votes": null }, { "id": "3150245", "postDate": "03/15/2025 08:16:24", "content": "

I haven't made any submissions yet, but I have couple ideas. I think It could be related to either different scales i.e., voxel spacing or too many false positives. We know that test set only has 0 or 1 motor on each tomogram and also there could unseen scales on test set.

", "rawMarkdown": "I haven't made any submissions yet, but I have couple ideas. I think It could be related to either different scales i.e., voxel spacing or too many false positives. We know that test set only has 0 or 1 motor on each tomogram and also there could unseen scales on test set.", "votes": null }, { "id": "3150309", "postDate": "03/15/2025 10:36:57", "content": "

In the training starter pipeline provided by the host, the split is done at tomogram level to avoid this issue of having slices from the same tomogram on different folds.

\n

In terms of cross-validation, I have also experienced this issue, where my models inferring on the validation set scored above 0.9 on the competition metric while hovering around 0.7 on the lb. For last week's efforts, I discarded that and moved my attention to dataset creation for training and post processing.

\n

One of my findings when playing around with extra augmentations during training is that when adding it, the training and validation loss are nicely coupled, and they stabilize at a lower validation loss.

\n

However, this always translated with a worse score on the LB (without changing any of the inference hyper-params, which could surely play a role).

\n

I believe these changes are caused by the domain shift, we do not know if there's other resolutions, there's no access to the voxel spacing to adapt post processing based on that, even weird aspect ratios could impact YOLO's letterboxing and thus object size (which I would have expected to benefit from heavy augs?).

", "rawMarkdown": "In the training starter pipeline provided by the host, the split is done at tomogram level to avoid this issue of having slices from the same tomogram on different folds.\n\nIn terms of cross-validation, I have also experienced this issue, where my models inferring on the validation set scored above 0.9 on the competition metric while hovering around 0.7 on the lb. For last week's efforts, I discarded that and moved my attention to dataset creation for training and post processing.\n\nOne of my findings when playing around with extra augmentations during training is that when adding it, the training and validation loss are nicely coupled, and they stabilize at a lower validation loss.\n\nHowever, this always translated with a worse score on the LB (without changing any of the inference hyper-params, which could surely play a role).\n\nI believe these changes are caused by the domain shift, we do not know if there's other resolutions, there's no access to the voxel spacing to adapt post processing based on that, even weird aspect ratios could impact YOLO's letterboxing and thus object size (which I would have expected to benefit from heavy augs?).", "votes": null }, { "id": "3150318", "postDate": "03/15/2025 10:47:57", "content": "

How long time does it take for your submission? It seems to be very slow …

", "rawMarkdown": "How long time does it take for your submission? It seems to be very slow ...", "votes": null }, { "id": "3150321", "postDate": "03/15/2025 10:53:35", "content": "

Around 3 hours

", "rawMarkdown": "Around 3 hours", "votes": null }, { "id": "3150383", "postDate": "03/15/2025 12:51:33", "content": "

At first I thought the CV/LB gap could be explained by the difference in tomogram ratio with and without motors, but as you point out, it cannot explain why data augmentation breaks the correlation, so I also feel that a domain shift is taking place.

", "rawMarkdown": "At first I thought the CV/LB gap could be explained by the difference in tomogram ratio with and without motors, but as you point out, it cannot explain why data augmentation breaks the correlation, so I also feel that a domain shift is taking place.", "votes": null }, { "id": "3150386", "postDate": "03/15/2025 12:52:19", "content": "

Playing with yolo

", "rawMarkdown": "Playing with yolo", "votes": null }, { "id": "3150389", "postDate": "03/15/2025 12:54:17", "content": "

Still can't get a decent CV score :(
\n(was hoping to get it days ago, but still having some trouble)

", "rawMarkdown": "Still can't get a decent CV score :(\n(was hoping to get it days ago, but still having some trouble)", "votes": null }, { "id": "3150398", "postDate": "03/15/2025 13:09:07", "content": "

The situation become very complicated when LB and CV are not consistent. It is hard to validate trained model and hard to decide LB/CV which should we trust.

", "rawMarkdown": "The situation become very complicated when LB and CV are not consistent. It is hard to validate trained model and hard to decide LB/CV which should we trust.", "votes": null }, { "id": "3151764", "postDate": "03/17/2025 04:24:37", "content": "

Tried to submit slightly different yolo models the scores vary a lot.

", "rawMarkdown": "Tried to submit slightly different yolo models the scores vary a lot.", "votes": null }, { "id": "3151765", "postDate": "03/17/2025 04:25:07", "content": "

I use only 0/1-motor tomograms for validation and currently having an AUC of 0.79

", "rawMarkdown": "I use only 0/1-motor tomograms for validation and currently having an AUC of 0.79", "votes": null }, { "id": "3155430", "postDate": "03/21/2025 03:03:44", "content": "

CV:1.0, LB:0.0 ¯_(ツ)_/¯

", "rawMarkdown": "CV:1.0, LB:0.0 ¯\\_(ツ)_/¯", "votes": null }, { "id": "3156969", "postDate": "03/22/2025 19:16:43", "content": "

Got exactly the same problem. 0.8 CV 0.25 LB. No data leakage. Probably hard augmetations will help, I used the basic ones for a baseline

", "rawMarkdown": "Got exactly the same problem. 0.8 CV 0.25 LB. No data leakage. Probably hard augmetations will help, I used the basic ones for a baseline", "votes": null }, { "id": "3157873", "postDate": "03/24/2025 01:17:48", "content": "

Me too :( best wish

", "rawMarkdown": "Me too :( best wish", "votes": null } ], "comments": [ { "id": 3146073, "author_name": "christofhenkel", "author_url": "", "post_date": "03/10/2025 13:50:32", "content": "

haha, battle of giants

", "votes": null, "replies": [ { "id": 3146101, "author_name": "saidkoussi", "author_url": "", "post_date": "03/10/2025 14:13:24", "content": "

battle for ranking 1

", "votes": null, "replies": [] }, { "id": 3150389, "author_name": "cnumber", "author_url": "", "post_date": "03/15/2025 12:54:17", "content": "

Still can't get a decent CV score :(
\n(was hoping to get it days ago, but still having some trouble)

", "votes": null, "replies": [] } ] }, { "id": 3146113, "author_name": "switch9527", "author_url": "", "post_date": "03/10/2025 14:27:58", "content": "

Do you use 3D U-Net?

", "votes": null, "replies": [ { "id": 3150386, "author_name": "cnumber", "author_url": "", "post_date": "03/15/2025 12:52:19", "content": "

Playing with yolo

", "votes": null, "replies": [] } ] }, { "id": 3146753, "author_name": "shtljw", "author_url": "", "post_date": "03/11/2025 08:07:16", "content": "

one fold, CV 0.84, but LB only scored 0.018, seems like data leakage, but I can't find any leakage in my training process till now. :(

", "votes": null, "replies": [ { "id": 3146758, "author_name": "shtljw", "author_url": "", "post_date": "03/11/2025 08:10:04", "content": "

It can't be the domain gap, can it?

", "votes": null, "replies": [ { "id": 3147550, "author_name": "zxcvbn369z", "author_url": "", "post_date": "03/12/2025 05:54:20", "content": "

I have CV 0.87 and LB 0.55, there is a gap, but not that huge. Maybe you can check your inference pipeline or subission format.

", "votes": null, "replies": [ { "id": 3147558, "author_name": "shtljw", "author_url": "", "post_date": "03/12/2025 06:10:08", "content": "

I have checked for many times, now I suspect the reason is domain gap and my model overfit to trian set, but I am not sure about it. I wonder if others encounter this CV-LB not same problem. If so, the problem maybe caused by domain gap between train and hidden test set.

", "votes": null, "replies": [] }, { "id": 3147560, "author_name": "shtljw", "author_url": "", "post_date": "03/12/2025 06:18:13", "content": "

And host mention that there is population shift between train set and test set in this disscussion, if I understand correctly, the population shift means domain gap in train and hidden test set.

", "votes": null, "replies": [ { "id": 3147581, "author_name": "zxcvbn369z", "author_url": "", "post_date": "03/12/2025 06:53:19", "content": "

Thank you, that explain why i'm getting ~0.3 gap between CV、LB. You can try using yolov8n and train for 30 epochs like the starting notebook, I can get similar LB result with starting notebooks setup.

", "votes": null, "replies": [] } ] } ] }, { "id": 3150217, "author_name": "cnumber", "author_url": "", "post_date": "03/15/2025 07:19:53", "content": "

I'm having the same problem right now.

", "votes": null, "replies": [ { "id": 3150230, "author_name": "gunesevitan", "author_url": "", "post_date": "03/15/2025 07:47:57", "content": "

Do you guys have adjacent slices from same tomograms on different folds?

", "votes": null, "replies": [] } ] }, { "id": 3150240, "author_name": "cnumber", "author_url": "", "post_date": "03/15/2025 08:03:30", "content": "

No, I'm being careful with that.

", "votes": null, "replies": [ { "id": 3150245, "author_name": "gunesevitan", "author_url": "", "post_date": "03/15/2025 08:16:24", "content": "

I haven't made any submissions yet, but I have couple ideas. I think It could be related to either different scales i.e., voxel spacing or too many false positives. We know that test set only has 0 or 1 motor on each tomogram and also there could unseen scales on test set.

", "votes": null, "replies": [ { "id": 3150309, "author_name": "andreizamfir", "author_url": "", "post_date": "03/15/2025 10:36:57", "content": "

In the training starter pipeline provided by the host, the split is done at tomogram level to avoid this issue of having slices from the same tomogram on different folds.

\n

In terms of cross-validation, I have also experienced this issue, where my models inferring on the validation set scored above 0.9 on the competition metric while hovering around 0.7 on the lb. For last week's efforts, I discarded that and moved my attention to dataset creation for training and post processing.

\n

One of my findings when playing around with extra augmentations during training is that when adding it, the training and validation loss are nicely coupled, and they stabilize at a lower validation loss.

\n

However, this always translated with a worse score on the LB (without changing any of the inference hyper-params, which could surely play a role).

\n

I believe these changes are caused by the domain shift, we do not know if there's other resolutions, there's no access to the voxel spacing to adapt post processing based on that, even weird aspect ratios could impact YOLO's letterboxing and thus object size (which I would have expected to benefit from heavy augs?).

", "votes": null, "replies": [ { "id": 3150318, "author_name": "yuanzhezhou", "author_url": "", "post_date": "03/15/2025 10:47:57", "content": "

How long time does it take for your submission? It seems to be very slow …

", "votes": null, "replies": [ { "id": 3150321, "author_name": "andreizamfir", "author_url": "", "post_date": "03/15/2025 10:53:35", "content": "

Around 3 hours

", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 3150383, "author_name": "cnumber", "author_url": "", "post_date": "03/15/2025 12:51:33", "content": "

At first I thought the CV/LB gap could be explained by the difference in tomogram ratio with and without motors, but as you point out, it cannot explain why data augmentation breaks the correlation, so I also feel that a domain shift is taking place.

", "votes": null, "replies": [] }, { "id": 3150398, "author_name": "shtljw", "author_url": "", "post_date": "03/15/2025 13:09:07", "content": "

The situation become very complicated when LB and CV are not consistent. It is hard to validate trained model and hard to decide LB/CV which should we trust.

", "votes": null, "replies": [ { "id": 3156969, "author_name": "ivashnyov", "author_url": "", "post_date": "03/22/2025 19:16:43", "content": "

Got exactly the same problem. 0.8 CV 0.25 LB. No data leakage. Probably hard augmetations will help, I used the basic ones for a baseline

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3151764, "author_name": "yuanzhezhou", "author_url": "", "post_date": "03/17/2025 04:24:37", "content": "

Tried to submit slightly different yolo models the scores vary a lot.

", "votes": null, "replies": [] }, { "id": 3151765, "author_name": "zacchaeus", "author_url": "", "post_date": "03/17/2025 04:25:07", "content": "

I use only 0/1-motor tomograms for validation and currently having an AUC of 0.79

", "votes": null, "replies": [] }, { "id": 3155430, "author_name": "itsuki9180", "author_url": "", "post_date": "03/21/2025 03:03:44", "content": "

CV:1.0, LB:0.0 ¯_(ツ)_/¯

", "votes": null, "replies": [] }, { "id": 3157873, "author_name": "garyzhao13", "author_url": "", "post_date": "03/24/2025 01:17:48", "content": "

Me too :( best wish

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3145212": "~~I'll start updating my results when I join this competition. (Currently running LuxAI)\n~~\n\nI haven't made a table yet, but I'm having a CV of over 0.9, while my LB is stll wondering around 0.5~0.6.\nThe correlation is very very weak, and I'm begining to think that this competition is a \"trust LB\" type of competition, which means that build a CV/LB table is almost meaningless.\nThere is still the possibility that my CV method is not good, so I'd be happy to hear from anyone who has good CV/LB correlation!\n\n| model | cv | lb |\n| --- | --- | --- |", "3146073": "haha, battle of giants", "3146101": "battle for ranking 1", "3146113": "Do you use 3D U-Net?", "3146753": "one fold, CV 0.84, but LB only scored 0.018, seems like data leakage, but I can't find any leakage in my training process till now. :(", "3146758": "It can't be the domain gap, can it?", "3147550": "I have CV 0.87 and LB 0.55, there is a gap, but not that huge. Maybe you can check your inference pipeline or subission format.", "3147558": "I have checked for many times, now I suspect the reason is domain gap and my model overfit to trian set, but I am not sure about it. I wonder if others encounter this CV-LB not same problem. If so, the problem maybe caused by domain gap between train and hidden test set.", "3147560": "And host mention that there is population shift between train set and test set in [this disscussion](https://www.kaggle.com/competitions/byu-locating-bacterial-flagellar-motors-2025/discussion/566137#3142976), if I understand correctly, the population shift means domain gap in train and hidden test set.", "3147581": "Thank you, that explain why i'm getting ~0.3 gap between CV、LB. You can try using yolov8n and train for 30 epochs like the starting notebook, I can get similar LB result with starting notebooks setup.", "3150217": "I'm having the same problem right now.", "3150230": "Do you guys have adjacent slices from same tomograms on different folds?", "3150240": "No, I'm being careful with that.", "3150245": "I haven't made any submissions yet, but I have couple ideas. I think It could be related to either different scales i.e., voxel spacing or too many false positives. We know that test set only has 0 or 1 motor on each tomogram and also there could unseen scales on test set.", "3150309": "In the training starter pipeline provided by the host, the split is done at tomogram level to avoid this issue of having slices from the same tomogram on different folds.\n\nIn terms of cross-validation, I have also experienced this issue, where my models inferring on the validation set scored above 0.9 on the competition metric while hovering around 0.7 on the lb. For last week's efforts, I discarded that and moved my attention to dataset creation for training and post processing.\n\nOne of my findings when playing around with extra augmentations during training is that when adding it, the training and validation loss are nicely coupled, and they stabilize at a lower validation loss.\n\nHowever, this always translated with a worse score on the LB (without changing any of the inference hyper-params, which could surely play a role).\n\nI believe these changes are caused by the domain shift, we do not know if there's other resolutions, there's no access to the voxel spacing to adapt post processing based on that, even weird aspect ratios could impact YOLO's letterboxing and thus object size (which I would have expected to benefit from heavy augs?).", "3150318": "How long time does it take for your submission? It seems to be very slow ...", "3150321": "Around 3 hours", "3150383": "At first I thought the CV/LB gap could be explained by the difference in tomogram ratio with and without motors, but as you point out, it cannot explain why data augmentation breaks the correlation, so I also feel that a domain shift is taking place.", "3150386": "Playing with yolo", "3150389": "Still can't get a decent CV score :(\n(was hoping to get it days ago, but still having some trouble)", "3150398": "The situation become very complicated when LB and CV are not consistent. It is hard to validate trained model and hard to decide LB/CV which should we trust.", "3151764": "Tried to submit slightly different yolo models the scores vary a lot.", "3151765": "I use only 0/1-motor tomograms for validation and currently having an AUC of 0.79", "3155430": "CV:1.0, LB:0.0 ¯\\_(ツ)_/¯", "3156969": "Got exactly the same problem. 0.8 CV 0.25 LB. No data leakage. Probably hard augmetations will help, I used the basic ones for a baseline", "3157873": "Me too :( best wish" }, "source": "meta" }