![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F33bc2aabb91c7ed3c0a4922f63758776%2FSelection_999(4722).png?generation=1706703897494348&alt=media\")
What does this tell me? What scenario would result in such wildly different optimal thresholds?
\nThanks!
What does this tell me? What scenario would result in such wildly different optimal thresholds?
\nThanks!
single model or ensembles? single model maybe more sensitive .
", "rawMarkdown": "single model or ensembles? single model maybe more sensitive .", "votes": null }, { "id": "2628336", "postDate": "01/31/2024 08:35:58", "content": "Single model. Sensitive to what? I mean what's the mechanism behind it? Why the model underpredicts so much? I have never experienced such a massive AND consistent difference between LB and CV thresholds before.
", "rawMarkdown": "Single model. Sensitive to what? I mean what's the mechanism behind it? Why the model underpredicts so much? I have never experienced such a massive AND consistent difference between LB and CV thresholds before.", "votes": null }, { "id": "2628377", "postDate": "01/31/2024 09:04:22", "content": "It means you overfitted the public leaderboard! 😅
", "rawMarkdown": "It means you overfitted the public leaderboard! 😅", "votes": null }, { "id": "2628401", "postDate": "01/31/2024 09:27:15", "content": "I know, but I want to understand what causes such huge threshold discrepancies between two seemingly similar images. Like I usually see something like 0.35-0.65 optimal threshold range. But here it's a 10x difference.
", "rawMarkdown": "I know, but I want to understand what causes such huge threshold discrepancies between two seemingly similar images. Like I usually see something like 0.35-0.65 optimal threshold range. But here it's a 10x difference.", "votes": null }, { "id": "2628417", "postDate": "01/31/2024 09:37:58", "content": "I see a few possibilities:
\nBUG?
\ncan't tell unless we know the score values. if the score values differs by too much, could be bug.
\nif there is no bug, here is the reason:
Not much of a difference in absolute values, only in thresholds
\nCV @ th 0.5 = 0.875
\nLB @ th 0.05 = 0.870
Is that on kidney 3 dense ?
\nAlso what is your LB score @0.5?
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fb55c3f75d514baca24e9a8008a74c258%2FSelection_999(4731).png?generation=1706720105023319&alt=media\")
we are more used to region dice.
\nfor boundary dice, it is different.
If it's 0.5, what's the public score? These are some local results I have, which show different patterns. I believe, besides the data itself, it's greatly related to our model architecture and data augmentation.![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F856355%2F16a9585642af5d3a685307435b8350bf%2F5741706751861_.pic.jpg?generation=1706751937233619&alt=media\")
@lihaoweicvch
\nFor me, such differences in the threshold value for the public lb gives really different scores. Do you have the same case?
@mohammad2012191 My single convolution based model is the same case, ensemble or ViT are much better
", "rawMarkdown": "mohammad2012191 My single convolution based model is the same case, ensemble or ViT are much better", "votes": null }, { "id": "2631958", "postDate": "02/02/2024 04:18:42", "content": "\nThese different patterns look interesting! What differences in the architecture, augmentations or other may have caused these?
", "rawMarkdown": "lihaoweicvch \n\nThese different patterns look interesting! What differences in the architecture, augmentations or other may have caused these?", "votes": null } ], "comments": [ { "id": 2628309, "author_name": "lihaoweicvch", "author_url": "", "post_date": "01/31/2024 08:09:01", "content": "single model or ensembles? single model maybe more sensitive .
", "votes": null, "replies": [ { "id": 2628336, "author_name": "sakvaua", "author_url": "", "post_date": "01/31/2024 08:35:58", "content": "Single model. Sensitive to what? I mean what's the mechanism behind it? Why the model underpredicts so much? I have never experienced such a massive AND consistent difference between LB and CV thresholds before.
", "votes": null, "replies": [ { "id": 2628638, "author_name": "hengck23", "author_url": "", "post_date": "01/31/2024 12:25:01", "content": "BUG?
\ncan't tell unless we know the score values. if the score values differs by too much, could be bug.
\nif there is no bug, here is the reason:
Not much of a difference in absolute values, only in thresholds
\nCV @ th 0.5 = 0.875
\nLB @ th 0.05 = 0.870
Is that on kidney 3 dense ?
\nAlso what is your LB score @0.5?
we are more used to region dice.
\nfor boundary dice, it is different.
If it's 0.5, what's the public score? These are some local results I have, which show different patterns. I believe, besides the data itself, it's greatly related to our model architecture and data augmentation.
@lihaoweicvch
\nFor me, such differences in the threshold value for the public lb gives really different scores. Do you have the same case?
@mohammad2012191 My single convolution based model is the same case, ensemble or ViT are much better
", "votes": null, "replies": [] } ] }, { "id": 2631958, "author_name": "crustacean", "author_url": "", "post_date": "02/02/2024 04:18:42", "content": "\nThese different patterns look interesting! What differences in the architecture, augmentations or other may have caused these?
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2628377, "author_name": "optimo", "author_url": "", "post_date": "01/31/2024 09:04:22", "content": "It means you overfitted the public leaderboard! 😅
", "votes": null, "replies": [ { "id": 2628401, "author_name": "sakvaua", "author_url": "", "post_date": "01/31/2024 09:27:15", "content": "I know, but I want to understand what causes such huge threshold discrepancies between two seemingly similar images. Like I usually see something like 0.35-0.65 optimal threshold range. But here it's a 10x difference.
", "votes": null, "replies": [ { "id": 2628417, "author_name": "optimo", "author_url": "", "post_date": "01/31/2024 09:37:58", "content": "I see a few possibilities:
\n