Hi. I'm relatively new here at this competition.
\nAfter reading some discussion, I realize train and test data are different in terms of overlapping.
\nIn my experiment, I use stratified groupkfold (n_splits=5), and CV and LB correlation of single fold is not so strong (not weak though).
\nConsidering public/private split ratio, I guess it is acceptable to trust LB with paying attention to CV score.
\nWhat's your opinion and strategy?
Thank you.
", "messages": [ { "id": 1577647, "postDate": "2021-11-10T09:14:05.023Z", "content": "Hi. I'm relatively new here at this competition.
\nAfter reading some discussion, I realize train and test data are different in terms of overlapping.
\nIn my experiment, I use stratified groupkfold (n_splits=5), and CV and LB correlation of single fold is not so strong (not weak though).
\nConsidering public/private split ratio, I guess it is acceptable to trust LB with paying attention to CV score.
\nWhat's your opinion and strategy?
Thank you.
", "rawMarkdown": "Hi. I'm relatively new here at this competition.\n\nAfter reading some discussion, I realize train and test data are different in terms of overlapping.\nIn my experiment, I use stratified groupkfold (n_splits=5), and CV and LB correlation of single fold is not so strong (not weak though).\nConsidering public/private split ratio, I guess it is acceptable to trust LB with paying attention to CV score.\nWhat's your opinion and strategy?\n\nThank you.", "votes": 10 }, { "id": 1581774, "postDate": "2021-11-14T08:33:22.257Z", "content": "what you probably want to do is test out your CV:
\nIf you find that there is a strong correlation between the LB and the CV, you can trust your CV.
\nBUT!!
\nOnly up to a point.
\nUsually, you will find out that as you get to the highest ranks on the leaderboard, your CV-LB correlation starts to lower. from there, you probably should look closely into the data to try and find the samples that are \"not aligned\" or the samples that mess up your models.
It all depends on the competition itself and the data. sometimes [often] if you did everything correct: you should trust your CV. whoever, there are some situations where coming up with a meaningful CV that won't break until the end of the competition is really really hard. when this is the case: you probably want to treat the LB as \"another fold\".
", "rawMarkdown": "what you probably want to do is test out your CV: \n1. Train multiple [different] models and get various CV scores.\n2. Submit them all to the LB. \n3. Plot [scatter] the CV scores you got vs the LB scores you got.\n4. Optional: measure [spearman / maybe also person] correlation of the LB scores to the CV scores.\n\nIf you find that there is a strong correlation between the LB and the CV, you can trust your CV.\n\n\n**BUT!!**\n\nOnly up to a point.\nUsually, you will find out that as you get to the highest ranks on the leaderboard, your CV-LB correlation starts to lower. from there, you probably should look closely into the data to try and find the samples that are \"not aligned\" or the samples that mess up your models. \n\nIt all depends on the competition itself and the data. sometimes [often] if you did everything correct: you should trust your CV. whoever, there are some situations where coming up with a meaningful CV that won't break until the end of the competition is really really hard. when this is the case: you probably want to treat the LB as \"another fold\". ", "votes": 5 }, { "id": 1581839, "postDate": "2021-11-14T09:27:37.203Z", "content": "In my case, ｗhen the CV (stratified kfold for cell_type) increased by 0.01, the LB also increased by 0.01.
\nCV 0.2717 -> LB 0.295
\nCV 0.2817 -> LB 0.304
Great correlation!!
\nI'll try kfolds (now single fold).
The above score is the result of single fold out of 10 folds.
", "rawMarkdown": "The above score is the result of single fold out of 10 folds.", "votes": 2 } ] }, { "id": 1578216, "postDate": "2021-11-10T18:52:08.353Z", "content": "My strategy:
\n1- High-risk submission ( Not CV but High in LB)
\n2- Low-risk submission ( CV but good in LB)
This is the strategy I often take.
", "rawMarkdown": "This is the strategy I often take." }, { "id": 1578767, "postDate": "2021-11-11T10:20:57.700Z", "rawMarkdown": "", "votes": 7, "isDeleted": true } ] }, { "id": 1578434, "postDate": "2021-11-11T03:26:37.183Z", "content": "Always trust CV!
", "rawMarkdown": "Always trust CV!", "replies": [ { "id": 1578639, "postDate": "2021-11-11T08:28:30.307Z", "content": "Not always correct, depends on the data setup. Some competitions have different test data than train data and then you might need to trust LB. I have no idea what's the case in this competition.
", "rawMarkdown": "Not always correct, depends on the data setup. Some competitions have different test data than train data and then you might need to trust LB. I have no idea what's the case in this competition.", "votes": 16 }, { "id": 1578677, "postDate": "2021-11-11T09:07:13.177Z", "content": "Thnak you @philippsinger
\nI have learned something new from your answer.
Always trust philippsinger!
", "rawMarkdown": "Always trust philippsinger!", "votes": 16 }, { "id": 1579012, "postDate": "2021-11-11T13:03:09.657Z", "content": "I'm very well aware of the uncertainties in this data set.
\nI don't see a straightforward way how to construct the cross validation. Once the problems above are addressed one should be able to trust the CV score.
\nI have seen some suggestions for the second one. The host needs to use conditional IoU thresholds instead of the entire range of IoU values.
\nFor the first one, one can perhaps identify the overlapping regions just by burning some predictions to see the changes in the public LB score. But then what's gonna happen for the private LB data?
I agree. Shake will happen to some extent, and more with non-random public/private split.
\nOne easy way is that we can use more less-overlapping images for validation dataset.
what you probably want to do is test out your CV:
\nIf you find that there is a strong correlation between the LB and the CV, you can trust your CV.
\nBUT!!
\nOnly up to a point.
\nUsually, you will find out that as you get to the highest ranks on the leaderboard, your CV-LB correlation starts to lower. from there, you probably should look closely into the data to try and find the samples that are \"not aligned\" or the samples that mess up your models.
It all depends on the competition itself and the data. sometimes [often] if you did everything correct: you should trust your CV. whoever, there are some situations where coming up with a meaningful CV that won't break until the end of the competition is really really hard. when this is the case: you probably want to treat the LB as \"another fold\".
", "votes": 5, "replies": [] }, { "id": 1581839, "author_name": "Yamame🐟", "author_url": "", "post_date": "2021-11-14T09:27:37.203000", "content": "In my case, ｗhen the CV (stratified kfold for cell_type) increased by 0.01, the LB also increased by 0.01.
\nCV 0.2717 -> LB 0.295
\nCV 0.2817 -> LB 0.304
Great correlation!!
\nI'll try kfolds (now single fold).
The above score is the result of single fold out of 10 folds.
", "votes": 2, "replies": [] } ] }, { "id": 1578216, "author_name": "Faisal Alsrheed", "author_url": "", "post_date": "2021-11-10T18:52:08.353000", "content": "My strategy:
\n1- High-risk submission ( Not CV but High in LB)
\n2- Low-risk submission ( CV but good in LB)
This is the strategy I often take.
", "votes": 0, "replies": [] }, { "id": 1578767, "author_name": "", "author_url": "", "post_date": "2021-11-11T10:20:57.700000", "content": "", "votes": 7, "replies": [] } ] }, { "id": 1578434, "author_name": "Tolga", "author_url": "", "post_date": "2021-11-11T03:26:37.183000", "content": "Always trust CV!
", "votes": 0, "replies": [ { "id": 1578639, "author_name": "Psi", "author_url": "", "post_date": "2021-11-11T08:28:30.307000", "content": "Not always correct, depends on the data setup. Some competitions have different test data than train data and then you might need to trust LB. I have no idea what's the case in this competition.
", "votes": 16, "replies": [] }, { "id": 1578677, "author_name": "Faisal Alsrheed", "author_url": "", "post_date": "2021-11-11T09:07:13.177000", "content": "Thnak you @philippsinger
\nI have learned something new from your answer.
Always trust philippsinger!
", "votes": 16, "replies": [] }, { "id": 1579012, "author_name": "Tolga", "author_url": "", "post_date": "2021-11-11T13:03:09.657000", "content": "I'm very well aware of the uncertainties in this data set.
\nI don't see a straightforward way how to construct the cross validation. Once the problems above are addressed one should be able to trust the CV score.
\nI have seen some suggestions for the second one. The host needs to use conditional IoU thresholds instead of the entire range of IoU values.
\nFor the first one, one can perhaps identify the overlapping regions just by burning some predictions to see the changes in the public LB score. But then what's gonna happen for the private LB data?
I agree. Shake will happen to some extent, and more with non-random public/private split.
\nOne easy way is that we can use more less-overlapping images for validation dataset.