I know its a much repeated adage! But trust your CV rather than Public LB should be the preference in this competition
\n1) There is a lot of difference between CV and Public LB.. for me they are also not coorelated.
\nIn the discussion thread (https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/206795) as well
Anurag's CV is 0.85. LB 0.90
\nJunYong's 0.965 CV and 0.942 LB
\nVIcky's CV is 95.2 and LB is 95.6
2) 25% of data is for public LB
\n\"The final results will be based on the other 75%, so the final standings may be different.\"
3) The scores are very tightly clustered
\nScore : 0,972 : Top 7 --- Prize Money
\nScore : 0,971 : Next 3 (rank 8-10) - Gold
\nScore : 0,970 : Next 6 (rank 11-16) - Silver
\nScore : 0,969 : Next 11 (rank 17-27) -Silver
\nScore : 0,968 : Next 38 (rank 28-65) --Silver; Bronze
\nScore : 0,967 : Next 51 (rank 66-116) -- Bronze; No medal
\nScore: 0,966 : Next 179 (rank 117-295) -- Has a Public Notebook ; around 15% of participants
\nScore: 0,965 : Next 298 (rank 296-593) - Has many public NBs ; around 25% of participants
Top 10% between 0.967-0.972
\ntop 50% between 0.965-972
\nTop 11%50% between 0,966-0,967 : Some with good CV may easily be in medal contention!
One of the tightest competition in recent times..
\nAs a result of these I expect a shakeup, Please focus on CV not Public LB
There are many strategies to improve and your final LB will depend on how good is your CV strategy.. I will suggest a few based on earlier discussions
\nA) Focus on augmentations - it has helped me and as per many discussions is key here as well
\nB) Increase number of folds - moving from 5 fold to 10 fold helped me
\n3) Chris Deotte's :How To CV and How To Ensemble OOF Files is a useful discussion
\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614
In case you have any suggestions for the last couple of weeks, please do share as well!
\nFinally, a shakeup is imminent, hope and pray we are in the right side of the shakeUP and not get ShakeOUT!!
", "messages": [ { "id": "1220488", "postDate": "02/28/2021 04:01:28", "content": "I know its a much repeated adage! But trust your CV rather than Public LB should be the preference in this competition
\n1) There is a lot of difference between CV and Public LB.. for me they are also not coorelated.
\nIn the discussion thread (https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/206795) as well
Anurag's CV is 0.85. LB 0.90
\nJunYong's 0.965 CV and 0.942 LB
\nVIcky's CV is 95.2 and LB is 95.6
2) 25% of data is for public LB
\n\"The final results will be based on the other 75%, so the final standings may be different.\"
3) The scores are very tightly clustered
\nScore : 0,972 : Top 7 --- Prize Money
\nScore : 0,971 : Next 3 (rank 8-10) - Gold
\nScore : 0,970 : Next 6 (rank 11-16) - Silver
\nScore : 0,969 : Next 11 (rank 17-27) -Silver
\nScore : 0,968 : Next 38 (rank 28-65) --Silver; Bronze
\nScore : 0,967 : Next 51 (rank 66-116) -- Bronze; No medal
\nScore: 0,966 : Next 179 (rank 117-295) -- Has a Public Notebook ; around 15% of participants
\nScore: 0,965 : Next 298 (rank 296-593) - Has many public NBs ; around 25% of participants
Top 10% between 0.967-0.972
\ntop 50% between 0.965-972
\nTop 11%50% between 0,966-0,967 : Some with good CV may easily be in medal contention!
One of the tightest competition in recent times..
\nAs a result of these I expect a shakeup, Please focus on CV not Public LB
There are many strategies to improve and your final LB will depend on how good is your CV strategy.. I will suggest a few based on earlier discussions
\nA) Focus on augmentations - it has helped me and as per many discussions is key here as well
\nB) Increase number of folds - moving from 5 fold to 10 fold helped me
\n3) Chris Deotte's :How To CV and How To Ensemble OOF Files is a useful discussion
\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614
In case you have any suggestions for the last couple of weeks, please do share as well!
\nFinally, a shakeup is imminent, hope and pray we are in the right side of the shakeUP and not get ShakeOUT!!
", "rawMarkdown": "I know its a much repeated adage! But trust your CV rather than Public LB should be the preference in this competition\n\n1) There is a lot of difference between CV and Public LB.. for me they are also not coorelated.\nIn the discussion thread (https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/206795) as well\n\nAnurag's CV is 0.85. LB 0.90 \nJunYong's 0.965 CV and 0.942 LB\nVIcky's CV is 95.2 and LB is 95.6\n\n2) 25% of data is for public LB\n\"The final results will be based on the other 75%, so the final standings may be different.\"\n\n3) The scores are very tightly clustered\nScore : 0,972 : Top 7 --- Prize Money\nScore : 0,971 : Next 3 (rank 8-10) - Gold\nScore : 0,970 : Next 6 (rank 11-16) - Silver\nScore : 0,969 : Next 11 (rank 17-27) -Silver\nScore : 0,968 : Next 38 (rank 28-65) --Silver; Bronze\nScore : 0,967 : Next 51 (rank 66-116) -- Bronze; No medal\nScore: 0,966 : Next 179 (rank 117-295) -- Has a Public Notebook ; around 15% of participants\nScore: 0,965 : Next 298 (rank 296-593) - Has many public NBs ; around 25% of participants\n\nTop 10% between 0.967-0.972\ntop 50% between 0.965-972 \nTop 11%50% between 0,966-0,967 : Some with good CV may easily be in medal contention!\n\nOne of the tightest competition in recent times..\nAs a result of these I expect a shakeup, Please focus on CV not Public LB\n\nThere are many strategies to improve and your final LB will depend on how good is your CV strategy.. I will suggest a few based on earlier discussions \n\nA) Focus on augmentations - it has helped me and as per many discussions is key here as well\nB) Increase number of folds - moving from 5 fold to 10 fold helped me\n3) Chris Deotte's :How To CV and How To Ensemble OOF Files is a useful discussion\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614\n\n\n\nIn case you have any suggestions for the last couple of weeks, please do share as well!\n\nFinally, a shakeup is imminent, hope and pray we are in the right side of the shakeUP and not get ShakeOUT!!", "votes": null }, { "id": "1220676", "postDate": "02/28/2021 08:47:34", "content": "I have observed that if cv increases,lb Increases,i personally Don't think there will be bad shakeup in this competition as i find correlation between cv and lb. Anurag,vicky and other people's name and cv that you mentioned.i would like to ask very important question \"are they all using 5fold cross validation\" as shown in sin's public kernel? If no \"then their cv strategy probably has leak and not good for this competition\"
\nWith sin's validation strategy, i always find cv lb correlation.. So i think \"if your cv strategy is NOT trustable then you will observe cv lb gap and good amount of shakeup\" but those who using cv that correlates with lb,should get good result in private data as well
very valid point @mobassir ..
\nFor me 10 fold helped more than 5 fold, my correlation with 5 fold CV and Public LB was not great
the problem is with the Public LB so thickly clustered and some likely using not great CV strategy, there may be a decent shakeup
\nFingers crossed dose not hurt us too much!
", "rawMarkdown": "very valid point @mobassir ..\nFor me 10 fold helped more than 5 fold, my correlation with 5 fold CV and Public LB was not great\n\n the problem is with the Public LB so thickly clustered and some likely using not great CV strategy, there may be a decent shakeup\n\n\nFingers crossed dose not hurt us too much!", "votes": null }, { "id": "1220707", "postDate": "02/28/2021 09:17:18", "content": "If we at least group same patient to one fold, The correlation should be somehow strong. If there are some leaks between each fold like similar image from same patient, CV is more unstable.
", "rawMarkdown": "If we at least group same patient to one fold, The correlation should be somehow strong. If there are some leaks between each fold like similar image from same patient, CV is more unstable.", "votes": null }, { "id": "1221090", "postDate": "02/28/2021 17:01:12", "content": "To clarify, the numbers that I mentioned there is the one-off largest inconsistency.
\nOther time, I do find CV and LB correlate well.
", "rawMarkdown": "To clarify, the numbers that I mentioned [there](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/206795) is the **one-off** largest inconsistency.\n\nOther time, I do find CV and LB correlate well.", "votes": null }, { "id": "1221111", "postDate": "02/28/2021 17:16:30", "content": "Thanks @jy2tong
\nsimilar with me, most of the CV and LB correlate well; but in some cases I have seen differences - so unsure how the mix is between public and private set. Also, agree that the difference may not be huge but some may be due to the folds i created in random
", "rawMarkdown": "Thanks @jy2tong \n\nsimilar with me, most of the CV and LB correlate well; but in some cases I have seen differences - so unsure how the mix is between public and private set. Also, agree that the difference may not be huge but some may be due to the folds i created in random", "votes": null }, { "id": "1221138", "postDate": "02/28/2021 17:34:02", "content": "\n\nB) Increase number of folds - moving from 5 fold to 10 fold helped me
\n
So you have a grand total of 8 ETT Abnormal images per fold, assuming you stratified them. That is probably the reason for fluctuating CV.
", "rawMarkdown": "> B) Increase number of folds - moving from 5 fold to 10 fold helped me\n\nSo you have a grand total of 8 ETT Abnormal images per fold, assuming you stratified them. That is probably the reason for fluctuating CV.", "votes": null }, { "id": "1221503", "postDate": "03/01/2021 03:41:38", "content": "Thanks for sharing this analysis. Yeah, this competition is very augmentation based from my experience, and very computational resourceful. As long as your CV strategy is solid as Mobassir stated, correlation between CV and LB is correlates pretty well. I suggest looking here. Haha, takes a long time to train as well if you're using kaggle notebooks, so don't rush it!
", "rawMarkdown": "Thanks for sharing this analysis. Yeah, this competition is very augmentation based from my experience, and very computational resourceful. As long as your CV strategy is solid as Mobassir stated, correlation between CV and LB is correlates pretty well. I suggest looking [here](https://www.kaggle.com/marisakamozz/cv-strategy-in-the-kaggle-environment). Haha, takes a long time to train as well if you're using kaggle notebooks, so don't rush it!", "votes": null } ], "comments": [ { "id": 1220676, "author_name": "mobassir", "author_url": "", "post_date": "02/28/2021 08:47:34", "content": "I have observed that if cv increases,lb Increases,i personally Don't think there will be bad shakeup in this competition as i find correlation between cv and lb. Anurag,vicky and other people's name and cv that you mentioned.i would like to ask very important question \"are they all using 5fold cross validation\" as shown in sin's public kernel? If no \"then their cv strategy probably has leak and not good for this competition\"
\nWith sin's validation strategy, i always find cv lb correlation.. So i think \"if your cv strategy is NOT trustable then you will observe cv lb gap and good amount of shakeup\" but those who using cv that correlates with lb,should get good result in private data as well
very valid point @mobassir ..
\nFor me 10 fold helped more than 5 fold, my correlation with 5 fold CV and Public LB was not great
the problem is with the Public LB so thickly clustered and some likely using not great CV strategy, there may be a decent shakeup
\nFingers crossed dose not hurt us too much!
", "votes": null, "replies": [] }, { "id": 1220707, "author_name": "steamedsheep", "author_url": "", "post_date": "02/28/2021 09:17:18", "content": "If we at least group same patient to one fold, The correlation should be somehow strong. If there are some leaks between each fold like similar image from same patient, CV is more unstable.
", "votes": null, "replies": [] } ] }, { "id": 1221090, "author_name": "jy2tong", "author_url": "", "post_date": "02/28/2021 17:01:12", "content": "To clarify, the numbers that I mentioned there is the one-off largest inconsistency.
\nOther time, I do find CV and LB correlate well.
", "votes": null, "replies": [ { "id": 1221111, "author_name": "kmldas", "author_url": "", "post_date": "02/28/2021 17:16:30", "content": "Thanks @jy2tong
\nsimilar with me, most of the CV and LB correlate well; but in some cases I have seen differences - so unsure how the mix is between public and private set. Also, agree that the difference may not be huge but some may be due to the folds i created in random
", "votes": null, "replies": [] } ] }, { "id": 1221138, "author_name": "fffrrt", "author_url": "", "post_date": "02/28/2021 17:34:02", "content": "\n\nB) Increase number of folds - moving from 5 fold to 10 fold helped me
\n
So you have a grand total of 8 ETT Abnormal images per fold, assuming you stratified them. That is probably the reason for fluctuating CV.
", "votes": null, "replies": [] }, { "id": 1221503, "author_name": "andyjianzhou", "author_url": "", "post_date": "03/01/2021 03:41:38", "content": "Thanks for sharing this analysis. Yeah, this competition is very augmentation based from my experience, and very computational resourceful. As long as your CV strategy is solid as Mobassir stated, correlation between CV and LB is correlates pretty well. I suggest looking here. Haha, takes a long time to train as well if you're using kaggle notebooks, so don't rush it!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1220488": "I know its a much repeated adage! But trust your CV rather than Public LB should be the preference in this competition\n\n1) There is a lot of difference between CV and Public LB.. for me they are also not coorelated.\nIn the discussion thread (https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/206795) as well\n\nAnurag's CV is 0.85. LB 0.90 \nJunYong's 0.965 CV and 0.942 LB\nVIcky's CV is 95.2 and LB is 95.6\n\n2) 25% of data is for public LB\n\"The final results will be based on the other 75%, so the final standings may be different.\"\n\n3) The scores are very tightly clustered\nScore : 0,972 : Top 7 --- Prize Money\nScore : 0,971 : Next 3 (rank 8-10) - Gold\nScore : 0,970 : Next 6 (rank 11-16) - Silver\nScore : 0,969 : Next 11 (rank 17-27) -Silver\nScore : 0,968 : Next 38 (rank 28-65) --Silver; Bronze\nScore : 0,967 : Next 51 (rank 66-116) -- Bronze; No medal\nScore: 0,966 : Next 179 (rank 117-295) -- Has a Public Notebook ; around 15% of participants\nScore: 0,965 : Next 298 (rank 296-593) - Has many public NBs ; around 25% of participants\n\nTop 10% between 0.967-0.972\ntop 50% between 0.965-972 \nTop 11%50% between 0,966-0,967 : Some with good CV may easily be in medal contention!\n\nOne of the tightest competition in recent times..\nAs a result of these I expect a shakeup, Please focus on CV not Public LB\n\nThere are many strategies to improve and your final LB will depend on how good is your CV strategy.. I will suggest a few based on earlier discussions \n\nA) Focus on augmentations - it has helped me and as per many discussions is key here as well\nB) Increase number of folds - moving from 5 fold to 10 fold helped me\n3) Chris Deotte's :How To CV and How To Ensemble OOF Files is a useful discussion\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614\n\n\n\nIn case you have any suggestions for the last couple of weeks, please do share as well!\n\nFinally, a shakeup is imminent, hope and pray we are in the right side of the shakeUP and not get ShakeOUT!!", "1220676": "I have observed that if cv increases,lb Increases,i personally Don't think there will be bad shakeup in this competition as i find correlation between cv and lb. Anurag,vicky and other people's name and cv that you mentioned.i would like to ask very important question \"are they all using 5fold cross validation\" as shown in sin's public kernel? If no \"then their cv strategy probably has leak and not good for this competition\"\nWith sin's validation strategy, i always find cv lb correlation.. So i think \"if your cv strategy is NOT trustable then you will observe cv lb gap and good amount of shakeup\" but those who using cv that correlates with lb,should get good result in private data as well", "1220703": "very valid point @mobassir ..\nFor me 10 fold helped more than 5 fold, my correlation with 5 fold CV and Public LB was not great\n\n the problem is with the Public LB so thickly clustered and some likely using not great CV strategy, there may be a decent shakeup\n\n\nFingers crossed dose not hurt us too much!", "1220707": "If we at least group same patient to one fold, The correlation should be somehow strong. If there are some leaks between each fold like similar image from same patient, CV is more unstable.", "1221090": "To clarify, the numbers that I mentioned [there](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/206795) is the **one-off** largest inconsistency.\n\nOther time, I do find CV and LB correlate well.", "1221111": "Thanks @jy2tong \n\nsimilar with me, most of the CV and LB correlate well; but in some cases I have seen differences - so unsure how the mix is between public and private set. Also, agree that the difference may not be huge but some may be due to the folds i created in random", "1221138": "> B) Increase number of folds - moving from 5 fold to 10 fold helped me\n\nSo you have a grand total of 8 ETT Abnormal images per fold, assuming you stratified them. That is probably the reason for fluctuating CV.", "1221503": "Thanks for sharing this analysis. Yeah, this competition is very augmentation based from my experience, and very computational resourceful. As long as your CV strategy is solid as Mobassir stated, correlation between CV and LB is correlates pretty well. I suggest looking [here](https://www.kaggle.com/marisakamozz/cv-strategy-in-the-kaggle-environment). Haha, takes a long time to train as well if you're using kaggle notebooks, so don't rush it!" }, "source": "meta" }