In this competition, we have about 21000 training images and 15000 test images.
\nPublic LB contains 31% test images (=4500 images).
KFold CV score +0.001 means that additional 21 images are predicted correctly.
\nOn the other hand, LB +0.001 means that the increase of correct prediction is only 4 or 5, which is about 4~5 times smaller number than CV +0.001.
Therefore, Trust CV is not bad strategy for selection of final submissions.
\nHow do you think about this?
In this competition, we have about 21000 training images and 15000 test images.
\nPublic LB contains 31% test images (=4500 images).
KFold CV score +0.001 means that additional 21 images are predicted correctly.
\nOn the other hand, LB +0.001 means that the increase of correct prediction is only 4 or 5, which is about 4~5 times smaller number than CV +0.001.
Therefore, Trust CV is not bad strategy for selection of final submissions.
\nHow do you think about this?
well we had some leaks of private lb so this should be helpful https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/207410
", "rawMarkdown": "well we had some leaks of private lb so this should be helpful https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/207410", "votes": null }, { "id": "1206500", "postDate": "02/17/2021 11:50:04", "content": "The number of images in LB doesn't matter since it's very noisy. Even if it had had 100k images, I would have highly not recommend to you to estimate your models according the LB. A good CV strategy usually is much better way to estimate your models even if it has a much smaller number of images. The Stratified K-fold is a good example. 5 folds should be the best. I got some interesting results after switching from stratified train test split to stratified K-fold.
", "rawMarkdown": "The number of images in LB doesn't matter since it's very noisy. Even if it had had 100k images, I would have highly not recommend to you to estimate your models according the LB. A good CV strategy usually is much better way to estimate your models even if it has a much smaller number of images. The Stratified K-fold is a good example. 5 folds should be the best. I got some interesting results after switching from stratified train test split to stratified K-fold.", "votes": null }, { "id": "1206749", "postDate": "02/17/2021 14:41:20", "content": "Very good point, trusting lb is basicaly having a validation strategy with less images therefore less robust. Completely agree with you!
", "rawMarkdown": "Very good point, trusting lb is basicaly having a validation strategy with less images therefore less robust. Completely agree with you!", "votes": null } ], "comments": [ { "id": 1206040, "author_name": "mithilsalunkhe", "author_url": "", "post_date": "02/17/2021 06:17:14", "content": "well we had some leaks of private lb so this should be helpful https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/207410
", "votes": null, "replies": [] }, { "id": 1206500, "author_name": "vadimtimakin", "author_url": "", "post_date": "02/17/2021 11:50:04", "content": "The number of images in LB doesn't matter since it's very noisy. Even if it had had 100k images, I would have highly not recommend to you to estimate your models according the LB. A good CV strategy usually is much better way to estimate your models even if it has a much smaller number of images. The Stratified K-fold is a good example. 5 folds should be the best. I got some interesting results after switching from stratified train test split to stratified K-fold.
", "votes": null, "replies": [] }, { "id": 1206749, "author_name": "yannmajewski", "author_url": "", "post_date": "02/17/2021 14:41:20", "content": "Very good point, trusting lb is basicaly having a validation strategy with less images therefore less robust. Completely agree with you!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1206021": "In this competition, we have about 21000 training images and 15000 test images. \nPublic LB contains 31% test images (=4500 images).\n\nKFold CV score +0.001 means that additional 21 images are predicted correctly.\nOn the other hand, LB +0.001 means that the increase of correct prediction is only 4 or 5, which is about 4~5 times smaller number than CV +0.001.\n\nTherefore, Trust CV is not bad strategy for selection of final submissions.\nHow do you think about this?", "1206040": "well we had some leaks of private lb so this should be helpful https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/207410", "1206500": "The number of images in LB doesn't matter since it's very noisy. Even if it had had 100k images, I would have highly not recommend to you to estimate your models according the LB. A good CV strategy usually is much better way to estimate your models even if it has a much smaller number of images. The Stratified K-fold is a good example. 5 folds should be the best. I got some interesting results after switching from stratified train test split to stratified K-fold.", "1206749": "Very good point, trusting lb is basicaly having a validation strategy with less images therefore less robust. Completely agree with you!" }, "source": "meta" }