{ "id": 214310, "title": "Gap between CV and LB.", "url": "/competitions/cassava-leaf-disease-classification/discussion/214310", "author_name": "", "post_date": "2021-01-26T06:44:45.136715900Z", "votes": 1, "comment_count": 2, "views": 0, "content": "

Based on these two threads: Best single model score (post re-score) and Best single model, I found it is common that CV and LB have big gaps.
\nI may wondering if the distribution of test data is different from training data? e.g., test data is balanced with evenly distributed categories. Is that possible?

", "messages": [ { "id": "1170342", "postDate": "01/26/2021 06:44:45", "content": "

Based on these two threads: Best single model score (post re-score) and Best single model, I found it is common that CV and LB have big gaps.
\nI may wondering if the distribution of test data is different from training data? e.g., test data is balanced with evenly distributed categories. Is that possible?

", "rawMarkdown": "Based on these two threads: [Best single model score (post re-score)](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/203111) and [Best single model](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198805), I found it is common that CV and LB have big gaps. \nI may wondering if the distribution of test data is different from training data? e.g., test data is balanced with evenly distributed categories. Is that possible?", "votes": null }, { "id": "1170580", "postDate": "01/26/2021 09:44:01", "content": "

Hello!

\n

It's not a big thing to prove that the test dataset (at least public) is as unbalanced as the training dataset (my Naive Baseline Notebook where I simply predict label 3 for each image scores 0.6). Maybe the hidden test dataset is balanced, but that's arguable.
\nAs for the difference between LB and CV, numerous experienced users, as well as the 2019 competition winner, suggest that CV is way more reliable.

", "rawMarkdown": "Hello!\n\nIt's not a big thing to prove that the test dataset (at least public) is as unbalanced as the training dataset (my [Naive Baseline Notebook](https://www.kaggle.com/nickuzmenkov/cassava-leaf-disease-na-ve-baseline) where I simply predict label `3` for each image scores 0.6). Maybe the hidden test dataset is balanced, but that's arguable.\nAs for the difference between LB and CV, numerous experienced users, as well as the [2019 competition winner](https://www.kaggle.com/c/cassava-disease/discussion/94114), suggest that CV is way more reliable.", "votes": null }, { "id": "1171136", "postDate": "01/26/2021 16:43:58", "content": "

Great! Thanks for your reply.

", "rawMarkdown": "Great! Thanks for your reply.", "votes": null } ], "comments": [ { "id": 1170580, "author_name": "nickuzmenkov", "author_url": "", "post_date": "01/26/2021 09:44:01", "content": "

Hello!

\n

It's not a big thing to prove that the test dataset (at least public) is as unbalanced as the training dataset (my Naive Baseline Notebook where I simply predict label 3 for each image scores 0.6). Maybe the hidden test dataset is balanced, but that's arguable.
\nAs for the difference between LB and CV, numerous experienced users, as well as the 2019 competition winner, suggest that CV is way more reliable.

", "votes": null, "replies": [ { "id": 1171136, "author_name": "leonshangguan", "author_url": "", "post_date": "01/26/2021 16:43:58", "content": "

Great! Thanks for your reply.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1170342": "Based on these two threads: [Best single model score (post re-score)](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/203111) and [Best single model](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198805), I found it is common that CV and LB have big gaps. \nI may wondering if the distribution of test data is different from training data? e.g., test data is balanced with evenly distributed categories. Is that possible?", "1170580": "Hello!\n\nIt's not a big thing to prove that the test dataset (at least public) is as unbalanced as the training dataset (my [Naive Baseline Notebook](https://www.kaggle.com/nickuzmenkov/cassava-leaf-disease-na-ve-baseline) where I simply predict label `3` for each image scores 0.6). Maybe the hidden test dataset is balanced, but that's arguable.\nAs for the difference between LB and CV, numerous experienced users, as well as the [2019 competition winner](https://www.kaggle.com/c/cassava-disease/discussion/94114), suggest that CV is way more reliable.", "1171136": "Great! Thanks for your reply." }, "source": "meta" }