{ "id": 217716, "title": "Noisy test, train and Public LB: what are you doing to prevent a shakeup?", "url": "/competitions/cassava-leaf-disease-classification/discussion/217716", "author_name": "Kamal Das", "post_date": "2021-02-08T04:13:50.540000", "votes": 8, "comment_count": 17, "views": 0, "content": "

Hi All,

\n

As has been discussed, the train data as well as test data is noisy

\n

The Public LB is also littered with submissions early in the competition where we had a different scoring system and showed the scores 0.02-0.03 higher

\n

Given all this, what are you doing to select your best models?

\n

Based on your CV? Highest LB score? What metrics are you using to evaluate your codes and their likely performance in the noise private data set?

\n

Thank you for your suggestion. I am getting somewhat lost in what to do here!

", "messages": [ { "id": 1192789, "postDate": "2021-02-09T10:14:00.867Z", "content": "
\n

what are you doing to prevent a shakeup?

\n
\n

I pray.

", "rawMarkdown": "> what are you doing to prevent a shakeup?\n\nI pray.", "votes": 12, "replies": [ { "id": 1193197, "postDate": "2021-02-09T14:02:54.630Z", "content": "

😂 kaggle will make up all (re)gain faith in the Almighty!

", "rawMarkdown": "😂 kaggle will make up all (re)gain faith in the Almighty! " } ] }, { "id": 1190835, "postDate": "2021-02-08T04:13:50.540Z", "content": "

Hi All,

\n

As has been discussed, the train data as well as test data is noisy

\n

The Public LB is also littered with submissions early in the competition where we had a different scoring system and showed the scores 0.02-0.03 higher

\n

Given all this, what are you doing to select your best models?

\n

Based on your CV? Highest LB score? What metrics are you using to evaluate your codes and their likely performance in the noise private data set?

\n

Thank you for your suggestion. I am getting somewhat lost in what to do here!

", "rawMarkdown": "Hi All,\n\nAs has been discussed, the train data as well as test data is noisy\n\nThe Public LB is also littered with submissions early in the competition where we had a different scoring system and showed the scores 0.02-0.03 higher\n\nGiven all this, what are you doing to select your best models?\n\nBased on your CV? Highest LB score? What metrics are you using to evaluate your codes and their likely performance in the noise private data set?\n\nThank you for your suggestion. I am getting somewhat lost in what to do here!", "votes": 8 }, { "id": 1196170, "postDate": "2021-02-11T09:50:33.507Z", "content": "

No tta used. It might overfit the 31% test data and may not generalise well for the remaining 69%.

", "rawMarkdown": "No tta used. It might overfit the 31% test data and may not generalise well for the remaining 69%.", "votes": 1 }, { "id": 1196112, "postDate": "2021-02-11T09:04:29.467Z", "content": "

I removed every non deterministic TTA from my submission, but I think it will a mess anyway, so enjoy the fiesta 🎉

", "rawMarkdown": "I removed every non deterministic TTA from my submission, but I think it will a mess anyway, so enjoy the fiesta 🎉", "votes": 1, "replies": [ { "id": 1196114, "postDate": "2021-02-11T09:07:41.480Z", "content": "

as @nroman said, we may pray 😄

", "rawMarkdown": "as @nroman said, we may pray 😄", "votes": 1 } ] }, { "id": 1193424, "postDate": "2021-02-09T16:24:11.733Z", "content": "
\n

The Public LB is also littered with submissions early in the competition where we had a different scoring system and showed the scores 0.02-0.03 higher

\n
\n

I thought all public LB scores were re-calculated after we switched to the new scoring system. Am I wrong?

", "rawMarkdown": "> The Public LB is also littered with submissions early in the competition where we had a different scoring system and showed the scores 0.02-0.03 higher\n\nI thought all public LB scores were re-calculated after we switched to the new scoring system. Am I wrong?", "votes": 1, "replies": [ { "id": 1193934, "postDate": "2021-02-10T01:36:30.457Z", "content": "

When did that happen?

", "rawMarkdown": "When did that happen?\n", "votes": 1 }, { "id": 1193938, "postDate": "2021-02-10T01:45:08.457Z", "content": "

@hammaadali A couple of month ago: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200803

", "rawMarkdown": "@hammaadali A couple of month ago: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200803", "votes": 1 }, { "id": 1193949, "postDate": "2021-02-10T02:08:31.490Z", "content": "

I think there may be a bug in the Public LB, one of my submissions was 0.901 as per old scoreboard and it showed 0.901 in the Public LB but 0.898 in the code/notebook page
\nin the public LB I still see submissions which are 2 months old that are ranked higher than me giving me hope that I may sneak into a bronze position…

", "rawMarkdown": "I think there may be a bug in the Public LB, one of my submissions was 0.901 as per old scoreboard and it showed 0.901 in the Public LB but 0.898 in the code/notebook page\nin the public LB I still see submissions which are 2 months old that are ranked higher than me giving me hope that I may sneak into a bronze position...", "votes": 2 }, { "id": 1193961, "postDate": "2021-02-10T02:22:49.567Z", "content": "

For instance
\nhttps://www.kaggle.com/hengck23/notebook6262cd4cd3 shows a Public Score: 0.897

\n

https://www.kaggle.com/underwearfitting/clean-inference-kernel-8xtta-lb902 shows a Public Score
\n0.896

\n

But if we go to the code section and sort by best score, https://www.kaggle.com/hengck23/notebook6262cd4cd3 shows a Score: 0.903 (vs 0.897)

\n

and https://www.kaggle.com/underwearfitting/clean-inference-kernel-8xtta-lb902 shows a Score 0.902 (vs 0.896)

\n

\n

Both these are 2 month old, anyone who copied from them initially and did not get a better score later would be showing high in Pubic LB but will get shaken out when the results come

", "rawMarkdown": "For instance\nhttps://www.kaggle.com/hengck23/notebook6262cd4cd3 shows a Public Score: 0.897\n\nhttps://www.kaggle.com/underwearfitting/clean-inference-kernel-8xtta-lb902 shows a Public Score\n0.896\n\nBut if we go to the code section and sort by best score, https://www.kaggle.com/hengck23/notebook6262cd4cd3 shows a Score: 0.903 (vs 0.897)\n\nand https://www.kaggle.com/underwearfitting/clean-inference-kernel-8xtta-lb902 shows a Score 0.902 (vs 0.896)\n\n[](https://storage.googleapis.com/kagglesdsdata/datasets/1149344/1926956/shake.JPG?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=databundle-worker-v2%40kaggle-161607.iam.gserviceaccount.com%2F20210210%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20210210T022208Z&X-Goog-Expires=172799&X-Goog-SignedHeaders=host&X-Goog-Signature=37b8cfd445c06749fa0c6395f86a848445f05cd456774f7eedb214848f68b33f90be439acfc276f0c1958ba61beb4c68cf32a696f1fecdb910c41acd71d368cc1884754fcac7f838c7101d41fa3a5f4ebf5b18b230531be279c92b5d93ffdc72f94d845a8521a9694658a19ab4ea16abc3f600aeeaf5966dacbea3f59e3c0bac5a7799025189cf22b638fcfb9d410c4263e8d17723e94520f92d3027ac8047253d85dbeff39742d2c5dbd19c130a7b8546b0d94ff8e530192d94716114419cc4568ac39154b1d68f2bbd69a779dbe2eb59ff6e9d44fdfccffc8f5d7c9faf3e46ee412dc10b16cb8917d21d754113d3a9d8f9bfcb667f24e60a165e974ca1f6ad)\n\nBoth these are 2 month old, anyone who copied from them initially and did not get a better score later would be showing high in Pubic LB but will get shaken out when the results come\n\n\n", "votes": 1 }, { "id": 1194014, "postDate": "2021-02-10T02:52:45.257Z", "content": "

Interesting…

", "rawMarkdown": "Interesting... ", "votes": 1 }, { "id": 1195483, "postDate": "2021-02-10T20:49:13.523Z", "content": "

Hi @kmldas

\n

The difference between those scores is caused by rescoring at the beginning of the competition. The link is here. This was due to some images in the test set which were used in in-class competition before. The scores in the notebook section is not representative for the old notebooks before the rescoring.

", "rawMarkdown": "Hi @kmldas \n\nThe difference between those scores is caused by rescoring at the beginning of the competition. The link is [here](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200803). This was due to some images in the test set which were used in in-class competition before. The scores in the notebook section is not representative for the old notebooks before the rescoring. ", "votes": 1 }, { "id": 1195701, "postDate": "2021-02-11T02:47:27.037Z", "content": "

I agree @snnclsr ! My point here is that some older notebooks in the Public LB have these submissions and scores (which I feel are not updated) . ..Those who copied the earlier notebooks and did not work on new submissions may be showing high public LB score but may not be anywhere close to those ranks…and this may lead to these being shaked out.

\n

As a result, we may see benefits to participants with better CVs; and who have continued to work on the competition and improve their codes

", "rawMarkdown": "I agree @snnclsr ! My point here is that some older notebooks in the Public LB have these submissions and scores (which I feel are not updated) . ..Those who copied the earlier notebooks and did not work on new submissions may be showing high public LB score but may not be anywhere close to those ranks...and this may lead to these being shaked out.\n\nAs a result, we may see benefits to participants with better CVs; and who have continued to work on the competition and improve their codes\n", "votes": 1 }, { "id": 1196841, "postDate": "2021-02-11T17:39:22.943Z", "content": "

Hi again @kmldas

\n

Actually, all of the LB scores were recalculated. So the situation you said is not possible technically. For example, if someone submitted 0.903 notebook before the rescorring, his/her score didnot stay 0.903 in the LB. If the score is dropped to let's say 0.897, the new score on the LB for that particular user became 0.897 as well not 0.903. The scores in the notebooks section are just misleading for the ones before the rescorring.

", "rawMarkdown": "Hi again @kmldas \n\nActually, all of the LB scores were recalculated. So the situation you said is not possible technically. For example, if someone submitted 0.903 notebook before the rescorring, his/her score didnot stay 0.903 in the LB. If the score is dropped to let's say 0.897, the new score on the LB for that particular user became 0.897 as well not 0.903. The scores in the notebooks section are just misleading for the ones before the rescorring.", "votes": 2 } ] }, { "id": 1191467, "postDate": "2021-02-08T13:20:31.597Z", "content": "

The CV and LB scores are being really weird and not sure what should we trust more.
\nThough I guess we will pick one best CV model and one best LB model (yes both are different models). Anyone better opinion, please enlighten.

", "rawMarkdown": "The CV and LB scores are being really weird and not sure what should we trust more. \nThough I guess we will pick one best CV model and one best LB model (yes both are different models). Anyone better opinion, please enlighten.", "votes": 2, "replies": [ { "id": 1198541, "postDate": "2021-02-13T06:03:55.763Z", "content": "

Empirically speaking，CV is more trustworthy

", "rawMarkdown": "Empirically speaking，CV is more trustworthy" } ] }, { "id": 1192788, "postDate": "2021-02-09T10:14:00.867Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1192789, "author_name": "Roman", "author_url": "", "post_date": "2021-02-09T10:14:00.867000", "content": "
\n

what are you doing to prevent a shakeup?

\n
\n

I pray.

", "votes": 12, "replies": [ { "id": 1193197, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-02-09T14:02:54.630000", "content": "

😂 kaggle will make up all (re)gain faith in the Almighty!

", "votes": 0, "replies": [] } ] }, { "id": 1196170, "author_name": "Vicky Goyal", "author_url": "", "post_date": "2021-02-11T09:50:33.507000", "content": "

No tta used. It might overfit the 31% test data and may not generalise well for the remaining 69%.

", "votes": 1, "replies": [] }, { "id": 1196112, "author_name": "Riccardo", "author_url": "", "post_date": "2021-02-11T09:04:29.467000", "content": "

I removed every non deterministic TTA from my submission, but I think it will a mess anyway, so enjoy the fiesta 🎉

", "votes": 1, "replies": [ { "id": 1196114, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-02-11T09:07:41.480000", "content": "

as @nroman said, we may pray 😄

", "votes": 1, "replies": [] } ] }, { "id": 1193424, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2021-02-09T16:24:11.733000", "content": "
\n

The Public LB is also littered with submissions early in the competition where we had a different scoring system and showed the scores 0.02-0.03 higher

\n
\n

I thought all public LB scores were re-calculated after we switched to the new scoring system. Am I wrong?

", "votes": 1, "replies": [ { "id": 1193934, "author_name": "Hammad Ali", "author_url": "", "post_date": "2021-02-10T01:36:30.457000", "content": "

When did that happen?

", "votes": 1, "replies": [] }, { "id": 1193938, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2021-02-10T01:45:08.457000", "content": "

@hammaadali A couple of month ago: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200803

", "votes": 1, "replies": [] }, { "id": 1193949, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-02-10T02:08:31.490000", "content": "

I think there may be a bug in the Public LB, one of my submissions was 0.901 as per old scoreboard and it showed 0.901 in the Public LB but 0.898 in the code/notebook page
\nin the public LB I still see submissions which are 2 months old that are ranked higher than me giving me hope that I may sneak into a bronze position…

", "votes": 2, "replies": [] }, { "id": 1193961, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-02-10T02:22:49.567000", "content": "

For instance
\nhttps://www.kaggle.com/hengck23/notebook6262cd4cd3 shows a Public Score: 0.897

\n

https://www.kaggle.com/underwearfitting/clean-inference-kernel-8xtta-lb902 shows a Public Score
\n0.896

\n

But if we go to the code section and sort by best score, https://www.kaggle.com/hengck23/notebook6262cd4cd3 shows a Score: 0.903 (vs 0.897)

\n

and https://www.kaggle.com/underwearfitting/clean-inference-kernel-8xtta-lb902 shows a Score 0.902 (vs 0.896)

\n

\n

Both these are 2 month old, anyone who copied from them initially and did not get a better score later would be showing high in Pubic LB but will get shaken out when the results come

", "votes": 1, "replies": [] }, { "id": 1194014, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2021-02-10T02:52:45.257000", "content": "

Interesting…

", "votes": 1, "replies": [] }, { "id": 1195483, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2021-02-10T20:49:13.523000", "content": "

Hi @kmldas

\n

The difference between those scores is caused by rescoring at the beginning of the competition. The link is here. This was due to some images in the test set which were used in in-class competition before. The scores in the notebook section is not representative for the old notebooks before the rescoring.

", "votes": 1, "replies": [] }, { "id": 1195701, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-02-11T02:47:27.037000", "content": "

I agree @snnclsr ! My point here is that some older notebooks in the Public LB have these submissions and scores (which I feel are not updated) . ..Those who copied the earlier notebooks and did not work on new submissions may be showing high public LB score but may not be anywhere close to those ranks…and this may lead to these being shaked out.

\n

As a result, we may see benefits to participants with better CVs; and who have continued to work on the competition and improve their codes

", "votes": 1, "replies": [] }, { "id": 1196841, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2021-02-11T17:39:22.943000", "content": "

Hi again @kmldas

\n

Actually, all of the LB scores were recalculated. So the situation you said is not possible technically. For example, if someone submitted 0.903 notebook before the rescorring, his/her score didnot stay 0.903 in the LB. If the score is dropped to let's say 0.897, the new score on the LB for that particular user became 0.897 as well not 0.903. The scores in the notebooks section are just misleading for the ones before the rescorring.

", "votes": 2, "replies": [] } ] }, { "id": 1191467, "author_name": "Hammad Ali", "author_url": "", "post_date": "2021-02-08T13:20:31.597000", "content": "

The CV and LB scores are being really weird and not sure what should we trust more.
\nThough I guess we will pick one best CV model and one best LB model (yes both are different models). Anyone better opinion, please enlighten.

", "votes": 2, "replies": [ { "id": 1198541, "author_name": "Yiheng Peng", "author_url": "", "post_date": "2021-02-13T06:03:55.763000", "content": "

Empirically speaking，CV is more trustworthy

", "votes": 0, "replies": [] } ] }, { "id": 1192788, "author_name": "", "author_url": "", "post_date": "2021-02-09T10:14:00.867000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1192789": "> what are you doing to prevent a shakeup?\n\nI pray.", "1190835": "Hi All,\n\nAs has been discussed, the train data as well as test data is noisy\n\nThe Public LB is also littered with submissions early in the competition where we had a different scoring system and showed the scores 0.02-0.03 higher\n\nGiven all this, what are you doing to select your best models?\n\nBased on your CV? Highest LB score? What metrics are you using to evaluate your codes and their likely performance in the noise private data set?\n\nThank you for your suggestion. I am getting somewhat lost in what to do here!", "1196170": "No tta used. It might overfit the 31% test data and may not generalise well for the remaining 69%.", "1196112": "I removed every non deterministic TTA from my submission, but I think it will a mess anyway, so enjoy the fiesta 🎉", "1193424": "> The Public LB is also littered with submissions early in the competition where we had a different scoring system and showed the scores 0.02-0.03 higher\n\nI thought all public LB scores were re-calculated after we switched to the new scoring system. Am I wrong?", "1191467": "The CV and LB scores are being really weird and not sure what should we trust more. \nThough I guess we will pick one best CV model and one best LB model (yes both are different models). Anyone better opinion, please enlighten.", "1192788": "" } }