{ "id": 209318, "title": "Is this competition a lottery?", "url": "/competitions/cassava-leaf-disease-classification/discussion/209318", "author_name": "arutema47", "post_date": "2021-01-07T05:57:16.990000", "votes": 65, "comment_count": 36, "views": 0, "content": "

https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/201471

\n

Regarding the noise in training, public and private data, I'm feeling that this competition is a lottery. If there is noise in private (I'm sure there is due to the PB leak), what are we predicting in the models? Are we predicting the annotator's habits..?😷
\nThe competition will be far interesting if hosts intended to fix the noise in the private set (and the competition will be so much more meaningful!)

", "messages": [ { "id": 1142071, "postDate": "2021-01-07T05:57:16.990Z", "content": "

https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/201471

\n

Regarding the noise in training, public and private data, I'm feeling that this competition is a lottery. If there is noise in private (I'm sure there is due to the PB leak), what are we predicting in the models? Are we predicting the annotator's habits..?😷
\nThe competition will be far interesting if hosts intended to fix the noise in the private set (and the competition will be so much more meaningful!)

", "rawMarkdown": "https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/201471\n\nRegarding the noise in training, public and private data, I'm feeling that this competition is a lottery. If there is noise in private (I'm sure there is due to the PB leak), what are we predicting in the models? Are we predicting the annotator's habits..?😷 \nThe competition will be far interesting if hosts intended to fix the noise in the private set (and the competition will be so much more meaningful!)", "votes": 65 }, { "id": 1142756, "postDate": "2021-01-07T15:21:30.803Z", "content": "

This paper might be addressing few concerns around picking models/validating in the presence of noise: Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels.

\n
Under diagonally-dominant class-conditional label noise, the main conclusions are as follows.\n\nA classifier maximizing its accuracy on the noisy distribution is guaranteed to maximize the accuracy on clean distribution.\nWe can obtain an optimal classifier by maximizing training accuracy on sufficiently many noisy samples.\nA noisy validation set is reliable.\n
\n

My initial guess would be, if label noise is systematic, e.g class conditional noise when most of the leaves are healthy annotators tending to label an image as healthy even though there exists a single disease leaf at the corner, then models would be able to learn it without needing to apply any sophisticated robust training. But if this is not the case, e.g. noise is not predictable by the models than training a robust model with methods studied in literature should be our safe bet for private leaderboard.

", "rawMarkdown": "This paper might be addressing few concerns around picking models/validating in the presence of noise: [Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels](https://arxiv.org/pdf/2012.04193.pdf).\n\n```\nUnder diagonally-dominant class-conditional label noise, the main conclusions are as follows.\n\nA classifier maximizing its accuracy on the noisy distribution is guaranteed to maximize the accuracy on clean distribution.\nWe can obtain an optimal classifier by maximizing training accuracy on sufficiently many noisy samples.\nA noisy validation set is reliable.\n```\n\nMy initial guess would be, if label noise is systematic, e.g class conditional noise when most of the leaves are healthy annotators tending to label an image as healthy even though there exists a single disease leaf at the corner, then models would be able to learn it without needing to apply any sophisticated robust training. But if this is not the case, e.g. noise is not predictable by the models than training a robust model with methods studied in literature should be our safe bet for private leaderboard. ", "votes": 24, "replies": [ { "id": 1142805, "postDate": "2021-01-07T15:42:31.890Z", "content": "

This is a great paper I read few days ago.

\n

It clearly prove that methodology is all we need.

", "rawMarkdown": "This is a great paper I read few days ago. \n\nIt clearly prove that methodology is all we need. \n\n", "votes": 4 }, { "id": 1142815, "postDate": "2021-01-07T15:52:13.713Z", "content": "

Yes, it released a lot of stress out of me too :)

", "rawMarkdown": "Yes, it released a lot of stress out of me too :)", "votes": 1 }, { "id": 1142839, "postDate": "2021-01-07T16:12:58.150Z", "content": "

And it's the first paper I read to validate on noisy data. While taking care to distiguish and explain systematic and non-systematic noise.

", "rawMarkdown": "And it's the first paper I read to validate on noisy data. While taking care to distiguish and explain systematic and non-systematic noise. ", "votes": 1 }, { "id": 1142893, "postDate": "2021-01-07T16:44:33.753Z", "content": "

Exactly! Before coming to this paper I was always wondering if a robust learning paper reports their results on noisy validation or not. They usually use clean validation set, which is not very suitable for real world.

", "rawMarkdown": "Exactly! Before coming to this paper I was always wondering if a robust learning paper reports their results on noisy validation or not. They usually use clean validation set, which is not very suitable for real world.", "votes": 1 }, { "id": 1142920, "postDate": "2021-01-07T17:07:23.400Z", "content": "

Yep ! And as said by @hengck23 in this topic . It may be impossible in real life to collect only clean data to validate your model.

", "rawMarkdown": "Yep ! And as said by @hengck23 in this [topic](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/201471#1106726) . It may be impossible in real life to collect only clean data to validate your model. ", "votes": 2 } ] }, { "id": 1142305, "postDate": "2021-01-07T09:57:42.737Z", "content": "

Self denoising works . I mean denoising done in Self supervised manner during training (and not manually)
\nIt improved both my CV and LB. I dont know what will happen on the private LB though.

\n

Anyway , these noise are more likely annotation errors (inherent to humans) than general and biased pattern of the whole data (the latter would make non sense).Thus, a good and well trained model should give better generalisation and better accuracy in both clean data and clean+noisy data, than a model which overfitted on noisy training data.

\n

But the difficulty is to train a model which maximize both Robustness and Accuracy in the presence of noise and most of our models tend to quickly overfit on noise by just memorizing them. That why it's really difficult to get further improvement at some point. This is total contrary of lottery IMHO

", "rawMarkdown": "Self denoising works . I mean denoising done in Self supervised manner during training (and not manually)\nIt improved both my CV and LB. I dont know what will happen on the private LB though. \n \nAnyway , these noise are more likely annotation errors (inherent to humans) than general and biased pattern of the whole data (the latter would make non sense).Thus, a good and well trained model should give better generalisation and better accuracy in both clean data and clean+noisy data, than a model which overfitted on noisy training data.\n\nBut the difficulty is to train a model which maximize both Robustness and Accuracy in the presence of noise and most of our models tend to quickly overfit on noise by just **memorizing them**. That why it's really difficult to get further improvement at some point. This is total contrary of lottery IMHO\n", "votes": 12, "replies": [ { "id": 1142324, "postDate": "2021-01-07T10:11:20.490Z", "content": "

Thanks for the deep insights. I tried self denoising and that's my best model right now but quite far from the top team's scores. Guess I need to dig deeper.

\n

I liked the previous PANDA competition where the private set noise was guaranteed to be much lower than the train/public sets, this competition would have been interesting for me if the randomness were comforted.

", "rawMarkdown": "Thanks for the deep insights. I tried self denoising and that's my best model right now but quite far from the top team's scores. Guess I need to dig deeper.\n\nI liked the previous PANDA competition where the private set noise was guaranteed to be much lower than the train/public sets, this competition would have been interesting for me if the randomness were comforted." }, { "id": 1142373, "postDate": "2021-01-07T10:42:31.873Z", "content": "
\n

I liked the previous PANDA competition where the private set noise was guaranteed to be much lower than the train/public sets

\n
\n

I didn't participate on PANDA competition . But did Kaggle confirm this while the competition was still ongoing ?

\n

I hope there will be much less noise on private LB here too.

", "rawMarkdown": "> I liked the previous PANDA competition where the private set noise was guaranteed to be much lower than the train/public sets\n\n\nI didn't participate on PANDA competition . But did Kaggle confirm this while the competition was still ongoing ? \n\nI hope there will be much less noise on private LB here too. " }, { "id": 1142391, "postDate": "2021-01-07T10:56:01.003Z", "content": "

Yes, the hosts released a paper on how the data were labeled before the competition.
\nFor public, the labeling was done by one expert and for private, the labeling was a voting between three experts.

", "rawMarkdown": "Yes, the hosts released a paper on how the data were labeled before the competition.\nFor public, the labeling was done by one expert and for private, the labeling was a voting between three experts.", "votes": 2 }, { "id": 1143704, "postDate": "2021-01-08T02:36:30.867Z", "content": "

Do you mind sharing what are some of the methods available for self-supervised denoising?

", "rawMarkdown": "Do you mind sharing what are some of the methods available for self-supervised denoising?" }, { "id": 1143882, "postDate": "2021-01-08T05:46:04.640Z", "content": "

You can read our solution in PANDA
\nhttps://github.com/kentaroy47/Kaggle-PANDA-1st-place-solution

", "rawMarkdown": "You can read our solution in PANDA\nhttps://github.com/kentaroy47/Kaggle-PANDA-1st-place-solution", "votes": 2 }, { "id": 1143913, "postDate": "2021-01-08T06:04:35.327Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1143914, "postDate": "2021-01-08T06:08:29.883Z", "content": "

Thanks a lot :)

", "rawMarkdown": "Thanks a lot :)" }, { "id": 1144188, "postDate": "2021-01-08T09:48:52.457Z", "content": "

@kyoshioka47 could you share the paper on how the data were labeled :D thanks

", "rawMarkdown": "@kyoshioka47 could you share the paper on how the data were labeled :D thanks" }, { "id": 1144350, "postDate": "2021-01-08T12:14:44.330Z", "content": "

that paper I mentioned was for PANDA, you can find it in the competition site :)

", "rawMarkdown": "that paper I mentioned was for PANDA, you can find it in the competition site :)", "votes": 1 }, { "id": 1145352, "postDate": "2021-01-09T04:52:51.780Z", "content": "

@serigne Can you kindly link some resource on the self supervised data denoising? Coming across image denoising task when searching on google.

", "rawMarkdown": "@serigne Can you kindly link some resource on the self supervised data denoising? Coming across image denoising task when searching on google." } ] }, { "id": 1144896, "postDate": "2021-01-08T18:27:45.633Z", "content": "

Does anyone know why the best public notebooks say they score LB 0.900+ when in list view, but when we go inside none score over 0.900?
\n\"\"
\n\"\"

", "rawMarkdown": "Does anyone know why the best public notebooks say they score `LB 0.900+` when in list view, but when we go inside none score over 0.900?\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2F24faacd71988322f89f808ff36e102e5%2Fone.png?generation=1610130442382506&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2F2e9febb1bf3d4910ee2af18711d7a55d%2Ftwo.png?generation=1610130452419433&alt=media)", "votes": 10, "replies": [ { "id": 1144904, "postDate": "2021-01-08T18:32:45.843Z", "content": "

@cdeotte I think its because some images where removed from the public/private test set and they did a leaderboard rescore

", "rawMarkdown": "@cdeotte I think its because some images where removed from the public/private test set and they did a leaderboard rescore", "votes": 5 }, { "id": 1144931, "postDate": "2021-01-08T18:50:37.533Z", "content": "

https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200803

\n

If you check the discussion above, you can see that public lb has been rescored.
\n(But the criteria for sorting notebooks have not been reset.)
\n@cdeotte

", "rawMarkdown": "https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200803\n\nIf you check the discussion above, you can see that public lb has been rescored.\n(But the criteria for sorting notebooks have not been reset.)\n@cdeotte ", "votes": 4 }, { "id": 1145041, "postDate": "2021-01-08T20:31:33.730Z", "content": "

thanks Yann and Heroseo

", "rawMarkdown": "thanks Yann and Heroseo", "votes": 2 } ] }, { "id": 1144182, "postDate": "2021-01-08T09:43:49.423Z", "content": "

In the wheat detection competition, the data set also had noise, but the host responded positively. But the host seldom uttered this game, which is very bad for us. I hope the host can come out and answer some questions.
\ne.g.
\n1.Is the label really wrong or because we are not experts and we can’t make a correct judgment
\n2.division criteria for the coexistence of multiple diseases

", "rawMarkdown": "In the wheat detection competition, the data set also had noise, but the host responded positively. But the host seldom uttered this game, which is very bad for us. I hope the host can come out and answer some questions.\ne.g. \n1.Is the label really wrong or because we are not experts and we can’t make a correct judgment\n2.division criteria for the coexistence of multiple diseases", "votes": 6, "replies": [ { "id": 1149468, "postDate": "2021-01-11T21:17:44.057Z", "content": "

This is very good question and should be answered. Is there a way we can tag host of the competition?

", "rawMarkdown": "This is very good question and should be answered. Is there a way we can tag host of the competition?", "votes": 1 } ] }, { "id": 1150915, "postDate": "2021-01-13T00:43:08.947Z", "content": "

So far I feel like my experiments have suggested:

\n\n

i dont know if im making the correct interpretation here but it seems to me that this suggests there is some level of 'noise' or blind luck at least in terms of predicting a small portion of borderline images. im not sure if this necessarily says anything for sure about errors in test labeling. i guess it would be surprising if the host could say with confidence that there cannot be a single error in test image labels as i figure it would usually be difficult to diagnose with complete accuracy from photos?

\n

hopefully there are still plenty of things people will figure out (with or without labeling noise) to make models more accurate overall.

", "rawMarkdown": "So far I feel like my experiments have suggested:\n\n- filtering out images which look suspiciously labeled from the training has not really had a big positive effect (though am already using some label smoothing). i think with original loss i managed to get around +0.3 or similar compared to a lower starting point.\n\n- making minor changes to inference...some of which surely should not be negative e.g. more tta cycles...has in some cases lowered LB score by 0.1-0.3. \n\ni dont know if im making the correct interpretation here but it seems to me that this suggests there is some level of 'noise' or blind luck at least in terms of predicting a small portion of borderline images. im not sure if this necessarily says anything for sure about errors in test labeling. i guess it would be surprising if the host could say with confidence that there cannot be a single error in test image labels as i figure it would usually be difficult to diagnose with complete accuracy from photos?\n\nhopefully there are still plenty of things people will figure out (with or without labeling noise) to make models more accurate overall.", "votes": 3 }, { "id": 1144121, "postDate": "2021-01-08T08:45:35.813Z", "content": "

It will be a lottery in terms of the leaderboard rank

\n

1) categorical accuracy
\n2) imbalance classes
\n3) noisy data
\n4) too narrow score between people

", "rawMarkdown": "It will be a lottery in terms of the leaderboard rank\n\n1) categorical accuracy\n2) imbalance classes\n3) noisy data\n4) too narrow score between people", "votes": 3 }, { "id": 1142497, "postDate": "2021-01-07T12:28:22.227Z", "content": "

What we need is clarification from the hosts about LB and private LB data. If it similar PANDA competition, we need change our model and Pretreatment🤥🤥🤥
\nThe gold miner competition will lose the meaning of competition. We want to defeat our opponents with wisdom and skill, not luck.(Although I like gold miners🙈

", "rawMarkdown": "What we need is clarification from the hosts about LB and private LB data. If it similar PANDA competition, we need change our model and Pretreatment🤥🤥🤥\nThe gold miner competition will lose the meaning of competition. We want to defeat our opponents with wisdom and skill, not luck.(Although I like gold miners🙈", "votes": 1 }, { "id": 1142262, "postDate": "2021-01-07T09:03:51.317Z", "content": "

How do you know that public and private labels are noisy?

", "rawMarkdown": "How do you know that public and private labels are noisy?", "votes": 2, "replies": [ { "id": 1142283, "postDate": "2021-01-07T09:32:04.537Z", "content": "

I'm speculating since the LB and PB were consistent when monitored during the data bleach

", "rawMarkdown": "I'm speculating since the LB and PB were consistent when monitored during the data bleach" }, { "id": 1142286, "postDate": "2021-01-07T09:37:21.290Z", "content": "

It only tells that public and private are probably random split. Why do you think that labels are noisy?

", "rawMarkdown": "It only tells that public and private are probably random split. Why do you think that labels are noisy?", "votes": 2 }, { "id": 1142292, "postDate": "2021-01-07T09:46:41.380Z", "content": "

I feel that labels are somewhat \"random\", when analyzing the confusion between disease and healthy.

\n

https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/202673
\nAs pointed out in this thread, the label annotation strategy may not be consistent especially when there is both disease and healthy leaf inside the image.

\n

But it's just my opinion, would love to hear your views too.

", "rawMarkdown": "I feel that labels are somewhat \"random\", when analyzing the confusion between disease and healthy.\n\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/202673\nAs pointed out in this thread, the label annotation strategy may not be consistent especially when there is both disease and healthy leaf inside the image.\n\nBut it's just my opinion, would love to hear your views too.", "votes": 4 }, { "id": 1142297, "postDate": "2021-01-07T09:50:55.513Z", "content": "

Ok, so training set labels are noisy, we still don't know if test set labels are noisy. Even if they are, it is the nature of the problem unless the test set noise is biased.

", "rawMarkdown": "Ok, so training set labels are noisy, we still don't know if test set labels are noisy. Even if they are, it is the nature of the problem unless the test set noise is biased.", "votes": 1 } ] }, { "id": 1142186, "postDate": "2021-01-07T08:11:09.043Z", "content": "

Well said. Because of this, I lost motivation after achieving 0.898 ☹️

", "rawMarkdown": "Well said. Because of this, I lost motivation after achieving 0.898 ☹️", "votes": 2 }, { "id": 1142133, "postDate": "2021-01-07T07:12:50.080Z", "content": "

yes, it is.

", "rawMarkdown": "yes, it is." }, { "id": 1145777, "postDate": "2021-01-09T10:27:28.587Z", "content": "

It would be nice if we could get some information on the quality of the test set. Eg. is it as noisy as the training set?

", "rawMarkdown": "It would be nice if we could get some information on the quality of the test set. Eg. is it as noisy as the training set?" }, { "id": 1142580, "postDate": "2021-01-07T13:33:57.483Z", "content": "

Same feelings here. Especially after the PB leak.

", "rawMarkdown": "Same feelings here. Especially after the PB leak." }, { "id": 1142107, "postDate": "2021-01-07T06:44:39.303Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1142756, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2021-01-07T15:21:30.803000", "content": "

This paper might be addressing few concerns around picking models/validating in the presence of noise: Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels.

\n
Under diagonally-dominant class-conditional label noise, the main conclusions are as follows.\n\nA classifier maximizing its accuracy on the noisy distribution is guaranteed to maximize the accuracy on clean distribution.\nWe can obtain an optimal classifier by maximizing training accuracy on sufficiently many noisy samples.\nA noisy validation set is reliable.\n
\n

My initial guess would be, if label noise is systematic, e.g class conditional noise when most of the leaves are healthy annotators tending to label an image as healthy even though there exists a single disease leaf at the corner, then models would be able to learn it without needing to apply any sophisticated robust training. But if this is not the case, e.g. noise is not predictable by the models than training a robust model with methods studied in literature should be our safe bet for private leaderboard.

", "votes": 24, "replies": [ { "id": 1142805, "author_name": "Serigne ", "author_url": "", "post_date": "2021-01-07T15:42:31.890000", "content": "

This is a great paper I read few days ago.

\n

It clearly prove that methodology is all we need.

", "votes": 4, "replies": [] }, { "id": 1142815, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2021-01-07T15:52:13.713000", "content": "

Yes, it released a lot of stress out of me too :)

", "votes": 1, "replies": [] }, { "id": 1142839, "author_name": "Serigne ", "author_url": "", "post_date": "2021-01-07T16:12:58.150000", "content": "

And it's the first paper I read to validate on noisy data. While taking care to distiguish and explain systematic and non-systematic noise.

", "votes": 1, "replies": [] }, { "id": 1142893, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2021-01-07T16:44:33.753000", "content": "

Exactly! Before coming to this paper I was always wondering if a robust learning paper reports their results on noisy validation or not. They usually use clean validation set, which is not very suitable for real world.

", "votes": 1, "replies": [] }, { "id": 1142920, "author_name": "Serigne ", "author_url": "", "post_date": "2021-01-07T17:07:23.400000", "content": "

Yep ! And as said by @hengck23 in this topic . It may be impossible in real life to collect only clean data to validate your model.

", "votes": 2, "replies": [] } ] }, { "id": 1142305, "author_name": "Serigne ", "author_url": "", "post_date": "2021-01-07T09:57:42.737000", "content": "

Self denoising works . I mean denoising done in Self supervised manner during training (and not manually)
\nIt improved both my CV and LB. I dont know what will happen on the private LB though.

\n

Anyway , these noise are more likely annotation errors (inherent to humans) than general and biased pattern of the whole data (the latter would make non sense).Thus, a good and well trained model should give better generalisation and better accuracy in both clean data and clean+noisy data, than a model which overfitted on noisy training data.

\n

But the difficulty is to train a model which maximize both Robustness and Accuracy in the presence of noise and most of our models tend to quickly overfit on noise by just memorizing them. That why it's really difficult to get further improvement at some point. This is total contrary of lottery IMHO

", "votes": 12, "replies": [ { "id": 1142324, "author_name": "arutema47", "author_url": "", "post_date": "2021-01-07T10:11:20.490000", "content": "

Thanks for the deep insights. I tried self denoising and that's my best model right now but quite far from the top team's scores. Guess I need to dig deeper.

\n

I liked the previous PANDA competition where the private set noise was guaranteed to be much lower than the train/public sets, this competition would have been interesting for me if the randomness were comforted.

", "votes": 0, "replies": [] }, { "id": 1142373, "author_name": "Serigne ", "author_url": "", "post_date": "2021-01-07T10:42:31.873000", "content": "
\n

I liked the previous PANDA competition where the private set noise was guaranteed to be much lower than the train/public sets

\n
\n

I didn't participate on PANDA competition . But did Kaggle confirm this while the competition was still ongoing ?

\n

I hope there will be much less noise on private LB here too.

", "votes": 0, "replies": [] }, { "id": 1142391, "author_name": "arutema47", "author_url": "", "post_date": "2021-01-07T10:56:01.003000", "content": "

Yes, the hosts released a paper on how the data were labeled before the competition.
\nFor public, the labeling was done by one expert and for private, the labeling was a voting between three experts.

", "votes": 2, "replies": [] }, { "id": 1143704, "author_name": "Sally80", "author_url": "", "post_date": "2021-01-08T02:36:30.867000", "content": "

Do you mind sharing what are some of the methods available for self-supervised denoising?

", "votes": 0, "replies": [] }, { "id": 1143882, "author_name": "arutema47", "author_url": "", "post_date": "2021-01-08T05:46:04.640000", "content": "

You can read our solution in PANDA
\nhttps://github.com/kentaroy47/Kaggle-PANDA-1st-place-solution

", "votes": 2, "replies": [] }, { "id": 1143913, "author_name": "", "author_url": "", "post_date": "2021-01-08T06:04:35.327000", "content": "", "votes": 0, "replies": [] }, { "id": 1143914, "author_name": "Sally80", "author_url": "", "post_date": "2021-01-08T06:08:29.883000", "content": "

Thanks a lot :)

", "votes": 0, "replies": [] }, { "id": 1144188, "author_name": "Benlei Cui", "author_url": "", "post_date": "2021-01-08T09:48:52.457000", "content": "

@kyoshioka47 could you share the paper on how the data were labeled :D thanks

", "votes": 0, "replies": [] }, { "id": 1144350, "author_name": "arutema47", "author_url": "", "post_date": "2021-01-08T12:14:44.330000", "content": "

that paper I mentioned was for PANDA, you can find it in the competition site :)

", "votes": 1, "replies": [] }, { "id": 1145352, "author_name": "Rahul Deora", "author_url": "", "post_date": "2021-01-09T04:52:51.780000", "content": "

@serigne Can you kindly link some resource on the self supervised data denoising? Coming across image denoising task when searching on google.

", "votes": 0, "replies": [] } ] }, { "id": 1144896, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-01-08T18:27:45.633000", "content": "

Does anyone know why the best public notebooks say they score LB 0.900+ when in list view, but when we go inside none score over 0.900?
\n\"\"
\n\"\"

", "votes": 10, "replies": [ { "id": 1144904, "author_name": "Yann Majewski", "author_url": "", "post_date": "2021-01-08T18:32:45.843000", "content": "

@cdeotte I think its because some images where removed from the public/private test set and they did a leaderboard rescore

", "votes": 5, "replies": [] }, { "id": 1144931, "author_name": "Heroseo", "author_url": "", "post_date": "2021-01-08T18:50:37.533000", "content": "

https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200803

\n

If you check the discussion above, you can see that public lb has been rescored.
\n(But the criteria for sorting notebooks have not been reset.)
\n@cdeotte

", "votes": 4, "replies": [] }, { "id": 1145041, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-01-08T20:31:33.730000", "content": "

thanks Yann and Heroseo

", "votes": 2, "replies": [] } ] }, { "id": 1144182, "author_name": "Benlei Cui", "author_url": "", "post_date": "2021-01-08T09:43:49.423000", "content": "

In the wheat detection competition, the data set also had noise, but the host responded positively. But the host seldom uttered this game, which is very bad for us. I hope the host can come out and answer some questions.
\ne.g.
\n1.Is the label really wrong or because we are not experts and we can’t make a correct judgment
\n2.division criteria for the coexistence of multiple diseases

", "votes": 6, "replies": [ { "id": 1149468, "author_name": "Sandeep", "author_url": "", "post_date": "2021-01-11T21:17:44.057000", "content": "

This is very good question and should be answered. Is there a way we can tag host of the competition?

", "votes": 1, "replies": [] } ] }, { "id": 1150915, "author_name": "Dave E", "author_url": "", "post_date": "2021-01-13T00:43:08.947000", "content": "

So far I feel like my experiments have suggested:

\n\n

i dont know if im making the correct interpretation here but it seems to me that this suggests there is some level of 'noise' or blind luck at least in terms of predicting a small portion of borderline images. im not sure if this necessarily says anything for sure about errors in test labeling. i guess it would be surprising if the host could say with confidence that there cannot be a single error in test image labels as i figure it would usually be difficult to diagnose with complete accuracy from photos?

\n

hopefully there are still plenty of things people will figure out (with or without labeling noise) to make models more accurate overall.

", "votes": 3, "replies": [] }, { "id": 1144121, "author_name": "kaggler", "author_url": "", "post_date": "2021-01-08T08:45:35.813000", "content": "

It will be a lottery in terms of the leaderboard rank

\n

1) categorical accuracy
\n2) imbalance classes
\n3) noisy data
\n4) too narrow score between people

", "votes": 3, "replies": [] }, { "id": 1142497, "author_name": "Mingjie Wang", "author_url": "", "post_date": "2021-01-07T12:28:22.227000", "content": "

What we need is clarification from the hosts about LB and private LB data. If it similar PANDA competition, we need change our model and Pretreatment🤥🤥🤥
\nThe gold miner competition will lose the meaning of competition. We want to defeat our opponents with wisdom and skill, not luck.(Although I like gold miners🙈

", "votes": 1, "replies": [] }, { "id": 1142262, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2021-01-07T09:03:51.317000", "content": "

How do you know that public and private labels are noisy?

", "votes": 2, "replies": [ { "id": 1142283, "author_name": "arutema47", "author_url": "", "post_date": "2021-01-07T09:32:04.537000", "content": "

I'm speculating since the LB and PB were consistent when monitored during the data bleach

", "votes": 0, "replies": [] }, { "id": 1142286, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2021-01-07T09:37:21.290000", "content": "

It only tells that public and private are probably random split. Why do you think that labels are noisy?

", "votes": 2, "replies": [] }, { "id": 1142292, "author_name": "arutema47", "author_url": "", "post_date": "2021-01-07T09:46:41.380000", "content": "

I feel that labels are somewhat \"random\", when analyzing the confusion between disease and healthy.

\n

https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/202673
\nAs pointed out in this thread, the label annotation strategy may not be consistent especially when there is both disease and healthy leaf inside the image.

\n

But it's just my opinion, would love to hear your views too.

", "votes": 4, "replies": [] }, { "id": 1142297, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2021-01-07T09:50:55.513000", "content": "

Ok, so training set labels are noisy, we still don't know if test set labels are noisy. Even if they are, it is the nature of the problem unless the test set noise is biased.

", "votes": 1, "replies": [] } ] }, { "id": 1142186, "author_name": "Kaushal Shah", "author_url": "", "post_date": "2021-01-07T08:11:09.043000", "content": "

Well said. Because of this, I lost motivation after achieving 0.898 ☹️

", "votes": 2, "replies": [] }, { "id": 1142133, "author_name": "Tian", "author_url": "", "post_date": "2021-01-07T07:12:50.080000", "content": "

yes, it is.

", "votes": 0, "replies": [] }, { "id": 1145777, "author_name": "SohamTamba", "author_url": "", "post_date": "2021-01-09T10:27:28.587000", "content": "

It would be nice if we could get some information on the quality of the test set. Eg. is it as noisy as the training set?

", "votes": 0, "replies": [] }, { "id": 1142580, "author_name": "Tsai29", "author_url": "", "post_date": "2021-01-07T13:33:57.483000", "content": "

Same feelings here. Especially after the PB leak.

", "votes": 0, "replies": [] }, { "id": 1142107, "author_name": "", "author_url": "", "post_date": "2021-01-07T06:44:39.303000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1142071": "https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/201471\n\nRegarding the noise in training, public and private data, I'm feeling that this competition is a lottery. If there is noise in private (I'm sure there is due to the PB leak), what are we predicting in the models? Are we predicting the annotator's habits..?😷 \nThe competition will be far interesting if hosts intended to fix the noise in the private set (and the competition will be so much more meaningful!)", "1142756": "This paper might be addressing few concerns around picking models/validating in the presence of noise: [Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels](https://arxiv.org/pdf/2012.04193.pdf).\n\n```\nUnder diagonally-dominant class-conditional label noise, the main conclusions are as follows.\n\nA classifier maximizing its accuracy on the noisy distribution is guaranteed to maximize the accuracy on clean distribution.\nWe can obtain an optimal classifier by maximizing training accuracy on sufficiently many noisy samples.\nA noisy validation set is reliable.\n```\n\nMy initial guess would be, if label noise is systematic, e.g class conditional noise when most of the leaves are healthy annotators tending to label an image as healthy even though there exists a single disease leaf at the corner, then models would be able to learn it without needing to apply any sophisticated robust training. But if this is not the case, e.g. noise is not predictable by the models than training a robust model with methods studied in literature should be our safe bet for private leaderboard. ", "1142305": "Self denoising works . I mean denoising done in Self supervised manner during training (and not manually)\nIt improved both my CV and LB. I dont know what will happen on the private LB though. \n \nAnyway , these noise are more likely annotation errors (inherent to humans) than general and biased pattern of the whole data (the latter would make non sense).Thus, a good and well trained model should give better generalisation and better accuracy in both clean data and clean+noisy data, than a model which overfitted on noisy training data.\n\nBut the difficulty is to train a model which maximize both Robustness and Accuracy in the presence of noise and most of our models tend to quickly overfit on noise by just **memorizing them**. That why it's really difficult to get further improvement at some point. This is total contrary of lottery IMHO\n", "1144896": "Does anyone know why the best public notebooks say they score `LB 0.900+` when in list view, but when we go inside none score over 0.900?\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2F24faacd71988322f89f808ff36e102e5%2Fone.png?generation=1610130442382506&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2F2e9febb1bf3d4910ee2af18711d7a55d%2Ftwo.png?generation=1610130452419433&alt=media)", "1144182": "In the wheat detection competition, the data set also had noise, but the host responded positively. But the host seldom uttered this game, which is very bad for us. I hope the host can come out and answer some questions.\ne.g. \n1.Is the label really wrong or because we are not experts and we can’t make a correct judgment\n2.division criteria for the coexistence of multiple diseases", "1150915": "So far I feel like my experiments have suggested:\n\n- filtering out images which look suspiciously labeled from the training has not really had a big positive effect (though am already using some label smoothing). i think with original loss i managed to get around +0.3 or similar compared to a lower starting point.\n\n- making minor changes to inference...some of which surely should not be negative e.g. more tta cycles...has in some cases lowered LB score by 0.1-0.3. \n\ni dont know if im making the correct interpretation here but it seems to me that this suggests there is some level of 'noise' or blind luck at least in terms of predicting a small portion of borderline images. im not sure if this necessarily says anything for sure about errors in test labeling. i guess it would be surprising if the host could say with confidence that there cannot be a single error in test image labels as i figure it would usually be difficult to diagnose with complete accuracy from photos?\n\nhopefully there are still plenty of things people will figure out (with or without labeling noise) to make models more accurate overall.", "1144121": "It will be a lottery in terms of the leaderboard rank\n\n1) categorical accuracy\n2) imbalance classes\n3) noisy data\n4) too narrow score between people", "1142497": "What we need is clarification from the hosts about LB and private LB data. If it similar PANDA competition, we need change our model and Pretreatment🤥🤥🤥\nThe gold miner competition will lose the meaning of competition. We want to defeat our opponents with wisdom and skill, not luck.(Although I like gold miners🙈", "1142262": "How do you know that public and private labels are noisy?", "1142186": "Well said. Because of this, I lost motivation after achieving 0.898 ☹️", "1142133": "yes, it is.", "1145777": "It would be nice if we could get some information on the quality of the test set. Eg. is it as noisy as the training set?", "1142580": "Same feelings here. Especially after the PB leak.", "1142107": "" } }