I has been joined this competition for like 2~3 weeks, and I think everybody already knew the data is quite noisy. So in these few weeks, I started to work on relabel the data and trained the model to see the result. I am not expert of the Leaf disease, so I can only relabel some very obvious mislabel instances. After relabeled partial of data, I got 0.93 cv on the partial clean dataset. So I decided to submit my first submission of this competition, then I got 0.883 LB. After I got this score, I realized maybe this will be a lottery competition for me. Here are few thoughts I would like to share about this result.
\nThe best scenario for me:
\nThe private testset is relatively clean, so we can train the model through some anti-noise methods to let the model has good generalization ability on clean data.
The worst scenario for me:
\nThe private testset is noisy as public train/test dataset. Then maybe the people whose model can learn the pattern of noisy label would have chance to win. But this is very hard and tricky, since your model might have to overfit on a very special spot(Not too much not too less). But I don't think in this scenario, the final winner's model can help the real world problem very effectively.
Conclusion:
\nFor people who want to win the competition, I would like to recommend you to train two independent model in above two scenarios for 2 final submissions. Then either the private testset is clean or not, you will have great chance to get the good final result.
Good luck :)
", "messages": [ { "id": "1118376", "postDate": "12/19/2020 02:45:12", "content": "I has been joined this competition for like 2~3 weeks, and I think everybody already knew the data is quite noisy. So in these few weeks, I started to work on relabel the data and trained the model to see the result. I am not expert of the Leaf disease, so I can only relabel some very obvious mislabel instances. After relabeled partial of data, I got 0.93 cv on the partial clean dataset. So I decided to submit my first submission of this competition, then I got 0.883 LB. After I got this score, I realized maybe this will be a lottery competition for me. Here are few thoughts I would like to share about this result.
\nThe best scenario for me:
\nThe private testset is relatively clean, so we can train the model through some anti-noise methods to let the model has good generalization ability on clean data.
The worst scenario for me:
\nThe private testset is noisy as public train/test dataset. Then maybe the people whose model can learn the pattern of noisy label would have chance to win. But this is very hard and tricky, since your model might have to overfit on a very special spot(Not too much not too less). But I don't think in this scenario, the final winner's model can help the real world problem very effectively.
Conclusion:
\nFor people who want to win the competition, I would like to recommend you to train two independent model in above two scenarios for 2 final submissions. Then either the private testset is clean or not, you will have great chance to get the good final result.
Good luck :)
", "rawMarkdown": "I has been joined this competition for like 2~3 weeks, and I think everybody already knew the data is quite noisy. So in these few weeks, I started to work on relabel the data and trained the model to see the result. I am not expert of the Leaf disease, so I can only relabel some very obvious mislabel instances. After relabeled partial of data, I got 0.93 cv on the partial clean dataset. So I decided to submit my first submission of this competition, then I got 0.883 LB. After I got this score, I realized maybe this will be a lottery competition for me. Here are few thoughts I would like to share about this result.\n\nThe best scenario for me:\nThe private testset is relatively clean, so we can train the model through some anti-noise methods to let the model has good generalization ability on clean data. \n\nThe worst scenario for me:\nThe private testset is noisy as public train/test dataset. Then maybe the people whose model can learn the pattern of noisy label would have chance to win. But this is very hard and tricky, since your model might have to overfit on a very special spot(Not too much not too less). But I don't think in this scenario, the final winner's model can help the real world problem very effectively. \n\n\nConclusion:\nFor people who want to win the competition, I would like to recommend you to train two independent model in above two scenarios for 2 final submissions. Then either the private testset is clean or not, you will have great chance to get the good final result.\n\nGood luck :)", "votes": null }, { "id": "1118381", "postDate": "12/19/2020 02:50:19", "content": "Hello,I think if the test is clean,you will get a higher score after relabel.
", "rawMarkdown": "Hello,I think if the test is clean,you will get a higher score after relabel.", "votes": null }, { "id": "1118384", "postDate": "12/19/2020 02:53:21", "content": "Yes, but right now we can only see the \"public\" testset result. Which means the private testset is clean or not remain unknown to us. But thanks for the remainder :)
", "rawMarkdown": "Yes, but right now we can only see the \"public\" testset result. Which means the private testset is clean or not remain unknown to us. But thanks for the remainder :)", "votes": null }, { "id": "1118386", "postDate": "12/19/2020 02:58:52", "content": "At the end of Global Wheat Detection,they only clean the label in private test set.Will they do that in this game?
", "rawMarkdown": "At the end of Global Wheat Detection,they only clean the label in private test set.Will they do that in this game?", "votes": null }, { "id": "1118388", "postDate": "12/19/2020 03:01:13", "content": "I think we can only know this question after the competition, and that's why I recommend people to train models for two different scenarios.
", "rawMarkdown": "I think we can only know this question after the competition, and that's why I recommend people to train models for two different scenarios.", "votes": null }, { "id": "1118465", "postDate": "12/19/2020 05:07:59", "content": "Most competitions will release the private set after it ends, so don't worry!
", "rawMarkdown": "Most competitions will release the private set after it ends, so don't worry!", "votes": null }, { "id": "1118574", "postDate": "12/19/2020 07:52:15", "content": "I don't think it's lottery or more lottery than many other competitions.
\nOtherwise you would get jump on LB with just small change on the same model (like we've seen on some \"lotery competitions\")
\nDenoising manually the dataset is not necessarily the best idea. And you may make the same mistake that those who (supposed) wrongly labelled the dataset.
\nBeware when you modify labels of 2 images and put one in the training set and the other in the validation set, you may introduce a leak in some way. Thus, for me, all \"label correction\" should be done only in the training set.
", "rawMarkdown": "I don't think it's lottery or more lottery than many other competitions. \n\nOtherwise you would get jump on LB with just small change on the same model (like we've seen on some \"lotery competitions\") \n\nDenoising manually the dataset is not necessarily the best idea. And you may make the same mistake that those who (supposed) wrongly labelled the dataset.\n\nBeware when you modify labels of 2 images and put one in the training set and the other in the validation set, you may introduce a leak in some way. Thus, for me, all \"label correction\" should be done only in the training set.", "votes": null }, { "id": "1118586", "postDate": "12/19/2020 08:13:22", "content": "Yeah, \"lottery\" might not be the right word for this. Just I can't comp up with better word for the title.
\nI do afraid I will be affected by the mislabeled instances which might cause the wrongly relabeling, so I only relabeled the instances which are very obviously wrong. And of course, I didn't expect this will give me any good result on the LB. Since I am quite sure there are also other people did the same thing. So I guess the safe way to do is to fit the model on both scenarios.
(already changed the title)
", "rawMarkdown": "Yeah, \"lottery\" might not be the right word for this. Just I can't comp up with better word for the title.\nI do afraid I will be affected by the mislabeled instances which might cause the wrongly relabeling, so I only relabeled the instances which are very obviously wrong. And of course, I didn't expect this will give me any good result on the LB. Since I am quite sure there are also other people did the same thing. So I guess the safe way to do is to fit the model on both scenarios.\n\n(already changed the title)", "votes": null }, { "id": "1118593", "postDate": "12/19/2020 08:25:18", "content": "Unless there is a breakout from the current tight pack on the leader board I think it's 100% a lottery. Hopefully someone will do the math for us and share how many more positive images you need to move from your current 800th position to 1st.
\nReminds me of a recent competition where I jumped up 1400 places because I did not have the time to create my normal \"over fit\" models :)
", "rawMarkdown": "Unless there is a breakout from the current tight pack on the leader board I think it's 100% a lottery. Hopefully someone will do the math for us and share how many more positive images you need to move from your current 800th position to 1st. \n\nReminds me of a recent competition where I jumped up 1400 places because I did not have the time to create my normal \"over fit\" models :)", "votes": null }, { "id": "1118604", "postDate": "12/19/2020 08:36:08", "content": "I guess denoise by relabeling the wrong label is pretty much impossible to get the 1st of LB, at least under current situation. LOL
\nI do have the same experience about jumping 1xxx places in the private board by not overftting the public dataset. But I think the most safe way for me is to submit an overfitting model and a no overfitting model. Well, if I have enough time to do this :)
\n\nUnless there is a breakout from the current tight pack on the leader board I think it's 100% a lottery.
\n
You see nothing contradictory here ?
\nHow a \"tight pack\" can be constant and not quickly broken if there is any lottery ?
\nAlso, beware there may be huge shake up in private LB like other similar competitions. That wouldn't mean it's lottery either. Trust methodology beyond CV and LB (to paraphrase a kaggler) ^^
", "rawMarkdown": "> Unless there is a breakout from the current tight pack on the leader board I think it's 100% a lottery. \n\nYou see nothing contradictory here ? \n\nHow a \"tight pack\" can be constant and not quickly broken if there is any lottery ? \n\n\nAlso, beware there may be huge shake up in private LB like other similar competitions. That wouldn't mean it's lottery either. Trust methodology beyond CV and LB (to paraphrase a kaggler) ^^", "votes": null }, { "id": "1118624", "postDate": "12/19/2020 08:50:39", "content": "I think the word \"lottery\" here means no matter how your model can predict correctly(truly correctly) on the instances, you might still get worse score than overfitted model.
", "rawMarkdown": "I think the word \"lottery\" here means no matter how your model can predict correctly(truly correctly) on the instances, you might still get worse score than overfitted model.", "votes": null }, { "id": "1118813", "postDate": "12/19/2020 12:22:36", "content": "The competition metric may have a weird effect here, there is little reward for a model to figure out that a certain type of image gets occasionally mislabeled - unless that teaches the model something about the continuum of pictures (i.e. the mislabeling is due to it really looking a bit like another class and the model can learn from that). However, learning to wrongly replicate mistakes that will also occur for the majority of cases in the test set would be useful (in terms of the competition, not really in the real world). Presumably, relabeling difficult mistakes near decision boundary would add the most value for improved real world performance, but will of course also be the hardest. Relabeling easy cases is likely to help the least, since presumably the model figures those out pretty well, but should not hurt.
", "rawMarkdown": "The competition metric may have a weird effect here, there is little reward for a model to figure out that a certain type of image gets occasionally mislabeled - unless that teaches the model something about the continuum of pictures (i.e. the mislabeling is due to it really looking a bit like another class and the model can learn from that). However, learning to wrongly replicate mistakes that will also occur for the majority of cases in the test set would be useful (in terms of the competition, not really in the real world). Presumably, relabeling difficult mistakes near decision boundary would add the most value for improved real world performance, but will of course also be the hardest. Relabeling easy cases is likely to help the least, since presumably the model figures those out pretty well, but should not hurt.", "votes": null }, { "id": "1118961", "postDate": "12/19/2020 15:24:21", "content": "I agree with you @serigne and technically all kaggle competitions are lottery ;)
", "rawMarkdown": "I agree with you @serigne and technically all kaggle competitions are lottery ;)", "votes": null }, { "id": "1119438", "postDate": "12/20/2020 04:26:34", "content": "all the deep learning stuffs can be boiled down to lottery…
", "rawMarkdown": "all the deep learning stuffs can be boiled down to lottery...", "votes": null }, { "id": "1119452", "postDate": "12/20/2020 04:49:37", "content": "I did the similar thing along the line, instead of re-labeling, I drop those obvious data the are incorrectly labeled. It seems giving me a very high CV, but not on the LB:(
", "rawMarkdown": "I did the similar thing along the line, instead of re-labeling, I drop those obvious data the are incorrectly labeled. It seems giving me a very high CV, but not on the LB:(", "votes": null }, { "id": "1119466", "postDate": "12/20/2020 05:06:10", "content": "Of course if you remove the hard examples it will give you a good CV, the public test set is not so accurate
", "rawMarkdown": "Of course if you remove the hard examples it will give you a good CV, the public test set is not so accurate", "votes": null }, { "id": "1119613", "postDate": "12/20/2020 08:31:03", "content": "Is that dropping them from training only or also from the validation (out of fold) set? In case of the latter, it's not surprising, in the former case (which would be the \"right\" way - to my mind - of trying this) it would seem odd to me.
", "rawMarkdown": "Is that dropping them from training only or also from the validation (out of fold) set? In case of the latter, it's not surprising, in the former case (which would be the \"right\" way - to my mind - of trying this) it would seem odd to me.", "votes": null } ], "comments": [ { "id": 1118381, "author_name": "zekunn", "author_url": "", "post_date": "12/19/2020 02:50:19", "content": "Hello,I think if the test is clean,you will get a higher score after relabel.
", "votes": null, "replies": [ { "id": 1118384, "author_name": "xiejialun", "author_url": "", "post_date": "12/19/2020 02:53:21", "content": "Yes, but right now we can only see the \"public\" testset result. Which means the private testset is clean or not remain unknown to us. But thanks for the remainder :)
", "votes": null, "replies": [] }, { "id": 1118386, "author_name": "zekunn", "author_url": "", "post_date": "12/19/2020 02:58:52", "content": "At the end of Global Wheat Detection,they only clean the label in private test set.Will they do that in this game?
", "votes": null, "replies": [] }, { "id": 1118388, "author_name": "xiejialun", "author_url": "", "post_date": "12/19/2020 03:01:13", "content": "I think we can only know this question after the competition, and that's why I recommend people to train models for two different scenarios.
", "votes": null, "replies": [] }, { "id": 1118465, "author_name": "aeryss", "author_url": "", "post_date": "12/19/2020 05:07:59", "content": "Most competitions will release the private set after it ends, so don't worry!
", "votes": null, "replies": [] } ] }, { "id": 1118574, "author_name": "serigne", "author_url": "", "post_date": "12/19/2020 07:52:15", "content": "I don't think it's lottery or more lottery than many other competitions.
\nOtherwise you would get jump on LB with just small change on the same model (like we've seen on some \"lotery competitions\")
\nDenoising manually the dataset is not necessarily the best idea. And you may make the same mistake that those who (supposed) wrongly labelled the dataset.
\nBeware when you modify labels of 2 images and put one in the training set and the other in the validation set, you may introduce a leak in some way. Thus, for me, all \"label correction\" should be done only in the training set.
", "votes": null, "replies": [ { "id": 1118586, "author_name": "xiejialun", "author_url": "", "post_date": "12/19/2020 08:13:22", "content": "Yeah, \"lottery\" might not be the right word for this. Just I can't comp up with better word for the title.
\nI do afraid I will be affected by the mislabeled instances which might cause the wrongly relabeling, so I only relabeled the instances which are very obviously wrong. And of course, I didn't expect this will give me any good result on the LB. Since I am quite sure there are also other people did the same thing. So I guess the safe way to do is to fit the model on both scenarios.
(already changed the title)
", "votes": null, "replies": [] }, { "id": 1118961, "author_name": "yannmajewski", "author_url": "", "post_date": "12/19/2020 15:24:21", "content": "I agree with you @serigne and technically all kaggle competitions are lottery ;)
", "votes": null, "replies": [] } ] }, { "id": 1118593, "author_name": "pcjimmmy", "author_url": "", "post_date": "12/19/2020 08:25:18", "content": "Unless there is a breakout from the current tight pack on the leader board I think it's 100% a lottery. Hopefully someone will do the math for us and share how many more positive images you need to move from your current 800th position to 1st.
\nReminds me of a recent competition where I jumped up 1400 places because I did not have the time to create my normal \"over fit\" models :)
", "votes": null, "replies": [ { "id": 1118604, "author_name": "xiejialun", "author_url": "", "post_date": "12/19/2020 08:36:08", "content": "I guess denoise by relabeling the wrong label is pretty much impossible to get the 1st of LB, at least under current situation. LOL
\nI do have the same experience about jumping 1xxx places in the private board by not overftting the public dataset. But I think the most safe way for me is to submit an overfitting model and a no overfitting model. Well, if I have enough time to do this :)
\n\nUnless there is a breakout from the current tight pack on the leader board I think it's 100% a lottery.
\n
You see nothing contradictory here ?
\nHow a \"tight pack\" can be constant and not quickly broken if there is any lottery ?
\nAlso, beware there may be huge shake up in private LB like other similar competitions. That wouldn't mean it's lottery either. Trust methodology beyond CV and LB (to paraphrase a kaggler) ^^
", "votes": null, "replies": [] }, { "id": 1118624, "author_name": "xiejialun", "author_url": "", "post_date": "12/19/2020 08:50:39", "content": "I think the word \"lottery\" here means no matter how your model can predict correctly(truly correctly) on the instances, you might still get worse score than overfitted model.
", "votes": null, "replies": [] } ] }, { "id": 1118813, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "12/19/2020 12:22:36", "content": "The competition metric may have a weird effect here, there is little reward for a model to figure out that a certain type of image gets occasionally mislabeled - unless that teaches the model something about the continuum of pictures (i.e. the mislabeling is due to it really looking a bit like another class and the model can learn from that). However, learning to wrongly replicate mistakes that will also occur for the majority of cases in the test set would be useful (in terms of the competition, not really in the real world). Presumably, relabeling difficult mistakes near decision boundary would add the most value for improved real world performance, but will of course also be the hardest. Relabeling easy cases is likely to help the least, since presumably the model figures those out pretty well, but should not hurt.
", "votes": null, "replies": [] }, { "id": 1119438, "author_name": "plugin1689", "author_url": "", "post_date": "12/20/2020 04:26:34", "content": "all the deep learning stuffs can be boiled down to lottery…
", "votes": null, "replies": [] }, { "id": 1119452, "author_name": "luqing2", "author_url": "", "post_date": "12/20/2020 04:49:37", "content": "I did the similar thing along the line, instead of re-labeling, I drop those obvious data the are incorrectly labeled. It seems giving me a very high CV, but not on the LB:(
", "votes": null, "replies": [ { "id": 1119466, "author_name": "yannmajewski", "author_url": "", "post_date": "12/20/2020 05:06:10", "content": "Of course if you remove the hard examples it will give you a good CV, the public test set is not so accurate
", "votes": null, "replies": [] }, { "id": 1119613, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "12/20/2020 08:31:03", "content": "Is that dropping them from training only or also from the validation (out of fold) set? In case of the latter, it's not surprising, in the former case (which would be the \"right\" way - to my mind - of trying this) it would seem odd to me.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1118376": "I has been joined this competition for like 2~3 weeks, and I think everybody already knew the data is quite noisy. So in these few weeks, I started to work on relabel the data and trained the model to see the result. I am not expert of the Leaf disease, so I can only relabel some very obvious mislabel instances. After relabeled partial of data, I got 0.93 cv on the partial clean dataset. So I decided to submit my first submission of this competition, then I got 0.883 LB. After I got this score, I realized maybe this will be a lottery competition for me. Here are few thoughts I would like to share about this result.\n\nThe best scenario for me:\nThe private testset is relatively clean, so we can train the model through some anti-noise methods to let the model has good generalization ability on clean data. \n\nThe worst scenario for me:\nThe private testset is noisy as public train/test dataset. Then maybe the people whose model can learn the pattern of noisy label would have chance to win. But this is very hard and tricky, since your model might have to overfit on a very special spot(Not too much not too less). But I don't think in this scenario, the final winner's model can help the real world problem very effectively. \n\n\nConclusion:\nFor people who want to win the competition, I would like to recommend you to train two independent model in above two scenarios for 2 final submissions. Then either the private testset is clean or not, you will have great chance to get the good final result.\n\nGood luck :)", "1118381": "Hello,I think if the test is clean,you will get a higher score after relabel.", "1118384": "Yes, but right now we can only see the \"public\" testset result. Which means the private testset is clean or not remain unknown to us. But thanks for the remainder :)", "1118386": "At the end of Global Wheat Detection,they only clean the label in private test set.Will they do that in this game?", "1118388": "I think we can only know this question after the competition, and that's why I recommend people to train models for two different scenarios.", "1118465": "Most competitions will release the private set after it ends, so don't worry!", "1118574": "I don't think it's lottery or more lottery than many other competitions. \n\nOtherwise you would get jump on LB with just small change on the same model (like we've seen on some \"lotery competitions\") \n\nDenoising manually the dataset is not necessarily the best idea. And you may make the same mistake that those who (supposed) wrongly labelled the dataset.\n\nBeware when you modify labels of 2 images and put one in the training set and the other in the validation set, you may introduce a leak in some way. Thus, for me, all \"label correction\" should be done only in the training set.", "1118586": "Yeah, \"lottery\" might not be the right word for this. Just I can't comp up with better word for the title.\nI do afraid I will be affected by the mislabeled instances which might cause the wrongly relabeling, so I only relabeled the instances which are very obviously wrong. And of course, I didn't expect this will give me any good result on the LB. Since I am quite sure there are also other people did the same thing. So I guess the safe way to do is to fit the model on both scenarios.\n\n(already changed the title)", "1118593": "Unless there is a breakout from the current tight pack on the leader board I think it's 100% a lottery. Hopefully someone will do the math for us and share how many more positive images you need to move from your current 800th position to 1st. \n\nReminds me of a recent competition where I jumped up 1400 places because I did not have the time to create my normal \"over fit\" models :)", "1118604": "I guess denoise by relabeling the wrong label is pretty much impossible to get the 1st of LB, at least under current situation. LOL\nI do have the same experience about jumping 1xxx places in the private board by not overftting the public dataset. But I think the most safe way for me is to submit an overfitting model and a no overfitting model. Well, if I have enough time to do this :)", "1118614": "> Unless there is a breakout from the current tight pack on the leader board I think it's 100% a lottery. \n\nYou see nothing contradictory here ? \n\nHow a \"tight pack\" can be constant and not quickly broken if there is any lottery ? \n\n\nAlso, beware there may be huge shake up in private LB like other similar competitions. That wouldn't mean it's lottery either. Trust methodology beyond CV and LB (to paraphrase a kaggler) ^^", "1118624": "I think the word \"lottery\" here means no matter how your model can predict correctly(truly correctly) on the instances, you might still get worse score than overfitted model.", "1118813": "The competition metric may have a weird effect here, there is little reward for a model to figure out that a certain type of image gets occasionally mislabeled - unless that teaches the model something about the continuum of pictures (i.e. the mislabeling is due to it really looking a bit like another class and the model can learn from that). However, learning to wrongly replicate mistakes that will also occur for the majority of cases in the test set would be useful (in terms of the competition, not really in the real world). Presumably, relabeling difficult mistakes near decision boundary would add the most value for improved real world performance, but will of course also be the hardest. Relabeling easy cases is likely to help the least, since presumably the model figures those out pretty well, but should not hurt.", "1118961": "I agree with you @serigne and technically all kaggle competitions are lottery ;)", "1119438": "all the deep learning stuffs can be boiled down to lottery...", "1119452": "I did the similar thing along the line, instead of re-labeling, I drop those obvious data the are incorrectly labeled. It seems giving me a very high CV, but not on the LB:(", "1119466": "Of course if you remove the hard examples it will give you a good CV, the public test set is not so accurate", "1119613": "Is that dropping them from training only or also from the validation (out of fold) set? In case of the latter, it's not surprising, in the former case (which would be the \"right\" way - to my mind - of trying this) it would seem odd to me." }, "source": "meta" }