This is my first real effort in a Kaggle competition and it has been incredibly rewarding. It has also been quite surpring to watch the leaderboard over the last couple of months.
\nLast weekend I was very happy to have gotten my position on the leaderboard in the top 1/3 with an 0.810 score. Not stellar, but I was happy. One week later, I'm nearer to the 50th percentile and I see the 0.812 scores are now dominating the leaderboard - 322 submissions have a score of 0.812!!
\nNice work to all of those who have achieved 0.812 and above, but I'm curious about the phenomenon. Obviously as the competition progresses, improvements in scores will be made, but I've seen bulk movements into particular ranges occur consistently. It seems there were step-function like changes in the number of submissions achieving 0.810 then 0.811 and now 0.812.
\nThis bulk movement of scores has to be, in my opinion, a result of shared knowledge or shared results - someone publishes a notebook with a new approach that demonstrates an improvement and many individuals replicate that notebook for their own submissions. I also see a number of discussions around gathering known public submissions and combining them. Again, it seems more an exercise in combining other works into an ensemble.
\nOr is this too pessimistic, assumnig such non-independence of scoring trends?
", "messages": [ { "id": "2009009", "postDate": "10/29/2022 14:41:24", "content": "This is my first real effort in a Kaggle competition and it has been incredibly rewarding. It has also been quite surpring to watch the leaderboard over the last couple of months.
\nLast weekend I was very happy to have gotten my position on the leaderboard in the top 1/3 with an 0.810 score. Not stellar, but I was happy. One week later, I'm nearer to the 50th percentile and I see the 0.812 scores are now dominating the leaderboard - 322 submissions have a score of 0.812!!
\nNice work to all of those who have achieved 0.812 and above, but I'm curious about the phenomenon. Obviously as the competition progresses, improvements in scores will be made, but I've seen bulk movements into particular ranges occur consistently. It seems there were step-function like changes in the number of submissions achieving 0.810 then 0.811 and now 0.812.
\nThis bulk movement of scores has to be, in my opinion, a result of shared knowledge or shared results - someone publishes a notebook with a new approach that demonstrates an improvement and many individuals replicate that notebook for their own submissions. I also see a number of discussions around gathering known public submissions and combining them. Again, it seems more an exercise in combining other works into an ensemble.
\nOr is this too pessimistic, assumnig such non-independence of scoring trends?
", "rawMarkdown": "This is my first real effort in a Kaggle competition and it has been incredibly rewarding. It has also been quite surpring to watch the leaderboard over the last couple of months.\n\nLast weekend I was very happy to have gotten my position on the leaderboard in the top 1/3 with an 0.810 score. Not stellar, but I was happy. One week later, I'm nearer to the 50th percentile and I see the 0.812 scores are now dominating the leaderboard - 322 submissions have a score of 0.812!!\n\nNice work to all of those who have achieved 0.812 and above, but I'm curious about the phenomenon. Obviously as the competition progresses, improvements in scores will be made, but I've seen bulk movements into particular ranges occur consistently. It seems there were step-function like changes in the number of submissions achieving 0.810 then 0.811 and now 0.812. \n\nThis bulk movement of scores has to be, in my opinion, a result of shared knowledge or shared results - someone publishes a notebook with a new approach that demonstrates an improvement and many individuals replicate that notebook for their own submissions. I also see a number of discussions around gathering known public submissions and combining them. Again, it seems more an exercise in combining other works into an ensemble.\n\nOr is this too pessimistic, assumnig such non-independence of scoring trends?", "votes": null }, { "id": "2009054", "postDate": "10/29/2022 15:52:38", "content": "Don't pay attention to the public leaderboard too much (but be sure to try top approaches on YOUR OWN CV)
\nThe top scoring notebook gets .812, and there is a submission file as an output. You could simply click that and submit it and score .812 without even reading the code. It's all good though, because that is just the public leaderboard! The private is all that matters. I am going to implement ALL the publicly shared knowledge that I can and try to improve it or prove it wrong using MY OWN cross-validation. If I can't improve the scores, then I will use the public knowledge as is.
\nLook at the public ideas and think \"what are they overlooking here?\" I am sure you can think of some things.
", "rawMarkdown": "Don't pay attention to the public leaderboard too much (but be sure to try top approaches on YOUR OWN CV)\n\nThe top scoring notebook gets .812, and there is a submission file as an output. You could simply click that and submit it and score .812 without even reading the code. It's all good though, because that is just the public leaderboard! The private is all that matters. I am going to implement ALL the publicly shared knowledge that I can and try to improve it or prove it wrong using MY OWN cross-validation. If I can't improve the scores, then I will use the public knowledge as is. \n\nLook at the public ideas and think \"what are they overlooking here?\" I am sure you can think of some things.", "votes": null }, { "id": "2009083", "postDate": "10/29/2022 16:12:19", "content": "Thanks for the response!
\n\n\nThe top scoring notebook gets .812, and there is a submission file as an output. You could simply click that and submit it and score .812 without even reading the code.
\n
This is exactly what I've been trying to avoid. From the start I've wanted this to be about doing and learning. It becomes a little discouraging to slip from 33rd percentile to 50th in a matter of days. But, understanding how that can happen is helpful.
", "rawMarkdown": "Thanks for the response!\n\n> The top scoring notebook gets .812, and there is a submission file as an output. You could simply click that and submit it and score .812 without even reading the code.\n\nThis is exactly what I've been trying to avoid. From the start I've wanted this to be about doing and learning. It becomes a little discouraging to slip from 33rd percentile to 50th in a matter of days. But, understanding how that can happen is helpful.", "votes": null }, { "id": "2009326", "postDate": "10/29/2022 22:10:46", "content": "You have a good approach about learning, I feel similar. All the bad feelings about going down on the public leaderboard are nothing compared to the good feelings when you jump in the private leaderboard. In fact, the lower your public score, the better your private position will look (it will have a ^400 next to your name).
\nThe main lesson I've learned is to try out public kernels and try to prove them wrong. I have ignored some aspects of public kernels and regretted it. If you try a public method on your cross-validation and it still beats all of your models, you should really try to figure out why. There is absolutely no shame in using public methods, especially if you work to understand them. Chris Deotte, one of the greatest kagglers of all time said that he used a public kernel in his recent solution explanation to the recent AMEX competition. Of course he said he tried to improve upon it first, but couldn't, and then used it to create oof (out of fold) samples for model stacking. I think his team also used and improved public models in their MOA competition solution.
", "rawMarkdown": "You have a good approach about learning, I feel similar. All the bad feelings about going down on the public leaderboard are nothing compared to the good feelings when you jump in the private leaderboard. In fact, the lower your public score, the better your private position will look (it will have a ^400 next to your name). \n\nThe main lesson I've learned is to try out public kernels and try to prove them wrong. I have ignored some aspects of public kernels and regretted it. If you try a public method on your cross-validation and it still beats all of your models, you should really try to figure out why. There is absolutely no shame in using public methods, especially if you work to understand them. Chris Deotte, one of the greatest kagglers of all time said that he used a public kernel in his recent solution explanation to the recent AMEX competition. Of course he said he tried to improve upon it first, but couldn't, and then used it to create oof (out of fold) samples for model stacking. I think his team also used and improved public models in their MOA competition solution.", "votes": null } ], "comments": [ { "id": 2009054, "author_name": "chrisrichardmiles", "author_url": "", "post_date": "10/29/2022 15:52:38", "content": "Don't pay attention to the public leaderboard too much (but be sure to try top approaches on YOUR OWN CV)
\nThe top scoring notebook gets .812, and there is a submission file as an output. You could simply click that and submit it and score .812 without even reading the code. It's all good though, because that is just the public leaderboard! The private is all that matters. I am going to implement ALL the publicly shared knowledge that I can and try to improve it or prove it wrong using MY OWN cross-validation. If I can't improve the scores, then I will use the public knowledge as is.
\nLook at the public ideas and think \"what are they overlooking here?\" I am sure you can think of some things.
", "votes": null, "replies": [ { "id": 2009083, "author_name": "kirkdco", "author_url": "", "post_date": "10/29/2022 16:12:19", "content": "Thanks for the response!
\n\n\nThe top scoring notebook gets .812, and there is a submission file as an output. You could simply click that and submit it and score .812 without even reading the code.
\n
This is exactly what I've been trying to avoid. From the start I've wanted this to be about doing and learning. It becomes a little discouraging to slip from 33rd percentile to 50th in a matter of days. But, understanding how that can happen is helpful.
", "votes": null, "replies": [] }, { "id": 2009326, "author_name": "chrisrichardmiles", "author_url": "", "post_date": "10/29/2022 22:10:46", "content": "You have a good approach about learning, I feel similar. All the bad feelings about going down on the public leaderboard are nothing compared to the good feelings when you jump in the private leaderboard. In fact, the lower your public score, the better your private position will look (it will have a ^400 next to your name).
\nThe main lesson I've learned is to try out public kernels and try to prove them wrong. I have ignored some aspects of public kernels and regretted it. If you try a public method on your cross-validation and it still beats all of your models, you should really try to figure out why. There is absolutely no shame in using public methods, especially if you work to understand them. Chris Deotte, one of the greatest kagglers of all time said that he used a public kernel in his recent solution explanation to the recent AMEX competition. Of course he said he tried to improve upon it first, but couldn't, and then used it to create oof (out of fold) samples for model stacking. I think his team also used and improved public models in their MOA competition solution.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2009009": "This is my first real effort in a Kaggle competition and it has been incredibly rewarding. It has also been quite surpring to watch the leaderboard over the last couple of months.\n\nLast weekend I was very happy to have gotten my position on the leaderboard in the top 1/3 with an 0.810 score. Not stellar, but I was happy. One week later, I'm nearer to the 50th percentile and I see the 0.812 scores are now dominating the leaderboard - 322 submissions have a score of 0.812!!\n\nNice work to all of those who have achieved 0.812 and above, but I'm curious about the phenomenon. Obviously as the competition progresses, improvements in scores will be made, but I've seen bulk movements into particular ranges occur consistently. It seems there were step-function like changes in the number of submissions achieving 0.810 then 0.811 and now 0.812. \n\nThis bulk movement of scores has to be, in my opinion, a result of shared knowledge or shared results - someone publishes a notebook with a new approach that demonstrates an improvement and many individuals replicate that notebook for their own submissions. I also see a number of discussions around gathering known public submissions and combining them. Again, it seems more an exercise in combining other works into an ensemble.\n\nOr is this too pessimistic, assumnig such non-independence of scoring trends?", "2009054": "Don't pay attention to the public leaderboard too much (but be sure to try top approaches on YOUR OWN CV)\n\nThe top scoring notebook gets .812, and there is a submission file as an output. You could simply click that and submit it and score .812 without even reading the code. It's all good though, because that is just the public leaderboard! The private is all that matters. I am going to implement ALL the publicly shared knowledge that I can and try to improve it or prove it wrong using MY OWN cross-validation. If I can't improve the scores, then I will use the public knowledge as is. \n\nLook at the public ideas and think \"what are they overlooking here?\" I am sure you can think of some things.", "2009083": "Thanks for the response!\n\n> The top scoring notebook gets .812, and there is a submission file as an output. You could simply click that and submit it and score .812 without even reading the code.\n\nThis is exactly what I've been trying to avoid. From the start I've wanted this to be about doing and learning. It becomes a little discouraging to slip from 33rd percentile to 50th in a matter of days. But, understanding how that can happen is helpful.", "2009326": "You have a good approach about learning, I feel similar. All the bad feelings about going down on the public leaderboard are nothing compared to the good feelings when you jump in the private leaderboard. In fact, the lower your public score, the better your private position will look (it will have a ^400 next to your name). \n\nThe main lesson I've learned is to try out public kernels and try to prove them wrong. I have ignored some aspects of public kernels and regretted it. If you try a public method on your cross-validation and it still beats all of your models, you should really try to figure out why. There is absolutely no shame in using public methods, especially if you work to understand them. Chris Deotte, one of the greatest kagglers of all time said that he used a public kernel in his recent solution explanation to the recent AMEX competition. Of course he said he tried to improve upon it first, but couldn't, and then used it to create oof (out of fold) samples for model stacking. I think his team also used and improved public models in their MOA competition solution." }, "source": "meta" }