I am just curious about people who dropped hundreds or thousands of places in ranking, what did they do wrong, and how were they overfitting?
\n\nAll the models we had in our team with good public LB score had a decent private LB (silver territory) even without adjusting for difference in distributions.
\n\nI would appreciate if people share what was the mistake they did that led to that severe public LB overfitting.
", "messages": [ { "id": "543870", "postDate": "06/04/2019 22:39:11", "content": "I am just curious about people who dropped hundreds or thousands of places in ranking, what did they do wrong, and how were they overfitting?
\n\nAll the models we had in our team with good public LB score had a decent private LB (silver territory) even without adjusting for difference in distributions.
\n\nI would appreciate if people share what was the mistake they did that led to that severe public LB overfitting.
", "rawMarkdown": "I am just curious about people who dropped hundreds or thousands of places in ranking, what did they do wrong, and how were they overfitting?\n\nAll the models we had in our team with good public LB score had a decent private LB (silver territory) even without adjusting for difference in distributions.\n\nI would appreciate if people share what was the mistake they did that led to that severe public LB overfitting.", "votes": null }, { "id": "546827", "postDate": "06/06/2019 23:54:19", "content": "I'm curious, too!
", "rawMarkdown": "I'm curious, too!", "votes": null }, { "id": "547285", "postDate": "06/07/2019 14:14:40", "content": "The private LB mean was quite high compared to public - if you came around 1000 just scale your ttfs by 1.1 and see what happens ;)
", "rawMarkdown": "The private LB mean was quite high compared to public - if you came around 1000 just scale your ttfs by 1.1 and see what happens ;)", "votes": null }, { "id": "547290", "postDate": "06/07/2019 14:22:54", "content": "What I meant is that even without any scaling, all our models were in the silver territory.
", "rawMarkdown": "What I meant is that even without any scaling, all our models were in the silver territory.", "votes": null }, { "id": "547295", "postDate": "06/07/2019 14:27:15", "content": "The same way you can tailor your predictions to private LB, you can overfit them towards public LB.
", "rawMarkdown": "The same way you can tailor your predictions to private LB, you can overfit them towards public LB.", "votes": null }, { "id": "547320", "postDate": "06/07/2019 15:16:15", "content": "Good for you! Optimizing on the public LB is suicidal in this competition due to the tiny amount of earthquakes. We know the Public Error was ~ 1.3 whereas the Private was around ~2.3 indicating these distributions weren't very similar. So anyone not using a decent CV would tank - me included ;)
\n\nHave a look at this https://www.kaggle.com/trentb/one-feature-no-ml-gold-medal-range
\n\nAre we really predicting good models or good means of the private data?
\n\nAlso using a few parameters is a great idea due to small amount of data.
\n\n\"With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.\" - John von Neumann
", "rawMarkdown": "Good for you! Optimizing on the public LB is suicidal in this competition due to the tiny amount of earthquakes. We know the Public Error was ~ 1.3 whereas the Private was around ~2.3 indicating these distributions weren't very similar. So anyone not using a decent CV would tank - me included ;)\n\nHave a look at this https://www.kaggle.com/trentb/one-feature-no-ml-gold-medal-range\n\nAre we really predicting good models or good means of the private data?\n\nAlso using a few parameters is a great idea due to small amount of data.\n\n\"With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.\" - John von Neumann", "votes": null }, { "id": "547324", "postDate": "06/07/2019 15:24:29", "content": "Our 18 top public LB subs are in silver or gold range in private. Not sure what to conclude from that ;) Surprisingly, this includes the few ones we purposely overfit to public LB :)
", "rawMarkdown": "Our 18 top public LB subs are in silver or gold range in private. Not sure what to conclude from that ;) Surprisingly, this includes the few ones we purposely overfit to public LB :)", "votes": null }, { "id": "547356", "postDate": "06/07/2019 16:03:19", "content": "So you mean that the people who dropped adjusted the submission mean to public LB mean ~ 4?
", "rawMarkdown": "So you mean that the people who dropped adjusted the submission mean to public LB mean ~ 4?", "votes": null }, { "id": "547359", "postDate": "06/07/2019 16:06:23", "content": "That what makes me wonder. My progress on public LB was hand in hand with private LB, except for two early solutions that had very good private LB, but I dismissed because of their bad public LB.
", "rawMarkdown": "That what makes me wonder. My progress on public LB was hand in hand with private LB, except for two early solutions that had very good private LB, but I dismissed because of their bad public LB.", "votes": null }, { "id": "547360", "postDate": "06/07/2019 16:06:25", "content": "I rather think that many used public LB to guide their model/feature/parameter selection. This leads to public LB overfit.
", "rawMarkdown": "I rather think that many used public LB to guide their model/feature/parameter selection. This leads to public LB overfit.", "votes": null }, { "id": "547363", "postDate": "06/07/2019 16:08:50", "content": "I am pretty sure that many tried that, at least they used public LB as guidance as @cpmpml correctly points out.
", "rawMarkdown": "I am pretty sure that many tried that, at least they used public LB as guidance as @cpmpml correctly points out.", "votes": null }, { "id": "547444", "postDate": "06/07/2019 18:05:57", "content": "As someone who dropped 1700 places some feedback:
\n\nMy trusted cv was 4-fold quake-based for faster computation (= 4 quakes in one fold, unshuffled), where i tried to get folds of roughly equal length and mean. With this, the last model that i wanted to submit scored 1.50 on public. Actually all models that i trained without shuffling (at the beginning simple 3-fold not quake-based) never scored below 2.0 cv, and almost all above 1.5 lb.
\n\nLooking at the low scores of top teams, i thought at least some of them must have been able to achieve such results with robust strategies and therefore have much better models/features. So for my final subs, i just took the model with the best public score that was trained with 10-fold shuffled cv and not really tested for overfitting and a neural net that i made shortly before without further optimization. I was simply hoping that one of them would be a lucky shot in the dark and didn't expect my models with ~1.5 lb to be competitive, even though i thought they'd be reliable an generalizing quite well.
\n\nIf i had simply taken my last model, it would've already been top 100, with the best model ~20 \n(but i don't not how exactly it looked anymore, should save everything next time, ...) .
\n\nMoral of the story: \nTrust your cv and the best models you built with it, even though public score might indicate that they won't lead to good positions on the final leaderboard. I'm still amazed that people got down to 1.2 lb, while i couldn't even break 1.4 lb trying to overfit to the test data.
", "rawMarkdown": "As someone who dropped 1700 places some feedback:\n\nMy trusted cv was 4-fold quake-based for faster computation (= 4 quakes in one fold, unshuffled), where i tried to get folds of roughly equal length and mean. With this, the last model that i wanted to submit scored 1.50 on public. Actually all models that i trained without shuffling (at the beginning simple 3-fold not quake-based) never scored below 2.0 cv, and almost all above 1.5 lb. \n\nLooking at the low scores of top teams, i thought at least some of them must have been able to achieve such results with robust strategies and therefore have much better models/features. So for my final subs, i just took the model with the best public score that was trained with 10-fold shuffled cv and not really tested for overfitting and a neural net that i made shortly before without further optimization. I was simply hoping that one of them would be a lucky shot in the dark and didn't expect my models with ~1.5 lb to be competitive, even though i thought they'd be reliable an generalizing quite well.\n\nIf i had simply taken my last model, it would've already been top 100, with the best model ~20 \n(but i don't not how exactly it looked anymore, should save everything next time, ...) .\n\nMoral of the story: \nTrust your cv and the best models you built with it, even though public score might indicate that they won't lead to good positions on the final leaderboard. I'm still amazed that people got down to 1.2 lb, while i couldn't even break 1.4 lb trying to overfit to the test data.", "votes": null }, { "id": "548096", "postDate": "06/08/2019 19:10:45", "content": "I'm not sure that everybody who took a dive was overfitting to the public LB. A fair number of public kernels show Private LB MAE's of 2.6-2.7, while their private LB's were 1.45 or so, so they would have suffered big drops. Maybe they were all LB-climbing, but I'm not so sure.
\n\nIt was easy to hit a brick wall with a CV of 2.1 or so, with a fit that gives a maximum TTF for every cycle of about 10, resulting in a Private LB of 2.6-2.7. From various discussion entries and graphs in some of the public kernels, I suspect a lot of other people hit this same brick wall. This led to my personal Walk of Shame of dropping 1153 positions.
\n\nI spent a huge amount of time in search of a feature that would give different values at the start of longer quake-cycles than short ones, but every feature I ever looked at started each cycle at about the same value and ended at about the same value (except some features where some of the shorter cycles end a bit differently).
\n\nI'm therefore puzzled at how the experts managed to build models to run all the way up to at least 12, without peeking at the test data. What is it that these models are latching onto that gets them to a higher mean (and higher max) TTF?
", "rawMarkdown": "I'm not sure that *everybody* who took a dive was overfitting to the public LB. A fair number of public kernels show Private LB MAE's of 2.6-2.7, while their private LB's were 1.45 or so, so they would have suffered big drops. Maybe they were all LB-climbing, but I'm not so sure.\n\nIt was easy to hit a brick wall with a CV of 2.1 or so, with a fit that gives a maximum TTF for every cycle of about 10, resulting in a Private LB of 2.6-2.7. From various discussion entries and graphs in some of the public kernels, I suspect a lot of other people hit this same brick wall. This led to my personal Walk of Shame of dropping 1153 positions. \n\nI spent a huge amount of time in search of a feature that would give different values at the start of longer quake-cycles than short ones, but every feature I ever looked at started each cycle at about the same value and ended at about the same value (except some features where some of the shorter cycles end a bit differently).\n\nI'm therefore puzzled at how the experts managed to build models to run all the way up to at least 12, without peeking at the test data. What is it that these models are latching onto that gets them to a higher mean (and higher max) TTF?", "votes": null }, { "id": "548379", "postDate": "06/09/2019 09:43:45", "content": "Thank you for sharing. There was peeking at test data in most cases. But even without peeking and with ~5.5 mean ttf submission, securing a top 100 position is achievable.
", "rawMarkdown": "Thank you for sharing. There was peeking at test data in most cases. But even without peeking and with ~5.5 mean ttf submission, securing a top 100 position is achievable.", "votes": null }, { "id": "548381", "postDate": "06/09/2019 09:45:22", "content": "Sorry for your drop and thank you for sharing
", "rawMarkdown": "Sorry for your drop and thank you for sharing", "votes": null }, { "id": "549331", "postDate": "06/10/2019 14:59:57", "content": "I was happy with my Public LB score so I felt safe, but I was trying to maximize my CV score only and these were the models I've chosen. CV was correlated with Public LB. I feel very disappointed with myself. The problem I had was probably the mean predictions - around 5.1. Even though I thought that the mean is not that important - it's just my model works well in some lower TTF areas.
", "rawMarkdown": "I was happy with my Public LB score so I felt safe, but I was trying to maximize my CV score only and these were the models I've chosen. CV was correlated with Public LB. I feel very disappointed with myself. The problem I had was probably the mean predictions - around 5.1. Even though I thought that the mean is not that important - it's just my model works well in some lower TTF areas.", "votes": null } ], "comments": [ { "id": 546827, "author_name": "tbmoon", "author_url": "", "post_date": "06/06/2019 23:54:19", "content": "I'm curious, too!
", "votes": null, "replies": [] }, { "id": 547285, "author_name": "scirpus", "author_url": "", "post_date": "06/07/2019 14:14:40", "content": "The private LB mean was quite high compared to public - if you came around 1000 just scale your ttfs by 1.1 and see what happens ;)
", "votes": null, "replies": [ { "id": 547290, "author_name": "amjad85", "author_url": "", "post_date": "06/07/2019 14:22:54", "content": "What I meant is that even without any scaling, all our models were in the silver territory.
", "votes": null, "replies": [] }, { "id": 547320, "author_name": "scirpus", "author_url": "", "post_date": "06/07/2019 15:16:15", "content": "Good for you! Optimizing on the public LB is suicidal in this competition due to the tiny amount of earthquakes. We know the Public Error was ~ 1.3 whereas the Private was around ~2.3 indicating these distributions weren't very similar. So anyone not using a decent CV would tank - me included ;)
\n\nHave a look at this https://www.kaggle.com/trentb/one-feature-no-ml-gold-medal-range
\n\nAre we really predicting good models or good means of the private data?
\n\nAlso using a few parameters is a great idea due to small amount of data.
\n\n\"With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.\" - John von Neumann
", "votes": null, "replies": [] } ] }, { "id": 547295, "author_name": "philippsinger", "author_url": "", "post_date": "06/07/2019 14:27:15", "content": "The same way you can tailor your predictions to private LB, you can overfit them towards public LB.
", "votes": null, "replies": [ { "id": 547356, "author_name": "amjad85", "author_url": "", "post_date": "06/07/2019 16:03:19", "content": "So you mean that the people who dropped adjusted the submission mean to public LB mean ~ 4?
", "votes": null, "replies": [] }, { "id": 547360, "author_name": "cpmpml", "author_url": "", "post_date": "06/07/2019 16:06:25", "content": "I rather think that many used public LB to guide their model/feature/parameter selection. This leads to public LB overfit.
", "votes": null, "replies": [] }, { "id": 547363, "author_name": "philippsinger", "author_url": "", "post_date": "06/07/2019 16:08:50", "content": "I am pretty sure that many tried that, at least they used public LB as guidance as @cpmpml correctly points out.
", "votes": null, "replies": [] } ] }, { "id": 547324, "author_name": "cpmpml", "author_url": "", "post_date": "06/07/2019 15:24:29", "content": "Our 18 top public LB subs are in silver or gold range in private. Not sure what to conclude from that ;) Surprisingly, this includes the few ones we purposely overfit to public LB :)
", "votes": null, "replies": [ { "id": 547359, "author_name": "amjad85", "author_url": "", "post_date": "06/07/2019 16:06:23", "content": "That what makes me wonder. My progress on public LB was hand in hand with private LB, except for two early solutions that had very good private LB, but I dismissed because of their bad public LB.
", "votes": null, "replies": [] } ] }, { "id": 547444, "author_name": "svenhinderer", "author_url": "", "post_date": "06/07/2019 18:05:57", "content": "As someone who dropped 1700 places some feedback:
\n\nMy trusted cv was 4-fold quake-based for faster computation (= 4 quakes in one fold, unshuffled), where i tried to get folds of roughly equal length and mean. With this, the last model that i wanted to submit scored 1.50 on public. Actually all models that i trained without shuffling (at the beginning simple 3-fold not quake-based) never scored below 2.0 cv, and almost all above 1.5 lb.
\n\nLooking at the low scores of top teams, i thought at least some of them must have been able to achieve such results with robust strategies and therefore have much better models/features. So for my final subs, i just took the model with the best public score that was trained with 10-fold shuffled cv and not really tested for overfitting and a neural net that i made shortly before without further optimization. I was simply hoping that one of them would be a lucky shot in the dark and didn't expect my models with ~1.5 lb to be competitive, even though i thought they'd be reliable an generalizing quite well.
\n\nIf i had simply taken my last model, it would've already been top 100, with the best model ~20 \n(but i don't not how exactly it looked anymore, should save everything next time, ...) .
\n\nMoral of the story: \nTrust your cv and the best models you built with it, even though public score might indicate that they won't lead to good positions on the final leaderboard. I'm still amazed that people got down to 1.2 lb, while i couldn't even break 1.4 lb trying to overfit to the test data.
", "votes": null, "replies": [ { "id": 548381, "author_name": "amjad85", "author_url": "", "post_date": "06/09/2019 09:45:22", "content": "Sorry for your drop and thank you for sharing
", "votes": null, "replies": [] } ] }, { "id": 548096, "author_name": "vgates", "author_url": "", "post_date": "06/08/2019 19:10:45", "content": "I'm not sure that everybody who took a dive was overfitting to the public LB. A fair number of public kernels show Private LB MAE's of 2.6-2.7, while their private LB's were 1.45 or so, so they would have suffered big drops. Maybe they were all LB-climbing, but I'm not so sure.
\n\nIt was easy to hit a brick wall with a CV of 2.1 or so, with a fit that gives a maximum TTF for every cycle of about 10, resulting in a Private LB of 2.6-2.7. From various discussion entries and graphs in some of the public kernels, I suspect a lot of other people hit this same brick wall. This led to my personal Walk of Shame of dropping 1153 positions.
\n\nI spent a huge amount of time in search of a feature that would give different values at the start of longer quake-cycles than short ones, but every feature I ever looked at started each cycle at about the same value and ended at about the same value (except some features where some of the shorter cycles end a bit differently).
\n\nI'm therefore puzzled at how the experts managed to build models to run all the way up to at least 12, without peeking at the test data. What is it that these models are latching onto that gets them to a higher mean (and higher max) TTF?
", "votes": null, "replies": [ { "id": 548379, "author_name": "amjad85", "author_url": "", "post_date": "06/09/2019 09:43:45", "content": "Thank you for sharing. There was peeking at test data in most cases. But even without peeking and with ~5.5 mean ttf submission, securing a top 100 position is achievable.
", "votes": null, "replies": [] } ] }, { "id": 549331, "author_name": "davids1992", "author_url": "", "post_date": "06/10/2019 14:59:57", "content": "I was happy with my Public LB score so I felt safe, but I was trying to maximize my CV score only and these were the models I've chosen. CV was correlated with Public LB. I feel very disappointed with myself. The problem I had was probably the mean predictions - around 5.1. Even though I thought that the mean is not that important - it's just my model works well in some lower TTF areas.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "543870": "I am just curious about people who dropped hundreds or thousands of places in ranking, what did they do wrong, and how were they overfitting?\n\nAll the models we had in our team with good public LB score had a decent private LB (silver territory) even without adjusting for difference in distributions.\n\nI would appreciate if people share what was the mistake they did that led to that severe public LB overfitting.", "546827": "I'm curious, too!", "547285": "The private LB mean was quite high compared to public - if you came around 1000 just scale your ttfs by 1.1 and see what happens ;)", "547290": "What I meant is that even without any scaling, all our models were in the silver territory.", "547295": "The same way you can tailor your predictions to private LB, you can overfit them towards public LB.", "547320": "Good for you! Optimizing on the public LB is suicidal in this competition due to the tiny amount of earthquakes. We know the Public Error was ~ 1.3 whereas the Private was around ~2.3 indicating these distributions weren't very similar. So anyone not using a decent CV would tank - me included ;)\n\nHave a look at this https://www.kaggle.com/trentb/one-feature-no-ml-gold-medal-range\n\nAre we really predicting good models or good means of the private data?\n\nAlso using a few parameters is a great idea due to small amount of data.\n\n\"With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.\" - John von Neumann", "547324": "Our 18 top public LB subs are in silver or gold range in private. Not sure what to conclude from that ;) Surprisingly, this includes the few ones we purposely overfit to public LB :)", "547356": "So you mean that the people who dropped adjusted the submission mean to public LB mean ~ 4?", "547359": "That what makes me wonder. My progress on public LB was hand in hand with private LB, except for two early solutions that had very good private LB, but I dismissed because of their bad public LB.", "547360": "I rather think that many used public LB to guide their model/feature/parameter selection. This leads to public LB overfit.", "547363": "I am pretty sure that many tried that, at least they used public LB as guidance as @cpmpml correctly points out.", "547444": "As someone who dropped 1700 places some feedback:\n\nMy trusted cv was 4-fold quake-based for faster computation (= 4 quakes in one fold, unshuffled), where i tried to get folds of roughly equal length and mean. With this, the last model that i wanted to submit scored 1.50 on public. Actually all models that i trained without shuffling (at the beginning simple 3-fold not quake-based) never scored below 2.0 cv, and almost all above 1.5 lb. \n\nLooking at the low scores of top teams, i thought at least some of them must have been able to achieve such results with robust strategies and therefore have much better models/features. So for my final subs, i just took the model with the best public score that was trained with 10-fold shuffled cv and not really tested for overfitting and a neural net that i made shortly before without further optimization. I was simply hoping that one of them would be a lucky shot in the dark and didn't expect my models with ~1.5 lb to be competitive, even though i thought they'd be reliable an generalizing quite well.\n\nIf i had simply taken my last model, it would've already been top 100, with the best model ~20 \n(but i don't not how exactly it looked anymore, should save everything next time, ...) .\n\nMoral of the story: \nTrust your cv and the best models you built with it, even though public score might indicate that they won't lead to good positions on the final leaderboard. I'm still amazed that people got down to 1.2 lb, while i couldn't even break 1.4 lb trying to overfit to the test data.", "548096": "I'm not sure that *everybody* who took a dive was overfitting to the public LB. A fair number of public kernels show Private LB MAE's of 2.6-2.7, while their private LB's were 1.45 or so, so they would have suffered big drops. Maybe they were all LB-climbing, but I'm not so sure.\n\nIt was easy to hit a brick wall with a CV of 2.1 or so, with a fit that gives a maximum TTF for every cycle of about 10, resulting in a Private LB of 2.6-2.7. From various discussion entries and graphs in some of the public kernels, I suspect a lot of other people hit this same brick wall. This led to my personal Walk of Shame of dropping 1153 positions. \n\nI spent a huge amount of time in search of a feature that would give different values at the start of longer quake-cycles than short ones, but every feature I ever looked at started each cycle at about the same value and ended at about the same value (except some features where some of the shorter cycles end a bit differently).\n\nI'm therefore puzzled at how the experts managed to build models to run all the way up to at least 12, without peeking at the test data. What is it that these models are latching onto that gets them to a higher mean (and higher max) TTF?", "548379": "Thank you for sharing. There was peeking at test data in most cases. But even without peeking and with ~5.5 mean ttf submission, securing a top 100 position is achievable.", "548381": "Sorry for your drop and thank you for sharing", "549331": "I was happy with my Public LB score so I felt safe, but I was trying to maximize my CV score only and these were the models I've chosen. CV was correlated with Public LB. I feel very disappointed with myself. The problem I had was probably the mean predictions - around 5.1. Even though I thought that the mean is not that important - it's just my model works well in some lower TTF areas." }, "source": "meta" }