I saw a lot of topics saying shake up will happen, but, how to avoid that?\nI used different features and models for this competition, and the LB is around 1.410-1.516. My best submission blend some of the top public kernels' submissions reaching 1.395.\nI really have questions about which should be selected as the final submission, best single model, or blend of all my models, or my best LB 1.395, or any other ideas?\nOr what I could do is only just wait and see? \nI will appreciate it if someone could give me suggestions.
", "messages": [ { "id": "538823", "postDate": "05/29/2019 06:45:37", "content": "I saw a lot of topics saying shake up will happen, but, how to avoid that?\nI used different features and models for this competition, and the LB is around 1.410-1.516. My best submission blend some of the top public kernels' submissions reaching 1.395.\nI really have questions about which should be selected as the final submission, best single model, or blend of all my models, or my best LB 1.395, or any other ideas?\nOr what I could do is only just wait and see? \nI will appreciate it if someone could give me suggestions.
", "rawMarkdown": "I saw a lot of topics saying shake up will happen, but, how to avoid that?\nI used different features and models for this competition, and the LB is around 1.410-1.516. My best submission blend some of the top public kernels' submissions reaching 1.395.\nI really have questions about which should be selected as the final submission, best single model, or blend of all my models, or my best LB 1.395, or any other ideas?\nOr what I could do is only just wait and see? \nI will appreciate it if someone could give me suggestions.", "votes": null }, { "id": "538831", "postDate": "05/29/2019 06:53:22", "content": "I am a novice in Kaggle competition, so I just have the same question. Could anyone offer some good advice to avoid the shake up? Thanks a lot!
", "rawMarkdown": "I am a novice in Kaggle competition, so I just have the same question. Could anyone offer some good advice to avoid the shake up? Thanks a lot!", "votes": null }, { "id": "538845", "postDate": "05/29/2019 07:17:43", "content": "It's better to say \"how to avoid shake down\", which means your submission scores worse in private LB than those placed below you on the public LB. People tend to say \"avoid overfitting\", but that's not right, because 100% of people are overfitting. The key thing is that the level of overfitting of each person.
\n\nDuring your course of submitting each day, if you focused more on public LB performance, you will tend to overfit it, and have more chance to drop, while people with a strong and sound CV setting trying to optimize it without looking on public LB, will more likely to jump.
\n\nOf course, if one's CV is incorrect in the first place, optimizing it is a disaster.
\n\nSo to answer your question: \"how to avoid shake down\", it's not too late to change your mindset, and select a submission that can generalize better on all kinds of test samples, with a slight penalty on public LB. That's also a psychological game: doing that you'll have no regret of being dropped, and will learn something from it, or you'll get sweet rewards. I think it's better than being surprisingly happy if you're unexpected rise on private LB (or accepting a drop) without understanding anything.
", "rawMarkdown": "It's better to say \"how to avoid shake down\", which means your submission scores worse in private LB than those placed below you on the public LB. People tend to say \"avoid overfitting\", but that's not right, because 100% of people are overfitting. The key thing is that the level of overfitting of each person.\n\nDuring your course of submitting each day, if you focused more on public LB performance, you will tend to overfit it, and have more chance to drop, while people with a strong and sound CV setting trying to optimize it without looking on public LB, will more likely to jump.\n\nOf course, if one's CV is incorrect in the first place, optimizing it is a disaster. \n\nSo to answer your question: \"how to avoid shake down\", it's not too late to change your mindset, and select a submission that can generalize better on all kinds of test samples, with a slight penalty on public LB. That's also a psychological game: doing that you'll have no regret of being dropped, and will learn something from it, or you'll get sweet rewards. I think it's better than being surprisingly happy if you're unexpected rise on private LB (or accepting a drop) without understanding anything.", "votes": null }, { "id": "538851", "postDate": "05/29/2019 07:28:02", "content": "Overfiting is defined by Kaggle this way:
\n\n\n\n\nIt’s very easy to overfit a model, creating something that performs very well on the public leaderboard, but very badly on the private one. This is called overfitting.
\n
I think we use it correctly here.
", "rawMarkdown": "Overfiting is [defined by Kaggle](https://www.kaggle.com/docs/competitions) this way:\n\n> It’s very easy to overfit a model, creating something that performs very well on the public leaderboard, but very badly on the private one. This is called overfitting.\n\nI think we use it correctly here.", "votes": null }, { "id": "538859", "postDate": "05/29/2019 07:43:11", "content": "Thanks for your kindly answer, I think you totally solved my question. I will not focus that much on LB, and I will choose my best single model and blend of my own models as the final submission, instead of the best public LB. To summarize, learning is more important than medals, I learned a lot from this competition and this community, and I will not regret even being dropped. Thanks again. Wish you a great result in this competition.
", "rawMarkdown": "Thanks for your kindly answer, I think you totally solved my question. I will not focus that much on LB, and I will choose my best single model and blend of my own models as the final submission, instead of the best public LB. To summarize, learning is more important than medals, I learned a lot from this competition and this community, and I will not regret even being dropped. Thanks again. Wish you a great result in this competition.", "votes": null }, { "id": "538885", "postDate": "05/29/2019 08:40:08", "content": "Besides what Kha Vo and CPMP said above, you can always pray or try to predict the private LB using alternative methods (e.g. fortune-teller, magic mushrooms, etc)
", "rawMarkdown": "Besides what Kha Vo and CPMP said above, you can always pray or try to predict the private LB using alternative methods (e.g. fortune-teller, magic mushrooms, etc)", "votes": null }, { "id": "538888", "postDate": "05/29/2019 08:48:46", "content": "LOL
", "rawMarkdown": "LOL", "votes": null }, { "id": "538900", "postDate": "05/29/2019 09:17:56", "content": "\n\n\nblend of my own models as the final submission, instead of the best public LB.
\n
You can choose 2 submissions before deadline. I recommend to choose the best public LB (maybe blend with top score kernels) as one submission and another one you wish.
", "rawMarkdown": "> blend of my own models as the final submission, instead of the best public LB. \n\nYou can choose 2 submissions before deadline. I recommend to choose the best public LB (maybe blend with top score kernels) as one submission and another one you wish.", "votes": null }, { "id": "538902", "postDate": "05/29/2019 09:20:50", "content": "Looking at your good score I'm not sure you need an advice. :) It seems you know what to do.
", "rawMarkdown": "Looking at your good score I'm not sure you need an advice. :) It seems you know what to do.", "votes": null }, { "id": "538909", "postDate": "05/29/2019 09:28:25", "content": "We are all trying to find a good answer to that question. We will know in few days who got the best answer.
\n\nHere, we have too little data, and differences in train/test data (eg mean), which makes it very difficult to have a reliable CV setting. I still think that relying on CV score is betetr than relying on LB score here, but you need to have a reasonable good CV setting. But there is no warranty at all that this will lead to a good result.
", "rawMarkdown": "We are all trying to find a good answer to that question. We will know in few days who got the best answer. \n\nHere, we have too little data, and differences in train/test data (eg mean), which makes it very difficult to have a reliable CV setting. I still think that relying on CV score is betetr than relying on LB score here, but you need to have a reasonable good CV setting. But there is no warranty at all that this will lead to a good result.", "votes": null }, { "id": "538910", "postDate": "05/29/2019 09:33:02", "content": "Is it a good method that blending the best LB model with the best CV model?
", "rawMarkdown": "Is it a good method that blending the best LB model with the best CV model?", "votes": null }, { "id": "538931", "postDate": "05/29/2019 10:05:34", "content": "It is the first time I take a Kaggle competition, so I fear that my score will drop a lot on private dataset.
", "rawMarkdown": "It is the first time I take a Kaggle competition, so I fear that my score will drop a lot on private dataset.", "votes": null }, { "id": "538933", "postDate": "05/29/2019 10:06:23", "content": "The best way is to submit an extremely bad score - you will then come last and have no shake up at all. ;)
\n\nSeriously though try to use a few independent models to reduce variance
\n\nIf the median of the private LB is significantly different than train data then expect everyone to tank (we will probably tank the same amount though!)
", "rawMarkdown": "The best way is to submit an extremely bad score - you will then come last and have no shake up at all. ;)\n\nSeriously though try to use a few independent models to reduce variance\n\nIf the median of the private LB is significantly different than train data then expect everyone to tank (we will probably tank the same amount though!)", "votes": null }, { "id": "538939", "postDate": "05/29/2019 10:11:23", "content": "Wish you get a great result in the end of this competition.
", "rawMarkdown": "Wish you get a great result in the end of this competition.", "votes": null }, { "id": "539018", "postDate": "05/29/2019 12:02:13", "content": "Wish you get good result too.
", "rawMarkdown": "Wish you get good result too.", "votes": null }, { "id": "539039", "postDate": "05/29/2019 12:37:49", "content": "\n\n\nIs it a good method that blending the best LB model with the best CV model?
\n
The answer to that is highly contingent. The logic behind choosing the best LB model and best CV model as separate submissions is that both give you a bit different, but ideally objective information about the performance of your model.
\n\nCV tells you how well you are able to fit the training set, but since you have to design CV yourself, its only as good as you make it. Public LB is a kind of weird thing as this sort of information does not exist in real applications. Its really a holdout set that was constructed by someone with unknown intentions, it may or may not be representative of the full test set and determining this is often a key to doing well. If its not, a good LB score is worse than meaningless (see my drop in ELO for a good example). If you are sure your validation strategy is sound and you have some idea of the composition of test, combining your best CV and best LB may be a good idea, otherwise keeping them separate is often a reasonable hedge.
\n\nGood luck!
", "rawMarkdown": "> Is it a good method that blending the best LB model with the best CV model?\n\nThe answer to that is highly contingent. The logic behind choosing the best LB model and best CV model as separate submissions is that both give you a bit different, but ideally objective information about the performance of your model.\n\nCV tells you how well you are able to fit the training set, but since you have to design CV yourself, its only as good as you make it. Public LB is a kind of weird thing as this sort of information does not exist in real applications. Its really a holdout set that was constructed by someone with unknown intentions, it may or may not be representative of the full test set and determining this is often a key to doing well. If its not, a good LB score is worse than meaningless (see my drop in ELO for a good example). If you are sure your validation strategy is sound and you have some idea of the composition of test, combining your best CV and best LB may be a good idea, otherwise keeping them separate is often a reasonable hedge. \n\nGood luck!", "votes": null }, { "id": "539167", "postDate": "05/29/2019 16:17:11", "content": "emmmm， Thanks~
", "rawMarkdown": "emmmm， Thanks~", "votes": null }, { "id": "539285", "postDate": "05/29/2019 20:27:36", "content": "You have 2 submissions. You can choose a safe model and a risky one.
", "rawMarkdown": "You have 2 submissions. You can choose a safe model and a risky one.", "votes": null }, { "id": "539378", "postDate": "05/30/2019 02:11:05", "content": "The problem is I don't know which is safe and which is risky.
", "rawMarkdown": "The problem is I don't know which is safe and which is risky.", "votes": null }, { "id": "539452", "postDate": "05/30/2019 03:55:04", "content": "thx a lot ;)
", "rawMarkdown": "thx a lot ;)", "votes": null }, { "id": "539455", "postDate": "05/30/2019 04:01:00", "content": "i share the same feeling. you are not alone
", "rawMarkdown": "i share the same feeling. you are not alone", "votes": null }, { "id": "539680", "postDate": "05/30/2019 11:41:58", "content": "\"safe model\" is a bit ironic in this competition where nobody is even sure about their own CV ;)
", "rawMarkdown": "\"safe model\" is a bit ironic in this competition where nobody is even sure about their own CV ;)", "votes": null }, { "id": "539728", "postDate": "05/30/2019 12:51:04", "content": "Trust your local CV rather than LB. Blending is also a good way.
", "rawMarkdown": "Trust your local CV rather than LB. Blending is also a good way.", "votes": null }, { "id": "539761", "postDate": "05/30/2019 13:46:34", "content": "I choose to use different kind of model to ensemble them, and give them equal weight to ensemble. It is a easy one. And choose the highest single model score.
", "rawMarkdown": "I choose to use different kind of model to ensemble them, and give them equal weight to ensemble. It is a easy one. And choose the highest single model score.", "votes": null }, { "id": "539812", "postDate": "05/30/2019 14:53:00", "content": "\n\n\nYou can choose a safe model
\n
I'd be interested in any help on this :)
", "rawMarkdown": "> You can choose a safe model \n\nI'd be interested in any help on this :)", "votes": null }, { "id": "539824", "postDate": "05/30/2019 15:17:33", "content": "That's a difficult thing to answer from person to person and model to model. Which is why \"trust your CV\" is so vague and ambiguous. Because you can trust a poorly constructed CV and be misled due to undiscovered leakage and be led down the wrong path due to \"trust your CV\". You can also have bias in constructing CV that may also lead you down the wrong path. I think there is also a potential for confirmation bias for using CV and LB moving in the same direction. We don't know how the public test set is selected. I feel like this can be a huge toss up. The best scoring model on private LB may end up being a ton of least expected submissions.
\n\nSafe model in this context would just be one that doesn't necessarily score the best on public LB but has well thought out CV, feature selection, and model selection(s).
", "rawMarkdown": "That's a difficult thing to answer from person to person and model to model. Which is why \"trust your CV\" is so vague and ambiguous. Because you can trust a poorly constructed CV and be misled due to undiscovered leakage and be led down the wrong path due to \"trust your CV\". You can also have bias in constructing CV that may also lead you down the wrong path. I think there is also a potential for confirmation bias for using CV and LB moving in the same direction. We don't know how the public test set is selected. I feel like this can be a huge toss up. The best scoring model on private LB may end up being a ton of least expected submissions.\n\nSafe model in this context would just be one that doesn't necessarily score the best on public LB but has well thought out CV, feature selection, and model selection(s).", "votes": null }, { "id": "539835", "postDate": "05/30/2019 15:32:45", "content": "@teeyee314 - Your idea of having a relatively poor score might be a clever idea.\nHow are you doing with your own custom rolling function - is it close to being finished?
", "rawMarkdown": "teeyee314 - Your idea of having a relatively poor score might be a clever idea.\nHow are you doing with your own custom rolling function - is it close to being finished?", "votes": null }, { "id": "539837", "postDate": "05/30/2019 15:34:56", "content": "@cpmpml you're welcome xD
", "rawMarkdown": "cpmpml you're welcome xD", "votes": null }, { "id": "539844", "postDate": "05/30/2019 15:46:16", "content": "@scirpus I gave up on the custom rolling function for the time being. It was for EDA for some work I thought was promising, but the model (public kernel) ended up having high bias. It was doing too good on CV and when I submitted it, I found out that the CV contained a newbie mistake that I should have seen a mile away. The rolling function was more of a visual aid for the model, not actual feature engineering. The other thing you can do is roll back some arbitrary date of submissions to see which one was best. That may be the one that was least overfits (the LB). Often times, towards the end of the competition a lot of people get carried away with what scores the best on LB and forget about the past 30 days or so of work. Just my opinion here.
", "rawMarkdown": "scirpus I gave up on the custom rolling function for the time being. It was for EDA for some work I thought was promising, but the model (public kernel) ended up having high bias. It was doing too good on CV and when I submitted it, I found out that the CV contained a newbie mistake that I should have seen a mile away. The rolling function was more of a visual aid for the model, not actual feature engineering. The other thing you can do is roll back some arbitrary date of submissions to see which one was best. That may be the one that was least overfits (the LB). Often times, towards the end of the competition a lot of people get carried away with what scores the best on LB and forget about the past 30 days or so of work. Just my opinion here.", "votes": null } ], "comments": [ { "id": 538831, "author_name": "lyf19950404", "author_url": "", "post_date": "05/29/2019 06:53:22", "content": "I am a novice in Kaggle competition, so I just have the same question. Could anyone offer some good advice to avoid the shake up? Thanks a lot!
", "votes": null, "replies": [ { "id": 538885, "author_name": "chechir", "author_url": "", "post_date": "05/29/2019 08:40:08", "content": "Besides what Kha Vo and CPMP said above, you can always pray or try to predict the private LB using alternative methods (e.g. fortune-teller, magic mushrooms, etc)
", "votes": null, "replies": [] }, { "id": 538888, "author_name": "lyf19950404", "author_url": "", "post_date": "05/29/2019 08:48:46", "content": "LOL
", "votes": null, "replies": [] }, { "id": 538902, "author_name": "sergeyzlobin", "author_url": "", "post_date": "05/29/2019 09:20:50", "content": "Looking at your good score I'm not sure you need an advice. :) It seems you know what to do.
", "votes": null, "replies": [] }, { "id": 538931, "author_name": "lyf19950404", "author_url": "", "post_date": "05/29/2019 10:05:34", "content": "It is the first time I take a Kaggle competition, so I fear that my score will drop a lot on private dataset.
", "votes": null, "replies": [] }, { "id": 538939, "author_name": "leonshangguan", "author_url": "", "post_date": "05/29/2019 10:11:23", "content": "Wish you get a great result in the end of this competition.
", "votes": null, "replies": [] }, { "id": 539018, "author_name": "lyf19950404", "author_url": "", "post_date": "05/29/2019 12:02:13", "content": "Wish you get good result too.
", "votes": null, "replies": [] }, { "id": 539167, "author_name": "leonshangguan", "author_url": "", "post_date": "05/29/2019 16:17:11", "content": "emmmm， Thanks~
", "votes": null, "replies": [] }, { "id": 539728, "author_name": "seefun", "author_url": "", "post_date": "05/30/2019 12:51:04", "content": "Trust your local CV rather than LB. Blending is also a good way.
", "votes": null, "replies": [] } ] }, { "id": 538845, "author_name": "khahuras", "author_url": "", "post_date": "05/29/2019 07:17:43", "content": "It's better to say \"how to avoid shake down\", which means your submission scores worse in private LB than those placed below you on the public LB. People tend to say \"avoid overfitting\", but that's not right, because 100% of people are overfitting. The key thing is that the level of overfitting of each person.
\n\nDuring your course of submitting each day, if you focused more on public LB performance, you will tend to overfit it, and have more chance to drop, while people with a strong and sound CV setting trying to optimize it without looking on public LB, will more likely to jump.
\n\nOf course, if one's CV is incorrect in the first place, optimizing it is a disaster.
\n\nSo to answer your question: \"how to avoid shake down\", it's not too late to change your mindset, and select a submission that can generalize better on all kinds of test samples, with a slight penalty on public LB. That's also a psychological game: doing that you'll have no regret of being dropped, and will learn something from it, or you'll get sweet rewards. I think it's better than being surprisingly happy if you're unexpected rise on private LB (or accepting a drop) without understanding anything.
", "votes": null, "replies": [ { "id": 538851, "author_name": "cpmpml", "author_url": "", "post_date": "05/29/2019 07:28:02", "content": "Overfiting is defined by Kaggle this way:
\n\n\n\n\nIt’s very easy to overfit a model, creating something that performs very well on the public leaderboard, but very badly on the private one. This is called overfitting.
\n
I think we use it correctly here.
", "votes": null, "replies": [] }, { "id": 538859, "author_name": "leonshangguan", "author_url": "", "post_date": "05/29/2019 07:43:11", "content": "Thanks for your kindly answer, I think you totally solved my question. I will not focus that much on LB, and I will choose my best single model and blend of my own models as the final submission, instead of the best public LB. To summarize, learning is more important than medals, I learned a lot from this competition and this community, and I will not regret even being dropped. Thanks again. Wish you a great result in this competition.
", "votes": null, "replies": [] }, { "id": 538900, "author_name": "sergeyzlobin", "author_url": "", "post_date": "05/29/2019 09:17:56", "content": "\n\n\nblend of my own models as the final submission, instead of the best public LB.
\n
You can choose 2 submissions before deadline. I recommend to choose the best public LB (maybe blend with top score kernels) as one submission and another one you wish.
", "votes": null, "replies": [] }, { "id": 538910, "author_name": "takeiy", "author_url": "", "post_date": "05/29/2019 09:33:02", "content": "Is it a good method that blending the best LB model with the best CV model?
", "votes": null, "replies": [] }, { "id": 539039, "author_name": "interneuron", "author_url": "", "post_date": "05/29/2019 12:37:49", "content": "\n\n\nIs it a good method that blending the best LB model with the best CV model?
\n
The answer to that is highly contingent. The logic behind choosing the best LB model and best CV model as separate submissions is that both give you a bit different, but ideally objective information about the performance of your model.
\n\nCV tells you how well you are able to fit the training set, but since you have to design CV yourself, its only as good as you make it. Public LB is a kind of weird thing as this sort of information does not exist in real applications. Its really a holdout set that was constructed by someone with unknown intentions, it may or may not be representative of the full test set and determining this is often a key to doing well. If its not, a good LB score is worse than meaningless (see my drop in ELO for a good example). If you are sure your validation strategy is sound and you have some idea of the composition of test, combining your best CV and best LB may be a good idea, otherwise keeping them separate is often a reasonable hedge.
\n\nGood luck!
", "votes": null, "replies": [] }, { "id": 539452, "author_name": "takeiy", "author_url": "", "post_date": "05/30/2019 03:55:04", "content": "thx a lot ;)
", "votes": null, "replies": [] } ] }, { "id": 538909, "author_name": "cpmpml", "author_url": "", "post_date": "05/29/2019 09:28:25", "content": "We are all trying to find a good answer to that question. We will know in few days who got the best answer.
\n\nHere, we have too little data, and differences in train/test data (eg mean), which makes it very difficult to have a reliable CV setting. I still think that relying on CV score is betetr than relying on LB score here, but you need to have a reasonable good CV setting. But there is no warranty at all that this will lead to a good result.
", "votes": null, "replies": [] }, { "id": 538933, "author_name": "scirpus", "author_url": "", "post_date": "05/29/2019 10:06:23", "content": "The best way is to submit an extremely bad score - you will then come last and have no shake up at all. ;)
\n\nSeriously though try to use a few independent models to reduce variance
\n\nIf the median of the private LB is significantly different than train data then expect everyone to tank (we will probably tank the same amount though!)
", "votes": null, "replies": [] }, { "id": 539285, "author_name": "felipefonte99", "author_url": "", "post_date": "05/29/2019 20:27:36", "content": "You have 2 submissions. You can choose a safe model and a risky one.
", "votes": null, "replies": [ { "id": 539378, "author_name": "leonshangguan", "author_url": "", "post_date": "05/30/2019 02:11:05", "content": "The problem is I don't know which is safe and which is risky.
", "votes": null, "replies": [] }, { "id": 539455, "author_name": "returnofsputnik", "author_url": "", "post_date": "05/30/2019 04:01:00", "content": "i share the same feeling. you are not alone
", "votes": null, "replies": [] }, { "id": 539680, "author_name": "ilu000", "author_url": "", "post_date": "05/30/2019 11:41:58", "content": "\"safe model\" is a bit ironic in this competition where nobody is even sure about their own CV ;)
", "votes": null, "replies": [] }, { "id": 539812, "author_name": "cpmpml", "author_url": "", "post_date": "05/30/2019 14:53:00", "content": "\n\n\nYou can choose a safe model
\n
I'd be interested in any help on this :)
", "votes": null, "replies": [] }, { "id": 539824, "author_name": "teeyee314", "author_url": "", "post_date": "05/30/2019 15:17:33", "content": "That's a difficult thing to answer from person to person and model to model. Which is why \"trust your CV\" is so vague and ambiguous. Because you can trust a poorly constructed CV and be misled due to undiscovered leakage and be led down the wrong path due to \"trust your CV\". You can also have bias in constructing CV that may also lead you down the wrong path. I think there is also a potential for confirmation bias for using CV and LB moving in the same direction. We don't know how the public test set is selected. I feel like this can be a huge toss up. The best scoring model on private LB may end up being a ton of least expected submissions.
\n\nSafe model in this context would just be one that doesn't necessarily score the best on public LB but has well thought out CV, feature selection, and model selection(s).
", "votes": null, "replies": [] }, { "id": 539835, "author_name": "scirpus", "author_url": "", "post_date": "05/30/2019 15:32:45", "content": "@teeyee314 - Your idea of having a relatively poor score might be a clever idea.\nHow are you doing with your own custom rolling function - is it close to being finished?
", "votes": null, "replies": [] }, { "id": 539837, "author_name": "davids1992", "author_url": "", "post_date": "05/30/2019 15:34:56", "content": "@cpmpml you're welcome xD
", "votes": null, "replies": [] }, { "id": 539844, "author_name": "teeyee314", "author_url": "", "post_date": "05/30/2019 15:46:16", "content": "@scirpus I gave up on the custom rolling function for the time being. It was for EDA for some work I thought was promising, but the model (public kernel) ended up having high bias. It was doing too good on CV and when I submitted it, I found out that the CV contained a newbie mistake that I should have seen a mile away. The rolling function was more of a visual aid for the model, not actual feature engineering. The other thing you can do is roll back some arbitrary date of submissions to see which one was best. That may be the one that was least overfits (the LB). Often times, towards the end of the competition a lot of people get carried away with what scores the best on LB and forget about the past 30 days or so of work. Just my opinion here.
", "votes": null, "replies": [] } ] }, { "id": 539761, "author_name": "dyyalex", "author_url": "", "post_date": "05/30/2019 13:46:34", "content": "I choose to use different kind of model to ensemble them, and give them equal weight to ensemble. It is a easy one. And choose the highest single model score.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "538823": "I saw a lot of topics saying shake up will happen, but, how to avoid that?\nI used different features and models for this competition, and the LB is around 1.410-1.516. My best submission blend some of the top public kernels' submissions reaching 1.395.\nI really have questions about which should be selected as the final submission, best single model, or blend of all my models, or my best LB 1.395, or any other ideas?\nOr what I could do is only just wait and see? \nI will appreciate it if someone could give me suggestions.", "538831": "I am a novice in Kaggle competition, so I just have the same question. Could anyone offer some good advice to avoid the shake up? Thanks a lot!", "538845": "It's better to say \"how to avoid shake down\", which means your submission scores worse in private LB than those placed below you on the public LB. People tend to say \"avoid overfitting\", but that's not right, because 100% of people are overfitting. The key thing is that the level of overfitting of each person.\n\nDuring your course of submitting each day, if you focused more on public LB performance, you will tend to overfit it, and have more chance to drop, while people with a strong and sound CV setting trying to optimize it without looking on public LB, will more likely to jump.\n\nOf course, if one's CV is incorrect in the first place, optimizing it is a disaster. \n\nSo to answer your question: \"how to avoid shake down\", it's not too late to change your mindset, and select a submission that can generalize better on all kinds of test samples, with a slight penalty on public LB. That's also a psychological game: doing that you'll have no regret of being dropped, and will learn something from it, or you'll get sweet rewards. I think it's better than being surprisingly happy if you're unexpected rise on private LB (or accepting a drop) without understanding anything.", "538851": "Overfiting is [defined by Kaggle](https://www.kaggle.com/docs/competitions) this way:\n\n> It’s very easy to overfit a model, creating something that performs very well on the public leaderboard, but very badly on the private one. This is called overfitting.\n\nI think we use it correctly here.", "538859": "Thanks for your kindly answer, I think you totally solved my question. I will not focus that much on LB, and I will choose my best single model and blend of my own models as the final submission, instead of the best public LB. To summarize, learning is more important than medals, I learned a lot from this competition and this community, and I will not regret even being dropped. Thanks again. Wish you a great result in this competition.", "538885": "Besides what Kha Vo and CPMP said above, you can always pray or try to predict the private LB using alternative methods (e.g. fortune-teller, magic mushrooms, etc)", "538888": "LOL", "538900": "> blend of my own models as the final submission, instead of the best public LB. \n\nYou can choose 2 submissions before deadline. I recommend to choose the best public LB (maybe blend with top score kernels) as one submission and another one you wish.", "538902": "Looking at your good score I'm not sure you need an advice. :) It seems you know what to do.", "538909": "We are all trying to find a good answer to that question. We will know in few days who got the best answer. \n\nHere, we have too little data, and differences in train/test data (eg mean), which makes it very difficult to have a reliable CV setting. I still think that relying on CV score is betetr than relying on LB score here, but you need to have a reasonable good CV setting. But there is no warranty at all that this will lead to a good result.", "538910": "Is it a good method that blending the best LB model with the best CV model?", "538931": "It is the first time I take a Kaggle competition, so I fear that my score will drop a lot on private dataset.", "538933": "The best way is to submit an extremely bad score - you will then come last and have no shake up at all. ;)\n\nSeriously though try to use a few independent models to reduce variance\n\nIf the median of the private LB is significantly different than train data then expect everyone to tank (we will probably tank the same amount though!)", "538939": "Wish you get a great result in the end of this competition.", "539018": "Wish you get good result too.", "539039": "> Is it a good method that blending the best LB model with the best CV model?\n\nThe answer to that is highly contingent. The logic behind choosing the best LB model and best CV model as separate submissions is that both give you a bit different, but ideally objective information about the performance of your model.\n\nCV tells you how well you are able to fit the training set, but since you have to design CV yourself, its only as good as you make it. Public LB is a kind of weird thing as this sort of information does not exist in real applications. Its really a holdout set that was constructed by someone with unknown intentions, it may or may not be representative of the full test set and determining this is often a key to doing well. If its not, a good LB score is worse than meaningless (see my drop in ELO for a good example). If you are sure your validation strategy is sound and you have some idea of the composition of test, combining your best CV and best LB may be a good idea, otherwise keeping them separate is often a reasonable hedge. \n\nGood luck!", "539167": "emmmm， Thanks~", "539285": "You have 2 submissions. You can choose a safe model and a risky one.", "539378": "The problem is I don't know which is safe and which is risky.", "539452": "thx a lot ;)", "539455": "i share the same feeling. you are not alone", "539680": "\"safe model\" is a bit ironic in this competition where nobody is even sure about their own CV ;)", "539728": "Trust your local CV rather than LB. Blending is also a good way.", "539761": "I choose to use different kind of model to ensemble them, and give them equal weight to ensemble. It is a easy one. And choose the highest single model score.", "539812": "> You can choose a safe model \n\nI'd be interested in any help on this :)", "539824": "That's a difficult thing to answer from person to person and model to model. Which is why \"trust your CV\" is so vague and ambiguous. Because you can trust a poorly constructed CV and be misled due to undiscovered leakage and be led down the wrong path due to \"trust your CV\". You can also have bias in constructing CV that may also lead you down the wrong path. I think there is also a potential for confirmation bias for using CV and LB moving in the same direction. We don't know how the public test set is selected. I feel like this can be a huge toss up. The best scoring model on private LB may end up being a ton of least expected submissions.\n\nSafe model in this context would just be one that doesn't necessarily score the best on public LB but has well thought out CV, feature selection, and model selection(s).", "539835": "teeyee314 - Your idea of having a relatively poor score might be a clever idea.\nHow are you doing with your own custom rolling function - is it close to being finished?", "539837": "cpmpml you're welcome xD", "539844": "scirpus I gave up on the custom rolling function for the time being. It was for EDA for some work I thought was promising, but the model (public kernel) ended up having high bias. It was doing too good on CV and when I submitted it, I found out that the CV contained a newbie mistake that I should have seen a mile away. The rolling function was more of a visual aid for the model, not actual feature engineering. The other thing you can do is roll back some arbitrary date of submissions to see which one was best. That may be the one that was least overfits (the LB). Often times, towards the end of the competition a lot of people get carried away with what scores the best on LB and forget about the past 30 days or so of work. Just my opinion here." }, "source": "meta" }