![\"\"](\"https://pp.userapi.com/c854220/v854220535/29d4e/_RML2HOjHhc.jpg\")
\nCan someone please explain to me what did I do, what's the nature of this CV-LB correlation and why plots are so weird? Am I overfitting like hell?\nThanks in advance!It's great in any competition to know how others are approaching to the solution without much of details. Let's share with each other which model we are using.
\n\nI am using a LightGBM model.
", "messages": [ { "id": 493196, "postDate": "2019-03-18T12:06:28.017Z", "content": "It's great in any competition to know how others are approaching to the solution without much of details. Let's share with each other which model we are using.
\n\nI am using a LightGBM model.
", "rawMarkdown": "It's great in any competition to know how others are approaching to the solution without much of details. Let's share with each other which model we are using.\n\nI am using a LightGBM model.", "votes": 35 }, { "id": 500613, "postDate": "2019-03-26T09:51:33.933Z", "content": "CatBoost - 1.87 LB: 1.441 on FFT features
", "rawMarkdown": "CatBoost - 1.87 LB: 1.441 on FFT features", "votes": 11, "replies": [ { "id": 516841, "postDate": "2019-04-15T04:26:32.580Z", "content": "use FFT feature only ?
", "rawMarkdown": "use FFT feature only ?" } ] }, { "id": 519221, "postDate": "2019-04-18T14:57:31.013Z", "content": "Starting with lgb and 7 features: cv 1.901, lb 1.444
", "rawMarkdown": "Starting with lgb and 7 features: cv 1.901, lb 1.444", "votes": 9, "replies": [ { "id": 519249, "postDate": "2019-04-18T15:57:19.120Z", "content": "I have 300 features and my lgb reaches 1.475. Clearly I'm taking the wrong approach!
", "rawMarkdown": "I have 300 features and my lgb reaches 1.475. Clearly I'm taking the wrong approach!", "votes": 1 }, { "id": 519259, "postDate": "2019-04-18T16:19:29.660Z", "content": "Well, I added 3 features and LB went to 2.5 ... I may have goofed, but I may just discover the challenge we all face here.
", "rawMarkdown": "Well, I added 3 features and LB went to 2.5 ... I may have goofed, but I may just discover the challenge we all face here.", "votes": 3 }, { "id": 519265, "postDate": "2019-04-18T16:27:40.197Z", "content": "Well, I did goof, let's see tomorrow if CV and LB evolve in the same direction.
", "rawMarkdown": "Well, I did goof, let's see tomorrow if CV and LB evolve in the same direction.", "votes": 1 }, { "id": 519276, "postDate": "2019-04-18T16:52:17.220Z", "content": "Are you using KFold or LOGO? I've found KFold gets better LB score but LOGO (and other quake-wise splits) get a much, much lower CV, around 1.5. As for my features, I'm pretty certain many of them are redundant due to multicolinearity. I'm otherwise pleased with them since the correlations to the target are very strong.
", "rawMarkdown": "Are you using KFold or LOGO? I've found KFold gets better LB score but LOGO (and other quake-wise splits) get a much, much lower CV, around 1.5. As for my features, I'm pretty certain many of them are redundant due to multicolinearity. I'm otherwise pleased with them since the correlations to the target are very strong. ", "votes": 1 }, { "id": 519285, "postDate": "2019-04-18T17:03:38.027Z", "content": "What is LOGO?
", "rawMarkdown": "What is LOGO?" }, { "id": 519288, "postDate": "2019-04-18T17:11:04.170Z", "content": "I guess it's leave-one-group-out. So meaning CV by each earthquake group.
", "rawMarkdown": "I guess it's leave-one-group-out. So meaning CV by each earthquake group." }, { "id": 519294, "postDate": "2019-04-18T17:21:26.360Z", "content": "Yes, Leave One Group Out. You'll find that earthquake periods 8 and 15 produce a notably high MAE with this approach, while many of the other quakes get an MAE of around 1 using a simple model. I got a poor LB with this approach but I'm not discounting it entirely.
", "rawMarkdown": "Yes, Leave One Group Out. You'll find that earthquake periods 8 and 15 produce a notably high MAE with this approach, while many of the other quakes get an MAE of around 1 using a simple model. I got a poor LB with this approach but I'm not discounting it entirely." }, { "id": 519325, "postDate": "2019-04-18T18:30:47.287Z", "content": "How do you make your final predictions with your models? Retrain on the whole set? Average all model predictions for the k-folds?\nI think the much lower cv of your LOGO models could be due to overfitting on the relatively small validation set, which is much larger if you use e.g. 3-fold or 5-fold cross-validation. Could also be a really good model, but comparing it with other cv-scores and accounting the lower lb score i guess it's unlikely. Maybe grouping 2-3 earthquakes together into one validation split could improve this.
", "rawMarkdown": "How do you make your final predictions with your models? Retrain on the whole set? Average all model predictions for the k-folds?\nI think the much lower cv of your LOGO models could be due to overfitting on the relatively small validation set, which is much larger if you use e.g. 3-fold or 5-fold cross-validation. Could also be a really good model, but comparing it with other cv-scores and accounting the lower lb score i guess it's unlikely. Maybe grouping 2-3 earthquakes together into one validation split could improve this.", "votes": 1 }, { "id": 519331, "postDate": "2019-04-18T18:45:50.877Z", "content": "Definitely. However, since we're only permitted 2 submissions per day it'd take more than a week to evaluate each model individually! Taking the mean prediction from LOGO gave me much worse results than a simple KFold.
", "rawMarkdown": "Definitely. However, since we're only permitted 2 submissions per day it'd take more than a week to evaluate each model individually! Taking the mean prediction from LOGO gave me much worse results than a simple KFold." }, { "id": 519372, "postDate": "2019-04-18T20:41:59.140Z", "content": "I didn't mean to eval each single model. I'm sure it's possible to \"cheat\" a better lb score by manipulating the train set to have a comparable mean ttf as the test set or simply taking the folds for training which seem to fit the test set distribution best, but that's another topic. I just wanted to say that you'd probably get a more realistic score if you didn't \"leave-one-group(=earthquake)-out\" but \"leave-multiple-groups-out\". Hope it's a little bit clearer now.
", "rawMarkdown": "I didn't mean to eval each single model. I'm sure it's possible to \"cheat\" a better lb score by manipulating the train set to have a comparable mean ttf as the test set or simply taking the folds for training which seem to fit the test set distribution best, but that's another topic. I just wanted to say that you'd probably get a more realistic score if you didn't \"leave-one-group(=earthquake)-out\" but \"leave-multiple-groups-out\". Hope it's a little bit clearer now.\n" }, { "id": 519414, "postDate": "2019-04-18T22:49:46.920Z", "content": "That's what I've been working on, disappointingly I can get anything to work better than simple KFolds.
", "rawMarkdown": "That's what I've been working on, disappointingly I can get anything to work better than simple KFolds." }, { "id": 519477, "postDate": "2019-04-19T03:20:35.463Z", "content": "Only 7 features?
", "rawMarkdown": "Only 7 features?" }, { "id": 519530, "postDate": "2019-04-19T06:31:34.183Z", "content": "Yes only 7
", "rawMarkdown": "Yes only 7", "votes": 2 }, { "id": 519539, "postDate": "2019-04-19T07:02:00.433Z", "content": "sounds like there are magic features here.
", "rawMarkdown": "sounds like there are magic features here." }, { "id": 519544, "postDate": "2019-04-19T07:12:41.103Z", "content": "No magic no leak ;)
\n\nupdate: lgb 10 features cv 1.866 lb 1.416
", "rawMarkdown": "No magic no leak ;)\n\nupdate: lgb 10 features cv 1.866 lb 1.416", "votes": 8 }, { "id": 519550, "postDate": "2019-04-19T07:18:22.803Z", "content": "cool
", "rawMarkdown": "cool", "votes": 1 }, { "id": 519564, "postDate": "2019-04-19T07:41:07.427Z", "content": "LGB 200+ features, LB 1.48, CV 2.01...\n@cpmpml, how do you do get 1.416 with 10 features?! Is it some Uncle-ish dark magic?
", "rawMarkdown": "LGB 200+ features, LB 1.48, CV 2.01...\n@cpmpml, how do you do get 1.416 with 10 features?! Is it some Uncle-ish dark magic?", "votes": 1 }, { "id": 519585, "postDate": "2019-04-19T08:32:28.763Z", "content": "Looks good, what kind of CV setup are you using @cpmpml if you mind telling.
", "rawMarkdown": "Looks good, what kind of CV setup are you using @cpmpml if you mind telling.", "votes": 2 }, { "id": 519655, "postDate": "2019-04-19T11:48:49.837Z", "content": "Psi,
\n\nI don't plan to share much in this competition after the bad experience in the Santander competition. I do remember you were on my side there, thanks for that, you're not the reason why I don't want to share anymore.
\n\nAnyway, I already shared more than what you shared during Santander ;)
", "rawMarkdown": "Psi, \n\nI don't plan to share much in this competition after the bad experience in the Santander competition. I do remember you were on my side there, thanks for that, you're not the reason why I don't want to share anymore.\n\nAnyway, I already shared more than what you shared during Santander ;)\n", "votes": 5 }, { "id": 519670, "postDate": "2019-04-19T12:25:59.127Z", "content": "What happened in the Santander competition? Couldn't find any discussions of bad experience, everybody is thanking each other. Did it get deleted?
", "rawMarkdown": "What happened in the Santander competition? Couldn't find any discussions of bad experience, everybody is thanking each other. Did it get deleted?" }, { "id": 519683, "postDate": "2019-04-19T13:13:14.937Z", "content": "Don't bother, if you weren't exposed to it, then don't look for it. In a nutshell, I, and few others like Psi's team, were accused of purposely misleading participants because we used the word 'magic' to describe what we were doing. People argued it was rather a leak exploit and that we should have said it. Hence my current team name ;)
", "rawMarkdown": "Don't bother, if you weren't exposed to it, then don't look for it. In a nutshell, I, and few others like Psi's team, were accused of purposely misleading participants because we used the word 'magic' to describe what we were doing. People argued it was rather a leak exploit and that we should have said it. Hence my current team name ;)", "votes": 2 }, { "id": 519685, "postDate": "2019-04-19T13:16:47.923Z", "content": "Fair enough :)
", "rawMarkdown": "Fair enough :)", "votes": 1 }, { "id": 519697, "postDate": "2019-04-19T13:36:37.970Z", "content": "@cpmpml, I am sorry for your experience. But I would rather want you and others to continue with spreading the \"magic\" because 1) this competition is much harder than Santander and 2) it is so much difficult to learn feature engineering just based on online articles or public kernels. I learnt from Santander that feature engineering through EDA is really important.
\n\nAgain, it is your choice, and I'll respect whatever decision you make.
", "rawMarkdown": "@cpmpml, I am sorry for your experience. But I would rather want you and others to continue with spreading the \"magic\" because 1) this competition is much harder than Santander and 2) it is so much difficult to learn feature engineering just based on online articles or public kernels. I learnt from Santander that feature engineering through EDA is really important. \n\nAgain, it is your choice, and I'll respect whatever decision you make." }, { "id": 519702, "postDate": "2019-04-19T13:55:06.480Z", "content": "To offer a counter opinion, I'd be careful about what you share. In case of any non-trivial findings, it's best to keep quiet. I find it selfish when people share major things for vanity's sake or for points. It spoils the discovery for others. Spoonfeeding is also condescending.\nIn addition to that, if you talk on the forum in language that is understood only by a part of the public, e.g. using the word \"magic\" to refer to something that you and some others have found, and discussing it without revealing it, it may be construed as \"private sharing outside teams\" and get your team banned. Why risk that, and for what?
", "rawMarkdown": "To offer a counter opinion, I'd be careful about what you share. In case of any non-trivial findings, it's best to keep quiet. I find it selfish when people share major things for vanity's sake or for points. It spoils the discovery for others. Spoonfeeding is also condescending.\nIn addition to that, if you talk on the forum in language that is understood only by a part of the public, e.g. using the word \"magic\" to refer to something that you and some others have found, and discussing it without revealing it, it may be construed as \"private sharing outside teams\" and get your team banned. Why risk that, and for what?", "votes": 1 }, { "id": 519770, "postDate": "2019-04-19T16:06:31.430Z", "content": "Sharing has its pros and conses, I'm not going to argue for one or the other here, but I want to react to one thing, namely:
\n\n\n\n\nif you talk on the forum in language that is understood only by a part of the public, e.g. using the word \"magic\" to refer to something that you and some others have found, and discussing it without revealing it, it may be construed as \"private sharing outside teams\" and get your team banned.
\n
I don't get how you can conclude using the word 'magic' is private sharing. By definition, something shared in the competition forum is not private sharing. Please explain.
", "rawMarkdown": "Sharing has its pros and conses, I'm not going to argue for one or the other here, but I want to react to one thing, namely:\n\n> if you talk on the forum in language that is understood only by a part of the public, e.g. using the word \"magic\" to refer to something that you and some others have found, and discussing it without revealing it, it may be construed as \"private sharing outside teams\" and get your team banned. \n\nI don't get how you can conclude using the word 'magic' is private sharing. By definition, something shared in the competition forum is not private sharing. Please explain.", "votes": 1 }, { "id": 519782, "postDate": "2019-04-19T16:17:44.107Z", "content": "Sure. You can share something only with people with whom you already share a secret, like the meaning of the word \"magic\". It's a basic form of encryption. For example, you can say \"I made a new feature by multiplying the first and the second magic features\". Only the people who know the magic features can understand that.
", "rawMarkdown": "Sure. You can share something only with people with whom you already share a secret, like the meaning of the word \"magic\". It's a basic form of encryption. For example, you can say \"I made a new feature by multiplying the first and the second magic features\". Only the people who know the magic features can understand that." }, { "id": 519798, "postDate": "2019-04-19T16:47:42.960Z", "content": "Good, you then agree that there is no private sharing if all communication goes via the forum.
", "rawMarkdown": "Good, you then agree that there is no private sharing if all communication goes via the forum.", "votes": 2 }, { "id": 519949, "postDate": "2019-04-19T21:15:52.743Z", "content": "I can understand your reluctance to share given the Santander experience. But I don't think it should put you off sharing in general. Santander was (in terms of the data itself) a relatively simple competition, and likely full of frustrated beginners hoping to stumble on a single clever trick to make them progress, growing bitter at their lack of success. This is a far more interesting challenge and I've found the discussions here to be better-natured as well. I'm currently writing a one-function, plug-in-and-play Hyperopt kernel for LGBM, XGB and CatBoost. It's not in my best interest to share it, but I try to learn from Chris Deotte's example - sharing is what makes Kaggle such a great place!
", "rawMarkdown": "I can understand your reluctance to share given the Santander experience. But I don't think it should put you off sharing in general. Santander was (in terms of the data itself) a relatively simple competition, and likely full of frustrated beginners hoping to stumble on a single clever trick to make them progress, growing bitter at their lack of success. This is a far more interesting challenge and I've found the discussions here to be better-natured as well. I'm currently writing a one-function, plug-in-and-play Hyperopt kernel for LGBM, XGB and CatBoost. It's not in my best interest to share it, but I try to learn from Chris Deotte's example - sharing is what makes Kaggle such a great place! ", "votes": 3 }, { "id": 519953, "postDate": "2019-04-19T21:20:22.217Z", "content": "Also, you got feedback for sharing your stuff - good, bad or ugly.
", "rawMarkdown": "Also, you got feedback for sharing your stuff - good, bad or ugly." }, { "id": 520097, "postDate": "2019-04-20T06:46:28.507Z", "content": "There are 36 teams with better things to share than me ;) They are mostly silent. Why don't you bug them a bit ? ;)
", "rawMarkdown": "There are 36 teams with better things to share than me ;) They are mostly silent. Why don't you bug them a bit ? ;)", "votes": 5 }, { "id": 520232, "postDate": "2019-04-20T13:36:10.997Z", "content": "@bigironsphere I agree with you. I am discussing a bit, and I'll keep doing it as long as it is around understanding the problem better. Sharing generic code as you plan to do is fine too, and I may do it here again if I have something to share. What I don't plan to share is how I approach the problem. I see it benefit to people who end up in front of me in the LB in every competition I enter. This time I want to see how it goes without this sharing.
\n\n@pukkinming I did get some thanks indeed, which is why I share in general.
\n\nAnd to those who down vote me, the more down vote the less sharing from me. Keep going.
\n\nAnyway, here is a bit of sharing as I can't help helping ;)
\n\n\n\n\nHow do you make your final predictions with your models? Retrain on the whole set? Average all model predictions for the k-folds?
\n
I do the latter. I find it to be better in general, but not always. ideally one should try both and pick what's best.
", "rawMarkdown": "@bigironsphere I agree with you. I am discussing a bit, and I'll keep doing it as long as it is around understanding the problem better. Sharing generic code as you plan to do is fine too, and I may do it here again if I have something to share. What I don't plan to share is how I approach the problem. I see it benefit to people who end up in front of me in the LB in every competition I enter. This time I want to see how it goes without this sharing.\n\n@pukkinming I did get some thanks indeed, which is why I share in general. \n\nAnd to those who down vote me, the more down vote the less sharing from me. Keep going.\n\nAnyway, here is a bit of sharing as I can't help helping ;)\n\n> How do you make your final predictions with your models? Retrain on the whole set? Average all model predictions for the k-folds?\n\nI do the latter. I find it to be better in general, but not always. ideally one should try both and pick what's best.", "votes": 4 }, { "id": 520268, "postDate": "2019-04-20T15:52:50.713Z", "content": "@cpmpml, let me thank you again for your sharing.
", "rawMarkdown": "@cpmpml, let me thank you again for your sharing.", "votes": 1 }, { "id": 520347, "postDate": "2019-04-20T19:02:32.597Z", "content": "Well, I'll share something with you since I've learned so much from you in the past.
\n\nWhen I realised that LOGO was leading to massive overfitting, I tried a custom validation strategy. There are three quakes that exhibit a 'mini-quake' halfway through that tend to confuse the models and make them predict a lower TTF than usual, hence why unshuffled KFold leads to such high variance among CV scores. These are the TTF periods 3, 8 and 15. To account for them, I would sample one of these quakes at random, along with 5 others without a mini-quake, and use this as the validation set. This process was repeated many times so that the complicated 'mini-quakes' segments were always present in a 2:1 ratio in the training and validation sets in every fold.
\n\nUltimately this lead to a good CV but a worse LB score, leading me to believe that the current LB test set includes simpler earthquake cases without the presence of 'mini-quake' segments. If you're obtaining good scores from a small number of features, it's likely that your model isn't overfitting based on these three complex cases - I've personally found that decreasing the number of features makes my models less sensitive to the mini-quakes when viewing their predictions graphically. Unfortunately, as ever, we have no way of knowing if this performance boost will be reflected in the private data.
", "rawMarkdown": "Well, I'll share something with you since I've learned so much from you in the past. \n\nWhen I realised that LOGO was leading to massive overfitting, I tried a custom validation strategy. There are three quakes that exhibit a 'mini-quake' halfway through that tend to confuse the models and make them predict a lower TTF than usual, hence why unshuffled KFold leads to such high variance among CV scores. These are the TTF periods 3, 8 and 15. To account for them, I would sample one of these quakes at random, along with 5 others without a mini-quake, and use this as the validation set. This process was repeated many times so that the complicated 'mini-quakes' segments were always present in a 2:1 ratio in the training and validation sets in every fold.\n\nUltimately this lead to a good CV but a worse LB score, leading me to believe that the current LB test set includes simpler earthquake cases without the presence of 'mini-quake' segments. If you're obtaining good scores from a small number of features, it's likely that your model isn't overfitting based on these three complex cases - I've personally found that decreasing the number of features makes my models less sensitive to the mini-quakes when viewing their predictions graphically. Unfortunately, as ever, we have no way of knowing if this performance boost will be reflected in the private data. ", "votes": 4 }, { "id": 523047, "postDate": "2019-04-25T13:14:31.503Z", "content": "did you choose your 10 features with top feature importance from huge statistic features ?
", "rawMarkdown": "did you choose your 10 features with top feature importance from huge statistic features ?" }, { "id": 524198, "postDate": "2019-04-28T07:42:27.573Z", "content": "I crafted the first 10 features without reading any of the forum or kernels. Then I selected the subset that appeared to work well on my cv setting. I submitted them in my first sub. The other 3 are cherry picked from public kernels. They improved my cv hence I submitted with them. Since then I tried lots of public kernel features and could find any that improve my cv. I didn’t submit any of them. That’s a long way to say I select feature by training models and looking at cv score. I never understood the use of statistics for feature selection. These statistics are only useful when you use generalized linear models IMHO.
", "rawMarkdown": "I crafted the first 10 features without reading any of the forum or kernels. Then I selected the subset that appeared to work well on my cv setting. I submitted them in my first sub. The other 3 are cherry picked from public kernels. They improved my cv hence I submitted with them. Since then I tried lots of public kernel features and could find any that improve my cv. I didn’t submit any of them. That’s a long way to say I select feature by training models and looking at cv score. I never understood the use of statistics for feature selection. These statistics are only useful when you use generalized linear models IMHO.", "votes": 3 }, { "id": 524836, "postDate": "2019-04-29T16:08:02.503Z", "content": "@bigironsphere Maybe I missed something - why do you think LOGO \"was leading to massive overfitting\"?
", "rawMarkdown": "@bigironsphere Maybe I missed something - why do you think LOGO \"was leading to massive overfitting\"?" }, { "id": 524844, "postDate": "2019-04-29T16:12:03.677Z", "content": "Perhaps overfitting was the wrong word, but each individual 'fold' in LOGO is being optimised towards a very specific validation set. I prefer using grouped quakes for validation, but maybe LOGO is better than I thought. We'll have to see.
", "rawMarkdown": "Perhaps overfitting was the wrong word, but each individual 'fold' in LOGO is being optimised towards a very specific validation set. I prefer using grouped quakes for validation, but maybe LOGO is better than I thought. We'll have to see." }, { "id": 524846, "postDate": "2019-04-29T16:18:39.967Z", "content": "@bigironsphere That should only happen if you early stop, right? If you just let your models converge and average over all fold errors (maybe scaled with the quake length), i think it's pretty safe.
\n\n@cpmpml Any thoughts on sequential feature selection? I was too lazy to do it all by hand and apply sffs after calculating f-scores of features. Seems to work quite well but i'm not sure if selection by hand would be better.
", "rawMarkdown": "@bigironsphere That should only happen if you early stop, right? If you just let your models converge and average over all fold errors (maybe scaled with the quake length), i think it's pretty safe.\n\n@cpmpml Any thoughts on sequential feature selection? I was too lazy to do it all by hand and apply sffs after calculating f-scores of features. Seems to work quite well but i'm not sure if selection by hand would be better.", "votes": 1 }, { "id": 524868, "postDate": "2019-04-29T17:16:13.173Z", "content": "Convergence would only increase the specificity of the model for that fold, early stopping would mitigate it. I'm not saying LOGO isn't safe, just that I don't think the best approach is to average a bunch of highly-specified models - I would prefer a more varied validation set. What do you mean by scaling fold errors by quake length?
", "rawMarkdown": "Convergence would only increase the specificity of the model for that fold, early stopping would mitigate it. I'm not saying LOGO isn't safe, just that I don't think the best approach is to average a bunch of highly-specified models - I would prefer a more varied validation set. What do you mean by scaling fold errors by quake length? " }, { "id": 524934, "postDate": "2019-04-29T19:37:30.390Z", "content": "@bigironsphere \nHow would it increase the specificity of the model? Maybe i'm wrong, but i'd say it's the opposite since you don't use any information from the validation data while training as opposed to training with early stopping. You'll still have some leakage by tuning hyperparameters but it guess that's inevitable.
\n\nFor the scaling part: I just meant that you take the number of samples per quake into account when calculating the cv score. Let's say you have 100 samples in total, do an earthquake-split and your first earthquake only consist of 2 samples, then you scale the error you get when validating it by factor 2/100 and so on. There probably are better options, but it's better than doing an unweighted average for sure.
", "rawMarkdown": "@bigironsphere \nHow would it increase the specificity of the model? Maybe i'm wrong, but i'd say it's the opposite since you don't use any information from the validation data while training as opposed to training with early stopping. You'll still have some leakage by tuning hyperparameters but it guess that's inevitable.\n\nFor the scaling part: I just meant that you take the number of samples per quake into account when calculating the cv score. Let's say you have 100 samples in total, do an earthquake-split and your first earthquake only consist of 2 samples, then you scale the error you get when validating it by factor 2/100 and so on. There probably are better options, but it's better than doing an unweighted average for sure." }, { "id": 524961, "postDate": "2019-04-29T21:15:06.577Z", "content": "The validation data isn't used for training but it has an implicit effect on the overall model's weights, since training only stops when the validation data can be optimally predicted. Hence why I advise against using small validation sets, as it limits the ability of models to generalise. Hyperparameter tuning won't cause leakage since it doesn't introduce any data, but it may well lead to overfitting. The main cause of leakage for our purposes is shuffling, since data from the same quakes are dispersed among the training and validation folds.
\n\nI can see the logic behind the earthquake scaling, but rather than manually play around with the errors, I would alter the weight of that model's predictions on the test set.
", "rawMarkdown": "The validation data isn't used for training but it has an implicit effect on the overall model's weights, since training only stops when the validation data can be optimally predicted. Hence why I advise against using small validation sets, as it limits the ability of models to generalise. Hyperparameter tuning won't cause leakage since it doesn't introduce any data, but it may well lead to overfitting. The main cause of leakage for our purposes is shuffling, since data from the same quakes are dispersed among the training and validation folds.\n\nI can see the logic behind the earthquake scaling, but rather than manually play around with the errors, I would alter the weight of that model's predictions on the test set.\n" }, { "id": 524985, "postDate": "2019-04-30T00:10:01.353Z", "content": "> The validation data isn't used for training but it has an implicit effect on the overall model's weights, since training only stops when the validation data can be optimally predicted.
\n\nThat's why i said i think it's better to not use early stopping with quakewise split without shuffling. If you don't use early stopping, the model won't stop training when the validation data is optimally predicted, because it doesn't see the validation data. It's just given a set of training data and parameters, trained the same way for each fold and evaluated with given parameters.
\n\n> Hyperparameter tuning won't cause leakage since it doesn't introduce any data.
\n\nIf you tune hyperparameters, you evaluate the models performance not only on the train, but mostly on the validation data. Since you can tweak training parameters to also fit your validation data better, you use information hidden in the validation data to tune your model, the result can be overfitting to the validation data. Thought that's called leakage here (leaking data from the val/test set into the training procedure) but maybe i misunderstood the term.
\n\nGuess the scaling part it wrote was bs. You can just save the predictions in a matrix and don't have to rescale anything to get equal weights for every sample in the cv-score. Sorry haven't slept that much :/
", "rawMarkdown": "> The validation data isn't used for training but it has an implicit effect on the overall model's weights, since training only stops when the validation data can be optimally predicted.\n\nThat's why i said i think it's better to not use early stopping with quakewise split without shuffling. If you don't use early stopping, the model won't stop training when the validation data is optimally predicted, because it doesn't see the validation data. It's just given a set of training data and parameters, trained the same way for each fold and evaluated with given parameters.\n\n> Hyperparameter tuning won't cause leakage since it doesn't introduce any data.\n\nIf you tune hyperparameters, you evaluate the models performance not only on the train, but mostly on the validation data. Since you can tweak training parameters to also fit your validation data better, you use information hidden in the validation data to tune your model, the result can be overfitting to the validation data. Thought that's called leakage here (leaking data from the val/test set into the training procedure) but maybe i misunderstood the term. \n\nGuess the scaling part it wrote was bs. You can just save the predictions in a matrix and don't have to rescale anything to get equal weights for every sample in the cv-score. Sorry haven't slept that much :/\n", "votes": 1 } ] }, { "id": 529992, "postDate": "2019-05-11T12:05:02.623Z", "content": "Update, lgb 14 features, LB 1.316
\n\nI am not reporting CV anymore because it is hard to compare. We know early stopping yields more optimistic CV than when not using it, Similarly shuffling yields better CV values. I still think this sharing has value as it shows one can get good result with a small number of features.
", "rawMarkdown": "Update, lgb 14 features, LB 1.316\n\nI am not reporting CV anymore because it is hard to compare. We know early stopping yields more optimistic CV than when not using it, Similarly shuffling yields better CV values. I still think this sharing has value as it shows one can get good result with a small number of features.\n", "votes": 8, "replies": [ { "id": 529993, "postDate": "2019-05-11T12:08:10.373Z", "content": "I agree, comparing CV scores within this thread is basically useless, it tells you roughly though what kind-of CV strategy people are doing.
", "rawMarkdown": "I agree, comparing CV scores within this thread is basically useless, it tells you roughly though what kind-of CV strategy people are doing.", "votes": 1 }, { "id": 530723, "postDate": "2019-05-13T14:16:40.403Z", "content": "@cpmpml Guess time to update :) Super Impressive!
", "rawMarkdown": "@cpmpml Guess time to update :) Super Impressive!", "votes": 1 }, { "id": 530778, "postDate": "2019-05-13T16:41:22.720Z", "content": "@cpmpml Are you planning to select your best public LBs for your submissions
", "rawMarkdown": "@cpmpml Are you planning to select your best public LBs for your submissions" }, { "id": 530782, "postDate": "2019-05-13T16:49:00.500Z", "content": "@returnofsputnik one of my final sub will most probably be the best public LB indeed.
", "rawMarkdown": "@returnofsputnik one of my final sub will most probably be the best public LB indeed.", "votes": 1 }, { "id": 530888, "postDate": "2019-05-13T22:22:13.437Z", "content": "Well, I just remember someone had said the public LB is useless, or meaningless...
", "rawMarkdown": "Well, I just remember someone had said the public LB is useless, or meaningless...", "votes": 2 }, { "id": 530894, "postDate": "2019-05-13T22:40:51.113Z", "content": "I remember that too :D \nBut to be fair, I do believe his best LB has also the best CV, so it's no brainer to choose that submission
", "rawMarkdown": "I remember that too :D \nBut to be fair, I do believe his best LB has also the best CV, so it's no brainer to choose that submission", "votes": 2 }, { "id": 530952, "postDate": "2019-05-14T03:14:19.387Z", "content": "thanks for sharing ;)
", "rawMarkdown": "thanks for sharing ;)" }, { "id": 530983, "postDate": "2019-05-14T04:52:11.853Z", "content": "\n\n\nI do believe his best LB has also the best CV,
\n
+1
\n\nThat's why i wrote 'most probably', as I can't be sure this will be true by end of competition. I will select the best CV submission for sure.
", "rawMarkdown": "> I do believe his best LB has also the best CV, \n\n+1\n\nThat's why i wrote 'most probably', as I can't be sure this will be true by end of competition. I will select the best CV submission for sure.", "votes": 1 }, { "id": 531062, "postDate": "2019-05-14T08:14:58.183Z", "content": "\n\n\nI just remember someone had said the public LB is useless, or meaningless…
\n
You have selective memories I'm afraid ;) , the same person also wrote this:
\n\n\n\n\nRight, saying it is totally useless is too strong. Looking at it as one fold is more accurate.
\n
See https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/91602#528378
", "rawMarkdown": ">I just remember someone had said the public LB is useless, or meaningless…\n\nYou have selective memories I'm afraid ;) , the same person also wrote this:\n\n> Right, saying it is totally useless is too strong. Looking at it as one fold is more accurate.\n\nSee https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/91602#528378" }, { "id": 531165, "postDate": "2019-05-14T12:21:46.967Z", "content": "Well, my bad!
", "rawMarkdown": "Well, my bad!", "votes": 1 } ] }, { "id": 524714, "postDate": "2019-04-29T12:15:01.080Z", "content": "LGBM 3Kfold Not Shuffled (I'm trying to keep it updated)\nCV: 2.084, LB: 1.526\nCV: 2.052, LB: 1.490\nCV: 2.014, LB: 1.433\nCV: 1.996, LB: 1.416 (6 features)
\n\nmy CV seems to match the LB direction quite well
", "rawMarkdown": "LGBM 3Kfold Not Shuffled (I'm trying to keep it updated)\nCV: 2.084, LB: 1.526\nCV: 2.052, LB: 1.490\nCV: 2.014, LB: 1.433\nCV: 1.996, LB: 1.416 (6 features)\n\nmy CV seems to match the LB direction quite well", "votes": 7 }, { "id": 493633, "postDate": "2019-03-18T22:34:24.800Z", "content": "LGB oof 1.99, LB 1.408. However I have another oof 1.94 but LB 1.460. I strongly believe the public LB (only around 300 samples) does not reflect the true standings, as scores of folds vary a lot.
", "rawMarkdown": "LGB oof 1.99, LB 1.408. However I have another oof 1.94 but LB 1.460. I strongly believe the public LB (only around 300 samples) does not reflect the true standings, as scores of folds vary a lot.", "votes": 5 }, { "id": 493622, "postDate": "2019-03-18T22:03:29.540Z", "content": "I'm using LightGBM too.
\n\nOOF: 2.042, LB: 1.468.
\n\nI found sensible feature engineering to be far more impactful than messing around with the training model.
", "rawMarkdown": "I'm using LightGBM too. \n\nOOF: 2.042, LB: 1.468. \n\nI found sensible feature engineering to be far more impactful than messing around with the training model.", "votes": 6 }, { "id": 522560, "postDate": "2019-04-24T16:39:42.793Z", "content": "LightGBM 23 features: CV 1.9793, LB 1.411
", "rawMarkdown": "LightGBM 23 features: CV 1.9793, LB 1.411", "votes": 3, "replies": [ { "id": 523004, "postDate": "2019-04-25T11:23:33.377Z", "content": "awesome！any idea abut identify valuable features？
", "rawMarkdown": "awesome！any idea abut identify valuable features？" } ] }, { "id": 522385, "postDate": "2019-04-24T11:10:18.343Z", "content": "CatBoost - oof 2.0176, LB 1.451
", "rawMarkdown": "CatBoost - oof 2.0176, LB 1.451", "votes": 3 }, { "id": 536036, "postDate": "2019-05-23T21:09:46.803Z", "content": "I tested single models only until now (no blending, no stacking). My best model has 1.278 on public LB.
\n\nA lot of algorithms do not work with my features (like XGB and LGBM). Maybe I have a problem setting them up.
", "rawMarkdown": "I tested single models only until now (no blending, no stacking). My best model has 1.278 on public LB.\n\nA lot of algorithms do not work with my features (like XGB and LGBM). Maybe I have a problem setting them up.", "votes": 4, "replies": [ { "id": 536057, "postDate": "2019-05-23T21:39:37.243Z", "content": "Very impressive! How many features are you working with?
", "rawMarkdown": "Very impressive! How many features are you working with?", "votes": 1 }, { "id": 536074, "postDate": "2019-05-23T22:26:11.263Z", "content": "8 features :))
", "rawMarkdown": "8 features :))", "votes": 5 }, { "id": 536319, "postDate": "2019-05-24T08:56:37.577Z", "content": "\n\n\nA lot of algorithms do not work with my features
\n
Interesting, I guess you won't disclose what model you use before competition end, but if you're inclined to do so now then you're most welcome!
", "rawMarkdown": "> A lot of algorithms do not work with my features \n\nInteresting, I guess you won't disclose what model you use before competition end, but if you're inclined to do so now then you're most welcome!\n\n", "votes": 1 }, { "id": 536437, "postDate": "2019-05-24T13:14:35.063Z", "content": "I use a tree based algorithms, but not XGB and LGBM implementation. But I don't believe my CV schema too much (there is some random noise).
", "rawMarkdown": "I use a tree based algorithms, but not XGB and LGBM implementation. But I don't believe my CV schema too much (there is some random noise).", "votes": 2 }, { "id": 537101, "postDate": "2019-05-26T07:17:37.907Z", "content": "Thanks, eager to see your solution after competition end.
", "rawMarkdown": "Thanks, eager to see your solution after competition end." } ] }, { "id": 493631, "postDate": "2019-03-18T22:22:38.450Z", "content": "catboost, oof: 1.92, lb: 1.462
", "rawMarkdown": "catboost, oof: 1.92, lb: 1.462", "votes": 3 }, { "id": 493629, "postDate": "2019-03-18T22:21:45.243Z", "content": "Just curious, does anyone use the over-sampling method? It’s giving me a very low CV but public LB is like 1.7, and I think the overlap between train and validation set is causing a problem. I have stopped using over-sampling since then.
", "rawMarkdown": "Just curious, does anyone use the over-sampling method? It’s giving me a very low CV but public LB is like 1.7, and I think the overlap between train and validation set is causing a problem. I have stopped using over-sampling since then.", "votes": 3, "replies": [ { "id": 493669, "postDate": "2019-03-19T00:07:41.317Z", "content": "I briefly tried it. It seemed to make a minor improvement in my CV score if I oversampled by a factor 2 (stride of 75000) but quickly deteriorates beyond that.
", "rawMarkdown": "I briefly tried it. It seemed to make a minor improvement in my CV score if I oversampled by a factor 2 (stride of 75000) but quickly deteriorates beyond that.", "votes": 3 }, { "id": 497004, "postDate": "2019-03-22T20:11:52.123Z", "content": "Tried that too 75000 oversampling is okay but anything beyond ruins the score
", "rawMarkdown": "Tried that too 75000 oversampling is okay but anything beyond ruins the score", "votes": 1 }, { "id": 503730, "postDate": "2019-03-30T13:05:35.247Z", "content": "do you find the solution to your problem? I have the same problem. very low CV but high LB
", "rawMarkdown": "do you find the solution to your problem? I have the same problem. very low CV but high LB", "votes": 1 }, { "id": 504082, "postDate": "2019-03-31T00:02:56.140Z", "content": "@wangzhaoxu, how low cv are u talking about?
", "rawMarkdown": "@wangzhaoxu, how low cv are u talking about?" }, { "id": 506069, "postDate": "2019-04-03T00:04:13.550Z", "content": "I got 0.3 for CV, and 1.7 for LB
", "rawMarkdown": "I got 0.3 for CV, and 1.7 for LB", "votes": 2 }, { "id": 506134, "postDate": "2019-04-03T02:48:50.820Z", "content": "0.3 is unrealistically low. You seem to overfit somehow. Are you shuffling your data before building the splits for cross-val? How do you optimize hyperparameters? (you can also overfit on the validation data)
", "rawMarkdown": "0.3 is unrealistically low. You seem to overfit somehow. Are you shuffling your data before building the splits for cross-val? How do you optimize hyperparameters? (you can also overfit on the validation data)", "votes": 2 }, { "id": 506142, "postDate": "2019-04-03T02:57:37.817Z", "content": "I am interested to know what model you used to train. Is it NN or RNN?
", "rawMarkdown": "I am interested to know what model you used to train. Is it NN or RNN?" }, { "id": 506483, "postDate": "2019-04-03T14:40:39.013Z", "content": "wangzhaoxu, surely either you have leakage or are massively overfitting a small training set. Did you engineer any new features that relied on the time_to_failure column by accident? Either that or my work is going far worse than I thought!
My current best single model is around 1.99 (oof score), but it only have a public score of 1.54, I’m also using lightgbm.
", "rawMarkdown": "My current best single model is around 1.99 (oof score), but it only have a public score of 1.54, I’m also using lightgbm.", "votes": 3 }, { "id": 504538, "postDate": "2019-03-31T19:42:42.263Z", "content": "cudnnlstm LB: 1.45
", "rawMarkdown": "cudnnlstm LB: 1.45", "votes": 4, "replies": [ { "id": 516853, "postDate": "2019-04-15T04:55:53.213Z", "content": "It's good to see this. May I ask if you do CV or you just fit 1 time with an appropriate iterations?
", "rawMarkdown": "It's good to see this. May I ask if you do CV or you just fit 1 time with an appropriate iterations?", "votes": 1 }, { "id": 517009, "postDate": "2019-04-15T11:08:14.803Z", "content": "Basically I used the model of:\n@mayer79\nhttps://www.kaggle.com/mayer79/rnn-starter-for-huge-time-series
\n\nI used cudnnLSTM instead of cudnngru, I played with different n_step and step_length, I got better n_step big results and few step-lengths. I also played with different features approx 50 was my best result.
\n\nOverlap data can help you with the overfitting.
", "rawMarkdown": "Basically I used the model of:\n@mayer79\nhttps://www.kaggle.com/mayer79/rnn-starter-for-huge-time-series\n\nI used cudnnLSTM instead of cudnngru, I played with different n_step and step_length, I got better n_step big results and few step-lengths. I also played with different features approx 50 was my best result.\n\nOverlap data can help you with the overfitting. ", "votes": 4 }, { "id": 517874, "postDate": "2019-04-16T16:19:46.200Z", "content": "I tried that kernel too, it seems that you need to make sure the train data and valid data are not overlapping if you want to make your CV trustworthy:
\n\ntrain_gen=generator(float_data,batch_size=batch_size, min_index=second_earthquake+1)\nvalid_gen=generator(float_data,batch_size=batch_size, max_index=second_earthquake)
It seems that using two layers of CuDNNGRU and dropout may help (my CV val_loss is around 1.98, LB at 1.48 with 54 features), but I still haven't found the optimal parameters for CuDNNLSTM. Also it's probably better to do a KFold validation.
", "rawMarkdown": "I tried that kernel too, it seems that you need to make sure the train data and valid data are not overlapping if you want to make your CV trustworthy:\n\n```train_gen=generator(float_data,batch_size=batch_size, min_index=second_earthquake+1)```\n```valid_gen=generator(float_data,batch_size=batch_size, max_index=second_earthquake)```\n\nIt seems that using two layers of CuDNNGRU and dropout may help (my CV val_loss is around 1.98, LB at 1.48 with 54 features), but I still haven't found the optimal parameters for CuDNNLSTM. Also it's probably better to do a KFold validation.", "votes": 3 }, { "id": 518065, "postDate": "2019-04-16T20:14:27.800Z", "content": "CV 1.48 is impressive!
", "rawMarkdown": "CV 1.48 is impressive!" }, { "id": 518275, "postDate": "2019-04-17T03:24:05.837Z", "content": "My bad, I meant LB around 1.48...
", "rawMarkdown": "My bad, I meant LB around 1.48...", "votes": 1 }, { "id": 518277, "postDate": "2019-04-17T03:28:44.510Z", "content": "That's impressive too!
", "rawMarkdown": "That's impressive too!" }, { "id": 518408, "postDate": "2019-04-17T07:03:24.350Z", "content": "Another trick, use loss \"logcosh\" with this your val_loss like very close to LB.
", "rawMarkdown": "Another trick, use loss \"logcosh\" with this your val_loss like very close to LB.", "votes": 1 }, { "id": 518658, "postDate": "2019-04-17T15:20:44.017Z", "content": "I doubt logcosh-loss helps, since the difference between lb and cv comes from the different data distributions in train and test set. I actually tried mae, logcosh and mse some time ago with the same approach and the lb result is pretty much the same.
\n\nThe comparable cv error with logcosh is coming from different (lower) error penalization (see https://cdn-images-1.medium.com/max/1600/1*BploIBOUrhbgdoB1BK_sOg.png for a graphical example).\nYou could get the same result by scaling down your mae-score with a factor<1 which doesn't really make sense.
", "rawMarkdown": "I doubt logcosh-loss helps, since the difference between lb and cv comes from the different data distributions in train and test set. I actually tried mae, logcosh and mse some time ago with the same approach and the lb result is pretty much the same.\n\nThe comparable cv error with logcosh is coming from different (lower) error penalization (see https://cdn-images-1.medium.com/max/1600/1*BploIBOUrhbgdoB1BK_sOg.png for a graphical example).\nYou could get the same result by scaling down your mae-score with a factor<1 which doesn't really make sense." } ] }, { "id": 494620, "postDate": "2019-03-20T03:44:17.373Z", "content": "Random Forest oof 1.89 and 1.512 public score.
", "rawMarkdown": "Random Forest oof 1.89 and 1.512 public score. ", "votes": 4 }, { "id": 493389, "postDate": "2019-03-18T16:01:45.460Z", "content": "Lgb, oof: 2.0644, lb: 1.532
\n\nEdit: oof 2.0695, lb 1.506. What's interesting, I managed to shrink the train/test gap to ~ 0.15 - e.g. on fold 5 I have 1.94887 train vs 2.08177 validation.
", "rawMarkdown": "Lgb, oof: 2.0644, lb: 1.532\n\nEdit: oof 2.0695, lb 1.506. What's interesting, I managed to shrink the train/test gap to ~ 0.15 - e.g. on fold 5 I have 1.94887 train vs 2.08177 validation.\n", "votes": 4 }, { "id": 529418, "postDate": "2019-05-09T20:54:49.073Z", "content": "Best LB score is LGB 18 features CV:2.00- LB:1.478.\nBest CV is 1.99 with 7 features but LB is 1.487.
", "rawMarkdown": "Best LB score is LGB 18 features CV:2.00- LB:1.478.\nBest CV is 1.99 with 7 features but LB is 1.487.", "votes": 1 }, { "id": 527530, "postDate": "2019-05-05T18:26:53.647Z", "content": "lgb with 70 features CV: 1.9721 LB: 1.407
", "rawMarkdown": "lgb with 70 features CV: 1.9721 LB: 1.407", "votes": 1 }, { "id": 527472, "postDate": "2019-05-05T16:03:39.273Z", "content": "LGB CV: 1.9723 LB: 1.414
", "rawMarkdown": "LGB CV: 1.9723 LB: 1.414", "votes": 1 }, { "id": 522627, "postDate": "2019-04-24T18:37:19.730Z", "content": "Hey there!\nPlease help me understand something.\nSituation: almost everyone here have CV about ~2 +- 0.03 - both top of leaderbord ( /summon @cpmpml) and not-so-top dudes like me. Today I played with one of my models that had a CV 2.03 and LB 1.51. I came up with some effed-up CV-scheme and after testing it I found out that it gives me CV 1.56 and LB 1.55. Features remain unchanged. And plot of y_train[:1500] and oof[:1500] looks like this:\n\nCan someone please explain to me what did I do, what's the nature of this CV-LB correlation and why plots are so weird? Am I overfitting like hell?\nThanks in advance!
How did you even manage to get the 14-ttf samples right? Seems like overfit, otherwise i'd like to know the feature that's able to do this :)
", "rawMarkdown": "How did you even manage to get the 14-ttf samples right? Seems like overfit, otherwise i'd like to know the feature that's able to do this :)", "votes": 1 }, { "id": 522871, "postDate": "2019-04-25T06:53:50.570Z", "content": "What is important is not the CV LB gap, which is very small in your case, but correlation. If you make a modification, say you add a feature, and your new CV score is improved, will LB score be improved as well?
\n\nLooking at the above, I bet you're using shuffling.
", "rawMarkdown": "What is important is not the CV LB gap, which is very small in your case, but correlation. If you make a modification, say you add a feature, and your new CV score is improved, will LB score be improved as well?\n\nLooking at the above, I bet you're using shuffling.", "votes": 4 }, { "id": 522889, "postDate": "2019-04-25T07:32:30.920Z", "content": "@svenhinderer, it's not really complicated. It isn't the feature and it was mentioned in discussions many times :)
\n\n@cpmpml, I didn't test this for correlation of CV-LB yet because I ran out of subs yesterday, so I'm gonna do it today if I'll have enough time.\nRegarding shuffling - yes, I'm using it. Does it have smth to do with this weird plot (i.e. oof jumps from some value to zero and back)?
", "rawMarkdown": "@svenhinderer, it's not really complicated. It isn't the feature and it was mentioned in discussions many times :)\n\n@cpmpml, I didn't test this for correlation of CV-LB yet because I ran out of subs yesterday, so I'm gonna do it today if I'll have enough time.\nRegarding shuffling - yes, I'm using it. Does it have smth to do with this weird plot (i.e. oof jumps from some value to zero and back)?", "votes": 1 }, { "id": 522909, "postDate": "2019-04-25T08:20:12.943Z", "content": "It is no problem to make your CV score close to public LB and even keep the correlation between them. Simple analysis of LB scores presented in a few comments proves that public dataset contains less high ttf segments than the train, so removing some high ttf segments from train you can make your train dataset more similar to public test than the raw one and obtain \"wanted\" results (low CV-LB distance, high correlation of CV/LB). The problem is that it is the better way to overfit, because the private dataset seems to contain significantly more high ttf segments than public one.\nThe way of improving CV described above is obviously wrong. I think most of Kagglers understand why and will not use it.\nTime for the punch line: \"Do not believe in your CV just because of you do not understand why it is close to LB score ;)\"
", "rawMarkdown": "It is no problem to make your CV score close to public LB and even keep the correlation between them. Simple analysis of LB scores presented in a few comments proves that public dataset contains less high ttf segments than the train, so removing some high ttf segments from train you can make your train dataset more similar to public test than the raw one and obtain \"wanted\" results (low CV-LB distance, high correlation of CV/LB). The problem is that it is the better way to overfit, because the private dataset seems to contain significantly more high ttf segments than public one.\nThe way of improving CV described above is obviously wrong. I think most of Kagglers understand why and will not use it.\nTime for the punch line: \"Do not believe in your CV just because of you do not understand why it is close to LB score ;)\"", "votes": 1 }, { "id": 522914, "postDate": "2019-04-25T08:32:05.740Z", "content": "\n\n\nbecause the private dataset seems to contain significantly more high ttf segments than public one.
\n
How do you know this? I asked in the post where a similar claim was made but got no answer.
", "rawMarkdown": "> because the private dataset seems to contain significantly more high ttf segments than public one.\n\nHow do you know this? I asked in the post where a similar claim was made but got no answer.", "votes": 2 }, { "id": 522919, "postDate": "2019-04-25T08:36:44.810Z", "content": "Because the LB is lower than cv too much ? and than assume public set are most consist of small ttf.
", "rawMarkdown": "Because the LB is lower than cv too much ? and than assume public set are most consist of small ttf.", "votes": 1 }, { "id": 522926, "postDate": "2019-04-25T08:40:49.393Z", "content": "@sionek , I understand why the CV method you described above is incorrect, but I didn't remove any ttf segments or anything else from train dataset, didn't change features, etc. Changes were only made in CV algorithm. Also, I'm 100% sure that my folds are not overlapping, and almost sure that there are no leaks.
", "rawMarkdown": "@sionek , I understand why the CV method you described above is incorrect, but I didn't remove any ttf segments or anything else from train dataset, didn't change features, etc. Changes were only made in CV algorithm. Also, I'm 100% sure that my folds are not overlapping, and almost sure that there are no leaks." }, { "id": 522928, "postDate": "2019-04-25T08:42:45.170Z", "content": "\n\n\nalmost sure that there are no leaks.
\n
Shuffling leaks time info as discussed elsewhere in this forum.
", "rawMarkdown": "> almost sure that there are no leaks.\n\nShuffling leaks time info as discussed elsewhere in this forum.", "votes": 3 }, { "id": 522935, "postDate": "2019-04-25T08:52:48.617Z", "content": "@cpmpml , thanks for info, I'll definitely try to find this topic and read it. 👍 \nBut still, I also used shuffling in earlier version of my model (which had CV 2.03 and LB 1.51). So I assume that shuffling is not the reason of CV dropping so significantly.
", "rawMarkdown": "@cpmpml , thanks for info, I'll definitely try to find this topic and read it. 👍 \nBut still, I also used shuffling in earlier version of my model (which had CV 2.03 and LB 1.51). So I assume that shuffling is not the reason of CV dropping so significantly." }, { "id": 522936, "postDate": "2019-04-25T08:55:03.207Z", "content": "https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/86530#
\n\n@CPMP, probably you did not get any answer, because it would be the repeating of the previous information. For me the aim, the method and the results and conclusion of the experiment are clear, however I did not check it, because I do not need that information.
", "rawMarkdown": "https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/86530#\n\n@CPMP, probably you did not get any answer, because it would be the repeating of the previous information. For me the aim, the method and the results and conclusion of the experiment are clear, however I did not check it, because I do not need that information." }, { "id": 522956, "postDate": "2019-04-25T09:31:48.357Z", "content": "Grzegorz, I disagree it was clear which is why I asked a question. It is not because one predicts a high ttf than the data has a high ttf. Either that experiment is fundamentally flawed, or I miss something. I interpret the lack of answer as a sign that I didn't miss much.
", "rawMarkdown": "Grzegorz, I disagree it was clear which is why I asked a question. It is not because one predicts a high ttf than the data has a high ttf. Either that experiment is fundamentally flawed, or I miss something. I interpret the lack of answer as a sign that I didn't miss much.\n" }, { "id": 522959, "postDate": "2019-04-25T09:34:30.627Z", "content": "\n\n\nI'll definitely try to find this topic and read it. 👍
\n
It has shuffling in the topic title: https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/89366#latest-521819
", "rawMarkdown": "> I'll definitely try to find this topic and read it. 👍 \n\nIt has shuffling in the topic title: https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/89366#latest-521819\n", "votes": 1 }, { "id": 523008, "postDate": "2019-04-25T11:32:24.490Z", "content": "Different CV strategies are not comparable to each other. But when you stick to a certain strategy and adjust other variables, you start to see the correlation with the LB scores. For example, my leave-quake-out CV gives a 2.2, and shuffled CV gives 1.88. Yet, both strategies give similar LB score.
", "rawMarkdown": "Different CV strategies are not comparable to each other. But when you stick to a certain strategy and adjust other variables, you start to see the correlation with the LB scores. For example, my leave-quake-out CV gives a 2.2, and shuffled CV gives 1.88. Yet, both strategies give similar LB score." }, { "id": 523168, "postDate": "2019-04-25T17:05:55.597Z", "content": "Wish i could say the same. If i shuffle and use early stopping, i also get <1.9 cv and <1.5 lb quite easily, but this approach is just begging to overfit imo. \nWith some form of quakewise split without early stopping, which i think is way more robust, lb never goes below 1,55, #feelsbad
", "rawMarkdown": "Wish i could say the same. If i shuffle and use early stopping, i also get <1.9 cv and <1.5 lb quite easily, but this approach is just begging to overfit imo. \nWith some form of quakewise split without early stopping, which i think is way more robust, lb never goes below 1,55, #feelsbad " }, { "id": 523235, "postDate": "2019-04-25T19:45:40.283Z", "content": "@amjad85 If you refit on whole data both results will always be the same on LB :P
", "rawMarkdown": "@amjad85 If you refit on whole data both results will always be the same on LB :P" }, { "id": 524095, "postDate": "2019-04-27T23:24:02.447Z", "content": "@philippsinger If you average the predictions from each fold and use early stopping, the results can be very different.
", "rawMarkdown": "@philippsinger If you average the predictions from each fold and use early stopping, the results can be very different." }, { "id": 524131, "postDate": "2019-04-28T02:46:05.870Z", "content": "@amjad85 \nDoes it even make sense to use early stopping with unshuffled data here? Doesn't that just give you a model that fits the validation data on each fold best, which is not optimal?
\n\n@stanislavblinov \nMaybe i'm blind. but can you elaborate on getting the high ttf-values right? I haven't seen anyone manage to do this, but maybe i missed it. The only way i can explain it is by overfitting to the training data.
", "rawMarkdown": "@amjad85 \nDoes it even make sense to use early stopping with unshuffled data here? Doesn't that just give you a model that fits the validation data on each fold best, which is not optimal?\n\n@stanislavblinov \nMaybe i'm blind. but can you elaborate on getting the high ttf-values right? I haven't seen anyone manage to do this, but maybe i missed it. The only way i can explain it is by overfitting to the training data." }, { "id": 524965, "postDate": "2019-04-29T21:28:33.653Z", "content": "@svenhinderer no it doesn't make sense IMO. I'm if favor of shuffling. The similar score I mentioned above with quake-based CV was done without early stopping.
", "rawMarkdown": "@svenhinderer no it doesn't make sense IMO. I'm if favor of shuffling. The similar score I mentioned above with quake-based CV was done without early stopping.", "votes": 1 } ] }, { "id": 520280, "postDate": "2019-04-20T16:06:25.163Z", "content": "@CPMP, did you get the answer for your question - what is LOGO? Also, you mentioned \"lgb 10 features cv 1.866 lb 1.416\" - I hope you didn't have any typos in this statement
", "rawMarkdown": "@CPMP, did you get the answer for your question - what is LOGO? Also, you mentioned \"lgb 10 features cv 1.866 lb 1.416\" - I hope you didn't have any typos in this statement", "votes": 1, "replies": [ { "id": 520349, "postDate": "2019-04-20T19:15:34.180Z", "content": "Yes and no typo ;)
", "rawMarkdown": "Yes and no typo ;)", "votes": 2 } ] }, { "id": 519870, "postDate": "2019-04-19T19:09:56.027Z", "content": "LGB CV 2.023 LB 1.533, CV 1.976 LB 1.436\n5 folds with shuffle
", "rawMarkdown": "LGB CV 2.023 LB 1.533, CV 1.976 LB 1.436\n5 folds with shuffle", "votes": 1 }, { "id": 501255, "postDate": "2019-03-27T05:25:09.623Z", "content": "LGB cv:2.03756 LB: 1.509
", "rawMarkdown": "LGB cv:2.03756 LB: 1.509", "votes": 1 }, { "id": 500532, "postDate": "2019-03-26T07:21:34.470Z", "content": "Edit: Lgbm: cv = 1.994, lb = 1.417\n(old) Lgbm: cv = 2.025, lb = 1.459
", "rawMarkdown": "Edit: Lgbm: cv = 1.994, lb = 1.417\n(old) Lgbm: cv = 2.025, lb = 1.459", "votes": 1 }, { "id": 500377, "postDate": "2019-03-25T23:17:39.067Z", "content": "RNN. MAE: 1.6896 L.B.: 1.533
", "rawMarkdown": "RNN. MAE: 1.6896 L.B.: 1.533", "votes": 1 }, { "id": 527468, "postDate": "2019-05-05T15:54:45.197Z", "content": "Update, lgb with 13 features, CV 1.772 LB 1.356
", "rawMarkdown": "Update, lgb with 13 features, CV 1.772 LB 1.356", "votes": 2, "replies": [ { "id": 527582, "postDate": "2019-05-05T21:45:42.227Z", "content": "LB within stdev? nice score btw :)
", "rawMarkdown": "LB within stdev? nice score btw :)", "votes": 2 }, { "id": 527605, "postDate": "2019-05-05T23:12:50.957Z", "content": "@cpmpml 13 high level features, right? I mean like pca or tsne features? Or just 13 features that you selected very well from many others statistical like those that has been shared? :)
", "rawMarkdown": "@cpmpml 13 high level features, right? I mean like pca or tsne features? Or just 13 features that you selected very well from many others statistical like those that has been shared? :)", "votes": 2 }, { "id": 527707, "postDate": "2019-05-06T06:17:52.130Z", "content": "3 I borrowed from public kernels and 10 I crafted. No tsne nor pca.
", "rawMarkdown": "3 I borrowed from public kernels and 10 I crafted. No tsne nor pca.", "votes": 3 }, { "id": 527823, "postDate": "2019-05-06T12:07:55.923Z", "content": "\n\n\nLB within stdev?
\n
LB is way lower than CV - CV std. It seems it is a general pattern, except for Kha Vo if I remember correctly.
", "rawMarkdown": "> LB within stdev? \n\nLB is way lower than CV - CV std. It seems it is a general pattern, except for Kha Vo if I remember correctly." }, { "id": 527846, "postDate": "2019-05-06T13:09:59.293Z", "content": "You remember incorrectly, I never said that.\nBy the way, I'm starting to realize what you are doing with your CV (how to reduce fold variance, how to add features). In my perspective, searching for more features from a small set of features (like what you're doing) is easier than most people do (remove bad features from a bunch), because it is much faster and more convenient to test. All we need is just a good CV strategy.
", "rawMarkdown": "You remember incorrectly, I never said that.\nBy the way, I'm starting to realize what you are doing with your CV (how to reduce fold variance, how to add features). In my perspective, searching for more features from a small set of features (like what you're doing) is easier than most people do (remove bad features from a bunch), because it is much faster and more convenient to test. All we need is just a good CV strategy.", "votes": 1 }, { "id": 527870, "postDate": "2019-05-06T13:45:23.237Z", "content": "\n\n\nYou remember incorrectly, I never said that.
\n
My bad, sorry.
", "rawMarkdown": "> You remember incorrectly, I never said that.\n\nMy bad, sorry.", "votes": 2 }, { "id": 529781, "postDate": "2019-05-10T18:31:15.607Z", "content": "@cpmpml When you mention your CV score, is that from your own CV method or a simple shuffled KFold score? Don't worry, I'm not going to ask what your CV strategy is!
", "rawMarkdown": "@cpmpml When you mention your CV score, is that from your own CV method or a simple shuffled KFold score? Don't worry, I'm not going to ask what your CV strategy is!", "votes": 1 }, { "id": 529933, "postDate": "2019-05-11T07:37:47.363Z", "content": "\n\n\nis that from your own CV method or a simple shuffled KFold score?
\n
It is from my CV setting, (what else?), a CV setting which could or could not be shuffled KFold.
", "rawMarkdown": "> is that from your own CV method or a simple shuffled KFold score? \n\nIt is from my CV setting, (what else?), a CV setting which could or could not be shuffled KFold.", "votes": 1 }, { "id": 529958, "postDate": "2019-05-11T09:41:56.443Z", "content": "Fair enough. I wasn't trying to pry insight into your methods, perhaps I had wrongly assumed that most people in this thread were using a shuffled-KFold when posting their CV scores for the sake of comparing baselines, since different methods can lead to varying results for the same data. Good luck with your work!
", "rawMarkdown": "Fair enough. I wasn't trying to pry insight into your methods, perhaps I had wrongly assumed that most people in this thread were using a shuffled-KFold when posting their CV scores for the sake of comparing baselines, since different methods can lead to varying results for the same data. Good luck with your work! ", "votes": 2 } ] }, { "id": 524343, "postDate": "2019-04-28T14:51:46.717Z", "content": "XGBoost 12 features: 5 fold CV=2.032, LB=1.461
", "rawMarkdown": "XGBoost 12 features: 5 fold CV=2.032, LB=1.461", "votes": 2, "replies": [ { "id": 524622, "postDate": "2019-04-29T07:50:37.487Z", "content": "Shuffled CV?
", "rawMarkdown": "Shuffled CV?" }, { "id": 524688, "postDate": "2019-04-29T11:18:49.530Z", "content": "Yep..shuffle=True
", "rawMarkdown": "Yep..shuffle=True" }, { "id": 524707, "postDate": "2019-04-29T11:53:31.040Z", "content": "Thx, and I guess LB prediction is average across folds?
", "rawMarkdown": "Thx, and I guess LB prediction is average across folds?" }, { "id": 524760, "postDate": "2019-04-29T13:26:09.427Z", "content": "Be careful with shuffled KFold. I can get CV below 1.8 using 27 features with a 5-fold, but the LB score is around 1.6. I think there is a lot of inherent randomness there. I gave up chasing the LB a while ago and just use a custom CV pipeline now.
", "rawMarkdown": "Be careful with shuffled KFold. I can get CV below 1.8 using 27 features with a 5-fold, but the LB score is around 1.6. I think there is a lot of inherent randomness there. I gave up chasing the LB a while ago and just use a custom CV pipeline now. ", "votes": 1 }, { "id": 524794, "postDate": "2019-04-29T14:33:11.870Z", "content": "Shuffled CV leaks the EQs, so it is problematic. But you also can find arguments for it. There is not a single CV setting that I am aware of that doesn't have some counter-arguments.
", "rawMarkdown": "Shuffled CV leaks the EQs, so it is problematic. But you also can find arguments for it. There is not a single CV setting that I am aware of that doesn't have some counter-arguments.", "votes": 2 }, { "id": 524804, "postDate": "2019-04-29T15:00:20.657Z", "content": "Psi, I use MAE of oof generated with 5 folds not the average MAE across folds.
", "rawMarkdown": "Psi, I use MAE of oof generated with 5 folds not the average MAE across folds." }, { "id": 524817, "postDate": "2019-04-29T15:16:09.597Z", "content": "BigIronSphere, How did you choose your 27 features? I personally like to add features one by one and check CV then LB. This usually takes a lot of time and also burns lots of submissions [it is hard to do it in this comp with 2 submission a day ;)] . For me the most important rule for keeping a feature is 1 ) if it lowers CV error then 2) checking if CV and LB are moving in the same direction.
", "rawMarkdown": "BigIronSphere, How did you choose your 27 features? I personally like to add features one by one and check CV then LB. This usually takes a lot of time and also burns lots of submissions [it is hard to do it in this comp with 2 submission a day ;)] . For me the most important rule for keeping a feature is 1 ) if it lowers CV error then 2) checking if CV and LB are moving in the same direction. ", "votes": 1 }, { "id": 524822, "postDate": "2019-04-29T15:28:17.167Z", "content": "The train data is very small and you can very quickly run a KFold inside a loop. I made an algorithm that randomly selects features from a pool and assigns them a normalised score based on their feature importance after a 5-fold unshuffled CV. After several thousand iterations I isolate the top quintile and see what the optimum number of features is in another loop (I'm simplifying a little). My method probably isn't fantastic since I'm pretty low on the LB, so perhaps it's not the best idea. I'm treating this competition more as a learning experience where I can prepare quick pipelines for future use.
", "rawMarkdown": "The train data is very small and you can very quickly run a KFold inside a loop. I made an algorithm that randomly selects features from a pool and assigns them a normalised score based on their feature importance after a 5-fold unshuffled CV. After several thousand iterations I isolate the top quintile and see what the optimum number of features is in another loop (I'm simplifying a little). My method probably isn't fantastic since I'm pretty low on the LB, so perhaps it's not the best idea. I'm treating this competition more as a learning experience where I can prepare quick pipelines for future use. ", "votes": 3 }, { "id": 524866, "postDate": "2019-04-29T17:11:09.667Z", "content": "@kainsama My question was rather how you produce final test prediction.
", "rawMarkdown": "@kainsama My question was rather how you produce final test prediction." }, { "id": 524878, "postDate": "2019-04-29T17:28:04.460Z", "content": "Sorry I misunderstood your question; and yes I average over folds to create the final test prediction.
", "rawMarkdown": "Sorry I misunderstood your question; and yes I average over folds to create the final test prediction." } ] }, { "id": 495428, "postDate": "2019-03-21T04:33:57.333Z", "content": "CNN. MAE: 2.084 LB: 1.616
", "rawMarkdown": "CNN. MAE: 2.084 LB: 1.616", "votes": 2, "replies": [ { "id": 495631, "postDate": "2019-03-21T10:38:39.160Z", "content": "Woah, I also had a CNN with exactly this score both local and LB :D
", "rawMarkdown": "Woah, I also had a CNN with exactly this score both local and LB :D", "votes": 1 }, { "id": 496190, "postDate": "2019-03-22T00:44:51.890Z", "content": "Crazy! I'm doing 2D convolutions over spectrograms, but I'm beginning to think it's not the best approach.
", "rawMarkdown": "Crazy! I'm doing 2D convolutions over spectrograms, but I'm beginning to think it's not the best approach.", "votes": 3 }, { "id": 496859, "postDate": "2019-03-22T17:12:59.133Z", "content": "Getting that score with your approach is quite impressive imo. I've tried the same but gave up pretty fast because it didn't seem to work well enough.\nIf you're interested, theres a kernel about it (not by me) called \"Spectrogram+Convolution=?\"
", "rawMarkdown": "Getting that score with your approach is quite impressive imo. I've tried the same but gave up pretty fast because it didn't seem to work well enough.\nIf you're interested, theres a kernel about it (not by me) called \"Spectrogram+Convolution=?\"", "votes": 1 } ] }, { "id": 493422, "postDate": "2019-03-18T16:20:18.323Z", "content": "Multiple Input NN ~1.5-1.6lb, what is oof score?
", "rawMarkdown": "Multiple Input NN ~1.5-1.6lb, what is oof score?", "votes": 2, "replies": [ { "id": 493452, "postDate": "2019-03-18T16:48:09.503Z", "content": "oof is out-of-fold cross validation
", "rawMarkdown": "oof is out-of-fold cross validation", "votes": 2 }, { "id": 493464, "postDate": "2019-03-18T17:00:03.603Z", "content": "Okay thanks, my oof mse is around 8 then, haven't tried mae in a while for training.
", "rawMarkdown": "Okay thanks, my oof mse is around 8 then, haven't tried mae in a while for training.", "votes": 1 } ] }, { "id": 502651, "postDate": "2019-03-28T21:49:50.647Z", "content": "CatBoost - 2.0273 CV, 1.475 LB
\n\nI'm noticing a couple of trends. Firstly, CatBoost seems to often outperform LGB which is strange since it was specifically designed for categorical data. I need to reread the CB docs and get a better understanding of its oblivious trees methods. Secondly, the relationship between CV and LB isn't very strong... I think we should trust CV much more. Bad news for me since I can't get oof below 2.0!
", "rawMarkdown": "CatBoost - 2.0273 CV, 1.475 LB\n\nI'm noticing a couple of trends. Firstly, CatBoost seems to often outperform LGB which is strange since it was specifically designed for categorical data. I need to reread the CB docs and get a better understanding of its oblivious trees methods. Secondly, the relationship between CV and LB isn't very strong... I think we should trust CV much more. Bad news for me since I can't get oof below 2.0!", "replies": [ { "id": 502662, "postDate": "2019-03-28T22:16:16.123Z", "content": "For some reason I always get better CV (best CV is around 1.98) with LGB as compared to either XGB or CatBoost (couldn't break 2), and XGB/CatBoost seem to run very slowly on my laptop.\nI agree with you on trusting CV in this competition: I've noticed that all my \"CV<2\" submissions got LB >1.5.
", "rawMarkdown": "For some reason I always get better CV (best CV is around 1.98) with LGB as compared to either XGB or CatBoost (couldn't break 2), and XGB/CatBoost seem to run very slowly on my laptop.\nI agree with you on trusting CV in this competition: I've noticed that all my \"CV<2\" submissions got LB >1.5.", "votes": 1 }, { "id": 502671, "postDate": "2019-03-28T22:50:10.010Z", "content": "CatBoost is extremely slow if you don't use a GPU, but this often comes at a cost of accuracy with smaller datasets. With Kaggle servers it's not really a problem but it still takes ~2.5 hours with CPU alone for a 5-fold. I think the public test dataset is misleading and has below-average TTF values which are easier to predict. I suspect this competition will hinge on extraction of features that indicate a TTF > 10, although they will be difficult to isolate. At a certain point it will likely become impossible given the stochastic physical processes involved.
\n\nInterestingly, my LGB gives better results for individual folds but not on total OOF. A lot of this may be random though...
", "rawMarkdown": "CatBoost is extremely slow if you don't use a GPU, but this often comes at a cost of accuracy with smaller datasets. With Kaggle servers it's not really a problem but it still takes ~2.5 hours with CPU alone for a 5-fold. I think the public test dataset is misleading and has below-average TTF values which are easier to predict. I suspect this competition will hinge on extraction of features that indicate a TTF > 10, although they will be difficult to isolate. At a certain point it will likely become impossible given the stochastic physical processes involved. \n\nInterestingly, my LGB gives better results for individual folds but not on total OOF. A lot of this may be random though...", "votes": 1 }, { "id": 502766, "postDate": "2019-03-29T03:42:00.520Z", "content": "I see, looks like stacking from multiple models is the way to go?
\n\nThe thing is that it's very difficult to predict if a segment TTF is in the high value or near 0, especially if the segment is close to the earthquake outbreak (TTF jumps from near 0 to high value), so LGB would take the middle ground, which is why we never see extremely low or high value from oof predictions.
\n\nOne thing I would suggest is to separate TTF jumps evenly in each fold, so that oof score becomes stable in each fold.
", "rawMarkdown": "I see, looks like stacking from multiple models is the way to go?\n\nThe thing is that it's very difficult to predict if a segment TTF is in the high value or near 0, especially if the segment is close to the earthquake outbreak (TTF jumps from near 0 to high value), so LGB would take the middle ground, which is why we never see extremely low or high value from oof predictions. \n\nOne thing I would suggest is to separate TTF jumps evenly in each fold, so that oof score becomes stable in each fold." }, { "id": 504089, "postDate": "2019-03-31T00:35:16.437Z", "content": "You could just skip the segments where earthquakes happen. It's still better to have 16 less samples than to have 16 completely wrong ones.
", "rawMarkdown": "You could just skip the segments where earthquakes happen. It's still better to have 16 less samples than to have 16 completely wrong ones.", "votes": 3 } ] }, { "id": 504083, "postDate": "2019-03-31T00:03:48.143Z", "rawMarkdown": "", "isDeleted": true }, { "id": 497085, "postDate": "2019-03-22T23:45:25.363Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ], "comments": [ { "id": 500613, "author_name": "", "author_url": "", "post_date": "2019-03-26T09:51:33.933000", "content": "CatBoost - 1.87 LB: 1.441 on FFT features
", "votes": 11, "replies": [ { "id": 516841, "author_name": "dinosaur", "author_url": "", "post_date": "2019-04-15T04:26:32.580000", "content": "use FFT feature only ?
", "votes": 0, "replies": [] } ] }, { "id": 519221, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-18T14:57:31.013000", "content": "Starting with lgb and 7 features: cv 1.901, lb 1.444
", "votes": 9, "replies": [ { "id": 519249, "author_name": "RNA", "author_url": "", "post_date": "2019-04-18T15:57:19.120000", "content": "I have 300 features and my lgb reaches 1.475. Clearly I'm taking the wrong approach!
", "votes": 1, "replies": [] }, { "id": 519259, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-18T16:19:29.660000", "content": "Well, I added 3 features and LB went to 2.5 ... I may have goofed, but I may just discover the challenge we all face here.
", "votes": 3, "replies": [] }, { "id": 519265, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-18T16:27:40.197000", "content": "Well, I did goof, let's see tomorrow if CV and LB evolve in the same direction.
", "votes": 1, "replies": [] }, { "id": 519276, "author_name": "RNA", "author_url": "", "post_date": "2019-04-18T16:52:17.220000", "content": "Are you using KFold or LOGO? I've found KFold gets better LB score but LOGO (and other quake-wise splits) get a much, much lower CV, around 1.5. As for my features, I'm pretty certain many of them are redundant due to multicolinearity. I'm otherwise pleased with them since the correlations to the target are very strong.
", "votes": 1, "replies": [] }, { "id": 519285, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-18T17:03:38.027000", "content": "What is LOGO?
", "votes": 0, "replies": [] }, { "id": 519288, "author_name": "FP", "author_url": "", "post_date": "2019-04-18T17:11:04.170000", "content": "I guess it's leave-one-group-out. So meaning CV by each earthquake group.
", "votes": 0, "replies": [] }, { "id": 519294, "author_name": "RNA", "author_url": "", "post_date": "2019-04-18T17:21:26.360000", "content": "Yes, Leave One Group Out. You'll find that earthquake periods 8 and 15 produce a notably high MAE with this approach, while many of the other quakes get an MAE of around 1 using a simple model. I got a poor LB with this approach but I'm not discounting it entirely.
", "votes": 0, "replies": [] }, { "id": 519325, "author_name": "Sven Hinderer", "author_url": "", "post_date": "2019-04-18T18:30:47.287000", "content": "How do you make your final predictions with your models? Retrain on the whole set? Average all model predictions for the k-folds?\nI think the much lower cv of your LOGO models could be due to overfitting on the relatively small validation set, which is much larger if you use e.g. 3-fold or 5-fold cross-validation. Could also be a really good model, but comparing it with other cv-scores and accounting the lower lb score i guess it's unlikely. Maybe grouping 2-3 earthquakes together into one validation split could improve this.
", "votes": 1, "replies": [] }, { "id": 519331, "author_name": "RNA", "author_url": "", "post_date": "2019-04-18T18:45:50.877000", "content": "Definitely. However, since we're only permitted 2 submissions per day it'd take more than a week to evaluate each model individually! Taking the mean prediction from LOGO gave me much worse results than a simple KFold.
", "votes": 0, "replies": [] }, { "id": 519372, "author_name": "Sven Hinderer", "author_url": "", "post_date": "2019-04-18T20:41:59.140000", "content": "I didn't mean to eval each single model. I'm sure it's possible to \"cheat\" a better lb score by manipulating the train set to have a comparable mean ttf as the test set or simply taking the folds for training which seem to fit the test set distribution best, but that's another topic. I just wanted to say that you'd probably get a more realistic score if you didn't \"leave-one-group(=earthquake)-out\" but \"leave-multiple-groups-out\". Hope it's a little bit clearer now.
", "votes": 0, "replies": [] }, { "id": 519414, "author_name": "RNA", "author_url": "", "post_date": "2019-04-18T22:49:46.920000", "content": "That's what I've been working on, disappointingly I can get anything to work better than simple KFolds.
", "votes": 0, "replies": [] }, { "id": 519477, "author_name": "Timmmmmms", "author_url": "", "post_date": "2019-04-19T03:20:35.463000", "content": "Only 7 features?
", "votes": 0, "replies": [] }, { "id": 519530, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-19T06:31:34.183000", "content": "Yes only 7
", "votes": 2, "replies": [] }, { "id": 519539, "author_name": "Marcus Lin", "author_url": "", "post_date": "2019-04-19T07:02:00.433000", "content": "sounds like there are magic features here.
", "votes": 0, "replies": [] }, { "id": 519544, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-19T07:12:41.103000", "content": "No magic no leak ;)
\n\nupdate: lgb 10 features cv 1.866 lb 1.416
", "votes": 8, "replies": [] }, { "id": 519550, "author_name": "Marcus Lin", "author_url": "", "post_date": "2019-04-19T07:18:22.803000", "content": "cool
", "votes": 1, "replies": [] }, { "id": 519564, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-04-19T07:41:07.427000", "content": "LGB 200+ features, LB 1.48, CV 2.01...\n@cpmpml, how do you do get 1.416 with 10 features?! Is it some Uncle-ish dark magic?
", "votes": 1, "replies": [] }, { "id": 519585, "author_name": "Psi", "author_url": "", "post_date": "2019-04-19T08:32:28.763000", "content": "Looks good, what kind of CV setup are you using @cpmpml if you mind telling.
", "votes": 2, "replies": [] }, { "id": 519655, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-19T11:48:49.837000", "content": "Psi,
\n\nI don't plan to share much in this competition after the bad experience in the Santander competition. I do remember you were on my side there, thanks for that, you're not the reason why I don't want to share anymore.
\n\nAnyway, I already shared more than what you shared during Santander ;)
", "votes": 5, "replies": [] }, { "id": 519670, "author_name": "redstr", "author_url": "", "post_date": "2019-04-19T12:25:59.127000", "content": "What happened in the Santander competition? Couldn't find any discussions of bad experience, everybody is thanking each other. Did it get deleted?
", "votes": 0, "replies": [] }, { "id": 519683, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-19T13:13:14.937000", "content": "Don't bother, if you weren't exposed to it, then don't look for it. In a nutshell, I, and few others like Psi's team, were accused of purposely misleading participants because we used the word 'magic' to describe what we were doing. People argued it was rather a leak exploit and that we should have said it. Hence my current team name ;)
", "votes": 2, "replies": [] }, { "id": 519685, "author_name": "Psi", "author_url": "", "post_date": "2019-04-19T13:16:47.923000", "content": "Fair enough :)
", "votes": 1, "replies": [] }, { "id": 519697, "author_name": "FP", "author_url": "", "post_date": "2019-04-19T13:36:37.970000", "content": "@cpmpml, I am sorry for your experience. But I would rather want you and others to continue with spreading the \"magic\" because 1) this competition is much harder than Santander and 2) it is so much difficult to learn feature engineering just based on online articles or public kernels. I learnt from Santander that feature engineering through EDA is really important.
\n\nAgain, it is your choice, and I'll respect whatever decision you make.
", "votes": 0, "replies": [] }, { "id": 519702, "author_name": "redstr", "author_url": "", "post_date": "2019-04-19T13:55:06.480000", "content": "To offer a counter opinion, I'd be careful about what you share. In case of any non-trivial findings, it's best to keep quiet. I find it selfish when people share major things for vanity's sake or for points. It spoils the discovery for others. Spoonfeeding is also condescending.\nIn addition to that, if you talk on the forum in language that is understood only by a part of the public, e.g. using the word \"magic\" to refer to something that you and some others have found, and discussing it without revealing it, it may be construed as \"private sharing outside teams\" and get your team banned. Why risk that, and for what?
", "votes": 1, "replies": [] }, { "id": 519770, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-19T16:06:31.430000", "content": "Sharing has its pros and conses, I'm not going to argue for one or the other here, but I want to react to one thing, namely:
\n\n\n\n\nif you talk on the forum in language that is understood only by a part of the public, e.g. using the word \"magic\" to refer to something that you and some others have found, and discussing it without revealing it, it may be construed as \"private sharing outside teams\" and get your team banned.
\n
I don't get how you can conclude using the word 'magic' is private sharing. By definition, something shared in the competition forum is not private sharing. Please explain.
", "votes": 1, "replies": [] }, { "id": 519782, "author_name": "redstr", "author_url": "", "post_date": "2019-04-19T16:17:44.107000", "content": "Sure. You can share something only with people with whom you already share a secret, like the meaning of the word \"magic\". It's a basic form of encryption. For example, you can say \"I made a new feature by multiplying the first and the second magic features\". Only the people who know the magic features can understand that.
", "votes": 0, "replies": [] }, { "id": 519798, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-19T16:47:42.960000", "content": "Good, you then agree that there is no private sharing if all communication goes via the forum.
", "votes": 2, "replies": [] }, { "id": 519949, "author_name": "RNA", "author_url": "", "post_date": "2019-04-19T21:15:52.743000", "content": "I can understand your reluctance to share given the Santander experience. But I don't think it should put you off sharing in general. Santander was (in terms of the data itself) a relatively simple competition, and likely full of frustrated beginners hoping to stumble on a single clever trick to make them progress, growing bitter at their lack of success. This is a far more interesting challenge and I've found the discussions here to be better-natured as well. I'm currently writing a one-function, plug-in-and-play Hyperopt kernel for LGBM, XGB and CatBoost. It's not in my best interest to share it, but I try to learn from Chris Deotte's example - sharing is what makes Kaggle such a great place!
", "votes": 3, "replies": [] }, { "id": 519953, "author_name": "FP", "author_url": "", "post_date": "2019-04-19T21:20:22.217000", "content": "Also, you got feedback for sharing your stuff - good, bad or ugly.
", "votes": 0, "replies": [] }, { "id": 520097, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-20T06:46:28.507000", "content": "There are 36 teams with better things to share than me ;) They are mostly silent. Why don't you bug them a bit ? ;)
", "votes": 5, "replies": [] }, { "id": 520232, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-20T13:36:10.997000", "content": "@bigironsphere I agree with you. I am discussing a bit, and I'll keep doing it as long as it is around understanding the problem better. Sharing generic code as you plan to do is fine too, and I may do it here again if I have something to share. What I don't plan to share is how I approach the problem. I see it benefit to people who end up in front of me in the LB in every competition I enter. This time I want to see how it goes without this sharing.
\n\n@pukkinming I did get some thanks indeed, which is why I share in general.
\n\nAnd to those who down vote me, the more down vote the less sharing from me. Keep going.
\n\nAnyway, here is a bit of sharing as I can't help helping ;)
\n\n\n\n\nHow do you make your final predictions with your models? Retrain on the whole set? Average all model predictions for the k-folds?
\n
I do the latter. I find it to be better in general, but not always. ideally one should try both and pick what's best.
", "votes": 4, "replies": [] }, { "id": 520268, "author_name": "FP", "author_url": "", "post_date": "2019-04-20T15:52:50.713000", "content": "@cpmpml, let me thank you again for your sharing.
", "votes": 1, "replies": [] }, { "id": 520347, "author_name": "RNA", "author_url": "", "post_date": "2019-04-20T19:02:32.597000", "content": "Well, I'll share something with you since I've learned so much from you in the past.
\n\nWhen I realised that LOGO was leading to massive overfitting, I tried a custom validation strategy. There are three quakes that exhibit a 'mini-quake' halfway through that tend to confuse the models and make them predict a lower TTF than usual, hence why unshuffled KFold leads to such high variance among CV scores. These are the TTF periods 3, 8 and 15. To account for them, I would sample one of these quakes at random, along with 5 others without a mini-quake, and use this as the validation set. This process was repeated many times so that the complicated 'mini-quakes' segments were always present in a 2:1 ratio in the training and validation sets in every fold.
\n\nUltimately this lead to a good CV but a worse LB score, leading me to believe that the current LB test set includes simpler earthquake cases without the presence of 'mini-quake' segments. If you're obtaining good scores from a small number of features, it's likely that your model isn't overfitting based on these three complex cases - I've personally found that decreasing the number of features makes my models less sensitive to the mini-quakes when viewing their predictions graphically. Unfortunately, as ever, we have no way of knowing if this performance boost will be reflected in the private data.
", "votes": 4, "replies": [] }, { "id": 523047, "author_name": "dinosaur", "author_url": "", "post_date": "2019-04-25T13:14:31.503000", "content": "did you choose your 10 features with top feature importance from huge statistic features ?
", "votes": 0, "replies": [] }, { "id": 524198, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-28T07:42:27.573000", "content": "I crafted the first 10 features without reading any of the forum or kernels. Then I selected the subset that appeared to work well on my cv setting. I submitted them in my first sub. The other 3 are cherry picked from public kernels. They improved my cv hence I submitted with them. Since then I tried lots of public kernel features and could find any that improve my cv. I didn’t submit any of them. That’s a long way to say I select feature by training models and looking at cv score. I never understood the use of statistics for feature selection. These statistics are only useful when you use generalized linear models IMHO.
", "votes": 3, "replies": [] }, { "id": 524836, "author_name": "DavidS", "author_url": "", "post_date": "2019-04-29T16:08:02.503000", "content": "@bigironsphere Maybe I missed something - why do you think LOGO \"was leading to massive overfitting\"?
", "votes": 0, "replies": [] }, { "id": 524844, "author_name": "RNA", "author_url": "", "post_date": "2019-04-29T16:12:03.677000", "content": "Perhaps overfitting was the wrong word, but each individual 'fold' in LOGO is being optimised towards a very specific validation set. I prefer using grouped quakes for validation, but maybe LOGO is better than I thought. We'll have to see.
", "votes": 0, "replies": [] }, { "id": 524846, "author_name": "Sven Hinderer", "author_url": "", "post_date": "2019-04-29T16:18:39.967000", "content": "@bigironsphere That should only happen if you early stop, right? If you just let your models converge and average over all fold errors (maybe scaled with the quake length), i think it's pretty safe.
\n\n@cpmpml Any thoughts on sequential feature selection? I was too lazy to do it all by hand and apply sffs after calculating f-scores of features. Seems to work quite well but i'm not sure if selection by hand would be better.
", "votes": 1, "replies": [] }, { "id": 524868, "author_name": "RNA", "author_url": "", "post_date": "2019-04-29T17:16:13.173000", "content": "Convergence would only increase the specificity of the model for that fold, early stopping would mitigate it. I'm not saying LOGO isn't safe, just that I don't think the best approach is to average a bunch of highly-specified models - I would prefer a more varied validation set. What do you mean by scaling fold errors by quake length?
", "votes": 0, "replies": [] }, { "id": 524934, "author_name": "Sven Hinderer", "author_url": "", "post_date": "2019-04-29T19:37:30.390000", "content": "@bigironsphere \nHow would it increase the specificity of the model? Maybe i'm wrong, but i'd say it's the opposite since you don't use any information from the validation data while training as opposed to training with early stopping. You'll still have some leakage by tuning hyperparameters but it guess that's inevitable.
\n\nFor the scaling part: I just meant that you take the number of samples per quake into account when calculating the cv score. Let's say you have 100 samples in total, do an earthquake-split and your first earthquake only consist of 2 samples, then you scale the error you get when validating it by factor 2/100 and so on. There probably are better options, but it's better than doing an unweighted average for sure.
", "votes": 0, "replies": [] }, { "id": 524961, "author_name": "RNA", "author_url": "", "post_date": "2019-04-29T21:15:06.577000", "content": "The validation data isn't used for training but it has an implicit effect on the overall model's weights, since training only stops when the validation data can be optimally predicted. Hence why I advise against using small validation sets, as it limits the ability of models to generalise. Hyperparameter tuning won't cause leakage since it doesn't introduce any data, but it may well lead to overfitting. The main cause of leakage for our purposes is shuffling, since data from the same quakes are dispersed among the training and validation folds.
\n\nI can see the logic behind the earthquake scaling, but rather than manually play around with the errors, I would alter the weight of that model's predictions on the test set.
", "votes": 0, "replies": [] }, { "id": 524985, "author_name": "Sven Hinderer", "author_url": "", "post_date": "2019-04-30T00:10:01.353000", "content": "> The validation data isn't used for training but it has an implicit effect on the overall model's weights, since training only stops when the validation data can be optimally predicted.
\n\nThat's why i said i think it's better to not use early stopping with quakewise split without shuffling. If you don't use early stopping, the model won't stop training when the validation data is optimally predicted, because it doesn't see the validation data. It's just given a set of training data and parameters, trained the same way for each fold and evaluated with given parameters.
\n\n> Hyperparameter tuning won't cause leakage since it doesn't introduce any data.
\n\nIf you tune hyperparameters, you evaluate the models performance not only on the train, but mostly on the validation data. Since you can tweak training parameters to also fit your validation data better, you use information hidden in the validation data to tune your model, the result can be overfitting to the validation data. Thought that's called leakage here (leaking data from the val/test set into the training procedure) but maybe i misunderstood the term.
\n\nGuess the scaling part it wrote was bs. You can just save the predictions in a matrix and don't have to rescale anything to get equal weights for every sample in the cv-score. Sorry haven't slept that much :/
", "votes": 1, "replies": [] } ] }, { "id": 529992, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-11T12:05:02.623000", "content": "Update, lgb 14 features, LB 1.316
\n\nI am not reporting CV anymore because it is hard to compare. We know early stopping yields more optimistic CV than when not using it, Similarly shuffling yields better CV values. I still think this sharing has value as it shows one can get good result with a small number of features.
", "votes": 8, "replies": [ { "id": 529993, "author_name": "Psi", "author_url": "", "post_date": "2019-05-11T12:08:10.373000", "content": "I agree, comparing CV scores within this thread is basically useless, it tells you roughly though what kind-of CV strategy people are doing.
", "votes": 1, "replies": [] }, { "id": 530723, "author_name": "student", "author_url": "", "post_date": "2019-05-13T14:16:40.403000", "content": "@cpmpml Guess time to update :) Super Impressive!
", "votes": 1, "replies": [] }, { "id": 530778, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2019-05-13T16:41:22.720000", "content": "@cpmpml Are you planning to select your best public LBs for your submissions
", "votes": 0, "replies": [] }, { "id": 530782, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-13T16:49:00.500000", "content": "@returnofsputnik one of my final sub will most probably be the best public LB indeed.
", "votes": 1, "replies": [] }, { "id": 530888, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-05-13T22:22:13.437000", "content": "Well, I just remember someone had said the public LB is useless, or meaningless...
", "votes": 2, "replies": [] }, { "id": 530894, "author_name": "Amjad", "author_url": "", "post_date": "2019-05-13T22:40:51.113000", "content": "I remember that too :D \nBut to be fair, I do believe his best LB has also the best CV, so it's no brainer to choose that submission
", "votes": 2, "replies": [] }, { "id": 530952, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2019-05-14T03:14:19.387000", "content": "thanks for sharing ;)
", "votes": 0, "replies": [] }, { "id": 530983, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-14T04:52:11.853000", "content": "\n\n\nI do believe his best LB has also the best CV,
\n
+1
\n\nThat's why i wrote 'most probably', as I can't be sure this will be true by end of competition. I will select the best CV submission for sure.
", "votes": 1, "replies": [] }, { "id": 531062, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-14T08:14:58.183000", "content": "\n\n\nI just remember someone had said the public LB is useless, or meaningless…
\n
You have selective memories I'm afraid ;) , the same person also wrote this:
\n\n\n\n\nRight, saying it is totally useless is too strong. Looking at it as one fold is more accurate.
\n
See https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/91602#528378
", "votes": 0, "replies": [] }, { "id": 531165, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-05-14T12:21:46.967000", "content": "Well, my bad!
", "votes": 1, "replies": [] } ] }, { "id": 524714, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2019-04-29T12:15:01.080000", "content": "LGBM 3Kfold Not Shuffled (I'm trying to keep it updated)\nCV: 2.084, LB: 1.526\nCV: 2.052, LB: 1.490\nCV: 2.014, LB: 1.433\nCV: 1.996, LB: 1.416 (6 features)
\n\nmy CV seems to match the LB direction quite well
", "votes": 7, "replies": [] }, { "id": 493633, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-03-18T22:34:24.800000", "content": "LGB oof 1.99, LB 1.408. However I have another oof 1.94 but LB 1.460. I strongly believe the public LB (only around 300 samples) does not reflect the true standings, as scores of folds vary a lot.
", "votes": 5, "replies": [] }, { "id": 493622, "author_name": "varun", "author_url": "", "post_date": "2019-03-18T22:03:29.540000", "content": "I'm using LightGBM too.
\n\nOOF: 2.042, LB: 1.468.
\n\nI found sensible feature engineering to be far more impactful than messing around with the training model.
", "votes": 6, "replies": [] }, { "id": 522560, "author_name": "Vicens Gaitan", "author_url": "", "post_date": "2019-04-24T16:39:42.793000", "content": "LightGBM 23 features: CV 1.9793, LB 1.411
", "votes": 3, "replies": [ { "id": 523004, "author_name": "dinosaur", "author_url": "", "post_date": "2019-04-25T11:23:33.377000", "content": "awesome！any idea abut identify valuable features？
", "votes": 0, "replies": [] } ] }, { "id": 522385, "author_name": "rafzy", "author_url": "", "post_date": "2019-04-24T11:10:18.343000", "content": "CatBoost - oof 2.0176, LB 1.451
", "votes": 3, "replies": [] }, { "id": 536036, "author_name": "Dre@mer", "author_url": "", "post_date": "2019-05-23T21:09:46.803000", "content": "I tested single models only until now (no blending, no stacking). My best model has 1.278 on public LB.
\n\nA lot of algorithms do not work with my features (like XGB and LGBM). Maybe I have a problem setting them up.
", "votes": 4, "replies": [ { "id": 536057, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-05-23T21:39:37.243000", "content": "Very impressive! How many features are you working with?
", "votes": 1, "replies": [] }, { "id": 536074, "author_name": "Dre@mer", "author_url": "", "post_date": "2019-05-23T22:26:11.263000", "content": "8 features :))
", "votes": 5, "replies": [] }, { "id": 536319, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-24T08:56:37.577000", "content": "\n\n\nA lot of algorithms do not work with my features
\n
Interesting, I guess you won't disclose what model you use before competition end, but if you're inclined to do so now then you're most welcome!
", "votes": 1, "replies": [] }, { "id": 536437, "author_name": "Dre@mer", "author_url": "", "post_date": "2019-05-24T13:14:35.063000", "content": "I use a tree based algorithms, but not XGB and LGBM implementation. But I don't believe my CV schema too much (there is some random noise).
", "votes": 2, "replies": [] }, { "id": 537101, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-26T07:17:37.907000", "content": "Thanks, eager to see your solution after competition end.
", "votes": 0, "replies": [] } ] }, { "id": 493631, "author_name": "Amjad", "author_url": "", "post_date": "2019-03-18T22:22:38.450000", "content": "catboost, oof: 1.92, lb: 1.462
", "votes": 3, "replies": [] }, { "id": 493629, "author_name": "JinCui", "author_url": "", "post_date": "2019-03-18T22:21:45.243000", "content": "Just curious, does anyone use the over-sampling method? It’s giving me a very low CV but public LB is like 1.7, and I think the overlap between train and validation set is causing a problem. I have stopped using over-sampling since then.
", "votes": 3, "replies": [ { "id": 493669, "author_name": "varun", "author_url": "", "post_date": "2019-03-19T00:07:41.317000", "content": "I briefly tried it. It seemed to make a minor improvement in my CV score if I oversampled by a factor 2 (stride of 75000) but quickly deteriorates beyond that.
", "votes": 3, "replies": [] }, { "id": 497004, "author_name": "Vasilis Stamatopoulos", "author_url": "", "post_date": "2019-03-22T20:11:52.123000", "content": "Tried that too 75000 oversampling is okay but anything beyond ruins the score
", "votes": 1, "replies": [] }, { "id": 503730, "author_name": "lemon", "author_url": "", "post_date": "2019-03-30T13:05:35.247000", "content": "do you find the solution to your problem? I have the same problem. very low CV but high LB
", "votes": 1, "replies": [] }, { "id": 504082, "author_name": "FP", "author_url": "", "post_date": "2019-03-31T00:02:56.140000", "content": "@wangzhaoxu, how low cv are u talking about?
", "votes": 0, "replies": [] }, { "id": 506069, "author_name": "lemon", "author_url": "", "post_date": "2019-04-03T00:04:13.550000", "content": "I got 0.3 for CV, and 1.7 for LB
", "votes": 2, "replies": [] }, { "id": 506134, "author_name": "Sven Hinderer", "author_url": "", "post_date": "2019-04-03T02:48:50.820000", "content": "0.3 is unrealistically low. You seem to overfit somehow. Are you shuffling your data before building the splits for cross-val? How do you optimize hyperparameters? (you can also overfit on the validation data)
", "votes": 2, "replies": [] }, { "id": 506142, "author_name": "FP", "author_url": "", "post_date": "2019-04-03T02:57:37.817000", "content": "I am interested to know what model you used to train. Is it NN or RNN?
", "votes": 0, "replies": [] }, { "id": 506483, "author_name": "RNA", "author_url": "", "post_date": "2019-04-03T14:40:39.013000", "content": "wangzhaoxu, surely either you have leakage or are massively overfitting a small training set. Did you engineer any new features that relied on the time_to_failure column by accident? Either that or my work is going far worse than I thought!
My current best single model is around 1.99 (oof score), but it only have a public score of 1.54, I’m also using lightgbm.
", "votes": 3, "replies": [] }, { "id": 504538, "author_name": "Alexus1000", "author_url": "", "post_date": "2019-03-31T19:42:42.263000", "content": "cudnnlstm LB: 1.45
", "votes": 4, "replies": [ { "id": 516853, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-04-15T04:55:53.213000", "content": "It's good to see this. May I ask if you do CV or you just fit 1 time with an appropriate iterations?
", "votes": 1, "replies": [] }, { "id": 517009, "author_name": "Alexus1000", "author_url": "", "post_date": "2019-04-15T11:08:14.803000", "content": "Basically I used the model of:\n@mayer79\nhttps://www.kaggle.com/mayer79/rnn-starter-for-huge-time-series
\n\nI used cudnnLSTM instead of cudnngru, I played with different n_step and step_length, I got better n_step big results and few step-lengths. I also played with different features approx 50 was my best result.
\n\nOverlap data can help you with the overfitting.
", "votes": 4, "replies": [] }, { "id": 517874, "author_name": "JinCui", "author_url": "", "post_date": "2019-04-16T16:19:46.200000", "content": "I tried that kernel too, it seems that you need to make sure the train data and valid data are not overlapping if you want to make your CV trustworthy:
\n\ntrain_gen=generator(float_data,batch_size=batch_size, min_index=second_earthquake+1)\nvalid_gen=generator(float_data,batch_size=batch_size, max_index=second_earthquake)
It seems that using two layers of CuDNNGRU and dropout may help (my CV val_loss is around 1.98, LB at 1.48 with 54 features), but I still haven't found the optimal parameters for CuDNNLSTM. Also it's probably better to do a KFold validation.
", "votes": 3, "replies": [] }, { "id": 518065, "author_name": "FP", "author_url": "", "post_date": "2019-04-16T20:14:27.800000", "content": "CV 1.48 is impressive!
", "votes": 0, "replies": [] }, { "id": 518275, "author_name": "JinCui", "author_url": "", "post_date": "2019-04-17T03:24:05.837000", "content": "My bad, I meant LB around 1.48...
", "votes": 1, "replies": [] }, { "id": 518277, "author_name": "FP", "author_url": "", "post_date": "2019-04-17T03:28:44.510000", "content": "That's impressive too!
", "votes": 0, "replies": [] }, { "id": 518408, "author_name": "Alexus1000", "author_url": "", "post_date": "2019-04-17T07:03:24.350000", "content": "Another trick, use loss \"logcosh\" with this your val_loss like very close to LB.
", "votes": 1, "replies": [] }, { "id": 518658, "author_name": "Sven Hinderer", "author_url": "", "post_date": "2019-04-17T15:20:44.017000", "content": "I doubt logcosh-loss helps, since the difference between lb and cv comes from the different data distributions in train and test set. I actually tried mae, logcosh and mse some time ago with the same approach and the lb result is pretty much the same.
\n\nThe comparable cv error with logcosh is coming from different (lower) error penalization (see https://cdn-images-1.medium.com/max/1600/1*BploIBOUrhbgdoB1BK_sOg.png for a graphical example).\nYou could get the same result by scaling down your mae-score with a factor<1 which doesn't really make sense.
", "votes": 0, "replies": [] } ] }, { "id": 494620, "author_name": "Aaron Harris", "author_url": "", "post_date": "2019-03-20T03:44:17.373000", "content": "Random Forest oof 1.89 and 1.512 public score.
", "votes": 4, "replies": [] }, { "id": 493389, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2019-03-18T16:01:45.460000", "content": "Lgb, oof: 2.0644, lb: 1.532
\n\nEdit: oof 2.0695, lb 1.506. What's interesting, I managed to shrink the train/test gap to ~ 0.15 - e.g. on fold 5 I have 1.94887 train vs 2.08177 validation.
", "votes": 4, "replies": [] }, { "id": 529418, "author_name": "Murat Korkmaz", "author_url": "", "post_date": "2019-05-09T20:54:49.073000", "content": "Best LB score is LGB 18 features CV:2.00- LB:1.478.\nBest CV is 1.99 with 7 features but LB is 1.487.
", "votes": 1, "replies": [] }, { "id": 527530, "author_name": "Philippe Lonjoux", "author_url": "", "post_date": "2019-05-05T18:26:53.647000", "content": "lgb with 70 features CV: 1.9721 LB: 1.407
", "votes": 1, "replies": [] }, { "id": 527472, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-05-05T16:03:39.273000", "content": "LGB CV: 1.9723 LB: 1.414
", "votes": 1, "replies": [] }, { "id": 522627, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-04-24T18:37:19.730000", "content": "Hey there!\nPlease help me understand something.\nSituation: almost everyone here have CV about ~2 +- 0.03 - both top of leaderbord ( /summon @cpmpml) and not-so-top dudes like me. Today I played with one of my models that had a CV 2.03 and LB 1.51. I came up with some effed-up CV-scheme and after testing it I found out that it gives me CV 1.56 and LB 1.55. Features remain unchanged. And plot of y_train[:1500] and oof[:1500] looks like this:\n\nCan someone please explain to me what did I do, what's the nature of this CV-LB correlation and why plots are so weird? Am I overfitting like hell?\nThanks in advance!
How did you even manage to get the 14-ttf samples right? Seems like overfit, otherwise i'd like to know the feature that's able to do this :)
", "votes": 1, "replies": [] }, { "id": 522871, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-25T06:53:50.570000", "content": "What is important is not the CV LB gap, which is very small in your case, but correlation. If you make a modification, say you add a feature, and your new CV score is improved, will LB score be improved as well?
\n\nLooking at the above, I bet you're using shuffling.
", "votes": 4, "replies": [] }, { "id": 522889, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-04-25T07:32:30.920000", "content": "@svenhinderer, it's not really complicated. It isn't the feature and it was mentioned in discussions many times :)
\n\n@cpmpml, I didn't test this for correlation of CV-LB yet because I ran out of subs yesterday, so I'm gonna do it today if I'll have enough time.\nRegarding shuffling - yes, I'm using it. Does it have smth to do with this weird plot (i.e. oof jumps from some value to zero and back)?
", "votes": 1, "replies": [] }, { "id": 522909, "author_name": "Grzegorz Sionkowski", "author_url": "", "post_date": "2019-04-25T08:20:12.943000", "content": "It is no problem to make your CV score close to public LB and even keep the correlation between them. Simple analysis of LB scores presented in a few comments proves that public dataset contains less high ttf segments than the train, so removing some high ttf segments from train you can make your train dataset more similar to public test than the raw one and obtain \"wanted\" results (low CV-LB distance, high correlation of CV/LB). The problem is that it is the better way to overfit, because the private dataset seems to contain significantly more high ttf segments than public one.\nThe way of improving CV described above is obviously wrong. I think most of Kagglers understand why and will not use it.\nTime for the punch line: \"Do not believe in your CV just because of you do not understand why it is close to LB score ;)\"
", "votes": 1, "replies": [] }, { "id": 522914, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-25T08:32:05.740000", "content": "\n\n\nbecause the private dataset seems to contain significantly more high ttf segments than public one.
\n
How do you know this? I asked in the post where a similar claim was made but got no answer.
", "votes": 2, "replies": [] }, { "id": 522919, "author_name": "Marcus Lin", "author_url": "", "post_date": "2019-04-25T08:36:44.810000", "content": "Because the LB is lower than cv too much ? and than assume public set are most consist of small ttf.
", "votes": 1, "replies": [] }, { "id": 522926, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-04-25T08:40:49.393000", "content": "@sionek , I understand why the CV method you described above is incorrect, but I didn't remove any ttf segments or anything else from train dataset, didn't change features, etc. Changes were only made in CV algorithm. Also, I'm 100% sure that my folds are not overlapping, and almost sure that there are no leaks.
", "votes": 0, "replies": [] }, { "id": 522928, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-25T08:42:45.170000", "content": "\n\n\nalmost sure that there are no leaks.
\n
Shuffling leaks time info as discussed elsewhere in this forum.
", "votes": 3, "replies": [] }, { "id": 522935, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-04-25T08:52:48.617000", "content": "@cpmpml , thanks for info, I'll definitely try to find this topic and read it. 👍 \nBut still, I also used shuffling in earlier version of my model (which had CV 2.03 and LB 1.51). So I assume that shuffling is not the reason of CV dropping so significantly.
", "votes": 0, "replies": [] }, { "id": 522936, "author_name": "Grzegorz Sionkowski", "author_url": "", "post_date": "2019-04-25T08:55:03.207000", "content": "https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/86530#
\n\n@CPMP, probably you did not get any answer, because it would be the repeating of the previous information. For me the aim, the method and the results and conclusion of the experiment are clear, however I did not check it, because I do not need that information.
", "votes": 0, "replies": [] }, { "id": 522956, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-25T09:31:48.357000", "content": "Grzegorz, I disagree it was clear which is why I asked a question. It is not because one predicts a high ttf than the data has a high ttf. Either that experiment is fundamentally flawed, or I miss something. I interpret the lack of answer as a sign that I didn't miss much.
", "votes": 0, "replies": [] }, { "id": 522959, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-25T09:34:30.627000", "content": "\n\n\nI'll definitely try to find this topic and read it. 👍
\n
It has shuffling in the topic title: https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/89366#latest-521819
", "votes": 1, "replies": [] }, { "id": 523008, "author_name": "Amjad", "author_url": "", "post_date": "2019-04-25T11:32:24.490000", "content": "Different CV strategies are not comparable to each other. But when you stick to a certain strategy and adjust other variables, you start to see the correlation with the LB scores. For example, my leave-quake-out CV gives a 2.2, and shuffled CV gives 1.88. Yet, both strategies give similar LB score.
", "votes": 0, "replies": [] }, { "id": 523168, "author_name": "Sven Hinderer", "author_url": "", "post_date": "2019-04-25T17:05:55.597000", "content": "Wish i could say the same. If i shuffle and use early stopping, i also get <1.9 cv and <1.5 lb quite easily, but this approach is just begging to overfit imo. \nWith some form of quakewise split without early stopping, which i think is way more robust, lb never goes below 1,55, #feelsbad
", "votes": 0, "replies": [] }, { "id": 523235, "author_name": "Psi", "author_url": "", "post_date": "2019-04-25T19:45:40.283000", "content": "@amjad85 If you refit on whole data both results will always be the same on LB :P
", "votes": 0, "replies": [] }, { "id": 524095, "author_name": "Amjad", "author_url": "", "post_date": "2019-04-27T23:24:02.447000", "content": "@philippsinger If you average the predictions from each fold and use early stopping, the results can be very different.
", "votes": 0, "replies": [] }, { "id": 524131, "author_name": "Sven Hinderer", "author_url": "", "post_date": "2019-04-28T02:46:05.870000", "content": "@amjad85 \nDoes it even make sense to use early stopping with unshuffled data here? Doesn't that just give you a model that fits the validation data on each fold best, which is not optimal?
\n\n@stanislavblinov \nMaybe i'm blind. but can you elaborate on getting the high ttf-values right? I haven't seen anyone manage to do this, but maybe i missed it. The only way i can explain it is by overfitting to the training data.
", "votes": 0, "replies": [] }, { "id": 524965, "author_name": "Amjad", "author_url": "", "post_date": "2019-04-29T21:28:33.653000", "content": "@svenhinderer no it doesn't make sense IMO. I'm if favor of shuffling. The similar score I mentioned above with quake-based CV was done without early stopping.
", "votes": 1, "replies": [] } ] }, { "id": 520280, "author_name": "AC_datascientist", "author_url": "", "post_date": "2019-04-20T16:06:25.163000", "content": "@CPMP, did you get the answer for your question - what is LOGO? Also, you mentioned \"lgb 10 features cv 1.866 lb 1.416\" - I hope you didn't have any typos in this statement
", "votes": 1, "replies": [ { "id": 520349, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-20T19:15:34.180000", "content": "Yes and no typo ;)
", "votes": 2, "replies": [] } ] }, { "id": 519870, "author_name": "mykper", "author_url": "", "post_date": "2019-04-19T19:09:56.027000", "content": "LGB CV 2.023 LB 1.533, CV 1.976 LB 1.436\n5 folds with shuffle
", "votes": 1, "replies": [] }, { "id": 501255, "author_name": "Liu123", "author_url": "", "post_date": "2019-03-27T05:25:09.623000", "content": "LGB cv:2.03756 LB: 1.509
", "votes": 1, "replies": [] }, { "id": 500532, "author_name": "bluetrain", "author_url": "", "post_date": "2019-03-26T07:21:34.470000", "content": "Edit: Lgbm: cv = 1.994, lb = 1.417\n(old) Lgbm: cv = 2.025, lb = 1.459
", "votes": 1, "replies": [] }, { "id": 500377, "author_name": "youness katraoui", "author_url": "", "post_date": "2019-03-25T23:17:39.067000", "content": "RNN. MAE: 1.6896 L.B.: 1.533
", "votes": 1, "replies": [] }, { "id": 527468, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-05T15:54:45.197000", "content": "Update, lgb with 13 features, CV 1.772 LB 1.356
", "votes": 2, "replies": [ { "id": 527582, "author_name": "raddar", "author_url": "", "post_date": "2019-05-05T21:45:42.227000", "content": "LB within stdev? nice score btw :)
", "votes": 2, "replies": [] }, { "id": 527605, "author_name": "Carlos Prades K.", "author_url": "", "post_date": "2019-05-05T23:12:50.957000", "content": "@cpmpml 13 high level features, right? I mean like pca or tsne features? Or just 13 features that you selected very well from many others statistical like those that has been shared? :)
", "votes": 2, "replies": [] }, { "id": 527707, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-06T06:17:52.130000", "content": "3 I borrowed from public kernels and 10 I crafted. No tsne nor pca.
", "votes": 3, "replies": [] }, { "id": 527823, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-06T12:07:55.923000", "content": "\n\n\nLB within stdev?
\n
LB is way lower than CV - CV std. It seems it is a general pattern, except for Kha Vo if I remember correctly.
", "votes": 0, "replies": [] }, { "id": 527846, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-05-06T13:09:59.293000", "content": "You remember incorrectly, I never said that.\nBy the way, I'm starting to realize what you are doing with your CV (how to reduce fold variance, how to add features). In my perspective, searching for more features from a small set of features (like what you're doing) is easier than most people do (remove bad features from a bunch), because it is much faster and more convenient to test. All we need is just a good CV strategy.
", "votes": 1, "replies": [] }, { "id": 527870, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-06T13:45:23.237000", "content": "\n\n\nYou remember incorrectly, I never said that.
\n
My bad, sorry.
", "votes": 2, "replies": [] }, { "id": 529781, "author_name": "RNA", "author_url": "", "post_date": "2019-05-10T18:31:15.607000", "content": "@cpmpml When you mention your CV score, is that from your own CV method or a simple shuffled KFold score? Don't worry, I'm not going to ask what your CV strategy is!
", "votes": 1, "replies": [] }, { "id": 529933, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-11T07:37:47.363000", "content": "\n\n\nis that from your own CV method or a simple shuffled KFold score?
\n
It is from my CV setting, (what else?), a CV setting which could or could not be shuffled KFold.
", "votes": 1, "replies": [] }, { "id": 529958, "author_name": "RNA", "author_url": "", "post_date": "2019-05-11T09:41:56.443000", "content": "Fair enough. I wasn't trying to pry insight into your methods, perhaps I had wrongly assumed that most people in this thread were using a shuffled-KFold when posting their CV scores for the sake of comparing baselines, since different methods can lead to varying results for the same data. Good luck with your work!
", "votes": 2, "replies": [] } ] }, { "id": 524343, "author_name": "Kain", "author_url": "", "post_date": "2019-04-28T14:51:46.717000", "content": "XGBoost 12 features: 5 fold CV=2.032, LB=1.461
", "votes": 2, "replies": [ { "id": 524622, "author_name": "Psi", "author_url": "", "post_date": "2019-04-29T07:50:37.487000", "content": "Shuffled CV?
", "votes": 0, "replies": [] }, { "id": 524688, "author_name": "Kain", "author_url": "", "post_date": "2019-04-29T11:18:49.530000", "content": "Yep..shuffle=True
", "votes": 0, "replies": [] }, { "id": 524707, "author_name": "Psi", "author_url": "", "post_date": "2019-04-29T11:53:31.040000", "content": "Thx, and I guess LB prediction is average across folds?
", "votes": 0, "replies": [] }, { "id": 524760, "author_name": "RNA", "author_url": "", "post_date": "2019-04-29T13:26:09.427000", "content": "Be careful with shuffled KFold. I can get CV below 1.8 using 27 features with a 5-fold, but the LB score is around 1.6. I think there is a lot of inherent randomness there. I gave up chasing the LB a while ago and just use a custom CV pipeline now.
", "votes": 1, "replies": [] }, { "id": 524794, "author_name": "Psi", "author_url": "", "post_date": "2019-04-29T14:33:11.870000", "content": "Shuffled CV leaks the EQs, so it is problematic. But you also can find arguments for it. There is not a single CV setting that I am aware of that doesn't have some counter-arguments.
", "votes": 2, "replies": [] }, { "id": 524804, "author_name": "Kain", "author_url": "", "post_date": "2019-04-29T15:00:20.657000", "content": "Psi, I use MAE of oof generated with 5 folds not the average MAE across folds.
", "votes": 0, "replies": [] }, { "id": 524817, "author_name": "Kain", "author_url": "", "post_date": "2019-04-29T15:16:09.597000", "content": "BigIronSphere, How did you choose your 27 features? I personally like to add features one by one and check CV then LB. This usually takes a lot of time and also burns lots of submissions [it is hard to do it in this comp with 2 submission a day ;)] . For me the most important rule for keeping a feature is 1 ) if it lowers CV error then 2) checking if CV and LB are moving in the same direction.
", "votes": 1, "replies": [] }, { "id": 524822, "author_name": "RNA", "author_url": "", "post_date": "2019-04-29T15:28:17.167000", "content": "The train data is very small and you can very quickly run a KFold inside a loop. I made an algorithm that randomly selects features from a pool and assigns them a normalised score based on their feature importance after a 5-fold unshuffled CV. After several thousand iterations I isolate the top quintile and see what the optimum number of features is in another loop (I'm simplifying a little). My method probably isn't fantastic since I'm pretty low on the LB, so perhaps it's not the best idea. I'm treating this competition more as a learning experience where I can prepare quick pipelines for future use.
", "votes": 3, "replies": [] }, { "id": 524866, "author_name": "Psi", "author_url": "", "post_date": "2019-04-29T17:11:09.667000", "content": "@kainsama My question was rather how you produce final test prediction.
", "votes": 0, "replies": [] }, { "id": 524878, "author_name": "Kain", "author_url": "", "post_date": "2019-04-29T17:28:04.460000", "content": "Sorry I misunderstood your question; and yes I average over folds to create the final test prediction.
", "votes": 0, "replies": [] } ] }, { "id": 495428, "author_name": "Daniel", "author_url": "", "post_date": "2019-03-21T04:33:57.333000", "content": "CNN. MAE: 2.084 LB: 1.616
", "votes": 2, "replies": [ { "id": 495631, "author_name": "Borys Tymchenko", "author_url": "", "post_date": "2019-03-21T10:38:39.160000", "content": "Woah, I also had a CNN with exactly this score both local and LB :D
", "votes": 1, "replies": [] }, { "id": 496190, "author_name": "Daniel", "author_url": "", "post_date": "2019-03-22T00:44:51.890000", "content": "Crazy! I'm doing 2D convolutions over spectrograms, but I'm beginning to think it's not the best approach.
", "votes": 3, "replies": [] }, { "id": 496859, "author_name": "Sven Hinderer", "author_url": "", "post_date": "2019-03-22T17:12:59.133000", "content": "Getting that score with your approach is quite impressive imo. I've tried the same but gave up pretty fast because it didn't seem to work well enough.\nIf you're interested, theres a kernel about it (not by me) called \"Spectrogram+Convolution=?\"
", "votes": 1, "replies": [] } ] }, { "id": 493422, "author_name": "Sven Hinderer", "author_url": "", "post_date": "2019-03-18T16:20:18.323000", "content": "Multiple Input NN ~1.5-1.6lb, what is oof score?
", "votes": 2, "replies": [ { "id": 493452, "author_name": "JinCui", "author_url": "", "post_date": "2019-03-18T16:48:09.503000", "content": "oof is out-of-fold cross validation
", "votes": 2, "replies": [] }, { "id": 493464, "author_name": "Sven Hinderer", "author_url": "", "post_date": "2019-03-18T17:00:03.603000", "content": "Okay thanks, my oof mse is around 8 then, haven't tried mae in a while for training.
", "votes": 1, "replies": [] } ] }, { "id": 502651, "author_name": "RNA", "author_url": "", "post_date": "2019-03-28T21:49:50.647000", "content": "CatBoost - 2.0273 CV, 1.475 LB
\n\nI'm noticing a couple of trends. Firstly, CatBoost seems to often outperform LGB which is strange since it was specifically designed for categorical data. I need to reread the CB docs and get a better understanding of its oblivious trees methods. Secondly, the relationship between CV and LB isn't very strong... I think we should trust CV much more. Bad news for me since I can't get oof below 2.0!
", "votes": 0, "replies": [ { "id": 502662, "author_name": "JinCui", "author_url": "", "post_date": "2019-03-28T22:16:16.123000", "content": "For some reason I always get better CV (best CV is around 1.98) with LGB as compared to either XGB or CatBoost (couldn't break 2), and XGB/CatBoost seem to run very slowly on my laptop.\nI agree with you on trusting CV in this competition: I've noticed that all my \"CV<2\" submissions got LB >1.5.
", "votes": 1, "replies": [] }, { "id": 502671, "author_name": "RNA", "author_url": "", "post_date": "2019-03-28T22:50:10.010000", "content": "CatBoost is extremely slow if you don't use a GPU, but this often comes at a cost of accuracy with smaller datasets. With Kaggle servers it's not really a problem but it still takes ~2.5 hours with CPU alone for a 5-fold. I think the public test dataset is misleading and has below-average TTF values which are easier to predict. I suspect this competition will hinge on extraction of features that indicate a TTF > 10, although they will be difficult to isolate. At a certain point it will likely become impossible given the stochastic physical processes involved.
\n\nInterestingly, my LGB gives better results for individual folds but not on total OOF. A lot of this may be random though...
", "votes": 1, "replies": [] }, { "id": 502766, "author_name": "JinCui", "author_url": "", "post_date": "2019-03-29T03:42:00.520000", "content": "I see, looks like stacking from multiple models is the way to go?
\n\nThe thing is that it's very difficult to predict if a segment TTF is in the high value or near 0, especially if the segment is close to the earthquake outbreak (TTF jumps from near 0 to high value), so LGB would take the middle ground, which is why we never see extremely low or high value from oof predictions.
\n\nOne thing I would suggest is to separate TTF jumps evenly in each fold, so that oof score becomes stable in each fold.
", "votes": 0, "replies": [] }, { "id": 504089, "author_name": "Sven Hinderer", "author_url": "", "post_date": "2019-03-31T00:35:16.437000", "content": "You could just skip the segments where earthquakes happen. It's still better to have 16 less samples than to have 16 completely wrong ones.
", "votes": 3, "replies": [] } ] }, { "id": 504083, "author_name": "", "author_url": "", "post_date": "2019-03-31T00:03:48.143000", "content": "", "votes": 0, "replies": [] }, { "id": 497085, "author_name": "", "author_url": "", "post_date": "2019-03-22T23:45:25.363000", "content": "", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "493196": "It's great in any competition to know how others are approaching to the solution without much of details. Let's share with each other which model we are using.\n\nI am using a LightGBM model.", "500613": "CatBoost - 1.87 LB: 1.441 on FFT features", "519221": "Starting with lgb and 7 features: cv 1.901, lb 1.444", "529992": "Update, lgb 14 features, LB 1.316\n\nI am not reporting CV anymore because it is hard to compare. We know early stopping yields more optimistic CV than when not using it, Similarly shuffling yields better CV values. I still think this sharing has value as it shows one can get good result with a small number of features.\n", "524714": "LGBM 3Kfold Not Shuffled (I'm trying to keep it updated)\nCV: 2.084, LB: 1.526\nCV: 2.052, LB: 1.490\nCV: 2.014, LB: 1.433\nCV: 1.996, LB: 1.416 (6 features)\n\nmy CV seems to match the LB direction quite well", "493633": "LGB oof 1.99, LB 1.408. However I have another oof 1.94 but LB 1.460. I strongly believe the public LB (only around 300 samples) does not reflect the true standings, as scores of folds vary a lot.", "493622": "I'm using LightGBM too. \n\nOOF: 2.042, LB: 1.468. \n\nI found sensible feature engineering to be far more impactful than messing around with the training model.", "522560": "LightGBM 23 features: CV 1.9793, LB 1.411", "522385": "CatBoost - oof 2.0176, LB 1.451", "536036": "I tested single models only until now (no blending, no stacking). My best model has 1.278 on public LB.\n\nA lot of algorithms do not work with my features (like XGB and LGBM). Maybe I have a problem setting them up.", "493631": "catboost, oof: 1.92, lb: 1.462", "493629": "Just curious, does anyone use the over-sampling method? It’s giving me a very low CV but public LB is like 1.7, and I think the overlap between train and validation set is causing a problem. I have stopped using over-sampling since then.", "493275": "My current best single model is around 1.99 (oof score), but it only have a public score of 1.54, I’m also using lightgbm.", "504538": "cudnnlstm LB: 1.45", "494620": "Random Forest oof 1.89 and 1.512 public score. ", "493389": "Lgb, oof: 2.0644, lb: 1.532\n\nEdit: oof 2.0695, lb 1.506. What's interesting, I managed to shrink the train/test gap to ~ 0.15 - e.g. on fold 5 I have 1.94887 train vs 2.08177 validation.\n", "529418": "Best LB score is LGB 18 features CV:2.00- LB:1.478.\nBest CV is 1.99 with 7 features but LB is 1.487.", "527530": "lgb with 70 features CV: 1.9721 LB: 1.407", "527472": "LGB CV: 1.9723 LB: 1.414", "522627": "Hey there!\nPlease help me understand something.\n**Situation:** almost everyone here have CV about ~2 +- 0.03 - both top of leaderbord ( /summon @cpmpml) and not-so-top dudes like me. Today I played with one of my models that had a CV 2.03 and LB 1.51. I came up with some effed-up CV-scheme and after testing it I found out that it gives me CV 1.56 and LB 1.55. Features remain unchanged. And plot of y_train[:1500] and oof[:1500] looks like this:\n\nCan someone please explain to me what did I do, what's the nature of this CV-LB correlation and why plots are so weird? Am I overfitting like hell?\nThanks in advance!", "520280": "@CPMP, did you get the answer for your question - what is LOGO? Also, you mentioned \"lgb 10 features cv 1.866 lb 1.416\" - I hope you didn't have any typos in this statement", "519870": "LGB CV 2.023 LB 1.533, CV 1.976 LB 1.436\n5 folds with shuffle", "501255": "LGB cv:2.03756 LB: 1.509", "500532": "Edit: Lgbm: cv = 1.994, lb = 1.417\n(old) Lgbm: cv = 2.025, lb = 1.459", "500377": "RNN. MAE: 1.6896 L.B.: 1.533", "527468": "Update, lgb with 13 features, CV 1.772 LB 1.356", "524343": "XGBoost 12 features: 5 fold CV=2.032, LB=1.461", "495428": "CNN. MAE: 2.084 LB: 1.616", "493422": "Multiple Input NN ~1.5-1.6lb, what is oof score?", "502651": "CatBoost - 2.0273 CV, 1.475 LB\n\nI'm noticing a couple of trends. Firstly, CatBoost seems to often outperform LGB which is strange since it was specifically designed for categorical data. I need to reread the CB docs and get a better understanding of its oblivious trees methods. Secondly, the relationship between CV and LB isn't very strong... I think we should trust CV much more. Bad news for me since I can't get oof below 2.0!", "504083": "", "497085": "" } }