![\"validation](\"https://i.imgur.com/rm9b2AL.png\")
I'm curious if anybody is willing to discuss their validation approach - I tried random split (loose relationship with LB), time based split based on similar time length (i.e. creating a validation set spanning a period of similar length as test set - too few observations left), split based on observation count (okayish, but nothing to write home about - local validation score 0.2231, lb 0.2291).
", "messages": [ { "id": 325763, "postDate": "2018-05-08T20:44:21.853Z", "content": "I'm curious if anybody is willing to discuss their validation approach - I tried random split (loose relationship with LB), time based split based on similar time length (i.e. creating a validation set spanning a period of similar length as test set - too few observations left), split based on observation count (okayish, but nothing to write home about - local validation score 0.2231, lb 0.2291).
", "rawMarkdown": "I'm curious if anybody is willing to discuss their validation approach - I tried random split (loose relationship with LB), time based split based on similar time length (i.e. creating a validation set spanning a period of similar length as test set - too few observations left), split based on observation count (okayish, but nothing to write home about - local validation score 0.2231, lb 0.2291).", "votes": 29 }, { "id": 325775, "postDate": "2018-05-08T21:03:55.713Z", "content": "We have 10 random KFold and constant difference between CV and LB near 0.005.
", "rawMarkdown": "We have 10 random KFold and constant difference between CV and LB near 0.005.", "votes": 15, "replies": [ { "id": 325777, "postDate": "2018-05-08T21:05:39.373Z", "content": "Interesting - I will give it a try, then. Thanks!
", "rawMarkdown": "Interesting - I will give it a try, then. Thanks!", "votes": 1 }, { "id": 325879, "postDate": "2018-05-09T02:13:51.677Z", "content": "Seems perfect
", "rawMarkdown": "Seems perfect" }, { "id": 326610, "postDate": "2018-05-10T02:39:03.840Z", "content": "same here, 5 random KFold and difference between CV and LB about 0.004-0.005, but when I performed stacking or using optim function, the difference is up to 0.006-0.008, weird.
", "rawMarkdown": "same here, 5 random KFold and difference between CV and LB about 0.004-0.005, but when I performed stacking or using optim function, the difference is up to 0.006-0.008, weird." }, { "id": 326615, "postDate": "2018-05-10T03:15:57.927Z", "content": "@Andrew-Chang: That's pretty normal in my experience as usually stacking overfits a little bit more.
", "rawMarkdown": "@Andrew-Chang: That's pretty normal in my experience as usually stacking overfits a little bit more." }, { "id": 326619, "postDate": "2018-05-10T03:20:56.703Z", "content": "You're right, but when I use k-folds and optim() function to determine the weight of every model, it also leads to big difference. It really confused me.
", "rawMarkdown": "You're right, but when I use k-folds and optim() function to determine the weight of every model, it also leads to big difference. It really confused me." }, { "id": 326820, "postDate": "2018-05-10T11:40:40.640Z", "content": "Update: so a 5-fold cv seems to give a nice and stable 0.005 difference between public and private LB. Thanks for the tip @Sergei.
", "rawMarkdown": "Update: so a 5-fold cv seems to give a nice and stable 0.005 difference between public and private LB. Thanks for the tip @Sergei.", "votes": 9 }, { "id": 348635, "postDate": "2018-06-27T03:10:23.683Z", "content": "@Peter Hurfold, does it mean for the stacking model using 10 fold cross validation might lead to more precise estimation than 5 fold?
\nAnd is it comparable for model using 5 fold CV and 10 fold? Just curious on how to find a best way to make the best use of CV. Thanks!
10 fold would be more precise than 5 fold, but maybe not by much. And it would be twice as time consuming. I personally stick with 5 fold, but I know some purists who do otherwise. As long as you're consistent throughout, you should be fine. :)
", "rawMarkdown": "10 fold would be more precise than 5 fold, but maybe not by much. And it would be twice as time consuming. I personally stick with 5 fold, but I know some purists who do otherwise. As long as you're consistent throughout, you should be fine. :)", "votes": 1 }, { "id": 348642, "postDate": "2018-06-27T03:33:45.167Z", "content": "Maybe another question from some purists might ask: usually how much the difference between CV and LB can be tolerated for the private LB?
", "rawMarkdown": "Maybe another question from some purists might ask: usually how much the difference between CV and LB can be tolerated for the private LB? " }, { "id": 348651, "postDate": "2018-06-27T03:53:27.997Z", "content": "It depends on the competition, which is why it is great to ask other people what they are experiencing and see if yours is similar.
", "rawMarkdown": "It depends on the competition, which is why it is great to ask other people what they are experiencing and see if yours is similar.", "votes": 1 }, { "id": 348652, "postDate": "2018-06-27T03:53:58.873Z", "content": "Another important aspect is how much variability there is in your gap between your submissions. The more consistency, the safer you can feel.
", "rawMarkdown": "Another important aspect is how much variability there is in your gap between your submissions. The more consistency, the safer you can feel.", "votes": 1 }, { "id": 348653, "postDate": "2018-06-27T03:54:23.890Z", "content": "Generally, you should trust your CV score most because it's based on the most data. Having a time-ordered LB can undermine that trust some, though.
", "rawMarkdown": "Generally, you should trust your CV score most because it's based on the most data. Having a time-ordered LB can undermine that trust some, though." } ] }, { "id": 325778, "postDate": "2018-05-08T21:06:14.847Z", "content": "I really like seeing this thread so I can see if I'm overfitting. :) Right now, I'm doing 5-fold CV and have a consistent gap between CV and LB between 0.0031 and 0.0038. I tried a time-based validation strategy once but it made my score worse.
", "rawMarkdown": "I really like seeing this thread so I can see if I'm overfitting. :) Right now, I'm doing 5-fold CV and have a consistent gap between CV and LB between 0.0031 and 0.0038. I tried a time-based validation strategy once but it made my score worse.", "votes": 9 }, { "id": 326945, "postDate": "2018-05-10T14:54:17.447Z", "content": "Be careful with the last days of training data set:
\n\ntraining[(training['activation_date']>=pd.to_datetime('2017-03-29'))].shape
\n\n(100, 17)
\n\ntraining[(training['activation_date']>=pd.to_datetime('2017-03-27'))].shape
\n\n(227848, 17)
\n\nOnly 100 samples after 2017-03-29.
", "rawMarkdown": "Be careful with the last days of training data set:\n\ntraining[(training['activation_date']>=pd.to_datetime('2017-03-29'))].shape\n\n(100, 17)\n\ntraining[(training['activation_date']>=pd.to_datetime('2017-03-27'))].shape\n\n(227848, 17)\n\nOnly 100 samples after 2017-03-29.", "votes": 3, "replies": [ { "id": 327002, "postDate": "2018-05-10T16:37:30.413Z", "content": "similar pattern can be seen in last days of test set.
\n\n2017-04-12 81824\n2017-04-13 77176\n2017-04-14 70366\n2017-04-15 58793\n2017-04-16 58909\n2017-04-17 80191\n2017-04-18 81114\n2017-04-19 64\n2017-04-20 1\nI am using this idea of stratified KFold on binned target:
\n\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\n\ndef stratified_train_valid_split(meta_train, target_column, target_bins, valid_size, random_state=1234):\n\n y = meta_train[target_column].values\n bins = np.linspace(0, y.shape[0], target_bins)\n y_binned = np.digitize(y, bins)\n\n return train_test_split(meta_train, test_size=valid_size, stratify=y_binned, random_state=random_state)\nIt seems that the target distribution in train/valid is similiar. So far only tested it on bad models but it seems to correlate with the LB improvements.
\n\nWhat do you think of this approach?
", "rawMarkdown": "I am using this idea of stratified KFold on binned target:\n\n\n import numpy as np\n from sklearn.model_selection import train_test_split\n \n def stratified_train_valid_split(meta_train, target_column, target_bins, valid_size, random_state=1234):\n\n y = meta_train[target_column].values\n bins = np.linspace(0, y.shape[0], target_bins)\n y_binned = np.digitize(y, bins)\n\n return train_test_split(meta_train, test_size=valid_size, stratify=y_binned, random_state=random_state)\n\nIt seems that the target distribution in train/valid is similiar. So far only tested it on bad models but it seems to correlate with the LB improvements. \n\nWhat do you think of this approach?", "votes": 4, "replies": [ { "id": 330811, "postDate": "2018-05-19T19:06:12.543Z", "content": "I'm doing the same thing, but I'm getting very different LB results. How similar are yours?
", "rawMarkdown": "I'm doing the same thing, but I'm getting very different LB results. How similar are yours?" }, { "id": 332461, "postDate": "2018-05-23T07:23:37.900Z", "content": "Finally ran new models, and it seems that 0.005 delta is what this approach gets me.
", "rawMarkdown": "Finally ran new models, and it seems that 0.005 delta is what this approach gets me. " } ] }, { "id": 336491, "postDate": "2018-05-31T20:37:14.747Z", "content": "I'm using March 26 - 28 as my validation set. Like most people here, I'm also getting ~0.005 delta. I'm still starting out though so not sure how well this will hold.
\n\n
That's a cool sheet. Thanks for sharing.
", "rawMarkdown": "That's a cool sheet. Thanks for sharing.", "votes": 1 } ] }, { "id": 336350, "postDate": "2018-05-31T13:43:33.810Z", "content": "I am on about 0.004 CV/LB delta, but I am only starting out - will be interesting to see how this gap will develop as I add more magic/non-magic into the mix :)
", "rawMarkdown": "I am on about 0.004 CV/LB delta, but I am only starting out - will be interesting to see how this gap will develop as I add more magic/non-magic into the mix :) ", "replies": [ { "id": 336361, "postDate": "2018-05-31T14:09:34.777Z", "content": "Now that I've added features from train_active / test_active and the images, I've found my gap has widened to ~0.0043.
", "rawMarkdown": "Now that I've added features from train_active / test_active and the images, I've found my gap has widened to ~0.0043." } ] }, { "id": 331960, "postDate": "2018-05-22T08:50:05.693Z", "content": "I use valid set as 10% random split from train set. It gives me ~0.0034 CV-LB delta
", "rawMarkdown": "I use valid set as 10% random split from train set. It gives me ~0.0034 CV-LB delta" }, { "id": 331033, "postDate": "2018-05-20T08:11:21.160Z", "content": "I confirm that using 5-fold random split results in a ~0.005 separation wrt LB. I tried decreasing and increasing the number of folds but 5 seems to be the optimal choice (also considering time).
\nI am also using the last 20% of the train dataset as validation, which results in a slightly smaller shift, around ~0.004
\n\n\njtrane87 wrote
\n \n\n\nI confirm that using 5-fold random split results in a ~0.005
\n
It may depend on the model
\n\n5-fold random split results in a 0.002x-0.003 CV-LB delta for me
", "rawMarkdown": "\n> **jtrane87 wrote**\n> \n> > I confirm that using 5-fold random split results in a ~0.005\n\nIt may depend on the model \n\n5-fold random split results in a 0.002x-0.003 CV-LB delta for me \n", "votes": 1 }, { "id": 331044, "postDate": "2018-05-20T08:34:54.263Z", "content": "Sure, it may depend on the model. Let's say: these are my observations so far.
", "rawMarkdown": "Sure, it may depend on the model. Let's say: these are my observations so far. " } ] }, { "id": 328702, "postDate": "2018-05-14T23:40:17.410Z", "content": "I don't think random spliting (or CV) would be a good approach, because you would be using future data to predict the past, which most of the time gives a more optimistic accuracy.
\n\nhttp://www.fast.ai/2017/11/13/validation-sets/
", "rawMarkdown": "I don't think random spliting (or CV) would be a good approach, because you would be using future data to predict the past, which most of the time gives a more optimistic accuracy.\n\nhttp://www.fast.ai/2017/11/13/validation-sets/", "replies": [ { "id": 328712, "postDate": "2018-05-15T00:08:55.970Z", "content": "In principle I agree, but: 1. this particular series seems to be pretty stationary 2. k-fold cv gives a more consistent estimate (smaller gap with public lb) then the time-based split.
", "rawMarkdown": "In principle I agree, but: 1. this particular series seems to be pretty stationary 2. k-fold cv gives a more consistent estimate (smaller gap with public lb) then the time-based split.", "votes": 2 }, { "id": 328713, "postDate": "2018-05-15T00:09:01.223Z", "content": "That's not always the case... Think about it this way. Every Kaggle problem is technically a time series problem, because the data was collected over time ...we just don't usually get to see the timestamps. What matters is not whether there is a timestamp or time-order to the data, but whether the target actually changes over time in a meaningful way. In this competition, it does not look like there is a reason so far to think that the time order matters. (But I'm still pretty scared of this and will keep hunting!)
", "rawMarkdown": "That's not always the case... Think about it this way. Every Kaggle problem is technically a time series problem, because the data was collected over time ...we just don't usually get to see the timestamps. What matters is not whether there is a timestamp or time-order to the data, but whether the target actually changes over time in a meaningful way. In this competition, it does not look like there is a reason so far to think that the time order matters. (But I'm still pretty scared of this and will keep hunting!)", "votes": 11 }, { "id": 329099, "postDate": "2018-05-15T17:57:23.560Z", "rawMarkdown": "", "isDeleted": true }, { "id": 348647, "postDate": "2018-06-27T03:48:53.650Z", "content": "@Konrad Banachewicz, treating time-series doesn't mean you cannot apply n-fold strategy. In fact, a more accurate estimation of time-series (or any sequence data where samples are dependent either on time or other order) could be \"leave-one-out \" or \"leave-n-out\"where you could leave n future samples for testing intentionally or incrementally if historical data give power of prediction. Because it is really time consuming, most people won't do such LOO or LnO CV with such big data.
\n\nMaybe some statics guru can shed some light about conducting time series CV effectively.
\n\nPlease see this post for more explanation: https://robjhyndman.com/hyndsight/tscv/
", "rawMarkdown": "@Konrad Banachewicz, treating time-series doesn't mean you cannot apply n-fold strategy. In fact, a more accurate estimation of time-series (or any sequence data where samples are dependent either on time or other order) could be \"leave-one-out \" or \"leave-n-out\"where you could leave n future samples for testing intentionally or incrementally if historical data give power of prediction. Because it is really time consuming, most people won't do such LOO or LnO CV with such big data.\n\nMaybe some statics guru can shed some light about conducting time series CV effectively. \n\nPlease see this post for more explanation: https://robjhyndman.com/hyndsight/tscv/" } ] }, { "id": 327057, "postDate": "2018-05-10T18:53:14.510Z", "content": "I tried 8:2 random sample many times with different seed, but Validation loss didn't change more than 0.0001.\nIt seems that the train data well distributed, not same as testset.\nWhat's more, as the loss going to be lower, the gap seems to be lower. but still shows difference about 0.005.(bcuz my LB is still high)\n
\n| model || Val_loss | LB |
\n| lgb | | 0.2351 | 0.2412 |
\n| NN | | 0.2588 | 0.3032 |
\n| NN | | 0.2475 | 0.2537 |
\n| NN | | 0.2430 | 0.2488 |
\n| NN | | 0.2407 | 0.2459 |
\nref : https://www.kaggle.com/dhznsdl/nn-model-adding-variables-step-by-step/code
Anyone treating it as a time series problem?
", "rawMarkdown": "Anyone treating it as a time series problem?", "replies": [ { "id": 326192, "postDate": "2018-05-09T12:35:18.880Z", "content": "Branden Murray mentioned he used activation_date to create train-validation split. Could be worth trying, athough I believe cross-validation would be a better approach this competition.
\n\nSee: https://www.kaggle.com/c/avito-demand-prediction/discussion/55587
", "rawMarkdown": "Branden Murray mentioned he used activation_date to create train-validation split. Could be worth trying, athough I believe cross-validation would be a better approach this competition.\n\nSee: https://www.kaggle.com/c/avito-demand-prediction/discussion/55587", "votes": 2 }, { "id": 326193, "postDate": "2018-05-09T12:35:54.933Z", "content": "I tried, but the lousy result is what prompted me to ask the original question...
", "rawMarkdown": "I tried, but the lousy result is what prompted me to ask the original question...", "votes": 3 }, { "id": 326398, "postDate": "2018-05-09T16:30:00.007Z", "content": "I'm more worried about this now., as we now know that the leaderboard is split by time with the public leaderboard coming from April 12th and 13th, and the private leaderboard coming from April 14-18th. But we also know that deal_probability seems to be quite stable across time... if this is a time series problem, it's a very stationary one. So it's unclear to me if we need to do time series or not.
We have 10 random KFold and constant difference between CV and LB near 0.005.
", "votes": 15, "replies": [ { "id": 325777, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-05-08T21:05:39.373000", "content": "Interesting - I will give it a try, then. Thanks!
", "votes": 1, "replies": [] }, { "id": 325879, "author_name": "Victor An", "author_url": "", "post_date": "2018-05-09T02:13:51.677000", "content": "Seems perfect
", "votes": 0, "replies": [] }, { "id": 326610, "author_name": "Angus Chang", "author_url": "", "post_date": "2018-05-10T02:39:03.840000", "content": "same here, 5 random KFold and difference between CV and LB about 0.004-0.005, but when I performed stacking or using optim function, the difference is up to 0.006-0.008, weird.
", "votes": 0, "replies": [] }, { "id": 326615, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-05-10T03:15:57.927000", "content": "@Andrew-Chang: That's pretty normal in my experience as usually stacking overfits a little bit more.
", "votes": 0, "replies": [] }, { "id": 326619, "author_name": "Angus Chang", "author_url": "", "post_date": "2018-05-10T03:20:56.703000", "content": "You're right, but when I use k-folds and optim() function to determine the weight of every model, it also leads to big difference. It really confused me.
", "votes": 0, "replies": [] }, { "id": 326820, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-05-10T11:40:40.640000", "content": "Update: so a 5-fold cv seems to give a nice and stable 0.005 difference between public and private LB. Thanks for the tip @Sergei.
", "votes": 9, "replies": [] }, { "id": 348635, "author_name": "Rene Wang", "author_url": "", "post_date": "2018-06-27T03:10:23.683000", "content": "@Peter Hurfold, does it mean for the stacking model using 10 fold cross validation might lead to more precise estimation than 5 fold?
\nAnd is it comparable for model using 5 fold CV and 10 fold? Just curious on how to find a best way to make the best use of CV. Thanks!
10 fold would be more precise than 5 fold, but maybe not by much. And it would be twice as time consuming. I personally stick with 5 fold, but I know some purists who do otherwise. As long as you're consistent throughout, you should be fine. :)
", "votes": 1, "replies": [] }, { "id": 348642, "author_name": "Rene Wang", "author_url": "", "post_date": "2018-06-27T03:33:45.167000", "content": "Maybe another question from some purists might ask: usually how much the difference between CV and LB can be tolerated for the private LB?
", "votes": 0, "replies": [] }, { "id": 348651, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-06-27T03:53:27.997000", "content": "It depends on the competition, which is why it is great to ask other people what they are experiencing and see if yours is similar.
", "votes": 1, "replies": [] }, { "id": 348652, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-06-27T03:53:58.873000", "content": "Another important aspect is how much variability there is in your gap between your submissions. The more consistency, the safer you can feel.
", "votes": 1, "replies": [] }, { "id": 348653, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-06-27T03:54:23.890000", "content": "Generally, you should trust your CV score most because it's based on the most data. Having a time-ordered LB can undermine that trust some, though.
", "votes": 0, "replies": [] } ] }, { "id": 325778, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-05-08T21:06:14.847000", "content": "I really like seeing this thread so I can see if I'm overfitting. :) Right now, I'm doing 5-fold CV and have a consistent gap between CV and LB between 0.0031 and 0.0038. I tried a time-based validation strategy once but it made my score worse.
", "votes": 9, "replies": [] }, { "id": 326945, "author_name": "Dimitris Leventis", "author_url": "", "post_date": "2018-05-10T14:54:17.447000", "content": "Be careful with the last days of training data set:
\n\ntraining[(training['activation_date']>=pd.to_datetime('2017-03-29'))].shape
\n\n(100, 17)
\n\ntraining[(training['activation_date']>=pd.to_datetime('2017-03-27'))].shape
\n\n(227848, 17)
\n\nOnly 100 samples after 2017-03-29.
", "votes": 3, "replies": [ { "id": 327002, "author_name": "Sohaib Omar", "author_url": "", "post_date": "2018-05-10T16:37:30.413000", "content": "similar pattern can be seen in last days of test set.
\n\n2017-04-12 81824\n2017-04-13 77176\n2017-04-14 70366\n2017-04-15 58793\n2017-04-16 58909\n2017-04-17 80191\n2017-04-18 81114\n2017-04-19 64\n2017-04-20 1\nI am using this idea of stratified KFold on binned target:
\n\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\n\ndef stratified_train_valid_split(meta_train, target_column, target_bins, valid_size, random_state=1234):\n\n y = meta_train[target_column].values\n bins = np.linspace(0, y.shape[0], target_bins)\n y_binned = np.digitize(y, bins)\n\n return train_test_split(meta_train, test_size=valid_size, stratify=y_binned, random_state=random_state)\nIt seems that the target distribution in train/valid is similiar. So far only tested it on bad models but it seems to correlate with the LB improvements.
\n\nWhat do you think of this approach?
", "votes": 4, "replies": [ { "id": 330811, "author_name": "ianchute", "author_url": "", "post_date": "2018-05-19T19:06:12.543000", "content": "I'm doing the same thing, but I'm getting very different LB results. How similar are yours?
", "votes": 0, "replies": [] }, { "id": 332461, "author_name": "Jakub Czakon", "author_url": "", "post_date": "2018-05-23T07:23:37.900000", "content": "Finally ran new models, and it seems that 0.005 delta is what this approach gets me.
", "votes": 0, "replies": [] } ] }, { "id": 336491, "author_name": "Lem Ko", "author_url": "", "post_date": "2018-05-31T20:37:14.747000", "content": "I'm using March 26 - 28 as my validation set. Like most people here, I'm also getting ~0.005 delta. I'm still starting out though so not sure how well this will hold.
\n\n
That's a cool sheet. Thanks for sharing.
", "votes": 1, "replies": [] } ] }, { "id": 336350, "author_name": "Yifan Xie", "author_url": "", "post_date": "2018-05-31T13:43:33.810000", "content": "I am on about 0.004 CV/LB delta, but I am only starting out - will be interesting to see how this gap will develop as I add more magic/non-magic into the mix :)
", "votes": 0, "replies": [ { "id": 336361, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-05-31T14:09:34.777000", "content": "Now that I've added features from train_active / test_active and the images, I've found my gap has widened to ~0.0043.
", "votes": 0, "replies": [] } ] }, { "id": 331960, "author_name": "AlexTru", "author_url": "", "post_date": "2018-05-22T08:50:05.693000", "content": "I use valid set as 10% random split from train set. It gives me ~0.0034 CV-LB delta
", "votes": 0, "replies": [] }, { "id": 331033, "author_name": "bluetrain", "author_url": "", "post_date": "2018-05-20T08:11:21.160000", "content": "I confirm that using 5-fold random split results in a ~0.005 separation wrt LB. I tried decreasing and increasing the number of folds but 5 seems to be the optimal choice (also considering time).
\nI am also using the last 20% of the train dataset as validation, which results in a slightly smaller shift, around ~0.004
\n\n\njtrane87 wrote
\n \n\n\nI confirm that using 5-fold random split results in a ~0.005
\n
It may depend on the model
\n\n5-fold random split results in a 0.002x-0.003 CV-LB delta for me
", "votes": 1, "replies": [] }, { "id": 331044, "author_name": "bluetrain", "author_url": "", "post_date": "2018-05-20T08:34:54.263000", "content": "Sure, it may depend on the model. Let's say: these are my observations so far.
", "votes": 0, "replies": [] } ] }, { "id": 328702, "author_name": "Axel Straminsky", "author_url": "", "post_date": "2018-05-14T23:40:17.410000", "content": "I don't think random spliting (or CV) would be a good approach, because you would be using future data to predict the past, which most of the time gives a more optimistic accuracy.
\n\nhttp://www.fast.ai/2017/11/13/validation-sets/
", "votes": 0, "replies": [ { "id": 328712, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-05-15T00:08:55.970000", "content": "In principle I agree, but: 1. this particular series seems to be pretty stationary 2. k-fold cv gives a more consistent estimate (smaller gap with public lb) then the time-based split.
", "votes": 2, "replies": [] }, { "id": 328713, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-05-15T00:09:01.223000", "content": "That's not always the case... Think about it this way. Every Kaggle problem is technically a time series problem, because the data was collected over time ...we just don't usually get to see the timestamps. What matters is not whether there is a timestamp or time-order to the data, but whether the target actually changes over time in a meaningful way. In this competition, it does not look like there is a reason so far to think that the time order matters. (But I'm still pretty scared of this and will keep hunting!)
", "votes": 11, "replies": [] }, { "id": 329099, "author_name": "", "author_url": "", "post_date": "2018-05-15T17:57:23.560000", "content": "", "votes": 0, "replies": [] }, { "id": 348647, "author_name": "Rene Wang", "author_url": "", "post_date": "2018-06-27T03:48:53.650000", "content": "@Konrad Banachewicz, treating time-series doesn't mean you cannot apply n-fold strategy. In fact, a more accurate estimation of time-series (or any sequence data where samples are dependent either on time or other order) could be \"leave-one-out \" or \"leave-n-out\"where you could leave n future samples for testing intentionally or incrementally if historical data give power of prediction. Because it is really time consuming, most people won't do such LOO or LnO CV with such big data.
\n\nMaybe some statics guru can shed some light about conducting time series CV effectively.
\n\nPlease see this post for more explanation: https://robjhyndman.com/hyndsight/tscv/
", "votes": 0, "replies": [] } ] }, { "id": 327057, "author_name": "YeongTaek Oh", "author_url": "", "post_date": "2018-05-10T18:53:14.510000", "content": "I tried 8:2 random sample many times with different seed, but Validation loss didn't change more than 0.0001.\nIt seems that the train data well distributed, not same as testset.\nWhat's more, as the loss going to be lower, the gap seems to be lower. but still shows difference about 0.005.(bcuz my LB is still high)\n
\n| model || Val_loss | LB |
\n| lgb | | 0.2351 | 0.2412 |
\n| NN | | 0.2588 | 0.3032 |
\n| NN | | 0.2475 | 0.2537 |
\n| NN | | 0.2430 | 0.2488 |
\n| NN | | 0.2407 | 0.2459 |
\nref : https://www.kaggle.com/dhznsdl/nn-model-adding-variables-step-by-step/code
Anyone treating it as a time series problem?
", "votes": 0, "replies": [ { "id": 326192, "author_name": "Kevin", "author_url": "", "post_date": "2018-05-09T12:35:18.880000", "content": "Branden Murray mentioned he used activation_date to create train-validation split. Could be worth trying, athough I believe cross-validation would be a better approach this competition.
\n\nSee: https://www.kaggle.com/c/avito-demand-prediction/discussion/55587
", "votes": 2, "replies": [] }, { "id": 326193, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-05-09T12:35:54.933000", "content": "I tried, but the lousy result is what prompted me to ask the original question...
", "votes": 3, "replies": [] }, { "id": 326398, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-05-09T16:30:00.007000", "content": "I'm more worried about this now., as we now know that the leaderboard is split by time with the public leaderboard coming from April 12th and 13th, and the private leaderboard coming from April 14-18th. But we also know that deal_probability seems to be quite stable across time... if this is a time series problem, it's a very stationary one. So it's unclear to me if we need to do time series or not.