{ "id": 90155, "title": "Time Serias Cross Validation", "url": "/competitions/LANL-Earthquake-Prediction/discussion/90155", "author_name": "", "post_date": "2019-04-21T07:30:14.824994200Z", "votes": 4, "comment_count": 11, "views": 0, "content": "

Many of the kernels (that I saw) used the K-Fold cross-validation method, but in my opinion, this is not the optimal cross-validation strategy for time series. In my opinion, walk-forward can produce more accurate results and prevent 'shake up' in a private dataset. What cross-validation methods are used in your solution and why?

", "messages": [ { "id": "520523", "postDate": "04/21/2019 07:30:14", "content": "

Many of the kernels (that I saw) used the K-Fold cross-validation method, but in my opinion, this is not the optimal cross-validation strategy for time series. In my opinion, walk-forward can produce more accurate results and prevent 'shake up' in a private dataset. What cross-validation methods are used in your solution and why?

", "rawMarkdown": "Many of the kernels (that I saw) used the K-Fold cross-validation method, but in my opinion, this is not the optimal cross-validation strategy for time series. In my opinion, [walk-forward](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html#sklearn.model_selection.TimeSeriesSplit) can produce more accurate results and prevent 'shake up' in a private dataset. What cross-validation methods are used in your solution and why?", "votes": null }, { "id": "520537", "postDate": "04/21/2019 08:17:08", "content": "

Is this a time series really? Does the state depend on what happened before the last quake? I wish I knew the answer to that question.

", "rawMarkdown": "Is this a time series really? Does the state depend on what happened before the last quake? I wish I knew the answer to that question.", "votes": null }, { "id": "520551", "postDate": "04/21/2019 09:25:04", "content": "

well, from the first place it seems like a timeseries problem, but in the end you have to predict the target variable of test segments which are shuffled 150k rows batches from the same experiment. So, in my mind Kfold is more viable in this context. You do not know if each segment id is 1, 2, 3, or n batches later than the last segment of training segment

", "rawMarkdown": "well, from the first place it seems like a timeseries problem, but in the end you have to predict the target variable of test segments which are shuffled 150k rows batches from the same experiment. So, in my mind Kfold is more viable in this context. You do not know if each segment id is 1, 2, 3, or n batches later than the last segment of training segment", "votes": null }, { "id": "520552", "postDate": "04/21/2019 09:28:02", "content": "

In my opinion, yes, these are time series. Because the current state depends on the previous state. Short-term dependence is obvious. But I cannot say the same about long-term addiction, for me it is not obvious.

", "rawMarkdown": "In my opinion, yes, these are time series. Because the current state depends on the previous state. Short-term dependence is obvious. But I cannot say the same about long-term addiction, for me it is not obvious.", "votes": null }, { "id": "520554", "postDate": "04/21/2019 09:31:11", "content": "

in my discussion thread \nhttps://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/89824#latest-519888\ni see that all my models can predict the upcoming new earthquakes much earlier than they happen. But this cannot be applied to the test set

", "rawMarkdown": "in my discussion thread \nhttps://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/89824#latest-519888\ni see that all my models can predict the upcoming new earthquakes much earlier than they happen. But this cannot be applied to the test set", "votes": null }, { "id": "520563", "postDate": "04/21/2019 09:59:00", "content": "

In their papers, the authors state that there is a small but noticable drift of the labquake setup during the entire test period due to the material being altered by the test itself.

", "rawMarkdown": "In their papers, the authors state that there is a small but noticable drift of the labquake setup during the entire test period due to the material being altered by the test itself.", "votes": null }, { "id": "520573", "postDate": "04/21/2019 10:19:45", "content": "

Thanks Ilu, that is what I was looking for.

\n\n

I should have been clearer in my question: is it one time series or a set of independent time series? Your comment says it is one time series.

\n\n

I should read their papers...

", "rawMarkdown": "Thanks Ilu, that is what I was looking for.\n\nI should have been clearer in my question: is it one time series or a set of independent time series? Your comment says it is one time series. \n\nI should read their papers...", "votes": null }, { "id": "521140", "postDate": "04/22/2019 12:10:42", "content": "

As we have no chance of ordering the test sets, I personally wouldn't try to model the long term drift.

", "rawMarkdown": "As we have no chance of ordering the test sets, I personally wouldn't try to model the long term drift.", "votes": null }, { "id": "521154", "postDate": "04/22/2019 12:53:17", "content": "
\n

I personally wouldn't try to model the long term drift.

\n
\n\n

We don't even know if test is after or before train...

", "rawMarkdown": "> I personally wouldn't try to model the long term drift. \n\nWe don't even know if test is after or before train...", "votes": null }, { "id": "521215", "postDate": "04/22/2019 14:59:41", "content": "

Exactly!

\n\n

Also, the test segments are not ordered so unless something changes I think treating them as independent is important.

\n\n

They may also not be contiguous.

\n\n

Train is one long recording, but test may not be. They did stress that these quakes are aperiodic as well, compared to what they studied in their papers, so finding temporal dependence here should be harder or less important.

", "rawMarkdown": "Exactly! \n\nAlso, the test segments are not ordered so unless something changes I think treating them as independent is important. \n\nThey may also not be contiguous. \n\nTrain is one long recording, but test may not be. They did stress that these quakes are aperiodic as well, compared to what they studied in their papers, so finding temporal dependence here should be harder or less important.", "votes": null }, { "id": "521416", "postDate": "04/22/2019 21:56:11", "content": "
\n

Train is one long recording, but test may not be. They did stress that these quakes are aperiodic as well, compared to what they studied in their papers, so finding temporal dependence here should be harder or less important.

\n
\n\n

from what I've read in the papers, there shouldn't be much of an issue.

\n\n
\n

Figure 2, the RF still does an excellent job in predicting failure time, showing that the approach can be generalized to aperiodic fault cycles. - Machine Learning Predicts Laboratory Earthquakes 2017

\n
", "rawMarkdown": "> Train is one long recording, but test may not be. They did stress that these quakes are aperiodic as well, compared to what they studied in their papers, so finding temporal dependence here should be harder or less important.\n\nfrom what I've read in the papers, there shouldn't be much of an issue. \n\n> Figure 2, the RF still does an excellent job in predicting failure time, showing that the approach can be generalized to aperiodic fault cycles. - Machine Learning Predicts Laboratory Earthquakes 2017", "votes": null }, { "id": "521522", "postDate": "04/23/2019 01:54:38", "content": "

@miklgr500, so how would you do walk-forward validation? Will you use each earthquake period/group to train and validate?

", "rawMarkdown": "miklgr500, so how would you do walk-forward validation? Will you use each earthquake period/group to train and validate?", "votes": null } ], "comments": [ { "id": 520537, "author_name": "cpmpml", "author_url": "", "post_date": "04/21/2019 08:17:08", "content": "

Is this a time series really? Does the state depend on what happened before the last quake? I wish I knew the answer to that question.

", "votes": null, "replies": [ { "id": 520552, "author_name": "miklgr500", "author_url": "", "post_date": "04/21/2019 09:28:02", "content": "

In my opinion, yes, these are time series. Because the current state depends on the previous state. Short-term dependence is obvious. But I cannot say the same about long-term addiction, for me it is not obvious.

", "votes": null, "replies": [] }, { "id": 520554, "author_name": "dkaraflos", "author_url": "", "post_date": "04/21/2019 09:31:11", "content": "

in my discussion thread \nhttps://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/89824#latest-519888\ni see that all my models can predict the upcoming new earthquakes much earlier than they happen. But this cannot be applied to the test set

", "votes": null, "replies": [] }, { "id": 520563, "author_name": "ilu000", "author_url": "", "post_date": "04/21/2019 09:59:00", "content": "

In their papers, the authors state that there is a small but noticable drift of the labquake setup during the entire test period due to the material being altered by the test itself.

", "votes": null, "replies": [] }, { "id": 520573, "author_name": "cpmpml", "author_url": "", "post_date": "04/21/2019 10:19:45", "content": "

Thanks Ilu, that is what I was looking for.

\n\n

I should have been clearer in my question: is it one time series or a set of independent time series? Your comment says it is one time series.

\n\n

I should read their papers...

", "votes": null, "replies": [] }, { "id": 521140, "author_name": "ilu000", "author_url": "", "post_date": "04/22/2019 12:10:42", "content": "

As we have no chance of ordering the test sets, I personally wouldn't try to model the long term drift.

", "votes": null, "replies": [] }, { "id": 521154, "author_name": "cpmpml", "author_url": "", "post_date": "04/22/2019 12:53:17", "content": "
\n

I personally wouldn't try to model the long term drift.

\n
\n\n

We don't even know if test is after or before train...

", "votes": null, "replies": [] }, { "id": 521215, "author_name": "interneuron", "author_url": "", "post_date": "04/22/2019 14:59:41", "content": "

Exactly!

\n\n

Also, the test segments are not ordered so unless something changes I think treating them as independent is important.

\n\n

They may also not be contiguous.

\n\n

Train is one long recording, but test may not be. They did stress that these quakes are aperiodic as well, compared to what they studied in their papers, so finding temporal dependence here should be harder or less important.

", "votes": null, "replies": [] }, { "id": 521416, "author_name": "teeyee314", "author_url": "", "post_date": "04/22/2019 21:56:11", "content": "
\n

Train is one long recording, but test may not be. They did stress that these quakes are aperiodic as well, compared to what they studied in their papers, so finding temporal dependence here should be harder or less important.

\n
\n\n

from what I've read in the papers, there shouldn't be much of an issue.

\n\n
\n

Figure 2, the RF still does an excellent job in predicting failure time, showing that the approach can be generalized to aperiodic fault cycles. - Machine Learning Predicts Laboratory Earthquakes 2017

\n
", "votes": null, "replies": [] } ] }, { "id": 520551, "author_name": "dkaraflos", "author_url": "", "post_date": "04/21/2019 09:25:04", "content": "

well, from the first place it seems like a timeseries problem, but in the end you have to predict the target variable of test segments which are shuffled 150k rows batches from the same experiment. So, in my mind Kfold is more viable in this context. You do not know if each segment id is 1, 2, 3, or n batches later than the last segment of training segment

", "votes": null, "replies": [] }, { "id": 521522, "author_name": "pukkinming", "author_url": "", "post_date": "04/23/2019 01:54:38", "content": "

@miklgr500, so how would you do walk-forward validation? Will you use each earthquake period/group to train and validate?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "520523": "Many of the kernels (that I saw) used the K-Fold cross-validation method, but in my opinion, this is not the optimal cross-validation strategy for time series. In my opinion, [walk-forward](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html#sklearn.model_selection.TimeSeriesSplit) can produce more accurate results and prevent 'shake up' in a private dataset. What cross-validation methods are used in your solution and why?", "520537": "Is this a time series really? Does the state depend on what happened before the last quake? I wish I knew the answer to that question.", "520551": "well, from the first place it seems like a timeseries problem, but in the end you have to predict the target variable of test segments which are shuffled 150k rows batches from the same experiment. So, in my mind Kfold is more viable in this context. You do not know if each segment id is 1, 2, 3, or n batches later than the last segment of training segment", "520552": "In my opinion, yes, these are time series. Because the current state depends on the previous state. Short-term dependence is obvious. But I cannot say the same about long-term addiction, for me it is not obvious.", "520554": "in my discussion thread \nhttps://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/89824#latest-519888\ni see that all my models can predict the upcoming new earthquakes much earlier than they happen. But this cannot be applied to the test set", "520563": "In their papers, the authors state that there is a small but noticable drift of the labquake setup during the entire test period due to the material being altered by the test itself.", "520573": "Thanks Ilu, that is what I was looking for.\n\nI should have been clearer in my question: is it one time series or a set of independent time series? Your comment says it is one time series. \n\nI should read their papers...", "521140": "As we have no chance of ordering the test sets, I personally wouldn't try to model the long term drift.", "521154": "> I personally wouldn't try to model the long term drift. \n\nWe don't even know if test is after or before train...", "521215": "Exactly! \n\nAlso, the test segments are not ordered so unless something changes I think treating them as independent is important. \n\nThey may also not be contiguous. \n\nTrain is one long recording, but test may not be. They did stress that these quakes are aperiodic as well, compared to what they studied in their papers, so finding temporal dependence here should be harder or less important.", "521416": "> Train is one long recording, but test may not be. They did stress that these quakes are aperiodic as well, compared to what they studied in their papers, so finding temporal dependence here should be harder or less important.\n\nfrom what I've read in the papers, there shouldn't be much of an issue. \n\n> Figure 2, the RF still does an excellent job in predicting failure time, showing that the approach can be generalized to aperiodic fault cycles. - Machine Learning Predicts Laboratory Earthquakes 2017", "521522": "miklgr500, so how would you do walk-forward validation? Will you use each earthquake period/group to train and validate?" }, "source": "meta" }