{
  "id": 580240,
  "title": "irregular gaps: How much of the problem is it?",
  "url": "/competitions/drw-crypto-market-prediction/discussion/580240",
  "author_name": "",
  "post_date": "2025-05-23T09:50:09.529046400Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi 🙋, </p>\n<p>I noticed that most of the time interval records are regular, however there are a few irregular gaps. </p>\n<p><code>print(train.index.to_series().diff().value_counts())</code></p>\n<pre><code>timestamp\n days :01:    \n days :02:       \n days :03:        \n days :04:        \n days :08:        \n days :06:        \n days :05:        \n days :07:         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days :09:         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n</code></pre>\n<p>looking at this, most (525613) of the data points are at 1 sec interval. However, gaps as long as 00:35:00 exist. </p>\n<p>Most of the irregular gaps are around a few moths. </p>\n<p><code>gaps = train.index.to_series().diff()\ngaps[gaps &gt; pd.Timedelta('1min')].plot(figsize=(15, 4), title='Time Gaps Over Time')</code> </p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F115891%2F52392e7038bc4457f470713b5dd823b0%2FScreenshot%202025-05-23%20110245.png?generation=1747994637841946&amp;alt=media\" alt=\"\"></p>\n<p>Though rare, but how much of the problem they pose in terms of accurate modelling?</p>\n<p>Thanks. </p>",
  "messages": [
    {
      "id": "3207835",
      "postDate": "05/23/2025 09:50:09",
      "content": "<p>Hi 🙋, </p>\n<p>I noticed that most of the time interval records are regular, however there are a few irregular gaps. </p>\n<p><code>print(train.index.to_series().diff().value_counts())</code></p>\n<pre><code>timestamp\n days :01:    \n days :02:       \n days :03:        \n days :04:        \n days :08:        \n days :06:        \n days :05:        \n days :07:         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days :09:         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n days ::         \n</code></pre>\n<p>looking at this, most (525613) of the data points are at 1 sec interval. However, gaps as long as 00:35:00 exist. </p>\n<p>Most of the irregular gaps are around a few moths. </p>\n<p><code>gaps = train.index.to_series().diff()\ngaps[gaps &gt; pd.Timedelta('1min')].plot(figsize=(15, 4), title='Time Gaps Over Time')</code> </p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F115891%2F52392e7038bc4457f470713b5dd823b0%2FScreenshot%202025-05-23%20110245.png?generation=1747994637841946&amp;alt=media\" alt=\"\"></p>\n<p>Though rare, but how much of the problem they pose in terms of accurate modelling?</p>\n<p>Thanks. </p>",
      "rawMarkdown": "Hi 🙋, \n\nI noticed that most of the time interval records are regular, however there are a few irregular gaps. \n\n`print(train.index.to_series().diff().value_counts()) `\n\n```python\ntimestamp\n0 days 00:01:00    525613\n0 days 00:02:00       117\n0 days 00:03:00        56\n0 days 00:04:00        18\n0 days 00:08:00        14\n0 days 00:06:00        13\n0 days 00:05:00        11\n0 days 00:07:00         7\n0 days 00:15:00         4\n0 days 00:20:00         4\n0 days 00:11:00         4\n0 days 00:13:00         3\n0 days 00:12:00         3\n0 days 00:22:00         2\n0 days 00:18:00         2\n0 days 00:10:00         2\n0 days 00:14:00         2\n0 days 00:09:00         2\n0 days 00:27:00         2\n0 days 00:32:00         1\n0 days 00:35:00         1\n0 days 00:30:00         1\n0 days 00:33:00         1\n0 days 00:31:00         1\n0 days 00:21:00         1\n0 days 00:17:00         1\n```\nlooking at this, most (525613) of the data points are at 1 sec interval. However, gaps as long as 00:35:00 exist. \n\nMost of the irregular gaps are around a few moths. \n\n`gaps = train.index.to_series().diff()\ngaps[gaps > pd.Timedelta('1min')].plot(figsize=(15, 4), title='Time Gaps Over Time')` \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F115891%2F52392e7038bc4457f470713b5dd823b0%2FScreenshot%202025-05-23%20110245.png?generation=1747994637841946&alt=media)\n\n\nThough rare, but how much of the problem they pose in terms of accurate modelling?\n\nThanks.",
      "votes": null
    },
    {
      "id": "3208475",
      "postDate": "05/24/2025 07:39:07",
      "content": "<p>I would say that because the test set does not (at least officially) recover time ordering, this should not be a problem because I would not use time neither explicitly or implicitly in the training here. In other words, if you completely shuffle your training set, the model you use should give the same score on validation and test set.</p>",
      "rawMarkdown": "I would say that because the test set does not (at least officially) recover time ordering, this should not be a problem because I would not use time neither explicitly or implicitly in the training here. In other words, if you completely shuffle your training set, the model you use should give the same score on validation and test set.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3208475,
      "author_name": "quentinadatte1307",
      "author_url": "",
      "post_date": "05/24/2025 07:39:07",
      "content": "<p>I would say that because the test set does not (at least officially) recover time ordering, this should not be a problem because I would not use time neither explicitly or implicitly in the training here. In other words, if you completely shuffle your training set, the model you use should give the same score on validation and test set.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3207835": "Hi 🙋, \n\nI noticed that most of the time interval records are regular, however there are a few irregular gaps. \n\n`print(train.index.to_series().diff().value_counts()) `\n\n```python\ntimestamp\n0 days 00:01:00    525613\n0 days 00:02:00       117\n0 days 00:03:00        56\n0 days 00:04:00        18\n0 days 00:08:00        14\n0 days 00:06:00        13\n0 days 00:05:00        11\n0 days 00:07:00         7\n0 days 00:15:00         4\n0 days 00:20:00         4\n0 days 00:11:00         4\n0 days 00:13:00         3\n0 days 00:12:00         3\n0 days 00:22:00         2\n0 days 00:18:00         2\n0 days 00:10:00         2\n0 days 00:14:00         2\n0 days 00:09:00         2\n0 days 00:27:00         2\n0 days 00:32:00         1\n0 days 00:35:00         1\n0 days 00:30:00         1\n0 days 00:33:00         1\n0 days 00:31:00         1\n0 days 00:21:00         1\n0 days 00:17:00         1\n```\nlooking at this, most (525613) of the data points are at 1 sec interval. However, gaps as long as 00:35:00 exist. \n\nMost of the irregular gaps are around a few moths. \n\n`gaps = train.index.to_series().diff()\ngaps[gaps > pd.Timedelta('1min')].plot(figsize=(15, 4), title='Time Gaps Over Time')` \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F115891%2F52392e7038bc4457f470713b5dd823b0%2FScreenshot%202025-05-23%20110245.png?generation=1747994637841946&alt=media)\n\n\nThough rare, but how much of the problem they pose in terms of accurate modelling?\n\nThanks.",
    "3208475": "I would say that because the test set does not (at least officially) recover time ordering, this should not be a problem because I would not use time neither explicitly or implicitly in the training here. In other words, if you completely shuffle your training set, the model you use should give the same score on validation and test set."
  },
  "source": "meta"
}