{
  "id": 541338,
  "title": "Is partitioning only for convenient data storage, or could it have other possible meanings？",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/541338",
  "author_name": "",
  "post_date": "2024-10-18T21:03:33.820442500Z",
  "votes": 1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I noticed that the date_id is continuous, but the maximum time_id varies at some date across different partitions. </p>",
  "messages": [
    {
      "id": "3021792",
      "postDate": "10/18/2024 21:03:33",
      "content": "<p>I noticed that the date_id is continuous, but the maximum time_id varies at some date across different partitions. </p>",
      "rawMarkdown": "I noticed that the date_id is continuous, but the maximum time_id varies at some date across different partitions.",
      "votes": null
    },
    {
      "id": "3021797",
      "postDate": "10/18/2024 21:07:18",
      "content": "<p>I think one can save the data with a partition_id like the way the host did here - I don't see any other intent here <a href=\"https://www.kaggle.com/sumenzhang\" target=\"_blank\">@sumenzhang</a> </p>",
      "rawMarkdown": "I think one can save the data with a partition_id like the way the host did here - I don't see any other intent here @sumenzhang",
      "votes": null
    },
    {
      "id": "3021799",
      "postDate": "10/18/2024 21:15:29",
      "content": "<p>Thank you for your input. I guess so as well since every partition seems to have the same number of date_ids, which suggests the partitioning is likely just for storage purposes.</p>",
      "rawMarkdown": "Thank you for your input. I guess so as well since every partition seems to have the same number of date_ids, which suggests the partitioning is likely just for storage purposes.",
      "votes": null
    },
    {
      "id": "3061961",
      "postDate": "12/03/2024 06:56:38",
      "content": "<p>Each partition contains about 170 days. <br>\ni.e. partition 1 first contains first 170 days, last partition contains last 169 days, or so.</p>\n<p>It would probably be interesting to find out what 170 days means, if anything; <br>\nI see a debate on time range <a href=\"https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/549223\" target=\"_blank\">https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/549223</a><br>\nMaybe there is a debate on partition days, too.</p>",
      "rawMarkdown": "Each partition contains about 170 days. \ni.e. partition 1 first contains first 170 days, last partition contains last 169 days, or so.\n\nIt would probably be interesting to find out what 170 days means, if anything; \nI see a debate on time range https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/549223\nMaybe there is a debate on partition days, too.",
      "votes": null
    },
    {
      "id": "3063491",
      "postDate": "12/04/2024 14:54:19",
      "content": "<p>At work, if I add a partition_by on a table, and then I query that table, it is almost always faster. Even if I don't filter or select anything related to the partitioned column. </p>",
      "rawMarkdown": "At work, if I add a partition_by on a table, and then I query that table, it is almost always faster. Even if I don't filter or select anything related to the partitioned column.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3021797,
      "author_name": "ravi20076",
      "author_url": "",
      "post_date": "10/18/2024 21:07:18",
      "content": "<p>I think one can save the data with a partition_id like the way the host did here - I don't see any other intent here <a href=\"https://www.kaggle.com/sumenzhang\" target=\"_blank\">@sumenzhang</a> </p>",
      "votes": null,
      "replies": [
        {
          "id": 3021799,
          "author_name": "sumenzhang",
          "author_url": "",
          "post_date": "10/18/2024 21:15:29",
          "content": "<p>Thank you for your input. I guess so as well since every partition seems to have the same number of date_ids, which suggests the partitioning is likely just for storage purposes.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3061961,
      "author_name": "icozma",
      "author_url": "",
      "post_date": "12/03/2024 06:56:38",
      "content": "<p>Each partition contains about 170 days. <br>\ni.e. partition 1 first contains first 170 days, last partition contains last 169 days, or so.</p>\n<p>It would probably be interesting to find out what 170 days means, if anything; <br>\nI see a debate on time range <a href=\"https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/549223\" target=\"_blank\">https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/549223</a><br>\nMaybe there is a debate on partition days, too.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3063491,
      "author_name": "romandovega",
      "author_url": "",
      "post_date": "12/04/2024 14:54:19",
      "content": "<p>At work, if I add a partition_by on a table, and then I query that table, it is almost always faster. Even if I don't filter or select anything related to the partitioned column. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3021792": "I noticed that the date_id is continuous, but the maximum time_id varies at some date across different partitions.",
    "3021797": "I think one can save the data with a partition_id like the way the host did here - I don't see any other intent here @sumenzhang",
    "3021799": "Thank you for your input. I guess so as well since every partition seems to have the same number of date_ids, which suggests the partitioning is likely just for storage purposes.",
    "3061961": "Each partition contains about 170 days. \ni.e. partition 1 first contains first 170 days, last partition contains last 169 days, or so.\n\nIt would probably be interesting to find out what 170 days means, if anything; \nI see a debate on time range https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/549223\nMaybe there is a debate on partition days, too.",
    "3063491": "At work, if I add a partition_by on a table, and then I query that table, it is almost always faster. Even if I don't filter or select anything related to the partitioned column."
  },
  "source": "meta"
}