{
  "id": 541687,
  "title": "How are you guys preprocessing data",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/541687",
  "author_name": "",
  "post_date": "2024-10-20T22:25:27.605296400Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi guys! I am trying to preprocess the dataset, but even when loading 3 of the 10 total parquet files, I get ETAs of over 80 hours. I am just formatting the data in a way that is easier for time series prediction(a list of symbols each with a list of features for each timestep). Am I doing something wrong?</p>",
  "messages": [
    {
      "id": "3023719",
      "postDate": "10/20/2024 22:25:27",
      "content": "<p>Hi guys! I am trying to preprocess the dataset, but even when loading 3 of the 10 total parquet files, I get ETAs of over 80 hours. I am just formatting the data in a way that is easier for time series prediction(a list of symbols each with a list of features for each timestep). Am I doing something wrong?</p>",
      "rawMarkdown": "Hi guys! I am trying to preprocess the dataset, but even when loading 3 of the 10 total parquet files, I get ETAs of over 80 hours. I am just formatting the data in a way that is easier for time series prediction(a list of symbols each with a list of features for each timestep). Am I doing something wrong?",
      "votes": null
    },
    {
      "id": "3023732",
      "postDate": "10/20/2024 23:08:36",
      "content": "<p>Use Polars or Dask, better to load it in lazyframe.</p>",
      "rawMarkdown": "Use Polars or Dask, better to load it in lazyframe.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3023732,
      "author_name": "jayshrivastava",
      "author_url": "",
      "post_date": "10/20/2024 23:08:36",
      "content": "<p>Use Polars or Dask, better to load it in lazyframe.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3023719": "Hi guys! I am trying to preprocess the dataset, but even when loading 3 of the 10 total parquet files, I get ETAs of over 80 hours. I am just formatting the data in a way that is easier for time series prediction(a list of symbols each with a list of features for each timestep). Am I doing something wrong?",
    "3023732": "Use Polars or Dask, better to load it in lazyframe."
  },
  "source": "meta"
}