{
  "id": 545015,
  "title": "Performance Tips?",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/545015",
  "author_name": "",
  "post_date": "2024-11-08T03:21:40.190416200Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Has anyone got any performance tips? I usually don't write Python and my submission is stupidly slow. I can't seem to get the ~150ms per prediction window to avoid a submission timeout.</p>\n<p><em>Note</em>: This is not an inference time problem. I profiled my code and the total time in inference for all symbols on a day is on average less than 0.3ms!<br>\n<code>profile:: model_predict = 0.00039995802217163146</code></p>\n<p>Profiling shows two main bottlenecks:<br>\n1.(t=~100ms):  Converting Polars dataframe to a <code>darts.TimeSeries</code> for use in predicting output. I actually managed to drop this from 130ms to about 90ms using a few tricks (not adding my own RangeIndex and also setting maintain_order=False on the partition_by) but am completely stuck on further optimisations</p>\n<pre><code>    \n    all_symbols = []\n     _, series  (df.partition_by(, maintain_order=)):\n        all_symbols.append(\n            TimeSeries.from_dataframe(\n                series.to_pandas(),\n                value_cols=,\n            ).astype(np.float32)\n        )\n</code></pre>\n<ol>\n<li>I keep track of the current time steps feature data along with my predictions so I can feed them in at t=t+1 to form part of the next prediction (e.g. each prediction has a lookback window of historical features + responders). </li>\n</ol>\n<pre><code>    predictions = predict_fn(_history, model)\n\n    \n    \n    \n    _history = _history.update(\n        predictions,\n        how=,\n        left_on=[],\n        right_on=[],\n    )\n</code></pre>\n<p>These two steps alone take ~125ms.</p>\n<p>I also for some reason have to recast my Polars dataframes a lot - possibly due to joining and concatenating not preserving the original datatypes? For example when concatenating the input features from one day to the next they often jump between Float32 and Float64. row_id also seems to jump between UInt32 and Int64 and a few other columns always seem to change type randomly. I have to recast datatypes 4-5 times per time step but the combined cost of these is ~3ms so I don't think its a big deal given the other bottlenecks I have.</p>\n<p><strong>Is the answer here to just not use Darts library?</strong></p>",
  "messages": [
    {
      "id": "3039417",
      "postDate": "11/08/2024 03:21:40",
      "content": "<p>Has anyone got any performance tips? I usually don't write Python and my submission is stupidly slow. I can't seem to get the ~150ms per prediction window to avoid a submission timeout.</p>\n<p><em>Note</em>: This is not an inference time problem. I profiled my code and the total time in inference for all symbols on a day is on average less than 0.3ms!<br>\n<code>profile:: model_predict = 0.00039995802217163146</code></p>\n<p>Profiling shows two main bottlenecks:<br>\n1.(t=~100ms):  Converting Polars dataframe to a <code>darts.TimeSeries</code> for use in predicting output. I actually managed to drop this from 130ms to about 90ms using a few tricks (not adding my own RangeIndex and also setting maintain_order=False on the partition_by) but am completely stuck on further optimisations</p>\n<pre><code>    \n    all_symbols = []\n     _, series  (df.partition_by(, maintain_order=)):\n        all_symbols.append(\n            TimeSeries.from_dataframe(\n                series.to_pandas(),\n                value_cols=,\n            ).astype(np.float32)\n        )\n</code></pre>\n<ol>\n<li>I keep track of the current time steps feature data along with my predictions so I can feed them in at t=t+1 to form part of the next prediction (e.g. each prediction has a lookback window of historical features + responders). </li>\n</ol>\n<pre><code>    predictions = predict_fn(_history, model)\n\n    \n    \n    \n    _history = _history.update(\n        predictions,\n        how=,\n        left_on=[],\n        right_on=[],\n    )\n</code></pre>\n<p>These two steps alone take ~125ms.</p>\n<p>I also for some reason have to recast my Polars dataframes a lot - possibly due to joining and concatenating not preserving the original datatypes? For example when concatenating the input features from one day to the next they often jump between Float32 and Float64. row_id also seems to jump between UInt32 and Int64 and a few other columns always seem to change type randomly. I have to recast datatypes 4-5 times per time step but the combined cost of these is ~3ms so I don't think its a big deal given the other bottlenecks I have.</p>\n<p><strong>Is the answer here to just not use Darts library?</strong></p>",
      "rawMarkdown": "Has anyone got any performance tips? I usually don't write Python and my submission is stupidly slow. I can't seem to get the ~150ms per prediction window to avoid a submission timeout.\n\n*Note*: This is not an inference time problem. I profiled my code and the total time in inference for all symbols on a day is on average less than 0.3ms!\n`profile:: model_predict = 0.00039995802217163146`\n\nProfiling shows two main bottlenecks:\n1.(t=~100ms):  Converting Polars dataframe to a `darts.TimeSeries` for use in predicting output. I actually managed to drop this from 130ms to about 90ms using a few tricks (not adding my own RangeIndex and also setting maintain_order=False on the partition_by) but am completely stuck on further optimisations\n\n```python\n    # TODO: SLOW!! ~110ms!!!\n    all_symbols = []\n    for _, series in enumerate(df.partition_by(\"symbol_id\", maintain_order=False)):\n        all_symbols.append(\n            TimeSeries.from_dataframe(\n                series.to_pandas(),\n                value_cols=\"responder_6\",\n            ).astype(np.float32)\n        )\n```\n\n2. I keep track of the current time steps feature data along with my predictions so I can feed them in at t=t+1 to form part of the next prediction (e.g. each prediction has a lookback window of historical features + responders). \n\n```python\n    predictions = predict_fn(_history, model)\n\n    # Join the predictions onto the lags DF for auto-regression (t+1 prediction)\n    # We then over-write these values with actual values on next days lags.\n    # TODO: SLOW ~25ms\n    _history = _history.update(\n        predictions,\n        how=\"left\",\n        left_on=[\"row_id\"],\n        right_on=[\"row_id\"],\n    )\n```\n\nThese two steps alone take ~125ms.\n\nI also for some reason have to recast my Polars dataframes a lot - possibly due to joining and concatenating not preserving the original datatypes? For example when concatenating the input features from one day to the next they often jump between Float32 and Float64. row_id also seems to jump between UInt32 and Int64 and a few other columns always seem to change type randomly. I have to recast datatypes 4-5 times per time step but the combined cost of these is ~3ms so I don't think its a big deal given the other bottlenecks I have.\n\n\n**Is the answer here to just not use Darts library?**",
      "votes": null
    },
    {
      "id": "3039421",
      "postDate": "11/08/2024 03:26:12",
      "content": "<p>I should also note that these benchmark times are from my local machine (M1 Max) that utilizes all 10 CPU cores at 100%. I can only imagine that the actual submission is a fair bit slower</p>",
      "rawMarkdown": "I should also note that these benchmark times are from my local machine (M1 Max) that utilizes all 10 CPU cores at 100%. I can only imagine that the actual submission is a fair bit slower",
      "votes": null
    },
    {
      "id": "3039428",
      "postDate": "11/08/2024 03:42:18",
      "content": "<h1>Update</h1>\n<p>I have sped the creation of time series up about 3x by moving to the <code>from_values</code> method and exporting my Polars df to a numpy array instead of a pandas dataframe. Possibly could have had similar results only exporting the columns I actually needed to a pandas dataframe too.</p>\n<pre><code>    \n    all_symbols = []\n     _, symbol_df  (df.partition_by(, maintain_order=)):\n        np_series = (\n            symbol_df.select().(- * lookback_window).to_numpy()\n        )\n\n        all_symbols.append(\n            TimeSeries.from_values(\n                values=np.pad(np_series, (lookback_window - (np_series), ))\n            ).astype(np.float32)\n        )\n</code></pre>",
      "rawMarkdown": "# Update\n\nI have sped the creation of time series up about 3x by moving to the `from_values` method and exporting my Polars df to a numpy array instead of a pandas dataframe. Possibly could have had similar results only exporting the columns I actually needed to a pandas dataframe too.\n\n```python\n    # Improved to ~30ms!\n    all_symbols = []\n    for _, symbol_df in enumerate(df.partition_by(\"symbol_id\", maintain_order=False)):\n        np_series = (\n            symbol_df.select(\"responder_6\").slice(-1 * lookback_window).to_numpy()\n        )\n\n        all_symbols.append(\n            TimeSeries.from_values(\n                values=np.pad(np_series, (lookback_window - len(np_series), 0))\n            ).astype(np.float32)\n        )\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3039421,
      "author_name": "michaeltimbs",
      "author_url": "",
      "post_date": "11/08/2024 03:26:12",
      "content": "<p>I should also note that these benchmark times are from my local machine (M1 Max) that utilizes all 10 CPU cores at 100%. I can only imagine that the actual submission is a fair bit slower</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3039428,
      "author_name": "michaeltimbs",
      "author_url": "",
      "post_date": "11/08/2024 03:42:18",
      "content": "<h1>Update</h1>\n<p>I have sped the creation of time series up about 3x by moving to the <code>from_values</code> method and exporting my Polars df to a numpy array instead of a pandas dataframe. Possibly could have had similar results only exporting the columns I actually needed to a pandas dataframe too.</p>\n<pre><code>    \n    all_symbols = []\n     _, symbol_df  (df.partition_by(, maintain_order=)):\n        np_series = (\n            symbol_df.select().(- * lookback_window).to_numpy()\n        )\n\n        all_symbols.append(\n            TimeSeries.from_values(\n                values=np.pad(np_series, (lookback_window - (np_series), ))\n            ).astype(np.float32)\n        )\n</code></pre>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3039417": "Has anyone got any performance tips? I usually don't write Python and my submission is stupidly slow. I can't seem to get the ~150ms per prediction window to avoid a submission timeout.\n\n*Note*: This is not an inference time problem. I profiled my code and the total time in inference for all symbols on a day is on average less than 0.3ms!\n`profile:: model_predict = 0.00039995802217163146`\n\nProfiling shows two main bottlenecks:\n1.(t=~100ms):  Converting Polars dataframe to a `darts.TimeSeries` for use in predicting output. I actually managed to drop this from 130ms to about 90ms using a few tricks (not adding my own RangeIndex and also setting maintain_order=False on the partition_by) but am completely stuck on further optimisations\n\n```python\n    # TODO: SLOW!! ~110ms!!!\n    all_symbols = []\n    for _, series in enumerate(df.partition_by(\"symbol_id\", maintain_order=False)):\n        all_symbols.append(\n            TimeSeries.from_dataframe(\n                series.to_pandas(),\n                value_cols=\"responder_6\",\n            ).astype(np.float32)\n        )\n```\n\n2. I keep track of the current time steps feature data along with my predictions so I can feed them in at t=t+1 to form part of the next prediction (e.g. each prediction has a lookback window of historical features + responders). \n\n```python\n    predictions = predict_fn(_history, model)\n\n    # Join the predictions onto the lags DF for auto-regression (t+1 prediction)\n    # We then over-write these values with actual values on next days lags.\n    # TODO: SLOW ~25ms\n    _history = _history.update(\n        predictions,\n        how=\"left\",\n        left_on=[\"row_id\"],\n        right_on=[\"row_id\"],\n    )\n```\n\nThese two steps alone take ~125ms.\n\nI also for some reason have to recast my Polars dataframes a lot - possibly due to joining and concatenating not preserving the original datatypes? For example when concatenating the input features from one day to the next they often jump between Float32 and Float64. row_id also seems to jump between UInt32 and Int64 and a few other columns always seem to change type randomly. I have to recast datatypes 4-5 times per time step but the combined cost of these is ~3ms so I don't think its a big deal given the other bottlenecks I have.\n\n\n**Is the answer here to just not use Darts library?**",
    "3039421": "I should also note that these benchmark times are from my local machine (M1 Max) that utilizes all 10 CPU cores at 100%. I can only imagine that the actual submission is a fair bit slower",
    "3039428": "# Update\n\nI have sped the creation of time series up about 3x by moving to the `from_values` method and exporting my Polars df to a numpy array instead of a pandas dataframe. Possibly could have had similar results only exporting the columns I actually needed to a pandas dataframe too.\n\n```python\n    # Improved to ~30ms!\n    all_symbols = []\n    for _, symbol_df in enumerate(df.partition_by(\"symbol_id\", maintain_order=False)):\n        np_series = (\n            symbol_df.select(\"responder_6\").slice(-1 * lookback_window).to_numpy()\n        )\n\n        all_symbols.append(\n            TimeSeries.from_values(\n                values=np.pad(np_series, (lookback_window - len(np_series), 0))\n            ).astype(np.float32)\n        )\n```"
  },
  "source": "meta"
}