{
  "id": 552235,
  "title": "Massive Confusion about online learning & online retrain | Meet time constraint but still fail on submission",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/552235",
  "author_name": "",
  "post_date": "2024-12-18T12:13:24.280734800Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi! All! I have tried to utilize online retrain or online learning on NN &amp; Tree model, when I test the overall time of each day on simulated data and they all meet the <strong>60s time constrain (less than 60s)</strong>, Finally the outcome shows up, however, it fails in 20 minutes when I submit the online learning version officially. Any expert can discuss about that? Thanks!</p>\n<p>Simulators that I use:<br>\n<a href=\"https://www.kaggle.com/code/shiyili/js24-rmf-submission-api-debug-with-synthetic-test\" target=\"_blank\">https://www.kaggle.com/code/shiyili/js24-rmf-submission-api-debug-with-synthetic-test</a><br>\n<a href=\"https://www.kaggle.com/code/chumajin/janestreet-updated-simulator-for-time-series-api\" target=\"_blank\">https://www.kaggle.com/code/chumajin/janestreet-updated-simulator-for-time-series-api</a></p>\n<p>The result shows like this:</p>\n<p><strong>Code:</strong></p>\n<p>if os.path.isfile('submission.parquet'):<br>\n    pl_sub = pl.read_parquet('submission.parquet')<br>\n    display(pl_sub)<br>\n    column_dtypes = {col: pl_sub[col].dtype for col in pl_sub.columns}<br>\n    print(column_dtypes)</p>\n<p><strong>Result:</strong></p>\n<p>shape: (714_384, 2)<br>\nrow_id    responder_6<br>\nu32    f64<br>\n1    0.026906<br>\n2    -0.049518<br>\n3    0.231084<br>\n4    0.238384<br>\n5    0.169966<br>\n…    …<br>\n714380    0.046166<br>\n714381    -0.071476<br>\n714382    -0.004296<br>\n714383    -0.004255<br>\n714384    -0.010947</p>\n<p>**Error type: **Notebook Inference Server Error</p>",
  "messages": [
    {
      "id": "3075042",
      "postDate": "12/18/2024 12:13:24",
      "content": "<p>Hi! All! I have tried to utilize online retrain or online learning on NN &amp; Tree model, when I test the overall time of each day on simulated data and they all meet the <strong>60s time constrain (less than 60s)</strong>, Finally the outcome shows up, however, it fails in 20 minutes when I submit the online learning version officially. Any expert can discuss about that? Thanks!</p>\n<p>Simulators that I use:<br>\n<a href=\"https://www.kaggle.com/code/shiyili/js24-rmf-submission-api-debug-with-synthetic-test\" target=\"_blank\">https://www.kaggle.com/code/shiyili/js24-rmf-submission-api-debug-with-synthetic-test</a><br>\n<a href=\"https://www.kaggle.com/code/chumajin/janestreet-updated-simulator-for-time-series-api\" target=\"_blank\">https://www.kaggle.com/code/chumajin/janestreet-updated-simulator-for-time-series-api</a></p>\n<p>The result shows like this:</p>\n<p><strong>Code:</strong></p>\n<p>if os.path.isfile('submission.parquet'):<br>\n    pl_sub = pl.read_parquet('submission.parquet')<br>\n    display(pl_sub)<br>\n    column_dtypes = {col: pl_sub[col].dtype for col in pl_sub.columns}<br>\n    print(column_dtypes)</p>\n<p><strong>Result:</strong></p>\n<p>shape: (714_384, 2)<br>\nrow_id    responder_6<br>\nu32    f64<br>\n1    0.026906<br>\n2    -0.049518<br>\n3    0.231084<br>\n4    0.238384<br>\n5    0.169966<br>\n…    …<br>\n714380    0.046166<br>\n714381    -0.071476<br>\n714382    -0.004296<br>\n714383    -0.004255<br>\n714384    -0.010947</p>\n<p>**Error type: **Notebook Inference Server Error</p>",
      "rawMarkdown": "Hi! All! I have tried to utilize online retrain or online learning on NN & Tree model, when I test the overall time of each day on simulated data and they all meet the **60s time constrain (less than 60s)**, Finally the outcome shows up, however, it fails in 20 minutes when I submit the online learning version officially. Any expert can discuss about that? Thanks!\n\nSimulators that I use:\nhttps://www.kaggle.com/code/shiyili/js24-rmf-submission-api-debug-with-synthetic-test\nhttps://www.kaggle.com/code/chumajin/janestreet-updated-simulator-for-time-series-api\n\nThe result shows like this:\n\n**Code:**\n\nif os.path.isfile('submission.parquet'):\n    pl_sub = pl.read_parquet('submission.parquet')\n    display(pl_sub)\n    column_dtypes = {col: pl_sub[col].dtype for col in pl_sub.columns}\n    print(column_dtypes)\n\n**Result:**\n\nshape: (714_384, 2)\nrow_id\tresponder_6\nu32\tf64\n1\t0.026906\n2\t-0.049518\n3\t0.231084\n4\t0.238384\n5\t0.169966\n…\t…\n714380\t0.046166\n714381\t-0.071476\n714382\t-0.004296\n714383\t-0.004255\n714384\t-0.010947\n\n**Error type: **Notebook Inference Server Error",
      "votes": null
    },
    {
      "id": "3077815",
      "postDate": "12/21/2024 12:38:03",
      "content": "<p>How many samples when you retrain the model everytime? There may be more and more data as the program run, and thus your notebook may cost more and more time.I keep the sample size for each retraining to about 40000 samples, it can run successfully.</p>",
      "rawMarkdown": "How many samples when you retrain the model everytime? There may be more and more data as the program run, and thus your notebook may cost more and more time.I keep the sample size for each retraining to about 40000 samples, it can run successfully.",
      "votes": null
    },
    {
      "id": "3077825",
      "postDate": "12/21/2024 12:46:04",
      "content": "<p>Hi! I would say it is definitely not this problem as I have tested the time is less than 30s and the cache does not crash</p>",
      "rawMarkdown": "Hi! I would say it is definitely not this problem as I have tested the time is less than 30s and the cache does not crash",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3077815,
      "author_name": "i2nfinit3y",
      "author_url": "",
      "post_date": "12/21/2024 12:38:03",
      "content": "<p>How many samples when you retrain the model everytime? There may be more and more data as the program run, and thus your notebook may cost more and more time.I keep the sample size for each retraining to about 40000 samples, it can run successfully.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3077825,
          "author_name": "larrylin666",
          "author_url": "",
          "post_date": "12/21/2024 12:46:04",
          "content": "<p>Hi! I would say it is definitely not this problem as I have tested the time is less than 30s and the cache does not crash</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3075042": "Hi! All! I have tried to utilize online retrain or online learning on NN & Tree model, when I test the overall time of each day on simulated data and they all meet the **60s time constrain (less than 60s)**, Finally the outcome shows up, however, it fails in 20 minutes when I submit the online learning version officially. Any expert can discuss about that? Thanks!\n\nSimulators that I use:\nhttps://www.kaggle.com/code/shiyili/js24-rmf-submission-api-debug-with-synthetic-test\nhttps://www.kaggle.com/code/chumajin/janestreet-updated-simulator-for-time-series-api\n\nThe result shows like this:\n\n**Code:**\n\nif os.path.isfile('submission.parquet'):\n    pl_sub = pl.read_parquet('submission.parquet')\n    display(pl_sub)\n    column_dtypes = {col: pl_sub[col].dtype for col in pl_sub.columns}\n    print(column_dtypes)\n\n**Result:**\n\nshape: (714_384, 2)\nrow_id\tresponder_6\nu32\tf64\n1\t0.026906\n2\t-0.049518\n3\t0.231084\n4\t0.238384\n5\t0.169966\n…\t…\n714380\t0.046166\n714381\t-0.071476\n714382\t-0.004296\n714383\t-0.004255\n714384\t-0.010947\n\n**Error type: **Notebook Inference Server Error",
    "3077815": "How many samples when you retrain the model everytime? There may be more and more data as the program run, and thus your notebook may cost more and more time.I keep the sample size for each retraining to about 40000 samples, it can run successfully.",
    "3077825": "Hi! I would say it is definitely not this problem as I have tested the time is less than 30s and the cache does not crash"
  },
  "source": "meta"
}