{
  "id": 548440,
  "title": "Online training",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/548440",
  "author_name": "",
  "post_date": "2024-11-26T20:55:53.618864100Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I'm training new model for each batch, locally it is running perfectly but submission gives, error: </p>\n<p>Notebook Threw Exception</p>\n<pre><code>temp=\nlags_: pl.DataFrame |  = \n () -&gt; pl.DataFrame | pd.DataFrame:\n     temp\n    selcol=[  idx  ()]+[,]\n    yp=\n    mdf_up=\n    model = NMDL()\n     temp   :\n        mdf_up=\n        model .fit(temp,lags[].to_numpy())\n\n    X=test[selcol].fill_null().to_numpy()\n    X_smoothed = apply_gaussian_filter(X)\n    X_scaled = scaler.transform(X_smoothed)\n    temp=X_scaled\n\n    (mdf_up):\n        yp=model .predict(X_scaled)\n\n        predictions = pl.DataFrame({\n            : test[],\n            : yp\n        })\n\n    :\n        predictions = pl.DataFrame({\n        : test[],\n        : [] * (test)\n        })\n     model \n\n    (predictions)\n     predictions\n</code></pre>\n<p>Note: NMDL is my custom model <br>\nWhat is wrong with my code here!!?</p>",
  "messages": [
    {
      "id": "3056337",
      "postDate": "11/26/2024 20:55:53",
      "content": "<p>I'm training new model for each batch, locally it is running perfectly but submission gives, error: </p>\n<p>Notebook Threw Exception</p>\n<pre><code>temp=\nlags_: pl.DataFrame |  = \n () -&gt; pl.DataFrame | pd.DataFrame:\n     temp\n    selcol=[  idx  ()]+[,]\n    yp=\n    mdf_up=\n    model = NMDL()\n     temp   :\n        mdf_up=\n        model .fit(temp,lags[].to_numpy())\n\n    X=test[selcol].fill_null().to_numpy()\n    X_smoothed = apply_gaussian_filter(X)\n    X_scaled = scaler.transform(X_smoothed)\n    temp=X_scaled\n\n    (mdf_up):\n        yp=model .predict(X_scaled)\n\n        predictions = pl.DataFrame({\n            : test[],\n            : yp\n        })\n\n    :\n        predictions = pl.DataFrame({\n        : test[],\n        : [] * (test)\n        })\n     model \n\n    (predictions)\n     predictions\n</code></pre>\n<p>Note: NMDL is my custom model <br>\nWhat is wrong with my code here!!?</p>",
      "rawMarkdown": "I'm training new model for each batch, locally it is running perfectly but submission gives, error: \n\nNotebook Threw Exception\n\n\n```\ntemp=None\nlags_: pl.DataFrame | None = None\ndef predict(test: pl.DataFrame, lags: pl.DataFrame | None) -> pl.DataFrame | pd.DataFrame:\n    global temp\n    selcol=[f\"feature_{idx:02d}\" for idx in range(79)]+[\"symbol_id\",\"time_id\"]\n    yp=None\n    mdf_up=False\n    model = NMDL(100)\n    if temp is not None:\n        mdf_up=True\n        model .fit(temp,lags[\"responder_6_lag_1\"].to_numpy())\n    \n    X=test[selcol].fill_null(3).to_numpy()\n    X_smoothed = apply_gaussian_filter(X)\n    X_scaled = scaler.transform(X_smoothed)\n    temp=X_scaled\n    \n    if(mdf_up):\n        yp=model .predict(X_scaled)\n   \n        predictions = pl.DataFrame({\n            'row_id': test['row_id'],\n            'responder_6': yp\n        })\n        \n    else:\n        predictions = pl.DataFrame({\n        'row_id': test['row_id'],\n        'responder_6': [0] * len(test)\n        })\n    del model \n\n    print(predictions)\n    return predictions\n```\nNote: NMDL is my custom model \nWhat is wrong with my code here!!?",
      "votes": null
    },
    {
      "id": "3056516",
      "postDate": "11/27/2024 04:11:49",
      "content": "<p>I believe the lags are updated as a global variable rather than the input to the function.</p>",
      "rawMarkdown": "I believe the lags are updated as a global variable rather than the input to the function.",
      "votes": null
    },
    {
      "id": "3060788",
      "postDate": "12/02/2024 05:38:21",
      "content": "<p>lags are None for all time steps except t=0. So only once per day will you have lag values passed that are not None.</p>\n<p>To do online training you want to do a check</p>\n<pre><code> lags   :\n    \n</code></pre>",
      "rawMarkdown": "lags are None for all time steps except t=0. So only once per day will you have lag values passed that are not None.\n\nTo do online training you want to do a check\n\n```python\nif lags is not None:\n    # Do training\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3056516,
      "author_name": "risanraja32",
      "author_url": "",
      "post_date": "11/27/2024 04:11:49",
      "content": "<p>I believe the lags are updated as a global variable rather than the input to the function.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3060788,
      "author_name": "michaeltimbs",
      "author_url": "",
      "post_date": "12/02/2024 05:38:21",
      "content": "<p>lags are None for all time steps except t=0. So only once per day will you have lag values passed that are not None.</p>\n<p>To do online training you want to do a check</p>\n<pre><code> lags   :\n    \n</code></pre>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3056337": "I'm training new model for each batch, locally it is running perfectly but submission gives, error: \n\nNotebook Threw Exception\n\n\n```\ntemp=None\nlags_: pl.DataFrame | None = None\ndef predict(test: pl.DataFrame, lags: pl.DataFrame | None) -> pl.DataFrame | pd.DataFrame:\n    global temp\n    selcol=[f\"feature_{idx:02d}\" for idx in range(79)]+[\"symbol_id\",\"time_id\"]\n    yp=None\n    mdf_up=False\n    model = NMDL(100)\n    if temp is not None:\n        mdf_up=True\n        model .fit(temp,lags[\"responder_6_lag_1\"].to_numpy())\n    \n    X=test[selcol].fill_null(3).to_numpy()\n    X_smoothed = apply_gaussian_filter(X)\n    X_scaled = scaler.transform(X_smoothed)\n    temp=X_scaled\n    \n    if(mdf_up):\n        yp=model .predict(X_scaled)\n   \n        predictions = pl.DataFrame({\n            'row_id': test['row_id'],\n            'responder_6': yp\n        })\n        \n    else:\n        predictions = pl.DataFrame({\n        'row_id': test['row_id'],\n        'responder_6': [0] * len(test)\n        })\n    del model \n\n    print(predictions)\n    return predictions\n```\nNote: NMDL is my custom model \nWhat is wrong with my code here!!?",
    "3056516": "I believe the lags are updated as a global variable rather than the input to the function.",
    "3060788": "lags are None for all time steps except t=0. So only once per day will you have lag values passed that are not None.\n\nTo do online training you want to do a check\n\n```python\nif lags is not None:\n    # Do training\n```"
  },
  "source": "meta"
}