{ "id": 542114, "title": "Question about Model Retraining during Prediction Phase within Time Limit", "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/542114", "author_name": "", "post_date": "2024-10-23T03:26:42.007179300Z", "votes": 2, "comment_count": 10, "views": 0, "content": "

I have a question regarding the prediction phase rules:

\n
    \n
  1. Is it allowed to retrain our model with newly obtained data during the prediction phase?
    2. \n
  2. Given the 9-hour time limit, would this scenario be permitted:
    3. \n
\n

I want to ensure this approach complies with competition rules before implementing it.
\nThank you for any clarification!

", "messages": [ { "id": "3025654", "postDate": "10/23/2024 03:26:42", "content": "

I have a question regarding the prediction phase rules:

\n
    \n
  1. Is it allowed to retrain our model with newly obtained data during the prediction phase?
    2. \n
  2. Given the 9-hour time limit, would this scenario be permitted:
    3. \n
\n

I want to ensure this approach complies with competition rules before implementing it.
\nThank you for any clarification!

", "rawMarkdown": "I have a question regarding the prediction phase rules:\n\n1. Is it allowed to retrain our model with newly obtained data during the prediction phase?\n2. Given the 9-hour time limit, would this scenario be permitted:\n - Use 1 hour for all predictions\n - Use remaining time (8 hours) for model retraining\n\nI want to ensure this approach complies with competition rules before implementing it.\nThank you for any clarification!", "votes": null }, { "id": "3025874", "postDate": "10/23/2024 08:19:00", "content": "
    \n
  1. It is allowed to re-train the model using newly obtained data.
    2. \n
  2. You can use your time as you want. BUT prediction for each batch has to be made within 10min after the data is served through the API. This means if you train the model between two batches, you need to ensure it takes less than 10min (plus inference).
    3. \n
", "rawMarkdown": "1. It is allowed to re-train the model using newly obtained data. \n2. You can use your time as you want. BUT **prediction for each batch has to be made within 10min after the data is served through the API.** This means if you train the model between two batches, you need to ensure it takes less than 10min (plus inference).", "votes": null }, { "id": "3026526", "postDate": "10/23/2024 23:23:42", "content": "

Thank you for the clear explanation! Your answer was very helpful.
\nConsidering this constraint, I think it would be safer to do any retraining after completing each batch's predictions rather than between batches to avoid timing risks.

", "rawMarkdown": "Thank you for the clear explanation! Your answer was very helpful.\nConsidering this constraint, I think it would be safer to do any retraining after completing each batch's predictions rather than between batches to avoid timing risks.", "votes": null }, { "id": "3026795", "postDate": "10/24/2024 08:35:12", "content": "
\n

retraining after completing each batch's predictions rather than between batches

\n
\n

curious how would you implement it.

", "rawMarkdown": ">retraining after completing each batch's predictions rather than between batches\n\ncurious how would you implement it.", "votes": null }, { "id": "3026894", "postDate": "10/24/2024 10:32:05", "content": "

If you trust the 10 minutes constratin and reply on it, you will experience the suffering I went through in last whole week. From my tons of tests, the real constraint the host set is 1 minute exactly and you can also see that explicitly in the kaggle evaluation code they provided.

", "rawMarkdown": "If you trust the 10 minutes constratin and reply on it, you will experience the suffering I went through in last whole week. From my tons of tests, the real constraint the host set is 1 minute exactly and you can also see that explicitly in the kaggle evaluation code they provided.", "votes": null }, { "id": "3027059", "postDate": "10/24/2024 13:45:50", "content": "

I also found this in the evaluation code. I hope this is a bug and they can fix it….

", "rawMarkdown": "I also found this in the evaluation code. I hope this is a bug and they can fix it....", "votes": null }, { "id": "3027329", "postDate": "10/24/2024 17:18:56", "content": "

indeed. I suffered from it too

", "rawMarkdown": "indeed. I suffered from it too", "votes": null }, { "id": "3027517", "postDate": "10/25/2024 00:28:13", "content": "

I'm considering implementing the retraining after the serve() process completes and returns control to the notebook. My understanding is:

\n
 os.getenv():\n    inference_server.serve()  \n    \n    retrain_models_with_new_data()  \n
\n

Is this approach feasible, or are there technical limitations that would prevent the execution of subsequent code after serve() completes?

", "rawMarkdown": "I'm considering implementing the retraining after the serve() process completes and returns control to the notebook. My understanding is:\n\n```python\nif os.getenv('KAGGLE_IS_COMPETITION_RERUN'):\n inference_server.serve() # Expect to take ~1 hour for predictions\n # After serve() completes, continue with retraining\n retrain_models_with_new_data() # Use remaining time (~8 hours)\n```\n\nIs this approach feasible, or are there technical limitations that would prevent the execution of subsequent code after serve() completes?", "votes": null }, { "id": "3027758", "postDate": "10/25/2024 08:45:34", "content": "

This would work. But when serve() is done, the submission is all generated. What do you do with the updated model?

", "rawMarkdown": "This would work. But when serve() is done, the submission is all generated. What do you do with the updated model?", "votes": null }, { "id": "3029202", "postDate": "10/27/2024 03:09:58", "content": "

Indeed, while all submissions for the current batch are generated, I'm planning to save the trained model for use in the next prediction cycle (2 weeks later). In other words, I'm always training models to be used for future predictions, not for the current submission.

", "rawMarkdown": "Indeed, while all submissions for the current batch are generated, I'm planning to save the trained model for use in the next prediction cycle (2 weeks later). In other words, I'm always training models to be used for future predictions, not for the current submission.", "votes": null }, { "id": "3029371", "postDate": "10/27/2024 07:47:48", "content": "

I’m not sure if the model file saved in this round can be kept in the next round. If they always start a new test environment, all files generated in the last round will be lost.

", "rawMarkdown": "I’m not sure if the model file saved in this round can be kept in the next round. If they always start a new test environment, all files generated in the last round will be lost.", "votes": null } ], "comments": [ { "id": 3025874, "author_name": "shiyili", "author_url": "", "post_date": "10/23/2024 08:19:00", "content": "
    \n
  1. It is allowed to re-train the model using newly obtained data.
    2. \n
  2. You can use your time as you want. BUT prediction for each batch has to be made within 10min after the data is served through the API. This means if you train the model between two batches, you need to ensure it takes less than 10min (plus inference).
    3. \n
", "votes": null, "replies": [ { "id": 3026526, "author_name": "hiroaaaa", "author_url": "", "post_date": "10/23/2024 23:23:42", "content": "

Thank you for the clear explanation! Your answer was very helpful.
\nConsidering this constraint, I think it would be safer to do any retraining after completing each batch's predictions rather than between batches to avoid timing risks.

", "votes": null, "replies": [ { "id": 3026795, "author_name": "shiyili", "author_url": "", "post_date": "10/24/2024 08:35:12", "content": "
\n

retraining after completing each batch's predictions rather than between batches

\n
\n

curious how would you implement it.

", "votes": null, "replies": [ { "id": 3027517, "author_name": "hiroaaaa", "author_url": "", "post_date": "10/25/2024 00:28:13", "content": "

I'm considering implementing the retraining after the serve() process completes and returns control to the notebook. My understanding is:

\n
 os.getenv():\n    inference_server.serve()  \n    \n    retrain_models_with_new_data()  \n
\n

Is this approach feasible, or are there technical limitations that would prevent the execution of subsequent code after serve() completes?

", "votes": null, "replies": [ { "id": 3027758, "author_name": "shiyili", "author_url": "", "post_date": "10/25/2024 08:45:34", "content": "

This would work. But when serve() is done, the submission is all generated. What do you do with the updated model?

", "votes": null, "replies": [ { "id": 3029202, "author_name": "hiroaaaa", "author_url": "", "post_date": "10/27/2024 03:09:58", "content": "

Indeed, while all submissions for the current batch are generated, I'm planning to save the trained model for use in the next prediction cycle (2 weeks later). In other words, I'm always training models to be used for future predictions, not for the current submission.

", "votes": null, "replies": [ { "id": 3029371, "author_name": "shiyili", "author_url": "", "post_date": "10/27/2024 07:47:48", "content": "

I’m not sure if the model file saved in this round can be kept in the next round. If they always start a new test environment, all files generated in the last round will be lost.

", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 3026894, "author_name": "lihaorocky", "author_url": "", "post_date": "10/24/2024 10:32:05", "content": "

If you trust the 10 minutes constratin and reply on it, you will experience the suffering I went through in last whole week. From my tons of tests, the real constraint the host set is 1 minute exactly and you can also see that explicitly in the kaggle evaluation code they provided.

", "votes": null, "replies": [ { "id": 3027059, "author_name": "shiyili", "author_url": "", "post_date": "10/24/2024 13:45:50", "content": "

I also found this in the evaluation code. I hope this is a bug and they can fix it….

", "votes": null, "replies": [] }, { "id": 3027329, "author_name": "thomaswang", "author_url": "", "post_date": "10/24/2024 17:18:56", "content": "

indeed. I suffered from it too

", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3025654": "I have a question regarding the prediction phase rules:\n\n1. Is it allowed to retrain our model with newly obtained data during the prediction phase?\n2. Given the 9-hour time limit, would this scenario be permitted:\n - Use 1 hour for all predictions\n - Use remaining time (8 hours) for model retraining\n\nI want to ensure this approach complies with competition rules before implementing it.\nThank you for any clarification!", "3025874": "1. It is allowed to re-train the model using newly obtained data. \n2. You can use your time as you want. BUT **prediction for each batch has to be made within 10min after the data is served through the API.** This means if you train the model between two batches, you need to ensure it takes less than 10min (plus inference).", "3026526": "Thank you for the clear explanation! Your answer was very helpful.\nConsidering this constraint, I think it would be safer to do any retraining after completing each batch's predictions rather than between batches to avoid timing risks.", "3026795": ">retraining after completing each batch's predictions rather than between batches\n\ncurious how would you implement it.", "3026894": "If you trust the 10 minutes constratin and reply on it, you will experience the suffering I went through in last whole week. From my tons of tests, the real constraint the host set is 1 minute exactly and you can also see that explicitly in the kaggle evaluation code they provided.", "3027059": "I also found this in the evaluation code. I hope this is a bug and they can fix it....", "3027329": "indeed. I suffered from it too", "3027517": "I'm considering implementing the retraining after the serve() process completes and returns control to the notebook. My understanding is:\n\n```python\nif os.getenv('KAGGLE_IS_COMPETITION_RERUN'):\n inference_server.serve() # Expect to take ~1 hour for predictions\n # After serve() completes, continue with retraining\n retrain_models_with_new_data() # Use remaining time (~8 hours)\n```\n\nIs this approach feasible, or are there technical limitations that would prevent the execution of subsequent code after serve() completes?", "3027758": "This would work. But when serve() is done, the submission is all generated. What do you do with the updated model?", "3029202": "Indeed, while all submissions for the current batch are generated, I'm planning to save the trained model for use in the next prediction cycle (2 weeks later). In other words, I'm always training models to be used for future predictions, not for the current submission.", "3029371": "I’m not sure if the model file saved in this round can be kept in the next round. If they always start a new test environment, all files generated in the last round will be lost." }, "source": "meta" }