The new API has limited the inference time of 10 min after each batch data being served. This would make it quite hard to perform online training and update the model using new data during the private leaderboard. Am I understanding it correctly?
", "messages": [ { "id": "3018611", "postDate": "10/15/2024 22:40:40", "content": "The new API has limited the inference time of 10 min after each batch data being served. This would make it quite hard to perform online training and update the model using new data during the private leaderboard. Am I understanding it correctly?
", "rawMarkdown": "The new API has limited the inference time of 10 min after each batch data being served. This would make it quite hard to perform online training and update the model using new data during the private leaderboard. Am I understanding it correctly?", "votes": null }, { "id": "3019291", "postDate": "10/16/2024 12:08:44", "content": "Yes, it looks like we can only train once.
", "rawMarkdown": "Yes, it looks like we can only train once.", "votes": null }, { "id": "3021031", "postDate": "10/18/2024 06:46:27", "content": "Sorry I thought 10 min is very long and a batch should not be very large, why it is not enough for online training? Since other ppl also agree with you, I must misunderstood something. Please tell me what did I miss?
", "rawMarkdown": "Sorry I thought 10 min is very long and a batch should not be very large, why it is not enough for online training? Since other ppl also agree with you, I must misunderstood something. Please tell me what did I miss?", "votes": null }, { "id": "3021079", "postDate": "10/18/2024 07:12:42", "content": "It’s plenty of time for inference but not enough for updating the model with new data.
", "rawMarkdown": "It’s plenty of time for inference but not enough for updating the model with new data.", "votes": null }, { "id": "3031368", "postDate": "10/29/2024 16:13:15", "content": "Training from scratch isn't possible but LightGBM refit and update could be useful.
", "rawMarkdown": "Training from scratch isn't possible but [LightGBM refit and update](https://stackoverflow.com/a/73669068/7975962) could be useful.", "votes": null }, { "id": "3031435", "postDate": "10/29/2024 17:23:05", "content": "Thanks! very helpful reference.
", "rawMarkdown": "Thanks! very helpful reference.", "votes": null }, { "id": "3031714", "postDate": "10/30/2024 03:04:01", "content": "just found this thread. I think its 1 min?
\nI am asking them here to increase the the timeout at least for the start of day batch
\nhttps://www.kaggle.com/code/ryanholbrook/jane-street-rmf-demo-submission/comments#3024352
the final testing data set should have ~125 days worth of data, even if they does increase it, I think because of the site-wide overall notebook timeout of 8/9 hours you only have ~3/4 minutes
", "rawMarkdown": "just found this thread. I think its 1 min? \nI am asking them here to increase the the timeout at least for the start of day batch\nhttps://www.kaggle.com/code/ryanholbrook/jane-street-rmf-demo-submission/comments#3024352\n\nthe final testing data set should have ~125 days worth of data, even if they does increase it, I think because of the site-wide overall notebook timeout of 8/9 hours you only have ~3/4 minutes", "votes": null }, { "id": "3033216", "postDate": "10/31/2024 22:56:23", "content": "If we do have 10 minutes it should be plenty of time. If we only have 1 minute (which seems more likely) then not so much. Each epoch of training for me takes about 45 seconds and each prediction is only 1-2. So if we had 10 minutes you could retrain 4-5 epochs on the previous days lags as fine tuning without any issues, just set a lower learning rate
", "rawMarkdown": "If we do have 10 minutes it should be plenty of time. If we only have 1 minute (which seems more likely) then not so much. Each epoch of training for me takes about 45 seconds and each prediction is only 1-2. So if we had 10 minutes you could retrain 4-5 epochs on the previous days lags as fine tuning without any issues, just set a lower learning rate", "votes": null }, { "id": "3033226", "postDate": "10/31/2024 23:53:19", "content": "Dear @sohier,
\nMay I clarify something here?
\nFirst, during the forecasting phase for the private LB, how often does the re-run with updated data occurs? In the Optiver Trading At Close competition, it was updated per month. Can we expect the same frequency?
\nSecond, when re-run the submitted notebook using the updated data, is it possible to use files saved in the last run? For example, if I have fine-tuned my model in the last re-run and saved it, can I somehow load the model in the next re-run?
\nMany thanks for your answer!
", "rawMarkdown": "Dear @sohier, \n\nMay I clarify something here? \n\nFirst, during the forecasting phase for the private LB, how often does the re-run with updated data occurs? In the Optiver Trading At Close competition, it was updated per month. Can we expect the same frequency?\n\nSecond, when re-run the submitted notebook using the updated data, is it possible to use files saved in the last run? For example, if I have fine-tuned my model in the last re-run and saved it, can I somehow load the model in the next re-run?\n\nMany thanks for your answer!", "votes": null } ], "comments": [ { "id": 3019291, "author_name": "sayoulala", "author_url": "", "post_date": "10/16/2024 12:08:44", "content": "Yes, it looks like we can only train once.
", "votes": null, "replies": [] }, { "id": 3021031, "author_name": "rocstone", "author_url": "", "post_date": "10/18/2024 06:46:27", "content": "Sorry I thought 10 min is very long and a batch should not be very large, why it is not enough for online training? Since other ppl also agree with you, I must misunderstood something. Please tell me what did I miss?
", "votes": null, "replies": [ { "id": 3021079, "author_name": "shiyili", "author_url": "", "post_date": "10/18/2024 07:12:42", "content": "It’s plenty of time for inference but not enough for updating the model with new data.
", "votes": null, "replies": [ { "id": 3033216, "author_name": "michaeltimbs", "author_url": "", "post_date": "10/31/2024 22:56:23", "content": "If we do have 10 minutes it should be plenty of time. If we only have 1 minute (which seems more likely) then not so much. Each epoch of training for me takes about 45 seconds and each prediction is only 1-2. So if we had 10 minutes you could retrain 4-5 epochs on the previous days lags as fine tuning without any issues, just set a lower learning rate
", "votes": null, "replies": [] } ] } ] }, { "id": 3031368, "author_name": "gunesevitan", "author_url": "", "post_date": "10/29/2024 16:13:15", "content": "Training from scratch isn't possible but LightGBM refit and update could be useful.
", "votes": null, "replies": [ { "id": 3031435, "author_name": "shiyili", "author_url": "", "post_date": "10/29/2024 17:23:05", "content": "Thanks! very helpful reference.
", "votes": null, "replies": [] } ] }, { "id": 3031714, "author_name": "dc260123", "author_url": "", "post_date": "10/30/2024 03:04:01", "content": "just found this thread. I think its 1 min?
\nI am asking them here to increase the the timeout at least for the start of day batch
\nhttps://www.kaggle.com/code/ryanholbrook/jane-street-rmf-demo-submission/comments#3024352
the final testing data set should have ~125 days worth of data, even if they does increase it, I think because of the site-wide overall notebook timeout of 8/9 hours you only have ~3/4 minutes
", "votes": null, "replies": [] }, { "id": 3033226, "author_name": "shiyili", "author_url": "", "post_date": "10/31/2024 23:53:19", "content": "Dear @sohier,
\nMay I clarify something here?
\nFirst, during the forecasting phase for the private LB, how often does the re-run with updated data occurs? In the Optiver Trading At Close competition, it was updated per month. Can we expect the same frequency?
\nSecond, when re-run the submitted notebook using the updated data, is it possible to use files saved in the last run? For example, if I have fine-tuned my model in the last re-run and saved it, can I somehow load the model in the next re-run?
\nMany thanks for your answer!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3018611": "The new API has limited the inference time of 10 min after each batch data being served. This would make it quite hard to perform online training and update the model using new data during the private leaderboard. Am I understanding it correctly?", "3019291": "Yes, it looks like we can only train once.", "3021031": "Sorry I thought 10 min is very long and a batch should not be very large, why it is not enough for online training? Since other ppl also agree with you, I must misunderstood something. Please tell me what did I miss?", "3021079": "It’s plenty of time for inference but not enough for updating the model with new data.", "3031368": "Training from scratch isn't possible but [LightGBM refit and update](https://stackoverflow.com/a/73669068/7975962) could be useful.", "3031435": "Thanks! very helpful reference.", "3031714": "just found this thread. I think its 1 min? \nI am asking them here to increase the the timeout at least for the start of day batch\nhttps://www.kaggle.com/code/ryanholbrook/jane-street-rmf-demo-submission/comments#3024352\n\nthe final testing data set should have ~125 days worth of data, even if they does increase it, I think because of the site-wide overall notebook timeout of 8/9 hours you only have ~3/4 minutes", "3033216": "If we do have 10 minutes it should be plenty of time. If we only have 1 minute (which seems more likely) then not so much. Each epoch of training for me takes about 45 seconds and each prediction is only 1-2. So if we had 10 minutes you could retrain 4-5 epochs on the previous days lags as fine tuning without any issues, just set a lower learning rate", "3033226": "Dear @sohier, \n\nMay I clarify something here? \n\nFirst, during the forecasting phase for the private LB, how often does the re-run with updated data occurs? In the Optiver Trading At Close competition, it was updated per month. Can we expect the same frequency?\n\nSecond, when re-run the submitted notebook using the updated data, is it possible to use files saved in the last run? For example, if I have fine-tuned my model in the last re-run and saved it, can I somehow load the model in the next re-run?\n\nMany thanks for your answer!" }, "source": "meta" }