{ "id": 543317, "title": "Memory constraints & online learning ", "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/543317", "author_name": "", "post_date": "2024-10-29T21:09:21.743523500Z", "votes": -7, "comment_count": 6, "views": 0, "content": "

Hi everyone,
\nI saw that many people are advising and saying that they are using their machines or cloud computing services due to the lack of memory in Kaggle notebooks. I would like to say if you aren't training your model using a Kaggle notebook due to memory issues, then say goodbye to online learning. If you didn't figure out a solution to the memory problem when it comes to the training data, you will not be able to overcome this problem during the inference phase when you will have even more data which means you will not be able to use online learning.

", "messages": [ { "id": "3031581", "postDate": "10/29/2024 21:09:21", "content": "

Hi everyone,
\nI saw that many people are advising and saying that they are using their machines or cloud computing services due to the lack of memory in Kaggle notebooks. I would like to say if you aren't training your model using a Kaggle notebook due to memory issues, then say goodbye to online learning. If you didn't figure out a solution to the memory problem when it comes to the training data, you will not be able to overcome this problem during the inference phase when you will have even more data which means you will not be able to use online learning.

", "rawMarkdown": "Hi everyone,\nI saw that many people are advising and saying that they are using their machines or cloud computing services due to the lack of memory in Kaggle notebooks. I would like to say if you aren't training your model using a Kaggle notebook due to memory issues, then say goodbye to online learning. If you didn't figure out a solution to the memory problem when it comes to the training data, you will not be able to overcome this problem during the inference phase when you will have even more data which means you will not be able to use online learning.", "votes": null }, { "id": "3031589", "postDate": "10/29/2024 21:43:08", "content": "

It is anyway not possible to use full data for online training due to the submission time constrain. No one would have such crazy idea. Only using the updated data for incremental online training is fine for the memory. Time constrain is the major issue.

", "rawMarkdown": "It is anyway not possible to use full data for online training due to the submission time constrain. No one would have such crazy idea. Only using the updated data for incremental online training is fine for the memory. Time constrain is the major issue.", "votes": null }, { "id": "3031595", "postDate": "10/29/2024 21:54:38", "content": "

I wrote with GBMs in mind, usually when people do online training for GBMs they retrain it which will require same amount of data used to train the initial model if you you which to maintain the original performance level

", "rawMarkdown": "I wrote with GBMs in mind, usually when people do online training for GBMs they retrain it which will require same amount of data used to train the initial model if you you which to maintain the original performance level", "votes": null }, { "id": "3031609", "postDate": "10/29/2024 22:36:09", "content": "

anyways, before worrying about memory, I would first figure out how to finish training within the 1-minute prediction window.

", "rawMarkdown": "anyways, before worrying about memory, I would first figure out how to finish training within the 1-minute prediction window.", "votes": null }, { "id": "3031613", "postDate": "10/29/2024 22:52:28", "content": "

Did Ryan confirm it only 1-min?

", "rawMarkdown": "Did Ryan confirm it only 1-min?", "votes": null }, { "id": "3031615", "postDate": "10/29/2024 22:59:17", "content": "

Not yet. Could be 10-min, still challenging

", "rawMarkdown": "Not yet. Could be 10-min, still challenging", "votes": null }, { "id": "3032055", "postDate": "10/30/2024 13:56:44", "content": "

Guys is it possible here train with TPU for longer time sequence?

", "rawMarkdown": "Guys is it possible here train with TPU for longer time sequence?", "votes": null } ], "comments": [ { "id": 3031589, "author_name": "shiyili", "author_url": "", "post_date": "10/29/2024 21:43:08", "content": "

It is anyway not possible to use full data for online training due to the submission time constrain. No one would have such crazy idea. Only using the updated data for incremental online training is fine for the memory. Time constrain is the major issue.

", "votes": null, "replies": [ { "id": 3031595, "author_name": "aymanallawi", "author_url": "", "post_date": "10/29/2024 21:54:38", "content": "

I wrote with GBMs in mind, usually when people do online training for GBMs they retrain it which will require same amount of data used to train the initial model if you you which to maintain the original performance level

", "votes": null, "replies": [ { "id": 3031609, "author_name": "shiyili", "author_url": "", "post_date": "10/29/2024 22:36:09", "content": "

anyways, before worrying about memory, I would first figure out how to finish training within the 1-minute prediction window.

", "votes": null, "replies": [ { "id": 3031613, "author_name": "aymanallawi", "author_url": "", "post_date": "10/29/2024 22:52:28", "content": "

Did Ryan confirm it only 1-min?

", "votes": null, "replies": [ { "id": 3031615, "author_name": "shiyili", "author_url": "", "post_date": "10/29/2024 22:59:17", "content": "

Not yet. Could be 10-min, still challenging

", "votes": null, "replies": [ { "id": 3032055, "author_name": "aifahim", "author_url": "", "post_date": "10/30/2024 13:56:44", "content": "

Guys is it possible here train with TPU for longer time sequence?

", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "3031581": "Hi everyone,\nI saw that many people are advising and saying that they are using their machines or cloud computing services due to the lack of memory in Kaggle notebooks. I would like to say if you aren't training your model using a Kaggle notebook due to memory issues, then say goodbye to online learning. If you didn't figure out a solution to the memory problem when it comes to the training data, you will not be able to overcome this problem during the inference phase when you will have even more data which means you will not be able to use online learning.", "3031589": "It is anyway not possible to use full data for online training due to the submission time constrain. No one would have such crazy idea. Only using the updated data for incremental online training is fine for the memory. Time constrain is the major issue.", "3031595": "I wrote with GBMs in mind, usually when people do online training for GBMs they retrain it which will require same amount of data used to train the initial model if you you which to maintain the original performance level", "3031609": "anyways, before worrying about memory, I would first figure out how to finish training within the 1-minute prediction window.", "3031613": "Did Ryan confirm it only 1-min?", "3031615": "Not yet. Could be 10-min, still challenging", "3032055": "Guys is it possible here train with TPU for longer time sequence?" }, "source": "meta" }