{ "id": 549681, "title": "Online Learning Gives Me Worse Results", "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/549681", "author_name": "", "post_date": "2024-12-03T12:17:59.687048600Z", "votes": 2, "comment_count": 11, "views": 0, "content": "

I’ve seen many people achieve some improvements in their scores through online learning, so I decided to give it a try. I trained a neural network model, but the results didn’t improve—in fact, they got worse. When I used my offline model for predictions, my public score was 0.0041. However, with the online learning approach, I updated the model daily using the data from the last two days, kept the same learning rate, and trained for 3 epochs each time. Surprisingly, the score dropped to -0.0142.

\n

I’d like to ask for advice: How should online learning be configured properly? Should I use only the new data completely? During training, should I freeze some parameters?

\n
", "messages": [ { "id": "3062246", "postDate": "12/03/2024 12:17:59", "content": "

I’ve seen many people achieve some improvements in their scores through online learning, so I decided to give it a try. I trained a neural network model, but the results didn’t improve—in fact, they got worse. When I used my offline model for predictions, my public score was 0.0041. However, with the online learning approach, I updated the model daily using the data from the last two days, kept the same learning rate, and trained for 3 epochs each time. Surprisingly, the score dropped to -0.0142.

\n

I’d like to ask for advice: How should online learning be configured properly? Should I use only the new data completely? During training, should I freeze some parameters?

\n
", "rawMarkdown": "I’ve seen many people achieve some improvements in their scores through online learning, so I decided to give it a try. I trained a neural network model, but the results didn’t improve—in fact, they got worse. When I used my offline model for predictions, my public score was 0.0041. However, with the online learning approach, I updated the model daily using the data from the last two days, kept the same learning rate, and trained for 3 epochs each time. Surprisingly, the score dropped to -0.0142.\n\nI’d like to ask for advice: How should online learning be configured properly? Should I use only the new data completely? During training, should I freeze some parameters?\n\n---------------", "votes": null }, { "id": "3062260", "postDate": "12/03/2024 12:36:03", "content": "

LR too high perhaps

", "rawMarkdown": "LR too high perhaps", "votes": null }, { "id": "3062288", "postDate": "12/03/2024 13:17:05", "content": "

Thank you for your comment:) LR is 5e-4, I'm not sure if this is too high, maybe I should try 1e-4

", "rawMarkdown": "Thank you for your comment:) LR is 5e-4, I'm not sure if this is too high, maybe I should try 1e-4", "votes": null }, { "id": "3062304", "postDate": "12/03/2024 13:42:13", "content": "

Yeah thats quite high if you are continuing training, try 1e-5 next and see how it looks

", "rawMarkdown": "Yeah thats quite high if you are continuing training, try 1e-5 next and see how it looks", "votes": null }, { "id": "3062925", "postDate": "12/04/2024 03:24:35", "content": "

We are in the similar condition,could you tell me your batch_size during oneline training?

", "rawMarkdown": "We are in the similar condition,could you tell me your batch_size during oneline training?", "votes": null }, { "id": "3063177", "postDate": "12/04/2024 08:11:16", "content": "

I keep it the same as offline training 8192

", "rawMarkdown": "I keep it the same as offline training 8192", "votes": null }, { "id": "3063291", "postDate": "12/04/2024 10:57:42", "content": "

Thank you for your sharing. Excuse me, can you tell me what this means? 'Using Mimic Data Streaming'

", "rawMarkdown": "Thank you for your sharing. Excuse me, can you tell me what this means? 'Using Mimic Data Streaming'", "votes": null }, { "id": "3063308", "postDate": "12/04/2024 11:28:08", "content": "

Since we cannot view the data after submission, I used the data before partition 9 as the training data and the data in partition 9 as our submission data. The advantage of this approach is that it allows us to quickly adjust parameters and estimate the model's actual performance on the real submission.

", "rawMarkdown": "Since we cannot view the data after submission, I used the data before partition 9 as the training data and the data in partition 9 as our submission data. The advantage of this approach is that it allows us to quickly adjust parameters and estimate the model's actual performance on the real submission.", "votes": null }, { "id": "3063381", "postDate": "12/04/2024 12:52:31", "content": "

What model structure did you use? How did you pretrain and valid the model before entering the online training mode?

", "rawMarkdown": "What model structure did you use? How did you pretrain and valid the model before entering the online training mode?", "votes": null }, { "id": "3063557", "postDate": "12/04/2024 15:52:28", "content": "

Just a simple mlp and use all data to retrain

", "rawMarkdown": "Just a simple mlp and use all data to retrain", "votes": null }, { "id": "3091468", "postDate": "01/08/2025 12:47:45", "content": "

i am running into the same negative score situation, does adjusting the lr help?

", "rawMarkdown": "i am running into the same negative score situation, does adjusting the lr help?", "votes": null }, { "id": "3091898", "postDate": "01/08/2025 21:49:33", "content": "

Yes, it has helped significantly in my case. 1e-5 seems like a good starting point, though I'm using 5e-6 personally.

", "rawMarkdown": "Yes, it has helped significantly in my case. 1e-5 seems like a good starting point, though I'm using 5e-6 personally.", "votes": null } ], "comments": [ { "id": 3062260, "author_name": "julianmukaj", "author_url": "", "post_date": "12/03/2024 12:36:03", "content": "

LR too high perhaps

", "votes": null, "replies": [ { "id": 3062288, "author_name": "xuliangxu", "author_url": "", "post_date": "12/03/2024 13:17:05", "content": "

Thank you for your comment:) LR is 5e-4, I'm not sure if this is too high, maybe I should try 1e-4

", "votes": null, "replies": [ { "id": 3062304, "author_name": "julianmukaj", "author_url": "", "post_date": "12/03/2024 13:42:13", "content": "

Yeah thats quite high if you are continuing training, try 1e-5 next and see how it looks

", "votes": null, "replies": [] }, { "id": 3091468, "author_name": "", "author_url": "", "post_date": "01/08/2025 12:47:45", "content": "

i am running into the same negative score situation, does adjusting the lr help?

", "votes": null, "replies": [ { "id": 3091898, "author_name": "johncaresio", "author_url": "", "post_date": "01/08/2025 21:49:33", "content": "

Yes, it has helped significantly in my case. 1e-5 seems like a good starting point, though I'm using 5e-6 personally.

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3062925, "author_name": "swufeleo", "author_url": "", "post_date": "12/04/2024 03:24:35", "content": "

We are in the similar condition,could you tell me your batch_size during oneline training?

", "votes": null, "replies": [ { "id": 3063177, "author_name": "xuliangxu", "author_url": "", "post_date": "12/04/2024 08:11:16", "content": "

I keep it the same as offline training 8192

", "votes": null, "replies": [] } ] }, { "id": 3063291, "author_name": "jaewook704", "author_url": "", "post_date": "12/04/2024 10:57:42", "content": "

Thank you for your sharing. Excuse me, can you tell me what this means? 'Using Mimic Data Streaming'

", "votes": null, "replies": [ { "id": 3063308, "author_name": "xuliangxu", "author_url": "", "post_date": "12/04/2024 11:28:08", "content": "

Since we cannot view the data after submission, I used the data before partition 9 as the training data and the data in partition 9 as our submission data. The advantage of this approach is that it allows us to quickly adjust parameters and estimate the model's actual performance on the real submission.

", "votes": null, "replies": [] } ] }, { "id": 3063381, "author_name": "shiyili", "author_url": "", "post_date": "12/04/2024 12:52:31", "content": "

What model structure did you use? How did you pretrain and valid the model before entering the online training mode?

", "votes": null, "replies": [ { "id": 3063557, "author_name": "xuliangxu", "author_url": "", "post_date": "12/04/2024 15:52:28", "content": "

Just a simple mlp and use all data to retrain

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3062246": "I’ve seen many people achieve some improvements in their scores through online learning, so I decided to give it a try. I trained a neural network model, but the results didn’t improve—in fact, they got worse. When I used my offline model for predictions, my public score was 0.0041. However, with the online learning approach, I updated the model daily using the data from the last two days, kept the same learning rate, and trained for 3 epochs each time. Surprisingly, the score dropped to -0.0142.\n\nI’d like to ask for advice: How should online learning be configured properly? Should I use only the new data completely? During training, should I freeze some parameters?\n\n---------------", "3062260": "LR too high perhaps", "3062288": "Thank you for your comment:) LR is 5e-4, I'm not sure if this is too high, maybe I should try 1e-4", "3062304": "Yeah thats quite high if you are continuing training, try 1e-5 next and see how it looks", "3062925": "We are in the similar condition,could you tell me your batch_size during oneline training?", "3063177": "I keep it the same as offline training 8192", "3063291": "Thank you for your sharing. Excuse me, can you tell me what this means? 'Using Mimic Data Streaming'", "3063308": "Since we cannot view the data after submission, I used the data before partition 9 as the training data and the data in partition 9 as our submission data. The advantage of this approach is that it allows us to quickly adjust parameters and estimate the model's actual performance on the real submission.", "3063381": "What model structure did you use? How did you pretrain and valid the model before entering the online training mode?", "3063557": "Just a simple mlp and use all data to retrain", "3091468": "i am running into the same negative score situation, does adjusting the lr help?", "3091898": "Yes, it has helped significantly in my case. 1e-5 seems like a good starting point, though I'm using 5e-6 personally." }, "source": "meta" }