![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F18244331%2Fc305c364f1be82f59b85694a97acdc9b%2F_20241221043223.png?generation=1734755559454056&alt=media\")
Following my previous post, where I failed to improve scores using partition 9 data as mock submission data, I’ve made some progress but also hit new roadblocks. Still, I’ve gained some insights that I’d like to share in the hope that they can help others. Best of luck to everyone in achieving great results!
\nCurrently, I’ve achieved some improvement on the partition 9 dataset, which I used as mock online learning data. The score increased from 0.0063 to 0.0157 through online learning. Here’s what I’ve learned:
\nI hope these points might be helpful to others.
\nHowever, I’ve now encountered a new issue (which is incredibly frustrating). When I continue to submit using the same strategy, the scores with and without online learning both end up being 0.0037. At least, unlike last time, the score didn’t get worse 😂, but I’m struggling to understand where the gap lies that caused this result.
", "messages": [ { "id": 3067144, "postDate": "2024-12-09T00:56:06.933Z", "content": "Following my previous post, where I failed to improve scores using partition 9 data as mock submission data, I’ve made some progress but also hit new roadblocks. Still, I’ve gained some insights that I’d like to share in the hope that they can help others. Best of luck to everyone in achieving great results!
\nCurrently, I’ve achieved some improvement on the partition 9 dataset, which I used as mock online learning data. The score increased from 0.0063 to 0.0157 through online learning. Here’s what I’ve learned:
\nI hope these points might be helpful to others.
\nHowever, I’ve now encountered a new issue (which is incredibly frustrating). When I continue to submit using the same strategy, the scores with and without online learning both end up being 0.0037. At least, unlike last time, the score didn’t get worse 😂, but I’m struggling to understand where the gap lies that caused this result.
", "rawMarkdown": "Following my previous post, where I failed to improve scores using partition 9 data as mock submission data, I’ve made some progress but also hit new roadblocks. Still, I’ve gained some insights that I’d like to share in the hope that they can help others. Best of luck to everyone in achieving great results! \n\nCurrently, I’ve achieved some improvement on the partition 9 dataset, which I used as mock online learning data. The score increased from 0.0063 to 0.0157 through online learning. Here’s what I’ve learned: \n\n1. **Set up a mock submission API.** \n2. **Avoid using only new data in the dataset.** For my model, maintaining a sliding window of data for updates proved effective (since I found that my model is very sensitive to small changes). \n3. **Pay special attention to the learning rate.** Again, because my model is sensitive to small changes, I’m currently using a learning rate of 1e-5, which is much smaller than the learning rate of my initial model. \n4. **Change the activation function of the final layer to linear.** I found this adjustment yielded a more robust outcome, and the results confirmed it. \n5. **Lock certain parameters during retraining** to prevent catastrophic forgetting in the model. \n\nI hope these points might be helpful to others. \n\nHowever, I’ve now encountered a new issue (which is incredibly frustrating). When I continue to submit using the same strategy, the scores with and without online learning both end up being 0.0037. At least, unlike last time, the score didn’t get worse 😂, but I’m struggling to understand where the gap lies that caused this result. ", "votes": 13 }, { "id": 3067558, "postDate": "2024-12-09T12:08:28.287Z", "content": "Sounds like you may forgot set online learning to True in your submission code? And 1e-5 is indeed very small. As a comparison, I used 1e-4 but I know this depend on also your model architecture and pretraining strategy.
", "rawMarkdown": "Sounds like you may forgot set online learning to True in your submission code? And 1e-5 is indeed very small. As a comparison, I used 1e-4 but I know this depend on also your model architecture and pretraining strategy. ", "votes": 3, "replies": [ { "id": 3067599, "postDate": "2024-12-09T12:59:24.323Z", "content": "Wow! Thank you very much for your comment! Could you please help me review this code as well? I've gone through it multiple times, but I really can't figure out where the issue might be. Here’s the partial process of my mock online learning:
\n date_id unique_date_ids:\n \n test = df.(pl.col() == date_id)\n test = preprocess(test)\n\n \n X_test = test.select(feature_columns).to_numpy()\n nn_pred = nn_model.predict(X_test)\n\n \n online_training_df = pl.concat([online_training_df, test], how=)\n online_date_ids = online_training_df.select().unique().sort()[].to_list()\n online_training_df = online_training_df.(pl.col()!=online_date_ids[])\n \n online_training()\nHere is my submission code:
\n lags :\n \n (test_tmp) > :\n test_date = pl.concat(test_tmp, how=)\n test_date = test_date.with_columns(pl.col().cast(pl.Int16))\n test_date = test_date.with_columns(pl.col().cast(pl.Int16))\n test_date = test_date.with_columns(pl.col().cast(pl.Int8))\n lags = lags.with_columns([(pl.col() - ).alias()])\n test_date = test_date.join(\n lags.select([, , , ]).rename({: }),\n on=[, , ],\n how=\n )\n test_date_tmp.append(test_date)\n test_tmp.clear()\n \n (test_date_tmp) > :\n \n new_data = pl.concat(test_date_tmp, how=).select(\n feature_columns + [weight_column, , target_column]\n )\n new_data = new_data.(\n ~pl.col(weight_column).is_null() & ~pl.col(target_column).is_null() &\n ~pl.col(weight_column).is_nan() & ~pl.col(target_column).is_nan()\n )\n new_data.height > :\n new_data = preprocess(new_data)\n window_df = pl.concat([window_df, new_data], how=)\n \n window_df = window_df.(\n pl.col() != window_df.select().head().to_series().item()\n )\n \n online_training()\n \n test_date_tmp.clear()\n :\n test_date_tmp.clear()\n\n test_predict = preprocess(test)\n test_tmp.append(test)\n\n X_test = test_predict.select(feature_columns).to_numpy()\nHard to say which part is wrong or it's not wrong at all. I would suggest you do some local test to make sure when you submit, the online training logic is indeed triggered. For example, in my case, I have prepared a dataset exactly with same same format to the test dataset at the very beginning and run the test run everytime before I submit to estimate how long the submission will take and make sure the logic is correctly triggered.
\ninference_server = kaggle_evaluation.jane_street_inference_server.JSInferenceServer(predict)\n\n os.getenv():\n inference_server.serve()\n:\n CONFIG.DEBUG:\n inference_server.run_local_gateway(\n (\n ,\n \n )\n )\n :\n inference_server.run_local_gateway(\n (\n ,\n ,\n )\n )\nThank you very much for your advice! I may have genuinely overlooked something. I will retest using this method today to identify where the issue lies.
", "rawMarkdown": "Thank you very much for your advice! I may have genuinely overlooked something. I will retest using this method today to identify where the issue lies." }, { "id": 3077510, "postDate": "2024-12-21T04:32:52.093Z", "content": "Hi!Hao, may I know if you have encountered of problem of doing online learning of NN? I have tested my data on this simulator, it works perfectly with a result but I failed immediately on the submission. Really appreciate your response!
I didn't see your code and which simulator do you mean? Could it be be simulator itself is problematic?
", "rawMarkdown": "I didn't see your code and which simulator do you mean? Could it be be simulator itself is problematic?", "votes": 1 }, { "id": 3077805, "postDate": "2024-12-21T12:20:35.020Z", "content": "I use this one:
\nhttps://www.kaggle.com/code/shiyili/js24-rmf-submission-api-debug-with-synthetic-test
\nand it runs well, I use trainer to retrain, my code is like this:
\n ds = NNDataset(X_nn_train, y_nn_train, w_nn_train)\n dl = DataLoader(ds, batch_size = 2048), =)\n\n trainer = Trainer(\n =CONFIG.nn_retrain_epochs,\n =,\n =,\n =\n )\n\n model nn_models:\n model.train()\n model.(device)\n trainer.fit(model, dl)\n model.eval()\n model.(device)\n ()\nMay I get your insight?
", "rawMarkdown": "I use this one:\n\nhttps://www.kaggle.com/code/shiyili/js24-rmf-submission-api-debug-with-synthetic-test\n\nand it runs well, I use trainer to retrain, my code is like this:\n\n ds = NNDataset(X_nn_train, y_nn_train, w_nn_train)\n dl = DataLoader(ds, batch_size = 2048), shuffle=False)\n \n trainer = Trainer(\n max_epochs=CONFIG.nn_retrain_epochs,\n enable_progress_bar=True,\n logger=False,\n enable_checkpointing=False\n )\n \n for model in nn_models:\n model.train()\n model.to(device)\n trainer.fit(model, dl)\n model.eval()\n model.to(device)\n print(\"[Retrain] NN models fine-tuned.\")\n\nMay I get your insight?" }, { "id": 3077843, "postDate": "2024-12-21T13:00:34.233Z", "content": "From what I see, it looks like some issues unrelated to model retraining, like missing values imputation, possible new symbol_ids, data alignment etc… You may check them out rather than focusing on model re-training. But that's just my guess.
", "rawMarkdown": "From what I see, it looks like some issues unrelated to model retraining, like missing values imputation, possible new symbol_ids, data alignment etc... You may check them out rather than focusing on model re-training. But that's just my guess. " }, { "id": 3077868, "postDate": "2024-12-21T13:21:49.697Z", "content": "I did not put the symbol id in my model lol, may I just ask do you use a integrated datamodule like DataModule of pytorchlightning in the retraining session or you write your own version?
", "rawMarkdown": " I did not put the symbol id in my model lol, may I just ask do you use a integrated datamodule like DataModule of pytorchlightning in the retraining session or you write your own version?" }, { "id": 3077878, "postDate": "2024-12-21T13:33:50.210Z", "content": "The retraining module is nothing but using the same training epoch function (used in offline training) with updated data, so no extra modules.
", "rawMarkdown": "The retraining module is nothing but using the same training epoch function (used in offline training) with updated data, so no extra modules. " } ] }, { "id": 3067610, "postDate": "2024-12-09T13:13:39.367Z", "content": "Regarding the learning rate, I have tested 1e-4, but the results were not good. It seems that my model is very sensitive. Thank you again for your comment!
", "rawMarkdown": "Regarding the learning rate, I have tested 1e-4, but the results were not good. It seems that my model is very sensitive. Thank you again for your comment!", "votes": 1 } ] }, { "id": 3067319, "postDate": "2024-12-09T06:34:08.483Z", "content": "Do you use some sort of sliding window normalization for features?
\nThx for sharing.
I haven't used the sliding window normalization strategy yet. I hope to first improve the online learning pipeline to achieve some improvements before making adjustments and optimizations.
", "rawMarkdown": "I haven't used the sliding window normalization strategy yet. I hope to first improve the online learning pipeline to achieve some improvements before making adjustments and optimizations." } ] }, { "id": 3067192, "postDate": "2024-12-09T02:52:02.850Z", "content": "So you used whole part 9 data as the validation set and incrementally learning on the set day by day?
\nIt is very interesting the strategy leading to such great improvement offline struggles online.
\nMaybe you should try early stop during incremental learning.
\nThx for sharing.
Actually, incrementally learning on the set day by day wasn't my original intention 😂. I just wanted to test the effectiveness of my online learning strategy on a partition of the data. Another pattern I've observed, which I can share, is that the score I get in the submission for my initial model is the same regardless of whether the model was trained using Part 4, 5, 6, 7, 8 or Part 4, 5, 6, 7, 8, 9.
", "rawMarkdown": "Actually, incrementally learning on the set day by day wasn't my original intention 😂. I just wanted to test the effectiveness of my online learning strategy on a partition of the data. Another pattern I've observed, which I can share, is that the score I get in the submission for my initial model is the same regardless of whether the model was trained using Part 4, 5, 6, 7, 8 or Part 4, 5, 6, 7, 8, 9." } ] }, { "id": 3067145, "postDate": "2024-12-09T01:06:59.600Z", "content": "i believe part reason the model is getting worse is because of batchnorms layer.
\nit reinitiate itself with a whole set of values that make the model forgets stuff.
Thank you so much for your insight! However, based on my understanding, since it's online learning, doesn't it inherently mean learning and forgetting simultaneously? So batch normalization doesn't seem to be an issue, right? Or am I misunderstanding something?
", "rawMarkdown": "Thank you so much for your insight! However, based on my understanding, since it's online learning, doesn't it inherently mean learning and forgetting simultaneously? So batch normalization doesn't seem to be an issue, right? Or am I misunderstanding something?", "replies": [ { "id": 3067423, "postDate": "2024-12-09T09:33:41.553Z", "content": "Depending on where you put the norm. In some cases norm can dramatically change the distribution of the input.
", "rawMarkdown": "Depending on where you put the norm. In some cases norm can dramatically change the distribution of the input.", "votes": 1 }, { "id": 3067518, "postDate": "2024-12-09T11:14:08.697Z", "content": "Thank you for your explanation. I still need to think more deeply about the operation of batch normalization. I just can’t understand why a strategy that works successfully in Part 9 would fail in the submission…
", "rawMarkdown": "Thank you for your explanation. I still need to think more deeply about the operation of batch normalization. I just can’t understand why a strategy that works successfully in Part 9 would fail in the submission..." } ] } ] }, { "id": 3068321, "postDate": "2024-12-10T06:32:55.240Z", "content": "I also have a much smaller LR for online learning, even smaller than yours. How many epochs do you use? I typically train for 10 epochs per day in the online portion. Your point #2 is interesting; I use a sliding window of data for online learning in LGBM models, but haven't used that in NN models yet. I haven't experimented with changing the output layer activation function; I presume that a linear function would increase the transmission of the gradient, and thus require a smaller LR again. On point #5, I guess the trick is to know \"which\" parameters? :)
", "rawMarkdown": "I also have a much smaller LR for online learning, even smaller than yours. How many epochs do you use? I typically train for 10 epochs per day in the online portion. Your point #2 is interesting; I use a sliding window of data for online learning in LGBM models, but haven't used that in NN models yet. I haven't experimented with changing the output layer activation function; I presume that a linear function would increase the transmission of the gradient, and thus require a smaller LR again. On point #5, I guess the trick is to know \"which\" parameters? :)", "votes": 2, "replies": [ { "id": 3068458, "postDate": "2024-12-10T09:36:34.397Z", "content": "Thank you very much for your comment and insight! I believe I have gained a new perspective on the problem. Considering points #2, #3, and #5, my overall approach is to update the model's parameters while maintaining its generalization ability and avoiding overfitting. I have frozen some parameters, so perhaps I can \"safely\" use a larger learning rate and more epochs (I used 20). Regarding which parameters to freeze, since I haven't achieved any good results on the leaderboard yet, it may not be very convincing. However, I think a simple principle is to freeze the earlier layers, as these layers typically learn more general and stable feature representations.
", "rawMarkdown": "Thank you very much for your comment and insight! I believe I have gained a new perspective on the problem. Considering points #2, #3, and #5, my overall approach is to update the model's parameters while maintaining its generalization ability and avoiding overfitting. I have frozen some parameters, so perhaps I can \"safely\" use a larger learning rate and more epochs (I used 20). Regarding which parameters to freeze, since I haven't achieved any good results on the leaderboard yet, it may not be very convincing. However, I think a simple principle is to freeze the earlier layers, as these layers typically learn more general and stable feature representations.", "votes": 1 } ] }, { "id": 3089282, "postDate": "2025-01-05T18:49:31.237Z", "content": "How did you setup mock submission API?
", "rawMarkdown": "How did you setup mock submission API?" }, { "id": 3083999, "postDate": "2024-12-30T09:00:44.413Z", "content": "What was the problem in the end? I am facing the same issue.
\nThank you very much for the discussion, nice one!
https://www.kaggle.com/code/chumajin/janestreet-updated-simulator-for-time-series-api
\nMaybe you can use the simulator to dubug your code.Besides, do you use any normalization method or neural network layer?Such as Batchnorm or Layernorm.
I used to do the same, make sure the code is bug free and adjust the learning rate and times to have the effect.
\nAs for your 5 point, I have also considered it, but if I freeze some parameters of the model, the effect will become worse, and the final effect will not be as good as updating the whole.
And, as far as I'm concerned, I feel that random sampling in data processing is a more \"robust\" method, at least for me, more useful than other methods… Just my feelings, of course, experiential replay is a good option.
", "rawMarkdown": "And, as far as I'm concerned, I feel that random sampling in data processing is a more \"robust\" method, at least for me, more useful than other methods... Just my feelings, of course, experiential replay is a good option." } ] } ], "comments": [ { "id": 3067558, "author_name": "HAO", "author_url": "", "post_date": "2024-12-09T12:08:28.287000", "content": "Sounds like you may forgot set online learning to True in your submission code? And 1e-5 is indeed very small. As a comparison, I used 1e-4 but I know this depend on also your model architecture and pretraining strategy.
", "votes": 3, "replies": [ { "id": 3067599, "author_name": "Xuliang Xu", "author_url": "", "post_date": "2024-12-09T12:59:24.323000", "content": "Wow! Thank you very much for your comment! Could you please help me review this code as well? I've gone through it multiple times, but I really can't figure out where the issue might be. Here’s the partial process of my mock online learning:
\n date_id unique_date_ids:\n \n test = df.(pl.col() == date_id)\n test = preprocess(test)\n\n \n X_test = test.select(feature_columns).to_numpy()\n nn_pred = nn_model.predict(X_test)\n\n \n online_training_df = pl.concat([online_training_df, test], how=)\n online_date_ids = online_training_df.select().unique().sort()[].to_list()\n online_training_df = online_training_df.(pl.col()!=online_date_ids[])\n \n online_training()\nHere is my submission code:
\n lags :\n \n (test_tmp) > :\n test_date = pl.concat(test_tmp, how=)\n test_date = test_date.with_columns(pl.col().cast(pl.Int16))\n test_date = test_date.with_columns(pl.col().cast(pl.Int16))\n test_date = test_date.with_columns(pl.col().cast(pl.Int8))\n lags = lags.with_columns([(pl.col() - ).alias()])\n test_date = test_date.join(\n lags.select([, , , ]).rename({: }),\n on=[, , ],\n how=\n )\n test_date_tmp.append(test_date)\n test_tmp.clear()\n \n (test_date_tmp) > :\n \n new_data = pl.concat(test_date_tmp, how=).select(\n feature_columns + [weight_column, , target_column]\n )\n new_data = new_data.(\n ~pl.col(weight_column).is_null() & ~pl.col(target_column).is_null() &\n ~pl.col(weight_column).is_nan() & ~pl.col(target_column).is_nan()\n )\n new_data.height > :\n new_data = preprocess(new_data)\n window_df = pl.concat([window_df, new_data], how=)\n \n window_df = window_df.(\n pl.col() != window_df.select().head().to_series().item()\n )\n \n online_training()\n \n test_date_tmp.clear()\n :\n test_date_tmp.clear()\n\n test_predict = preprocess(test)\n test_tmp.append(test)\n\n X_test = test_predict.select(feature_columns).to_numpy()\nHard to say which part is wrong or it's not wrong at all. I would suggest you do some local test to make sure when you submit, the online training logic is indeed triggered. For example, in my case, I have prepared a dataset exactly with same same format to the test dataset at the very beginning and run the test run everytime before I submit to estimate how long the submission will take and make sure the logic is correctly triggered.
\ninference_server = kaggle_evaluation.jane_street_inference_server.JSInferenceServer(predict)\n\n os.getenv():\n inference_server.serve()\n:\n CONFIG.DEBUG:\n inference_server.run_local_gateway(\n (\n ,\n \n )\n )\n :\n inference_server.run_local_gateway(\n (\n ,\n ,\n )\n )\nThank you very much for your advice! I may have genuinely overlooked something. I will retest using this method today to identify where the issue lies.
", "votes": 0, "replies": [] }, { "id": 3077510, "author_name": "Mr RRR", "author_url": "", "post_date": "2024-12-21T04:32:52.093000", "content": "Hi!Hao, may I know if you have encountered of problem of doing online learning of NN? I have tested my data on this simulator, it works perfectly with a result but I failed immediately on the submission. Really appreciate your response!
I didn't see your code and which simulator do you mean? Could it be be simulator itself is problematic?
", "votes": 1, "replies": [] }, { "id": 3077805, "author_name": "Mr RRR", "author_url": "", "post_date": "2024-12-21T12:20:35.020000", "content": "I use this one:
\nhttps://www.kaggle.com/code/shiyili/js24-rmf-submission-api-debug-with-synthetic-test
\nand it runs well, I use trainer to retrain, my code is like this:
\n ds = NNDataset(X_nn_train, y_nn_train, w_nn_train)\n dl = DataLoader(ds, batch_size = 2048), =)\n\n trainer = Trainer(\n =CONFIG.nn_retrain_epochs,\n =,\n =,\n =\n )\n\n model nn_models:\n model.train()\n model.(device)\n trainer.fit(model, dl)\n model.eval()\n model.(device)\n ()\nMay I get your insight?
", "votes": 0, "replies": [] }, { "id": 3077843, "author_name": "HAO", "author_url": "", "post_date": "2024-12-21T13:00:34.233000", "content": "From what I see, it looks like some issues unrelated to model retraining, like missing values imputation, possible new symbol_ids, data alignment etc… You may check them out rather than focusing on model re-training. But that's just my guess.
", "votes": 0, "replies": [] }, { "id": 3077868, "author_name": "Mr RRR", "author_url": "", "post_date": "2024-12-21T13:21:49.697000", "content": "I did not put the symbol id in my model lol, may I just ask do you use a integrated datamodule like DataModule of pytorchlightning in the retraining session or you write your own version?
", "votes": 0, "replies": [] }, { "id": 3077878, "author_name": "HAO", "author_url": "", "post_date": "2024-12-21T13:33:50.210000", "content": "The retraining module is nothing but using the same training epoch function (used in offline training) with updated data, so no extra modules.
", "votes": 0, "replies": [] } ] }, { "id": 3067610, "author_name": "Xuliang Xu", "author_url": "", "post_date": "2024-12-09T13:13:39.367000", "content": "Regarding the learning rate, I have tested 1e-4, but the results were not good. It seems that my model is very sensitive. Thank you again for your comment!
", "votes": 1, "replies": [] } ] }, { "id": 3067319, "author_name": "Alexey", "author_url": "", "post_date": "2024-12-09T06:34:08.483000", "content": "Do you use some sort of sliding window normalization for features?
\nThx for sharing.
I haven't used the sliding window normalization strategy yet. I hope to first improve the online learning pipeline to achieve some improvements before making adjustments and optimizations.
", "votes": 0, "replies": [] } ] }, { "id": 3067192, "author_name": "Pengqian Lu", "author_url": "", "post_date": "2024-12-09T02:52:02.850000", "content": "So you used whole part 9 data as the validation set and incrementally learning on the set day by day?
\nIt is very interesting the strategy leading to such great improvement offline struggles online.
\nMaybe you should try early stop during incremental learning.
\nThx for sharing.
Actually, incrementally learning on the set day by day wasn't my original intention 😂. I just wanted to test the effectiveness of my online learning strategy on a partition of the data. Another pattern I've observed, which I can share, is that the score I get in the submission for my initial model is the same regardless of whether the model was trained using Part 4, 5, 6, 7, 8 or Part 4, 5, 6, 7, 8, 9.
", "votes": 0, "replies": [] } ] }, { "id": 3067145, "author_name": "ZT", "author_url": "", "post_date": "2024-12-09T01:06:59.600000", "content": "i believe part reason the model is getting worse is because of batchnorms layer.
\nit reinitiate itself with a whole set of values that make the model forgets stuff.
Thank you so much for your insight! However, based on my understanding, since it's online learning, doesn't it inherently mean learning and forgetting simultaneously? So batch normalization doesn't seem to be an issue, right? Or am I misunderstanding something?
", "votes": 0, "replies": [ { "id": 3067423, "author_name": "SLi", "author_url": "", "post_date": "2024-12-09T09:33:41.553000", "content": "Depending on where you put the norm. In some cases norm can dramatically change the distribution of the input.
", "votes": 1, "replies": [] }, { "id": 3067518, "author_name": "Xuliang Xu", "author_url": "", "post_date": "2024-12-09T11:14:08.697000", "content": "Thank you for your explanation. I still need to think more deeply about the operation of batch normalization. I just can’t understand why a strategy that works successfully in Part 9 would fail in the submission…
", "votes": 0, "replies": [] } ] } ] }, { "id": 3068321, "author_name": "Maciej Zawadzki", "author_url": "", "post_date": "2024-12-10T06:32:55.240000", "content": "I also have a much smaller LR for online learning, even smaller than yours. How many epochs do you use? I typically train for 10 epochs per day in the online portion. Your point #2 is interesting; I use a sliding window of data for online learning in LGBM models, but haven't used that in NN models yet. I haven't experimented with changing the output layer activation function; I presume that a linear function would increase the transmission of the gradient, and thus require a smaller LR again. On point #5, I guess the trick is to know \"which\" parameters? :)
", "votes": 2, "replies": [ { "id": 3068458, "author_name": "Xuliang Xu", "author_url": "", "post_date": "2024-12-10T09:36:34.397000", "content": "Thank you very much for your comment and insight! I believe I have gained a new perspective on the problem. Considering points #2, #3, and #5, my overall approach is to update the model's parameters while maintaining its generalization ability and avoiding overfitting. I have frozen some parameters, so perhaps I can \"safely\" use a larger learning rate and more epochs (I used 20). Regarding which parameters to freeze, since I haven't achieved any good results on the leaderboard yet, it may not be very convincing. However, I think a simple principle is to freeze the earlier layers, as these layers typically learn more general and stable feature representations.
", "votes": 1, "replies": [] } ] }, { "id": 3089282, "author_name": "Pankaj Gupta", "author_url": "", "post_date": "2025-01-05T18:49:31.237000", "content": "How did you setup mock submission API?
", "votes": 0, "replies": [] }, { "id": 3083999, "author_name": "Lorenzo Campana", "author_url": "", "post_date": "2024-12-30T09:00:44.413000", "content": "What was the problem in the end? I am facing the same issue.
\nThank you very much for the discussion, nice one!
https://www.kaggle.com/code/chumajin/janestreet-updated-simulator-for-time-series-api
\nMaybe you can use the simulator to dubug your code.Besides, do you use any normalization method or neural network layer?Such as Batchnorm or Layernorm.
I used to do the same, make sure the code is bug free and adjust the learning rate and times to have the effect.
\nAs for your 5 point, I have also considered it, but if I freeze some parameters of the model, the effect will become worse, and the final effect will not be as good as updating the whole.
And, as far as I'm concerned, I feel that random sampling in data processing is a more \"robust\" method, at least for me, more useful than other methods… Just my feelings, of course, experiential replay is a good option.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "3067144": "Following my previous post, where I failed to improve scores using partition 9 data as mock submission data, I’ve made some progress but also hit new roadblocks. Still, I’ve gained some insights that I’d like to share in the hope that they can help others. Best of luck to everyone in achieving great results! \n\nCurrently, I’ve achieved some improvement on the partition 9 dataset, which I used as mock online learning data. The score increased from 0.0063 to 0.0157 through online learning. Here’s what I’ve learned: \n\n1. **Set up a mock submission API.** \n2. **Avoid using only new data in the dataset.** For my model, maintaining a sliding window of data for updates proved effective (since I found that my model is very sensitive to small changes). \n3. **Pay special attention to the learning rate.** Again, because my model is sensitive to small changes, I’m currently using a learning rate of 1e-5, which is much smaller than the learning rate of my initial model. \n4. **Change the activation function of the final layer to linear.** I found this adjustment yielded a more robust outcome, and the results confirmed it. \n5. **Lock certain parameters during retraining** to prevent catastrophic forgetting in the model. \n\nI hope these points might be helpful to others. \n\nHowever, I’ve now encountered a new issue (which is incredibly frustrating). When I continue to submit using the same strategy, the scores with and without online learning both end up being 0.0037. At least, unlike last time, the score didn’t get worse 😂, but I’m struggling to understand where the gap lies that caused this result. ", "3067558": "Sounds like you may forgot set online learning to True in your submission code? And 1e-5 is indeed very small. As a comparison, I used 1e-4 but I know this depend on also your model architecture and pretraining strategy. ", "3067319": "Do you use some sort of sliding window normalization for features?\nThx for sharing.", "3067192": "So you used whole part 9 data as the validation set and incrementally learning on the set day by day?\nIt is very interesting the strategy leading to such great improvement offline struggles online.\nMaybe you should try early stop during incremental learning.\nThx for sharing.", "3067145": "i believe part reason the model is getting worse is because of batchnorms layer.\nit reinitiate itself with a whole set of values that make the model forgets stuff. ", "3068321": "I also have a much smaller LR for online learning, even smaller than yours. How many epochs do you use? I typically train for 10 epochs per day in the online portion. Your point #2 is interesting; I use a sliding window of data for online learning in LGBM models, but haven't used that in NN models yet. I haven't experimented with changing the output layer activation function; I presume that a linear function would increase the transmission of the gradient, and thus require a smaller LR again. On point #5, I guess the trick is to know \"which\" parameters? :)", "3089282": "How did you setup mock submission API?", "3083999": "What was the problem in the end? I am facing the same issue. \nThank you very much for the discussion, nice one!", "3068501": "https://www.kaggle.com/code/chumajin/janestreet-updated-simulator-for-time-series-api\nMaybe you can use the simulator to dubug your code.Besides, do you use any normalization method or neural network layer?Such as Batchnorm or Layernorm.", "3067751": "I used to do the same, make sure the code is bug free and adjust the learning rate and times to have the effect. \nAs for your 5 point, I have also considered it, but if I freeze some parameters of the model, the effect will become worse, and the final effect will not be as good as updating the whole." } }