{ "id": 548978, "title": "Ways to improve R2", "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/548978", "author_name": "", "post_date": "2024-11-30T03:57:11.472552900Z", "votes": 10, "comment_count": 13, "views": 0, "content": "

Its been 2 month, and I haven't found anything that really working.

\n
    \n

  1. I have tried different model structures, stacking dense layers, LSTM, Conv, attention… but they are not as good as the baseline.

    2. \n

  2. Features engineering did not seemed to help either, nomalization, weights selection etc.. getting even worse score than my baseline.

    3. \n
\n

What are the possible ways to improve score offline? I am currently clueless…

", "messages": [ { "id": "3058839", "postDate": "11/30/2024 03:57:11", "content": "

Its been 2 month, and I haven't found anything that really working.

\n
    \n

  1. I have tried different model structures, stacking dense layers, LSTM, Conv, attention… but they are not as good as the baseline.

    2. \n

  2. Features engineering did not seemed to help either, nomalization, weights selection etc.. getting even worse score than my baseline.

    3. \n
\n

What are the possible ways to improve score offline? I am currently clueless…

", "rawMarkdown": "Its been 2 month, and I haven't found anything that really working.\n\n1. I have tried different model structures, stacking dense layers, LSTM, Conv, attention... but they are not as good as the baseline. \n\n2. Features engineering did not seemed to help either, nomalization, weights selection etc.. getting even worse score than my baseline.\n\nWhat are the possible ways to improve score offline? I am currently clueless...", "votes": null }, { "id": "3062065", "postDate": "12/03/2024 08:47:06", "content": "

Hey can u help me with this this error \"Notebook Inference Server Error\" when i try to submit
\nthis is the test notebook https://www.kaggle.com/code/sudhirsars/jn-tester
\nand this is the training notebook https://www.kaggle.com/code/sudhirsars/trainer-jn

\n

can you guyz please have a look and guide me where iam wrong

", "rawMarkdown": "Hey can u help me with this this error \"Notebook Inference Server Error\" when i try to submit\nthis is the test notebook https://www.kaggle.com/code/sudhirsars/jn-tester\nand this is the training notebook https://www.kaggle.com/code/sudhirsars/trainer-jn\n\ncan you guyz please have a look and guide me where iam wrong", "votes": null }, { "id": "3062075", "postDate": "12/03/2024 09:06:05", "content": "

inference error means you have something that it can not find.
\ndid you put up the right path to get the model etc.
\ndebug outside the predict function

", "rawMarkdown": "inference error means you have something that it can not find.\ndid you put up the right path to get the model etc.\ndebug outside the predict function", "votes": null }, { "id": "3062077", "postDate": "12/03/2024 09:12:54", "content": "

I would suggest sticking to one architecture and developing a robust pipeline for online learning. If everything you did is solid and makes sense, 0.006+ is definitely can be expected, no matter what architectures you are using.

", "rawMarkdown": "I would suggest sticking to one architecture and developing a robust pipeline for online learning. If everything you did is solid and makes sense, 0.006+ is definitely can be expected, no matter what architectures you are using.", "votes": null }, { "id": "3062085", "postDate": "12/03/2024 09:24:46", "content": "

Every thing is good, even iam able to print final prediction in predict function still I get this error every time you can look at logs of test notebook I have no logged error from inference server but still when I submit the notebook it get this error on the submissions panel

", "rawMarkdown": "Every thing is good, even iam able to print final prediction in predict function still I get this error every time you can look at logs of test notebook I have no logged error from inference server but still when I submit the notebook it get this error on the submissions panel", "votes": null }, { "id": "3062123", "postDate": "12/03/2024 10:05:00", "content": "

Congrats on your boost to the new rank!

\n

May I ask what have you seen that makes a big difference in your experiments? I understand that online learning is quite critical, but I have tested several model architectures with different online training scheme and found no luck. I must have overlooked something. Normalization? Imputation? Feature engineering?

\n

Thank you so much and it would be greatly appreciated if you could share some thoughts.

", "rawMarkdown": "Congrats on your boost to the new rank!\n\nMay I ask what have you seen that makes a big difference in your experiments? I understand that online learning is quite critical, but I have tested several model architectures with different online training scheme and found no luck. I must have overlooked something. Normalization? Imputation? Feature engineering?\n\nThank you so much and it would be greatly appreciated if you could share some thoughts.", "votes": null }, { "id": "3062169", "postDate": "12/03/2024 11:01:51", "content": "
\n

May I ask what have you seen that makes a big difference in your experiments?

\n
\n

This is a big quesition. Honestly, I'm also still exporing different possibilities. But from what see in my experiments, setting up robust local test to simulate the online env is very important. With a robust pipeline, you could test all the things you mentioned (models, normalisations, features etc.) very easily. One thing to notice here is aligning how you train your model with how data is feeded when you submit your code.

\n
\n

Model architectures

\n
\n

As I mentioned, from my experiments a very simple mlp with online learning can already lead to very far. You should expect to see 0.002+ improvement with same model by adding online learning. You don't need to worry about other things before you achieve this online training scheme.

\n
\n

Normalisation?

\n
\n

It's suppsed to be very important as discussed in another great post. But it's not really necessary to have a perfect normalisation method to have a decent score. At least at this moment, I'm still using global mean/std normalisation.

\n
\n

Imputation?

\n
\n

Same to normalisation. I'm just using 0 for imputation. Will explore other possibilities later.

\n
\n

Feature engineering?

\n
\n

NN is suposed to be great to do end-to-end learning, right? Try to design models to learn it.

", "rawMarkdown": ">May I ask what have you seen that makes a big difference in your experiments?\n\nThis is a big quesition. Honestly, I'm also still exporing different possibilities. But from what see in my experiments, setting up robust local test to simulate the online env is very important. With a robust pipeline, you could test all the things you mentioned (models, normalisations, features etc.) very easily. One thing to notice here is aligning how you train your model with how data is feeded when you submit your code.\n>Model architectures\n\nAs I mentioned, from my experiments a very simple mlp with online learning can already lead to very far. You should expect to see 0.002+ improvement with same model by adding online learning. You don't need to worry about other things before you achieve this online training scheme. \n>Normalisation? \n\nIt's suppsed to be very important as discussed in another great post. But it's not really necessary to have a perfect normalisation method to have a decent score. At least at this moment, I'm still using global mean/std normalisation.\n\n>Imputation? \n\nSame to normalisation. I'm just using 0 for imputation. Will explore other possibilities later. \n>Feature engineering? \n\nNN is suposed to be great to do end-to-end learning, right? Try to design models to learn it.", "votes": null }, { "id": "3062207", "postDate": "12/03/2024 11:33:05", "content": "

Are you able to share some more on online learning? Are you training an each days data/stacking history before training? I have a OL pipeline but I am not seeing these boosts everyone else is, I am only seeing 0.0005 on LB effectively. Even with local testing, online training on the last 150 days for example, the boost is a bit higher in CV but LB is not that drastic.. Not sure how others are approaching it, I tend to get catastrophic forgetting if I use Live data only so I mitigated this in current pipeline, but the Victor guy seems to use live data only.. 🤷‍♂️ - Are you training the model from scratch/initialization using OL ?

", "rawMarkdown": "Are you able to share some more on online learning? Are you training an each days data/stacking history before training? I have a OL pipeline but I am not seeing these boosts everyone else is, I am only seeing 0.0005 on LB effectively. Even with local testing, online training on the last 150 days for example, the boost is a bit higher in CV but LB is not that drastic.. Not sure how others are approaching it, I tend to get catastrophic forgetting if I use Live data only so I mitigated this in current pipeline, but the Victor guy seems to use live data only.. 🤷‍♂️ - Are you training the model from scratch/initialization using OL ?", "votes": null }, { "id": "3062441", "postDate": "12/03/2024 15:32:45", "content": "

Yes, the lift of OL is also depending on the training strategy and your models. Sometimes I also see only ~0.001 life from my local test.

", "rawMarkdown": "Yes, the lift of OL is also depending on the training strategy and your models. Sometimes I also see only ~0.001 life from my local test.", "votes": null }, { "id": "3062513", "postDate": "12/03/2024 16:21:24", "content": "

How did you vary your batch size / learning rate during online learning vs train? Did you retain the state of your optimizer from train or re-initialize it?

", "rawMarkdown": "How did you vary your batch size / learning rate during online learning vs train? Did you retain the state of your optimizer from train or re-initialize it?", "votes": null }, { "id": "3062605", "postDate": "12/03/2024 17:58:19", "content": "

You can tweak all the things you want by simulating the submission environment and test those things out to find the best option, right?

", "rawMarkdown": "You can tweak all the things you want by simulating the submission environment and test those things out to find the best option, right?", "votes": null }, { "id": "3062626", "postDate": "12/03/2024 18:15:53", "content": "
\n

You should expect to see 0.002+

\n
\n

Strong models don't improve as well? I would be above first place if online learning gave 0.002+. It's a bit discouraging.

", "rawMarkdown": ">You should expect to see 0.002+\n\nStrong models don't improve as well? I would be above first place if online learning gave 0.002+. It's a bit discouraging.", "votes": null }, { "id": "3063089", "postDate": "12/04/2024 06:33:05", "content": "
\n
\n

Normalisation?

\n
\n

It's suppsed to be very important as discussed in another great post. But it's not really necessary to have a perfect normalisation method to have a decent score. At least at this moment, I'm still using global mean/std normalisation.

\n
\n

Do you think normalize per symbol would be better than normal the whole thing?
\nI personally found normalize per symbol is worse than normalize the whole column and it take lots time to run.

", "rawMarkdown": "> >Normalisation? \n> \n> It's suppsed to be very important as discussed in another great post. But it's not really necessary to have a perfect normalisation method to have a decent score. At least at this moment, I'm still using global mean/std normalisation.\n\n\n\nDo you think normalize per symbol would be better than normal the whole thing? \nI personally found normalize per symbol is worse than normalize the whole column and it take lots time to run.", "votes": null }, { "id": "3063092", "postDate": "12/04/2024 06:35:04", "content": "

i didnt spot anything not right from just the code.
\ni only get this error when I still refering to for example model v2, where I loaded v3 in the notebook

", "rawMarkdown": "i didnt spot anything not right from just the code.\ni only get this error when I still refering to for example model v2, where I loaded v3 in the notebook", "votes": null } ], "comments": [ { "id": 3062065, "author_name": "sudhirsars", "author_url": "", "post_date": "12/03/2024 08:47:06", "content": "

Hey can u help me with this this error \"Notebook Inference Server Error\" when i try to submit
\nthis is the test notebook https://www.kaggle.com/code/sudhirsars/jn-tester
\nand this is the training notebook https://www.kaggle.com/code/sudhirsars/trainer-jn

\n

can you guyz please have a look and guide me where iam wrong

", "votes": null, "replies": [ { "id": 3062075, "author_name": "zoutain", "author_url": "", "post_date": "12/03/2024 09:06:05", "content": "

inference error means you have something that it can not find.
\ndid you put up the right path to get the model etc.
\ndebug outside the predict function

", "votes": null, "replies": [ { "id": 3062085, "author_name": "sudhirsars", "author_url": "", "post_date": "12/03/2024 09:24:46", "content": "

Every thing is good, even iam able to print final prediction in predict function still I get this error every time you can look at logs of test notebook I have no logged error from inference server but still when I submit the notebook it get this error on the submissions panel

", "votes": null, "replies": [ { "id": 3063092, "author_name": "zoutain", "author_url": "", "post_date": "12/04/2024 06:35:04", "content": "

i didnt spot anything not right from just the code.
\ni only get this error when I still refering to for example model v2, where I loaded v3 in the notebook

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3062077, "author_name": "lihaorocky", "author_url": "", "post_date": "12/03/2024 09:12:54", "content": "

I would suggest sticking to one architecture and developing a robust pipeline for online learning. If everything you did is solid and makes sense, 0.006+ is definitely can be expected, no matter what architectures you are using.

", "votes": null, "replies": [ { "id": 3062123, "author_name": "shiyili", "author_url": "", "post_date": "12/03/2024 10:05:00", "content": "

Congrats on your boost to the new rank!

\n

May I ask what have you seen that makes a big difference in your experiments? I understand that online learning is quite critical, but I have tested several model architectures with different online training scheme and found no luck. I must have overlooked something. Normalization? Imputation? Feature engineering?

\n

Thank you so much and it would be greatly appreciated if you could share some thoughts.

", "votes": null, "replies": [ { "id": 3062169, "author_name": "lihaorocky", "author_url": "", "post_date": "12/03/2024 11:01:51", "content": "
\n

May I ask what have you seen that makes a big difference in your experiments?

\n
\n

This is a big quesition. Honestly, I'm also still exporing different possibilities. But from what see in my experiments, setting up robust local test to simulate the online env is very important. With a robust pipeline, you could test all the things you mentioned (models, normalisations, features etc.) very easily. One thing to notice here is aligning how you train your model with how data is feeded when you submit your code.

\n
\n

Model architectures

\n
\n

As I mentioned, from my experiments a very simple mlp with online learning can already lead to very far. You should expect to see 0.002+ improvement with same model by adding online learning. You don't need to worry about other things before you achieve this online training scheme.

\n
\n

Normalisation?

\n
\n

It's suppsed to be very important as discussed in another great post. But it's not really necessary to have a perfect normalisation method to have a decent score. At least at this moment, I'm still using global mean/std normalisation.

\n
\n

Imputation?

\n
\n

Same to normalisation. I'm just using 0 for imputation. Will explore other possibilities later.

\n
\n

Feature engineering?

\n
\n

NN is suposed to be great to do end-to-end learning, right? Try to design models to learn it.

", "votes": null, "replies": [ { "id": 3062207, "author_name": "julianmukaj", "author_url": "", "post_date": "12/03/2024 11:33:05", "content": "

Are you able to share some more on online learning? Are you training an each days data/stacking history before training? I have a OL pipeline but I am not seeing these boosts everyone else is, I am only seeing 0.0005 on LB effectively. Even with local testing, online training on the last 150 days for example, the boost is a bit higher in CV but LB is not that drastic.. Not sure how others are approaching it, I tend to get catastrophic forgetting if I use Live data only so I mitigated this in current pipeline, but the Victor guy seems to use live data only.. 🤷‍♂️ - Are you training the model from scratch/initialization using OL ?

", "votes": null, "replies": [ { "id": 3062441, "author_name": "lihaorocky", "author_url": "", "post_date": "12/03/2024 15:32:45", "content": "

Yes, the lift of OL is also depending on the training strategy and your models. Sometimes I also see only ~0.001 life from my local test.

", "votes": null, "replies": [ { "id": 3062513, "author_name": "redfoongus", "author_url": "", "post_date": "12/03/2024 16:21:24", "content": "

How did you vary your batch size / learning rate during online learning vs train? Did you retain the state of your optimizer from train or re-initialize it?

", "votes": null, "replies": [ { "id": 3062605, "author_name": "lihaorocky", "author_url": "", "post_date": "12/03/2024 17:58:19", "content": "

You can tweak all the things you want by simulating the submission environment and test those things out to find the best option, right?

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3062626, "author_name": "sergeifironov", "author_url": "", "post_date": "12/03/2024 18:15:53", "content": "
\n

You should expect to see 0.002+

\n
\n

Strong models don't improve as well? I would be above first place if online learning gave 0.002+. It's a bit discouraging.

", "votes": null, "replies": [] }, { "id": 3063089, "author_name": "zoutain", "author_url": "", "post_date": "12/04/2024 06:33:05", "content": "
\n
\n

Normalisation?

\n
\n

It's suppsed to be very important as discussed in another great post. But it's not really necessary to have a perfect normalisation method to have a decent score. At least at this moment, I'm still using global mean/std normalisation.

\n
\n

Do you think normalize per symbol would be better than normal the whole thing?
\nI personally found normalize per symbol is worse than normalize the whole column and it take lots time to run.

", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "3058839": "Its been 2 month, and I haven't found anything that really working.\n\n1. I have tried different model structures, stacking dense layers, LSTM, Conv, attention... but they are not as good as the baseline. \n\n2. Features engineering did not seemed to help either, nomalization, weights selection etc.. getting even worse score than my baseline.\n\nWhat are the possible ways to improve score offline? I am currently clueless...", "3062065": "Hey can u help me with this this error \"Notebook Inference Server Error\" when i try to submit\nthis is the test notebook https://www.kaggle.com/code/sudhirsars/jn-tester\nand this is the training notebook https://www.kaggle.com/code/sudhirsars/trainer-jn\n\ncan you guyz please have a look and guide me where iam wrong", "3062075": "inference error means you have something that it can not find.\ndid you put up the right path to get the model etc.\ndebug outside the predict function", "3062077": "I would suggest sticking to one architecture and developing a robust pipeline for online learning. If everything you did is solid and makes sense, 0.006+ is definitely can be expected, no matter what architectures you are using.", "3062085": "Every thing is good, even iam able to print final prediction in predict function still I get this error every time you can look at logs of test notebook I have no logged error from inference server but still when I submit the notebook it get this error on the submissions panel", "3062123": "Congrats on your boost to the new rank!\n\nMay I ask what have you seen that makes a big difference in your experiments? I understand that online learning is quite critical, but I have tested several model architectures with different online training scheme and found no luck. I must have overlooked something. Normalization? Imputation? Feature engineering?\n\nThank you so much and it would be greatly appreciated if you could share some thoughts.", "3062169": ">May I ask what have you seen that makes a big difference in your experiments?\n\nThis is a big quesition. Honestly, I'm also still exporing different possibilities. But from what see in my experiments, setting up robust local test to simulate the online env is very important. With a robust pipeline, you could test all the things you mentioned (models, normalisations, features etc.) very easily. One thing to notice here is aligning how you train your model with how data is feeded when you submit your code.\n>Model architectures\n\nAs I mentioned, from my experiments a very simple mlp with online learning can already lead to very far. You should expect to see 0.002+ improvement with same model by adding online learning. You don't need to worry about other things before you achieve this online training scheme. \n>Normalisation? \n\nIt's suppsed to be very important as discussed in another great post. But it's not really necessary to have a perfect normalisation method to have a decent score. At least at this moment, I'm still using global mean/std normalisation.\n\n>Imputation? \n\nSame to normalisation. I'm just using 0 for imputation. Will explore other possibilities later. \n>Feature engineering? \n\nNN is suposed to be great to do end-to-end learning, right? Try to design models to learn it.", "3062207": "Are you able to share some more on online learning? Are you training an each days data/stacking history before training? I have a OL pipeline but I am not seeing these boosts everyone else is, I am only seeing 0.0005 on LB effectively. Even with local testing, online training on the last 150 days for example, the boost is a bit higher in CV but LB is not that drastic.. Not sure how others are approaching it, I tend to get catastrophic forgetting if I use Live data only so I mitigated this in current pipeline, but the Victor guy seems to use live data only.. 🤷‍♂️ - Are you training the model from scratch/initialization using OL ?", "3062441": "Yes, the lift of OL is also depending on the training strategy and your models. Sometimes I also see only ~0.001 life from my local test.", "3062513": "How did you vary your batch size / learning rate during online learning vs train? Did you retain the state of your optimizer from train or re-initialize it?", "3062605": "You can tweak all the things you want by simulating the submission environment and test those things out to find the best option, right?", "3062626": ">You should expect to see 0.002+\n\nStrong models don't improve as well? I would be above first place if online learning gave 0.002+. It's a bit discouraging.", "3063089": "> >Normalisation? \n> \n> It's suppsed to be very important as discussed in another great post. But it's not really necessary to have a perfect normalisation method to have a decent score. At least at this moment, I'm still using global mean/std normalisation.\n\n\n\nDo you think normalize per symbol would be better than normal the whole thing? \nI personally found normalize per symbol is worse than normalize the whole column and it take lots time to run.", "3063092": "i didnt spot anything not right from just the code.\ni only get this error when I still refering to for example model v2, where I loaded v3 in the notebook" }, "source": "meta" }