{ "id": 547587, "title": "What is your best single model score?", "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/547587", "author_name": "", "post_date": "2024-11-22T13:45:49.406445800Z", "votes": 2, "comment_count": 21, "views": 0, "content": "

My best NN model so far is 0.0043, without retraining/feature engineering.

", "messages": [ { "id": "3052491", "postDate": "11/22/2024 13:45:49", "content": "

My best NN model so far is 0.0043, without retraining/feature engineering.

", "rawMarkdown": "My best NN model so far is 0.0043, without retraining/feature engineering.", "votes": null }, { "id": "3052547", "postDate": "11/22/2024 14:40:14", "content": "

0.0046，lgb model with all training data and adding lags_1 feature.

\n

Can I ask for the basic architecture of your NN model？I have tried some simple MLP layers, but my validation score is not good.

", "rawMarkdown": "0.0046，lgb model with all training data and adding lags_1 feature.\n\nCan I ask for the basic architecture of your NN model？I have tried some simple MLP layers, but my validation score is not good.", "votes": null }, { "id": "3052554", "postDate": "11/22/2024 14:46:48", "content": "

AE + MLP ~291,583 parameters

", "rawMarkdown": "AE + MLP ~291,583 parameters", "votes": null }, { "id": "3052574", "postDate": "11/22/2024 15:08:22", "content": "

thanks for sharing.Do you use MSE as the loss function?

", "rawMarkdown": "thanks for sharing.Do you use MSE as the loss function?", "votes": null }, { "id": "3052578", "postDate": "11/22/2024 15:19:25", "content": "

0.0054 w/ NN
\n0.0050 w/ LGBM
\nBoth with no online training

", "rawMarkdown": "0.0054 w/ NN\n0.0050 w/ LGBM\nBoth with no online training", "votes": null }, { "id": "3052677", "postDate": "11/22/2024 17:25:28", "content": "

LogCosh or Huber.

", "rawMarkdown": "LogCosh or Huber.", "votes": null }, { "id": "3052876", "postDate": "11/22/2024 23:38:09", "content": "

Is online training possible??
\nI haven't tried it yet.

", "rawMarkdown": "Is online training possible??\nI haven't tried it yet.", "votes": null }, { "id": "3055709", "postDate": "11/26/2024 03:32:43", "content": "

May I ask what NN mdoel you use to get 0.0054?

", "rawMarkdown": "May I ask what NN mdoel you use to get 0.0054?", "votes": null }, { "id": "3056175", "postDate": "11/26/2024 16:25:01", "content": "

0.0055 with xgboost.

", "rawMarkdown": "0.0055 with xgboost.", "votes": null }, { "id": "3056182", "postDate": "11/26/2024 16:34:21", "content": "

LGB: 0.0059 (without online learning)
\nCatBoost: 0.0060 (with online learning) - But I think GBDT is a dead end. Switched to NN already.
\nNN: 0.0061 (with online learning)

", "rawMarkdown": "LGB: 0.0059 (without online learning)\nCatBoost: 0.0060 (with online learning) - But I think GBDT is a dead end. Switched to NN already. \nNN: 0.0061 (with online learning)", "votes": null }, { "id": "3056202", "postDate": "11/26/2024 17:02:58", "content": "
\n

Is online training possible??

\n
\n

Yes I'd say it's required to be in the top 10 for this competition.

", "rawMarkdown": ">Is online training possible??\n\nYes I'd say it's required to be in the top 10 for this competition.", "votes": null }, { "id": "3056206", "postDate": "11/26/2024 17:05:39", "content": "
\n

May I ask what NN mdoel you use to get 0.0054?

\n
\n

Tensorflow with a simple architecture to allow online learning. Can't provide more details without giving my \"secrets\".

", "rawMarkdown": "> May I ask what NN mdoel you use to get 0.0054?\n\nTensorflow with a simple architecture to allow online learning. Can't provide more details without giving my \"secrets\".", "votes": null }, { "id": "3056207", "postDate": "11/26/2024 17:06:45", "content": "

I agree tree models are a dead end but have you tried blending it with your NN?

", "rawMarkdown": "I agree tree models are a dead end but have you tried blending it with your NN?", "votes": null }, { "id": "3056215", "postDate": "11/26/2024 17:16:49", "content": "

No, it's not that time yet, right? 😃 But from the time consumption point, it shouldn't be a problem. NN model with online learning is super fast (within 2 hours, I'm using pytorch. Tensorflow should be much faster - will swich later), while the GBDT model with online learning cost about 4 hours. But I don't think GBDT models will even play a role in my final solution.

", "rawMarkdown": "No, it's not that time yet, right? 😃 But from the time consumption point, it shouldn't be a problem. NN model with online learning is super fast (within 2 hours, I'm using pytorch. Tensorflow should be much faster - will swich later), while the GBDT model with online learning cost about 4 hours. But I don't think GBDT models will even play a role in my final solution.", "votes": null }, { "id": "3056356", "postDate": "11/26/2024 21:43:55", "content": "

Can you clarify why you think TF would be much faster than Torch?

", "rawMarkdown": "Can you clarify why you think TF would be much faster than Torch?", "votes": null }, { "id": "3056406", "postDate": "11/27/2024 00:13:03", "content": "

Nice. Agreed, not that time! I have a reasonable LGBM and a good performing NN, will leave blending for last. Focusing on making a working LSTM now. I have a feeling the best solutions will be LSTMs

", "rawMarkdown": "Nice. Agreed, not that time! I have a reasonable LGBM and a good performing NN, will leave blending for last. Focusing on making a working LSTM now. I have a feeling the best solutions will be LSTMs", "votes": null }, { "id": "3064717", "postDate": "12/05/2024 23:39:25", "content": "

i used a simple NN model without feature egineering, got 0.0008.
\nand with online learning somehow became worse, got -0.43 ….
\ni have reduced LR to 1e-6, trained 2 epochs with small batch size
\nI had no clue what I did wrong….any suggestions?

", "rawMarkdown": "i used a simple NN model without feature egineering, got 0.0008.\nand with online learning somehow became worse, got -0.43 ....\ni have reduced LR to 1e-6, trained 2 epochs with small batch size\nI had no clue what I did wrong....any suggestions?", "votes": null }, { "id": "3066923", "postDate": "12/08/2024 16:00:41", "content": "

Did you do a lot of feature engineering?

", "rawMarkdown": "Did you do a lot of feature engineering?", "votes": null }, { "id": "3066924", "postDate": "12/08/2024 16:01:31", "content": "

May I ask if you did a lot of feature engineering on the GBDT models?

", "rawMarkdown": "May I ask if you did a lot of feature engineering on the GBDT models?", "votes": null }, { "id": "3066928", "postDate": "12/08/2024 16:04:10", "content": "

It all depends on the definition of \"a lot\". But, yes, I did some feature engineering.

", "rawMarkdown": "It all depends on the definition of \"a lot\". But, yes, I did some feature engineering.", "votes": null }, { "id": "3067400", "postDate": "12/09/2024 08:47:28", "content": "

Do you have the code we can comment?

", "rawMarkdown": "Do you have the code we can comment?", "votes": null }, { "id": "3067468", "postDate": "12/09/2024 10:19:04", "content": "

How are we looking on offline validation sets vs leaderboard?
\nE.g. a model that gets ~ 0.0135 in offline val (min 0.0095) does 0.0048 on the leaderboard, are people seeing similar results?

", "rawMarkdown": "How are we looking on offline validation sets vs leaderboard?\nE.g. a model that gets ~ 0.0135 in offline val (min 0.0095) does 0.0048 on the leaderboard, are people seeing similar results?", "votes": null } ], "comments": [ { "id": 3052547, "author_name": "i2nfinit3y", "author_url": "", "post_date": "11/22/2024 14:40:14", "content": "

0.0046，lgb model with all training data and adding lags_1 feature.

\n

Can I ask for the basic architecture of your NN model？I have tried some simple MLP layers, but my validation score is not good.

", "votes": null, "replies": [ { "id": 3052554, "author_name": "nandodmelo", "author_url": "", "post_date": "11/22/2024 14:46:48", "content": "

AE + MLP ~291,583 parameters

", "votes": null, "replies": [ { "id": 3052574, "author_name": "i2nfinit3y", "author_url": "", "post_date": "11/22/2024 15:08:22", "content": "

thanks for sharing.Do you use MSE as the loss function?

", "votes": null, "replies": [ { "id": 3052677, "author_name": "nandodmelo", "author_url": "", "post_date": "11/22/2024 17:25:28", "content": "

LogCosh or Huber.

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3052578, "author_name": "natanlabarrere", "author_url": "", "post_date": "11/22/2024 15:19:25", "content": "

0.0054 w/ NN
\n0.0050 w/ LGBM
\nBoth with no online training

", "votes": null, "replies": [ { "id": 3052876, "author_name": "jayshrivastava", "author_url": "", "post_date": "11/22/2024 23:38:09", "content": "

Is online training possible??
\nI haven't tried it yet.

", "votes": null, "replies": [ { "id": 3056202, "author_name": "natanlabarrere", "author_url": "", "post_date": "11/26/2024 17:02:58", "content": "
\n

Is online training possible??

\n
\n

Yes I'd say it's required to be in the top 10 for this competition.

", "votes": null, "replies": [] } ] }, { "id": 3055709, "author_name": "jiadiwang", "author_url": "", "post_date": "11/26/2024 03:32:43", "content": "

May I ask what NN mdoel you use to get 0.0054?

", "votes": null, "replies": [ { "id": 3056206, "author_name": "natanlabarrere", "author_url": "", "post_date": "11/26/2024 17:05:39", "content": "
\n

May I ask what NN mdoel you use to get 0.0054?

\n
\n

Tensorflow with a simple architecture to allow online learning. Can't provide more details without giving my \"secrets\".

", "votes": null, "replies": [] } ] } ] }, { "id": 3056175, "author_name": "jackvd", "author_url": "", "post_date": "11/26/2024 16:25:01", "content": "

0.0055 with xgboost.

", "votes": null, "replies": [ { "id": 3066923, "author_name": "xstargate", "author_url": "", "post_date": "12/08/2024 16:00:41", "content": "

Did you do a lot of feature engineering?

", "votes": null, "replies": [] } ] }, { "id": 3056182, "author_name": "lihaorocky", "author_url": "", "post_date": "11/26/2024 16:34:21", "content": "

LGB: 0.0059 (without online learning)
\nCatBoost: 0.0060 (with online learning) - But I think GBDT is a dead end. Switched to NN already.
\nNN: 0.0061 (with online learning)

", "votes": null, "replies": [ { "id": 3056207, "author_name": "natanlabarrere", "author_url": "", "post_date": "11/26/2024 17:06:45", "content": "

I agree tree models are a dead end but have you tried blending it with your NN?

", "votes": null, "replies": [ { "id": 3056215, "author_name": "lihaorocky", "author_url": "", "post_date": "11/26/2024 17:16:49", "content": "

No, it's not that time yet, right? 😃 But from the time consumption point, it shouldn't be a problem. NN model with online learning is super fast (within 2 hours, I'm using pytorch. Tensorflow should be much faster - will swich later), while the GBDT model with online learning cost about 4 hours. But I don't think GBDT models will even play a role in my final solution.

", "votes": null, "replies": [ { "id": 3056356, "author_name": "redfoongus", "author_url": "", "post_date": "11/26/2024 21:43:55", "content": "

Can you clarify why you think TF would be much faster than Torch?

", "votes": null, "replies": [] }, { "id": 3056406, "author_name": "natanlabarrere", "author_url": "", "post_date": "11/27/2024 00:13:03", "content": "

Nice. Agreed, not that time! I have a reasonable LGBM and a good performing NN, will leave blending for last. Focusing on making a working LSTM now. I have a feeling the best solutions will be LSTMs

", "votes": null, "replies": [] } ] } ] }, { "id": 3066924, "author_name": "xstargate", "author_url": "", "post_date": "12/08/2024 16:01:31", "content": "

May I ask if you did a lot of feature engineering on the GBDT models?

", "votes": null, "replies": [ { "id": 3066928, "author_name": "lihaorocky", "author_url": "", "post_date": "12/08/2024 16:04:10", "content": "

It all depends on the definition of \"a lot\". But, yes, I did some feature engineering.

", "votes": null, "replies": [] } ] } ] }, { "id": 3064717, "author_name": "zoutain", "author_url": "", "post_date": "12/05/2024 23:39:25", "content": "

i used a simple NN model without feature egineering, got 0.0008.
\nand with online learning somehow became worse, got -0.43 ….
\ni have reduced LR to 1e-6, trained 2 epochs with small batch size
\nI had no clue what I did wrong….any suggestions?

", "votes": null, "replies": [ { "id": 3067400, "author_name": "adam99", "author_url": "", "post_date": "12/09/2024 08:47:28", "content": "

Do you have the code we can comment?

", "votes": null, "replies": [] } ] }, { "id": 3067468, "author_name": "forecastingvibes", "author_url": "", "post_date": "12/09/2024 10:19:04", "content": "

How are we looking on offline validation sets vs leaderboard?
\nE.g. a model that gets ~ 0.0135 in offline val (min 0.0095) does 0.0048 on the leaderboard, are people seeing similar results?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3052491": "My best NN model so far is 0.0043, without retraining/feature engineering.", "3052547": "0.0046，lgb model with all training data and adding lags_1 feature.\n\nCan I ask for the basic architecture of your NN model？I have tried some simple MLP layers, but my validation score is not good.", "3052554": "AE + MLP ~291,583 parameters", "3052574": "thanks for sharing.Do you use MSE as the loss function?", "3052578": "0.0054 w/ NN\n0.0050 w/ LGBM\nBoth with no online training", "3052677": "LogCosh or Huber.", "3052876": "Is online training possible??\nI haven't tried it yet.", "3055709": "May I ask what NN mdoel you use to get 0.0054?", "3056175": "0.0055 with xgboost.", "3056182": "LGB: 0.0059 (without online learning)\nCatBoost: 0.0060 (with online learning) - But I think GBDT is a dead end. Switched to NN already. \nNN: 0.0061 (with online learning)", "3056202": ">Is online training possible??\n\nYes I'd say it's required to be in the top 10 for this competition.", "3056206": "> May I ask what NN mdoel you use to get 0.0054?\n\nTensorflow with a simple architecture to allow online learning. Can't provide more details without giving my \"secrets\".", "3056207": "I agree tree models are a dead end but have you tried blending it with your NN?", "3056215": "No, it's not that time yet, right? 😃 But from the time consumption point, it shouldn't be a problem. NN model with online learning is super fast (within 2 hours, I'm using pytorch. Tensorflow should be much faster - will swich later), while the GBDT model with online learning cost about 4 hours. But I don't think GBDT models will even play a role in my final solution.", "3056356": "Can you clarify why you think TF would be much faster than Torch?", "3056406": "Nice. Agreed, not that time! I have a reasonable LGBM and a good performing NN, will leave blending for last. Focusing on making a working LSTM now. I have a feeling the best solutions will be LSTMs", "3064717": "i used a simple NN model without feature egineering, got 0.0008.\nand with online learning somehow became worse, got -0.43 ....\ni have reduced LR to 1e-6, trained 2 epochs with small batch size\nI had no clue what I did wrong....any suggestions?", "3066923": "Did you do a lot of feature engineering?", "3066924": "May I ask if you did a lot of feature engineering on the GBDT models?", "3066928": "It all depends on the definition of \"a lot\". But, yes, I did some feature engineering.", "3067400": "Do you have the code we can comment?", "3067468": "How are we looking on offline validation sets vs leaderboard?\nE.g. a model that gets ~ 0.0135 in offline val (min 0.0095) does 0.0048 on the leaderboard, are people seeing similar results?" }, "source": "meta" }