![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F962168%2F2f5736390aba7bee2428621320aeab2f%2FWechatIMG1131.jpg?generation=1733841979312142&alt=media\")
I'm new to kaggle, but this competition piqued my interest. I figure the top solutions are very complex - online learning, ensembles of models, recurrent nets, reasoning across symbols, etc. Is this true?
\nI'm somewhat proud to say my 0.0064 solution is a simple 3 layer mlp with none of that at all. Just regressing the responders from the 79 features of a single symbol. I am extremely careful about how the network is trained, the specific architectural choices, and the generalization of the model, which I love playing around with.
\nHow complex is your solution?
", "messages": [ { "id": 3068027, "postDate": "2024-12-09T21:25:51.560Z", "content": "I'm new to kaggle, but this competition piqued my interest. I figure the top solutions are very complex - online learning, ensembles of models, recurrent nets, reasoning across symbols, etc. Is this true?
\nI'm somewhat proud to say my 0.0064 solution is a simple 3 layer mlp with none of that at all. Just regressing the responders from the 79 features of a single symbol. I am extremely careful about how the network is trained, the specific architectural choices, and the generalization of the model, which I love playing around with.
\nHow complex is your solution?
", "rawMarkdown": "I'm new to kaggle, but this competition piqued my interest. I figure the top solutions are very complex - online learning, ensembles of models, recurrent nets, reasoning across symbols, etc. Is this true?\n\nI'm somewhat proud to say my 0.0064 solution is a simple 3 layer mlp with none of that at all. Just regressing the responders from the 79 features of a single symbol. I am extremely careful about how the network is trained, the specific architectural choices, and the generalization of the model, which I love playing around with.\n\nHow complex is your solution?", "votes": 31 }, { "id": 3071606, "postDate": "2024-12-14T02:08:08.203Z", "content": "I am impressed a simple MLP can have such good results!
\nCould you tell me about hyperparameters in details?
That’s a great score for offline simple model, mine is 0.0062 but more complex, have not been very careful with the training though, unable to focus due to time.
\nWilling to share anything you learned in your more careful training?
", "rawMarkdown": "That’s a great score for offline simple model, mine is 0.0062 but more complex, have not been very careful with the training though, unable to focus due to time.\n\nWilling to share anything you learned in your more careful training? ", "votes": 3 }, { "id": 3068677, "postDate": "2024-12-10T14:46:59.313Z", "content": "Please make it clear if your score is before or after the data update.
\nHere I have a screenshot of the Leaderboard at 17:49 yesterday (Dec-09, central eu time). I couldn't find your score with 0.0064.
\n
Wow! Congrats! As I'm writing this, my LB score is just slightly ahead of yours, but my model is much more complex. Specifically, I'm using an ensemble of GBDT and MLP models with and without online learning. Any one model (type) performs well in CV but not that great in LB, especially the static GBDT model. What have you found to be the biggest bang for the buck in the training?
", "rawMarkdown": "Wow! Congrats! As I'm writing this, my LB score is just slightly ahead of yours, but my model is much more complex. Specifically, I'm using an ensemble of GBDT and MLP models with and without online learning. Any one model (type) performs well in CV but not that great in LB, especially the static GBDT model. What have you found to be the biggest bang for the buck in the training?", "votes": 2 }, { "id": 3068596, "postDate": "2024-12-10T12:59:21.240Z", "content": "I didn't believe that a single and simple model could score more than 0.006, thanks for sharing!
\nI've honestly used a blending solution of boosting trees and a NN model with also lag data but I couldn't get more than 0.0054.
\nI think the data to train the model is key.
\nOn my side I've considered the last 500 days of the train set provided in the competition.
\nIt would be nice to hear how you select the training data.
\nI'm surprised to see that you considered only the features as input data and you regressed the responders. Did you consider all the datetimes?
For this score, I used all datetimes except the last 5% of the data which I was using for eval.
", "rawMarkdown": "For this score, I used all datetimes except the last 5% of the data which I was using for eval. ", "votes": 1, "replies": [ { "id": 3069196, "postDate": "2024-12-11T06:59:42.673Z", "content": "I am new in kaggle too. I have few questions, you said you used all datetimes data. due to the memory limits, you uploaded a model which is trainned locally ? thanks for sharing.
", "rawMarkdown": "I am new in kaggle too. I have few questions, you said you used all datetimes data. due to the memory limits, you uploaded a model which is trainned locally ? thanks for sharing." } ] } ] }, { "id": 3068172, "postDate": "2024-12-10T03:22:05.400Z", "content": "I've been trying to build a simple MLP (3 layers with 256 hidden units, ReLU/SiLU activation, dropout 0.1-0.3, 5*tanh output) and trained that data on days 1200-1699 and 0.001 LR. For some reason my submission scores are all quite bad (0 to -0.05). Does anyone have any advice on what I might have done wrong? (maybe the training data or architecture is just too small)
", "rawMarkdown": "I've been trying to build a simple MLP (3 layers with 256 hidden units, ReLU/SiLU activation, dropout 0.1-0.3, 5*tanh output) and trained that data on days 1200-1699 and 0.001 LR. For some reason my submission scores are all quite bad (0 to -0.05). Does anyone have any advice on what I might have done wrong? (maybe the training data or architecture is just too small)", "votes": 2, "replies": [ { "id": 3068173, "postDate": "2024-12-10T03:32:43.127Z", "content": "Shuffling the data and adding some noise is giving +ve r2,
", "rawMarkdown": "Shuffling the data and adding some noise is giving +ve r2, ", "votes": 2, "replies": [ { "id": 3068186, "postDate": "2024-12-10T03:55:49.727Z", "content": "Interesting! I do use shuffling in the data loader but haven't thought of adding noise, thanks for the suggestion
", "rawMarkdown": "Interesting! I do use shuffling in the data loader but haven't thought of adding noise, thanks for the suggestion" } ] }, { "id": 3068174, "postDate": "2024-12-10T03:37:13.197Z", "content": "your dataset range might be too small and the model isnt able to generalise well ?
", "rawMarkdown": "your dataset range might be too small and the model isnt able to generalise well ?", "votes": 1, "replies": [ { "id": 3068185, "postDate": "2024-12-10T03:55:09.167Z", "content": "Yeah I wonder. I am trying to get by just using Kaggle kernels and the torch data loader can barely fit 1200-1699 (at 1100 start I get a memory error - maybe should just load the data on request? But that would be very slow..)
", "rawMarkdown": "Yeah I wonder. I am trying to get by just using Kaggle kernels and the torch data loader can barely fit 1200-1699 (at 1100 start I get a memory error - maybe should just load the data on request? But that would be very slow..)" }, { "id": 3068192, "postDate": "2024-12-10T04:12:13.057Z", "content": "i dont have the code for this but thinking out loud maybe u can try this to overcome the limited RAM situation
\nTraining neural nets properly requires getting 100s of little details right! There's so many things it could be. I've been obsessing about these details for over 8 years, and that experience helps here. Anyways, I'll happily share after competition is over.
", "rawMarkdown": "Training neural nets properly requires getting 100s of little details right! There's so many things it could be. I've been obsessing about these details for over 8 years, and that experience helps here. Anyways, I'll happily share after competition is over.", "votes": 3, "replies": [ { "id": 3068435, "postDate": "2024-12-10T08:58:34.617Z", "content": "Look forward to your precious experience in training neural networks.
", "rawMarkdown": "Look forward to your precious experience in training neural networks." } ] }, { "id": 3068868, "postDate": "2024-12-10T18:34:01.473Z", "content": "Update: Thanks all for your inputs so far, they have encouraged me to try again. I have retried submitting an MLP and managed to get 0.0064 score (offline). Here are some interesting lessons I learned:
\nThere is probably a lot of scope to improve.
", "rawMarkdown": "Update: Thanks all for your inputs so far, they have encouraged me to try again. I have retried submitting an MLP and managed to get 0.0064 score (offline). Here are some interesting lessons I learned:\n- Regularization (dropout, weight decay) has helped a lot\n- My MLP only trained on ~300 days and 25 epochs (pretty low) on Kaggle kernels.\n- I am just glad to not have a negative R2 anymore (on MLPs) at the moment 😅\n\nThere is probably a lot of scope to improve.", "votes": 5, "replies": [ { "id": 3069181, "postDate": "2024-12-11T06:44:13.120Z", "content": "Thanks Kevin, so your MLP validation R2 in your notebook is 0.0064 and your leaderboard R2 is 0.0070, right?
\nMay I know how did you get rid of the negative R2? My validation score is similar to yours too, but in the leaderboard it is just negative.
Did you use validation dataset for early stop?
", "rawMarkdown": "Did you use validation dataset for early stop?" }, { "id": 3069521, "postDate": "2024-12-11T15:24:41.157Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3069522, "postDate": "2024-12-11T15:25:05.720Z", "content": "For the MLP, my internal validation is at around 0.007 (+- 0.0005) and leaderboard score was 0.0064 (after the data update, so pre-update I reckon LB would have been 0.004-0.005). My current \"best\" score is an ensemble of GBDT + NN, though I was just playing around really..
\nI think the main issue that I experienced was that the MLP was overfitting really quickly to the training data (R2 -> 0.4-0.6 while on validation it became negative), so using some moderate regularization prevented that from happening, also leading to only ~25 epochs. I guess that helps with the Kaggle GPU allowance lol.
\nLeaderboard seems to be moving very quickly now!
", "rawMarkdown": "For the MLP, my internal validation is at around 0.007 (+- 0.0005) and leaderboard score was 0.0064 (after the data update, so pre-update I reckon LB would have been 0.004-0.005). My current \"best\" score is an ensemble of GBDT + NN, though I was just playing around really..\n\nI think the main issue that I experienced was that the MLP was overfitting really quickly to the training data (R2 -> 0.4-0.6 while on validation it became negative), so using some moderate regularization prevented that from happening, also leading to only ~25 epochs. I guess that helps with the Kaggle GPU allowance lol.\n\nLeaderboard seems to be moving very quickly now!" } ] }, { "id": 3075541, "postDate": "2024-12-19T02:08:16.090Z", "content": "I had the same problem as yours. your submission score is negative so it is not the problem of models. The negative score is because of the miss much of predictions and row id. make sure your prediction and row id match.
", "rawMarkdown": "I had the same problem as yours. your submission score is negative so it is not the problem of models. The negative score is because of the miss much of predictions and row id. make sure your prediction and row id match. \n" } ] }, { "id": 3068405, "postDate": "2024-12-10T07:58:38.943Z", "content": "You are comparing scores on different datasets. Yesterday, the public dataset was updated, 80 days were added. It seems that all models received a boost of 0.001-0.002. Therefore, 0.006 does not look impressive on the new dataset. We still need online learning.
", "rawMarkdown": "You are comparing scores on different datasets. Yesterday, [the public dataset was updated](https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/550790), 80 days were added. It seems that all models received a boost of 0.001-0.002. Therefore, 0.006 does not look impressive on the new dataset. We still need online learning.", "replies": [ { "id": 3068408, "postDate": "2024-12-10T08:08:34.297Z", "content": "This was posted before that change was even announced.. So 0.0064 on old dataset was good.
", "rawMarkdown": "This was posted before that change was even announced.. So 0.0064 on old dataset was good.", "votes": 2, "replies": [ { "id": 3068410, "postDate": "2024-12-10T08:13:39.943Z", "content": "No, it was posted about one hour after the dataset extension. But soon you will see that, since more \"high\" scores will show up in PB.
", "rawMarkdown": "No, it was posted about one hour after the dataset extension. But soon you will see that, since more \"high\" scores will show up in PB. " }, { "id": 3068415, "postDate": "2024-12-10T08:15:19.963Z", "content": "They literally said it will take several days for scores to update, and they closed submissions one hour before this post, so you think they ran his first and updated the score before anyone elses, all within that one hour? If I can get 0.0062 offline on old dataset, OP can get 0.0064. I saw his ranking at time of post
", "rawMarkdown": "They literally said it will take several days for scores to update, and they closed submissions one hour before this post, so you think they ran his first and updated the score before anyone elses, all within that one hour? If I can get 0.0062 offline on old dataset, OP can get 0.0064. I saw his ranking at time of post" }, { "id": 3068416, "postDate": "2024-12-10T08:17:42.917Z", "content": "Yep. I managed to make several submits on the new dataset. I got a boost of 0.0015 on one model, just resubmited it.
", "rawMarkdown": "Yep. I managed to make several submits on the new dataset. I got a boost of 0.0015 on one model, just resubmited it." }, { "id": 3068417, "postDate": "2024-12-10T08:20:05.050Z", "content": "@probablynobody can you confirm is your score a new re-run or not? Maybe I caught the post right on the switch over lol
", "rawMarkdown": "@probablynobody can you confirm is your score a new re-run or not? Maybe I caught the post right on the switch over lol" }, { "id": 3068661, "postDate": "2024-12-10T14:26:56.460Z", "content": "I improved my score from 0.0071 to 0.0089 by resubmitting the same notebook. Still no online learning, but I'll do that next.
", "rawMarkdown": "I improved my score from 0.0071 to 0.0089 by resubmitting the same notebook. Still no online learning, but I'll do that next.", "votes": 4 }, { "id": 3068666, "postDate": "2024-12-10T14:35:13.950Z", "content": "Wow! It's amazing! It's not a single model, is it?
", "rawMarkdown": "Wow! It's amazing! It's not a single model, is it? " }, { "id": 3068674, "postDate": "2024-12-10T14:43:48.500Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3068675, "postDate": "2024-12-10T14:44:12.737Z", "content": "No. I'm using an ensemble of models.
", "rawMarkdown": "No. I'm using an ensemble of models.", "votes": 1 }, { "id": 3068894, "postDate": "2024-12-10T19:20:55.260Z", "content": "Yes, this is before the dataset extension.
", "rawMarkdown": "Yes, this is before the dataset extension." } ] } ] }, { "id": 3075620, "postDate": "2024-12-19T04:24:28.500Z", "content": "I am impressed a simple MLP can have such good results!
", "rawMarkdown": "I am impressed a simple MLP can have such good results!", "votes": -1 }, { "id": 3083865, "postDate": "2024-12-30T05:44:37.450Z", "content": "I'm curious whether MLP or other NN model will overfitting the data now.
", "rawMarkdown": "I'm curious whether MLP or other NN model will overfitting the data now." }, { "id": 3077774, "postDate": "2024-12-21T11:42:55.360Z", "content": "0.0064 from mlp is so impressive. Thank you for sharing the tips!
", "rawMarkdown": "0.0064 from mlp is so impressive. Thank you for sharing the tips!" }, { "id": 3074651, "postDate": "2024-12-17T22:20:54.590Z", "content": "Im currently playing around with XGBoost and the different ways to implement it. I am either training a smaller model for each symbol s or one larger model for the whole data. This is my first competition and my first submission, that is why I had to get accustomed to the submission process first, which took up a little bit of my time. However, my first model for the whole dataset resulted in 0.0019, which is not perfect, but also not too bad for a first attempt. I am still trying to improve over time! Good luck to you all!
", "rawMarkdown": "Im currently playing around with XGBoost and the different ways to implement it. I am either training a smaller model for each symbol s or one larger model for the whole data. This is my first competition and my first submission, that is why I had to get accustomed to the submission process first, which took up a little bit of my time. However, my first model for the whole dataset resulted in 0.0019, which is not perfect, but also not too bad for a first attempt. I am still trying to improve over time! Good luck to you all!" }, { "id": 3072811, "postDate": "2024-12-15T16:41:30.227Z", "content": "I’m lucky to have your support—thank you!
", "rawMarkdown": "I’m lucky to have your support—thank you!" }, { "id": 3070376, "postDate": "2024-12-12T15:53:07.393Z", "content": "I am impressed a simple MLP can have such good results! May I ask any tips on how the network is trained, the specific architectural choices, and the generalization of the model as you mentioned?
", "rawMarkdown": "I am impressed a simple MLP can have such good results! May I ask any tips on how the network is trained, the specific architectural choices, and the generalization of the model as you mentioned? " }, { "id": 3069170, "postDate": "2024-12-11T06:28:59.333Z", "content": "I have not been able to train any NN architecture that results in a positive R2 so you are beating me!
", "rawMarkdown": "I have not been able to train any NN architecture that results in a positive R2 so you are beating me!" }, { "id": 3068949, "postDate": "2024-12-10T21:23:15.163Z", "content": "Has anyone tried sequential models? I tried to use LSTM and experiment on a bunch of seq lengths, but only get negative or 0 on r2
", "rawMarkdown": "Has anyone tried sequential models? I tried to use LSTM and experiment on a bunch of seq lengths, but only get negative or 0 on r2" }, { "id": 3068158, "postDate": "2024-12-10T02:50:01.787Z", "content": "I am curious about how you choose your local CV set, can you share with us?
", "rawMarkdown": "I am curious about how you choose your local CV set, can you share with us?" }, { "id": 3068135, "postDate": "2024-12-10T02:08:01.637Z", "content": "wow, amazing.
\nhow about feature engineering. Did you normalize or anything?
I am using the means and stds of the features and responders calculated over the whole train set in my model. No other normalization.
", "rawMarkdown": "I am using the means and stds of the features and responders calculated over the whole train set in my model. No other normalization.", "votes": 2, "replies": [ { "id": 3069685, "postDate": "2024-12-11T18:53:58.377Z", "content": "May I ask, how are you handling null/nan values?
", "rawMarkdown": "May I ask, how are you handling null/nan values?" } ] } ] }, { "id": 3068100, "postDate": "2024-12-10T00:55:45.570Z", "content": "Nice work! May I ask if that is from offline training only ? From what I’ve read so far in the discussion i haven’t seen many get much above .005-.006 with a single model offline
", "rawMarkdown": "Nice work! May I ask if that is from offline training only ? From what I’ve read so far in the discussion i haven’t seen many get much above .005-.006 with a single model offline" }, { "id": 3078457, "postDate": "2024-12-22T10:50:27.567Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3071139, "postDate": "2024-12-13T11:55:03.770Z", "content": "thanks for the nice information
", "rawMarkdown": "thanks for the nice information" } ], "comments": [ { "id": 3071606, "author_name": "Alex", "author_url": "", "post_date": "2024-12-14T02:08:08.203000", "content": "I am impressed a simple MLP can have such good results!
\nCould you tell me about hyperparameters in details?
That’s a great score for offline simple model, mine is 0.0062 but more complex, have not been very careful with the training though, unable to focus due to time.
\nWilling to share anything you learned in your more careful training?
", "votes": 3, "replies": [] }, { "id": 3068677, "author_name": "SLi", "author_url": "", "post_date": "2024-12-10T14:46:59.313000", "content": "Please make it clear if your score is before or after the data update.
\nHere I have a screenshot of the Leaderboard at 17:49 yesterday (Dec-09, central eu time). I couldn't find your score with 0.0064.
\n
Wow! Congrats! As I'm writing this, my LB score is just slightly ahead of yours, but my model is much more complex. Specifically, I'm using an ensemble of GBDT and MLP models with and without online learning. Any one model (type) performs well in CV but not that great in LB, especially the static GBDT model. What have you found to be the biggest bang for the buck in the training?
", "votes": 2, "replies": [] }, { "id": 3068596, "author_name": "Simone De Gasperis", "author_url": "", "post_date": "2024-12-10T12:59:21.240000", "content": "I didn't believe that a single and simple model could score more than 0.006, thanks for sharing!
\nI've honestly used a blending solution of boosting trees and a NN model with also lag data but I couldn't get more than 0.0054.
\nI think the data to train the model is key.
\nOn my side I've considered the last 500 days of the train set provided in the competition.
\nIt would be nice to hear how you select the training data.
\nI'm surprised to see that you considered only the features as input data and you regressed the responders. Did you consider all the datetimes?
For this score, I used all datetimes except the last 5% of the data which I was using for eval.
", "votes": 1, "replies": [ { "id": 3069196, "author_name": "unknown_trader", "author_url": "", "post_date": "2024-12-11T06:59:42.673000", "content": "I am new in kaggle too. I have few questions, you said you used all datetimes data. due to the memory limits, you uploaded a model which is trainned locally ? thanks for sharing.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3068172, "author_name": "Kevin Lam", "author_url": "", "post_date": "2024-12-10T03:22:05.400000", "content": "I've been trying to build a simple MLP (3 layers with 256 hidden units, ReLU/SiLU activation, dropout 0.1-0.3, 5*tanh output) and trained that data on days 1200-1699 and 0.001 LR. For some reason my submission scores are all quite bad (0 to -0.05). Does anyone have any advice on what I might have done wrong? (maybe the training data or architecture is just too small)
", "votes": 2, "replies": [ { "id": 3068173, "author_name": "Abhi", "author_url": "", "post_date": "2024-12-10T03:32:43.127000", "content": "Shuffling the data and adding some noise is giving +ve r2,
", "votes": 2, "replies": [ { "id": 3068186, "author_name": "Kevin Lam", "author_url": "", "post_date": "2024-12-10T03:55:49.727000", "content": "Interesting! I do use shuffling in the data loader but haven't thought of adding noise, thanks for the suggestion
", "votes": 0, "replies": [] } ] }, { "id": 3068174, "author_name": "just another tuesday", "author_url": "", "post_date": "2024-12-10T03:37:13.197000", "content": "your dataset range might be too small and the model isnt able to generalise well ?
", "votes": 1, "replies": [ { "id": 3068185, "author_name": "Kevin Lam", "author_url": "", "post_date": "2024-12-10T03:55:09.167000", "content": "Yeah I wonder. I am trying to get by just using Kaggle kernels and the torch data loader can barely fit 1200-1699 (at 1100 start I get a memory error - maybe should just load the data on request? But that would be very slow..)
", "votes": 0, "replies": [] }, { "id": 3068192, "author_name": "just another tuesday", "author_url": "", "post_date": "2024-12-10T04:12:13.057000", "content": "i dont have the code for this but thinking out loud maybe u can try this to overcome the limited RAM situation
\nTraining neural nets properly requires getting 100s of little details right! There's so many things it could be. I've been obsessing about these details for over 8 years, and that experience helps here. Anyways, I'll happily share after competition is over.
", "votes": 3, "replies": [ { "id": 3068435, "author_name": "Xiang Sheng", "author_url": "", "post_date": "2024-12-10T08:58:34.617000", "content": "Look forward to your precious experience in training neural networks.
", "votes": 0, "replies": [] } ] }, { "id": 3068868, "author_name": "Kevin Lam", "author_url": "", "post_date": "2024-12-10T18:34:01.473000", "content": "Update: Thanks all for your inputs so far, they have encouraged me to try again. I have retried submitting an MLP and managed to get 0.0064 score (offline). Here are some interesting lessons I learned:
\nThere is probably a lot of scope to improve.
", "votes": 5, "replies": [ { "id": 3069181, "author_name": "Ben Fung", "author_url": "", "post_date": "2024-12-11T06:44:13.120000", "content": "Thanks Kevin, so your MLP validation R2 in your notebook is 0.0064 and your leaderboard R2 is 0.0070, right?
\nMay I know how did you get rid of the negative R2? My validation score is similar to yours too, but in the leaderboard it is just negative.
Did you use validation dataset for early stop?
", "votes": 0, "replies": [] }, { "id": 3069521, "author_name": "", "author_url": "", "post_date": "2024-12-11T15:24:41.157000", "content": "", "votes": 0, "replies": [] }, { "id": 3069522, "author_name": "Kevin Lam", "author_url": "", "post_date": "2024-12-11T15:25:05.720000", "content": "For the MLP, my internal validation is at around 0.007 (+- 0.0005) and leaderboard score was 0.0064 (after the data update, so pre-update I reckon LB would have been 0.004-0.005). My current \"best\" score is an ensemble of GBDT + NN, though I was just playing around really..
\nI think the main issue that I experienced was that the MLP was overfitting really quickly to the training data (R2 -> 0.4-0.6 while on validation it became negative), so using some moderate regularization prevented that from happening, also leading to only ~25 epochs. I guess that helps with the Kaggle GPU allowance lol.
\nLeaderboard seems to be moving very quickly now!
", "votes": 0, "replies": [] } ] }, { "id": 3075541, "author_name": "YesGoBo", "author_url": "", "post_date": "2024-12-19T02:08:16.090000", "content": "I had the same problem as yours. your submission score is negative so it is not the problem of models. The negative score is because of the miss much of predictions and row id. make sure your prediction and row id match.
", "votes": 0, "replies": [] } ] }, { "id": 3068405, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2024-12-10T07:58:38.943000", "content": "You are comparing scores on different datasets. Yesterday, the public dataset was updated, 80 days were added. It seems that all models received a boost of 0.001-0.002. Therefore, 0.006 does not look impressive on the new dataset. We still need online learning.
", "votes": 0, "replies": [ { "id": 3068408, "author_name": "JM", "author_url": "", "post_date": "2024-12-10T08:08:34.297000", "content": "This was posted before that change was even announced.. So 0.0064 on old dataset was good.
", "votes": 2, "replies": [ { "id": 3068410, "author_name": "HAO", "author_url": "", "post_date": "2024-12-10T08:13:39.943000", "content": "No, it was posted about one hour after the dataset extension. But soon you will see that, since more \"high\" scores will show up in PB.
", "votes": 0, "replies": [] }, { "id": 3068415, "author_name": "JM", "author_url": "", "post_date": "2024-12-10T08:15:19.963000", "content": "They literally said it will take several days for scores to update, and they closed submissions one hour before this post, so you think they ran his first and updated the score before anyone elses, all within that one hour? If I can get 0.0062 offline on old dataset, OP can get 0.0064. I saw his ranking at time of post
", "votes": 0, "replies": [] }, { "id": 3068416, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2024-12-10T08:17:42.917000", "content": "Yep. I managed to make several submits on the new dataset. I got a boost of 0.0015 on one model, just resubmited it.
", "votes": 0, "replies": [] }, { "id": 3068417, "author_name": "JM", "author_url": "", "post_date": "2024-12-10T08:20:05.050000", "content": "@probablynobody can you confirm is your score a new re-run or not? Maybe I caught the post right on the switch over lol
", "votes": 0, "replies": [] }, { "id": 3068661, "author_name": "Thomas Dueholm Hansen", "author_url": "", "post_date": "2024-12-10T14:26:56.460000", "content": "I improved my score from 0.0071 to 0.0089 by resubmitting the same notebook. Still no online learning, but I'll do that next.
", "votes": 4, "replies": [] }, { "id": 3068666, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2024-12-10T14:35:13.950000", "content": "Wow! It's amazing! It's not a single model, is it?
", "votes": 0, "replies": [] }, { "id": 3068674, "author_name": "", "author_url": "", "post_date": "2024-12-10T14:43:48.500000", "content": "", "votes": 0, "replies": [] }, { "id": 3068675, "author_name": "Thomas Dueholm Hansen", "author_url": "", "post_date": "2024-12-10T14:44:12.737000", "content": "No. I'm using an ensemble of models.
", "votes": 1, "replies": [] }, { "id": 3068894, "author_name": "Jon", "author_url": "", "post_date": "2024-12-10T19:20:55.260000", "content": "Yes, this is before the dataset extension.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3075620, "author_name": "MrSimple", "author_url": "", "post_date": "2024-12-19T04:24:28.500000", "content": "I am impressed a simple MLP can have such good results!
", "votes": -1, "replies": [] }, { "id": 3083865, "author_name": "yc_kaggle", "author_url": "", "post_date": "2024-12-30T05:44:37.450000", "content": "I'm curious whether MLP or other NN model will overfitting the data now.
", "votes": 0, "replies": [] }, { "id": 3077774, "author_name": "Priscilla Ong", "author_url": "", "post_date": "2024-12-21T11:42:55.360000", "content": "0.0064 from mlp is so impressive. Thank you for sharing the tips!
", "votes": 0, "replies": [] }, { "id": 3074651, "author_name": "Timothy_brrll", "author_url": "", "post_date": "2024-12-17T22:20:54.590000", "content": "Im currently playing around with XGBoost and the different ways to implement it. I am either training a smaller model for each symbol s or one larger model for the whole data. This is my first competition and my first submission, that is why I had to get accustomed to the submission process first, which took up a little bit of my time. However, my first model for the whole dataset resulted in 0.0019, which is not perfect, but also not too bad for a first attempt. I am still trying to improve over time! Good luck to you all!
", "votes": 0, "replies": [] }, { "id": 3072811, "author_name": "Rudi Özgen", "author_url": "", "post_date": "2024-12-15T16:41:30.227000", "content": "I’m lucky to have your support—thank you!
", "votes": 0, "replies": [] }, { "id": 3070376, "author_name": "lele", "author_url": "", "post_date": "2024-12-12T15:53:07.393000", "content": "I am impressed a simple MLP can have such good results! May I ask any tips on how the network is trained, the specific architectural choices, and the generalization of the model as you mentioned?
", "votes": 0, "replies": [] }, { "id": 3069170, "author_name": "Michael Timbs", "author_url": "", "post_date": "2024-12-11T06:28:59.333000", "content": "I have not been able to train any NN architecture that results in a positive R2 so you are beating me!
", "votes": 0, "replies": [] }, { "id": 3068949, "author_name": "drxdaniel", "author_url": "", "post_date": "2024-12-10T21:23:15.163000", "content": "Has anyone tried sequential models? I tried to use LSTM and experiment on a bunch of seq lengths, but only get negative or 0 on r2
", "votes": 0, "replies": [] }, { "id": 3068158, "author_name": "ironrro", "author_url": "", "post_date": "2024-12-10T02:50:01.787000", "content": "I am curious about how you choose your local CV set, can you share with us?
", "votes": 0, "replies": [] }, { "id": 3068135, "author_name": "ZT", "author_url": "", "post_date": "2024-12-10T02:08:01.637000", "content": "wow, amazing.
\nhow about feature engineering. Did you normalize or anything?
I am using the means and stds of the features and responders calculated over the whole train set in my model. No other normalization.
", "votes": 2, "replies": [ { "id": 3069685, "author_name": "Danila Kurganov", "author_url": "", "post_date": "2024-12-11T18:53:58.377000", "content": "May I ask, how are you handling null/nan values?
", "votes": 0, "replies": [] } ] } ] }, { "id": 3068100, "author_name": "LGreig", "author_url": "", "post_date": "2024-12-10T00:55:45.570000", "content": "Nice work! May I ask if that is from offline training only ? From what I’ve read so far in the discussion i haven’t seen many get much above .005-.006 with a single model offline
", "votes": 0, "replies": [] }, { "id": 3078457, "author_name": "", "author_url": "", "post_date": "2024-12-22T10:50:27.567000", "content": "", "votes": 0, "replies": [] }, { "id": 3071139, "author_name": "wanyuan", "author_url": "", "post_date": "2024-12-13T11:55:03.770000", "content": "thanks for the nice information
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3068027": "I'm new to kaggle, but this competition piqued my interest. I figure the top solutions are very complex - online learning, ensembles of models, recurrent nets, reasoning across symbols, etc. Is this true?\n\nI'm somewhat proud to say my 0.0064 solution is a simple 3 layer mlp with none of that at all. Just regressing the responders from the 79 features of a single symbol. I am extremely careful about how the network is trained, the specific architectural choices, and the generalization of the model, which I love playing around with.\n\nHow complex is your solution?", "3071606": "I am impressed a simple MLP can have such good results!\nCould you tell me about hyperparameters in details?", "3068147": "That’s a great score for offline simple model, mine is 0.0062 but more complex, have not been very careful with the training though, unable to focus due to time.\n\nWilling to share anything you learned in your more careful training? ", "3068677": "Please make it clear if your score is before or after the data update. \n\nHere I have a screenshot of the Leaderboard at 17:49 yesterday (Dec-09, central eu time). I couldn't find your score with 0.0064. \n\n", "3068200": "Wow! Congrats! As I'm writing this, my LB score is just slightly ahead of yours, but my model is much more complex. Specifically, I'm using an ensemble of GBDT and MLP models with and without online learning. Any one model (type) performs well in CV but not that great in LB, especially the static GBDT model. What have you found to be the biggest bang for the buck in the training?", "3068596": "I didn't believe that a single and simple model could score more than 0.006, thanks for sharing!\nI've honestly used a blending solution of boosting trees and a NN model with also lag data but I couldn't get more than 0.0054.\nI think the data to train the model is key.\nOn my side I've considered the last 500 days of the train set provided in the competition.\nIt would be nice to hear how you select the training data.\nI'm surprised to see that you considered only the features as input data and you regressed the responders. Did you consider all the datetimes?", "3068172": "I've been trying to build a simple MLP (3 layers with 256 hidden units, ReLU/SiLU activation, dropout 0.1-0.3, 5*tanh output) and trained that data on days 1200-1699 and 0.001 LR. For some reason my submission scores are all quite bad (0 to -0.05). Does anyone have any advice on what I might have done wrong? (maybe the training data or architecture is just too small)", "3068405": "You are comparing scores on different datasets. Yesterday, [the public dataset was updated](https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/550790), 80 days were added. It seems that all models received a boost of 0.001-0.002. Therefore, 0.006 does not look impressive on the new dataset. We still need online learning.", "3075620": "I am impressed a simple MLP can have such good results!", "3083865": "I'm curious whether MLP or other NN model will overfitting the data now.", "3077774": "0.0064 from mlp is so impressive. Thank you for sharing the tips!", "3074651": "Im currently playing around with XGBoost and the different ways to implement it. I am either training a smaller model for each symbol s or one larger model for the whole data. This is my first competition and my first submission, that is why I had to get accustomed to the submission process first, which took up a little bit of my time. However, my first model for the whole dataset resulted in 0.0019, which is not perfect, but also not too bad for a first attempt. I am still trying to improve over time! Good luck to you all!", "3072811": "I’m lucky to have your support—thank you!", "3070376": "I am impressed a simple MLP can have such good results! May I ask any tips on how the network is trained, the specific architectural choices, and the generalization of the model as you mentioned? ", "3069170": "I have not been able to train any NN architecture that results in a positive R2 so you are beating me!", "3068949": "Has anyone tried sequential models? I tried to use LSTM and experiment on a bunch of seq lengths, but only get negative or 0 on r2", "3068158": "I am curious about how you choose your local CV set, can you share with us?", "3068135": "wow, amazing.\nhow about feature engineering. Did you normalize or anything?", "3068100": "Nice work! May I ask if that is from offline training only ? From what I’ve read so far in the discussion i haven’t seen many get much above .005-.006 with a single model offline", "3078457": "", "3071139": "thanks for the nice information" } }