Looking at the LB scores of this competition, the data seems to be very hard to model, and the variance of the target seems to be very high. I've been trying lgbm models (as the rest of the community it seems) and I'm currently thinking of creating an NN model (transformer or LSTM based), but my MLP baseline doesn't seem to be better than lgbm yet. Have you experimented with NNs on this data?
", "messages": [ { "id": "3042628", "postDate": "11/11/2024 16:20:00", "content": "Looking at the LB scores of this competition, the data seems to be very hard to model, and the variance of the target seems to be very high. I've been trying lgbm models (as the rest of the community it seems) and I'm currently thinking of creating an NN model (transformer or LSTM based), but my MLP baseline doesn't seem to be better than lgbm yet. Have you experimented with NNs on this data?
", "rawMarkdown": "Looking at the LB scores of this competition, the data seems to be very hard to model, and the variance of the target seems to be very high. I've been trying lgbm models (as the rest of the community it seems) and I'm currently thinking of creating an NN model (transformer or LSTM based), but my MLP baseline doesn't seem to be better than lgbm yet. Have you experimented with NNs on this data?", "votes": null }, { "id": "3042741", "postDate": "11/11/2024 17:59:23", "content": "I have been trying neural networks for the past week, and so far, my best result is 0.0038. However, I am facing some issues:
\nValidation R² instability: My R² score reaches its highest value too early, and after that, the results get worse.
\nValidation R² discrepancy with leaderboard (LB): The validation R² score is quite different from the LB score and is not stable. For example, an increase in the validation R² does not always lead to an improvement on the LB. This is probably my biggest issue right now, as submitting takes a lot of time, making it prohibitive to test many things.
I'm curios and I want to try MLP however I expected some of the difficulties your facing now which discourage me from trying to train MLP I think most of the time taken for each submission is spent on data cleaning specially on dealing with missing value and normalization and categorical feature encoding
", "rawMarkdown": "I'm curios and I want to try MLP however I expected some of the difficulties your facing now which discourage me from trying to train MLP I think most of the time taken for each submission is spent on data cleaning specially on dealing with missing value and normalization and categorical feature encoding", "votes": null }, { "id": "3042932", "postDate": "11/11/2024 23:35:58", "content": "\n\nMy R² score reaches its highest value too early, and after that, the results get worse.
\n
Seems like some overfitting is happening here. What is your model configuration?
", "rawMarkdown": ">My R² score reaches its highest value too early, and after that, the results get worse.\n\nSeems like some overfitting is happening here. What is your model configuration?", "votes": null }, { "id": "3043225", "postDate": "11/12/2024 07:45:02", "content": "Keep it up! I’m working on the same thing. After a week of effort, I’ve improved my MLP model’s score from 0.0031 to 0.0045.
", "rawMarkdown": "Keep it up! I’m working on the same thing. After a week of effort, I’ve improved my MLP model’s score from 0.0031 to 0.0045.", "votes": null }, { "id": "3043501", "postDate": "11/12/2024 12:56:49", "content": "I tried many configurations: pure NN, AE + NN, and in all cases, the network reaches its highest value (Val_R2) too early (around epochs 4-5), and after that, the val_r2 starts to decline (while val_loss keeps decreasing).
\nI have already tried different data splits, such as using files 1-7 for training and 8-9 for validation, using files 1-9 for training, and using the last 100 days for validation. However, this results in an inconsistent relationship between val_r2 and the leaderboard (LB).
", "rawMarkdown": "I tried many configurations: pure NN, AE + NN, and in all cases, the network reaches its highest value (Val_R2) too early (around epochs 4-5), and after that, the val_r2 starts to decline (while val_loss keeps decreasing).\n\nI have already tried different data splits, such as using files 1-7 for training and 8-9 for validation, using files 1-9 for training, and using the last 100 days for validation. However, this results in an inconsistent relationship between val_r2 and the leaderboard (LB).", "votes": null }, { "id": "3043665", "postDate": "11/12/2024 15:14:21", "content": "This notebook (MLP baseline) might be useful for Neural Networks.
", "rawMarkdown": "This [notebook](https://www.kaggle.com/code/voix97/jane-street-rmf-nn-with-pytorch-lightning) (MLP baseline) might be useful for Neural Networks.", "votes": null }, { "id": "3043799", "postDate": "11/12/2024 17:58:26", "content": "what is the size of your model ? For example, how many neurons and layers does your MLP have?
", "rawMarkdown": "what is the size of your model ? For example, how many neurons and layers does your MLP have?", "votes": null }, { "id": "3043888", "postDate": "11/12/2024 19:18:31", "content": "My best model so far (0.0043):
\nTotal params: 344,221 (1.31 MB)
\n Trainable params: 114,175 (446.00 KB)
\n Non-trainable params: 1,694 (6.62 KB)
\n Optimizer params: 228,352 (892.00 KB)
Total 7 Dense Layers, + dropout, BatchNormalization,
", "rawMarkdown": "My best model so far (0.0043):\n\n Total params: 344,221 (1.31 MB)\n Trainable params: 114,175 (446.00 KB)\n Non-trainable params: 1,694 (6.62 KB)\n Optimizer params: 228,352 (892.00 KB)\n\nTotal 7 Dense Layers, + dropout, BatchNormalization,", "votes": null }, { "id": "3043889", "postDate": "11/12/2024 19:18:37", "content": "I was using LGBM and with a few attempts got 0.0050. I'm trying NN now and my best score has been 0.0046 with it. The biggest issue here with NNs seems to be dealing with the time span overfitting - i.e. the model loses accuracy the more you go forward in time. I suspect the top scores have been able to overcome that.
", "rawMarkdown": "I was using LGBM and with a few attempts got 0.0050. I'm trying NN now and my best score has been 0.0046 with it. The biggest issue here with NNs seems to be dealing with the time span overfitting - i.e. the model loses accuracy the more you go forward in time. I suspect the top scores have been able to overcome that.", "votes": null }, { "id": "3043899", "postDate": "11/12/2024 19:28:37", "content": "Maybe your learning rate is too high? Have you tried smaller values?
", "rawMarkdown": "Maybe your learning rate is too high? Have you tried smaller values?", "votes": null }, { "id": "3043978", "postDate": "11/12/2024 21:45:43", "content": "try less layers & less params?
", "rawMarkdown": "try less layers & less params?", "votes": null }, { "id": "3044182", "postDate": "11/13/2024 06:18:24", "content": "and your dropout rate?
", "rawMarkdown": "and your dropout rate?", "votes": null }, { "id": "3044357", "postDate": "11/13/2024 10:03:20", "content": "Hello @voix97 thank you for sharing. Can you release your training pipeline. I am not used to pytorch, I use tensorflow.
", "rawMarkdown": "Hello @voix97 thank you for sharing. Can you release your training pipeline. I am not used to pytorch, I use tensorflow.", "votes": null }, { "id": "3044362", "postDate": "11/13/2024 10:12:55", "content": "And how about your validation split, which parts did you take?
", "rawMarkdown": "And how about your validation split, which parts did you take?", "votes": null }, { "id": "3044374", "postDate": "11/13/2024 10:28:33", "content": "Hi, I'm preparing for collecting training code. Could you please upvote the inference notebook?
", "rawMarkdown": "Hi, I'm preparing for collecting training code. Could you please upvote the inference notebook?", "votes": null }, { "id": "3044442", "postDate": "11/13/2024 12:28:38", "content": "Do you think I have too many parameters?
", "rawMarkdown": "Do you think I have too many parameters?", "votes": null }, { "id": "3044446", "postDate": "11/13/2024 12:31:11", "content": "Parquet 8 and 9 to val.
\nI've tried in the last 100 days, my results have gotten worse
Dropout after dense layers (0.1)
", "rawMarkdown": "Dropout after dense layers (0.1)", "votes": null }, { "id": "3044450", "postDate": "11/13/2024 12:32:13", "content": "5e-5.
\nI will try lr decay or reduce on plateau.
", "rawMarkdown": "5e-5.\n\nI will try lr decay or reduce on plateau.", "votes": null }, { "id": "3044542", "postDate": "11/13/2024 13:50:50", "content": "I had a similar val and similar issues. Now trying different splits and overfitting seems better on local CV. But didn't yet submitted to see how the private score will act…
", "rawMarkdown": "I had a similar val and similar issues. Now trying different splits and overfitting seems better on local CV. But didn't yet submitted to see how the private score will act...", "votes": null }, { "id": "3044578", "postDate": "11/13/2024 14:43:41", "content": "Yes, I would try 3 layers and reduce the number of hidden neurons. The public NN model used an MLP with only 3 layers of [512, 512, 256] hidden neurons and it works. I'd rather start from a simple model and gradually grow its parameter, instead of starting directly with a complicated one.
", "rawMarkdown": "Yes, I would try 3 layers and reduce the number of hidden neurons. The public NN model used an MLP with only 3 layers of [512, 512, 256] hidden neurons and it works. I'd rather start from a simple model and gradually grow its parameter, instead of starting directly with a complicated one.", "votes": null }, { "id": "3044615", "postDate": "11/13/2024 15:48:43", "content": "@carmenai This notebook has shown training pipeline.
", "rawMarkdown": "carmenai This [notebook](https://www.kaggle.com/code/voix97/jane-street-rmf-training-nn) has shown training pipeline.", "votes": null }, { "id": "3044749", "postDate": "11/13/2024 19:11:25", "content": "@voix97 Thanks! The notebook is super helpful.
", "rawMarkdown": "voix97 Thanks! The notebook is super helpful.", "votes": null }, { "id": "3046128", "postDate": "11/15/2024 06:31:38", "content": "great, i like this topic
", "rawMarkdown": "great, i like this topic", "votes": null } ], "comments": [ { "id": 3042741, "author_name": "nandodmelo", "author_url": "", "post_date": "11/11/2024 17:59:23", "content": "I have been trying neural networks for the past week, and so far, my best result is 0.0038. However, I am facing some issues:
\nValidation R² instability: My R² score reaches its highest value too early, and after that, the results get worse.
\nValidation R² discrepancy with leaderboard (LB): The validation R² score is quite different from the LB score and is not stable. For example, an increase in the validation R² does not always lead to an improvement on the LB. This is probably my biggest issue right now, as submitting takes a lot of time, making it prohibitive to test many things.
I'm curios and I want to try MLP however I expected some of the difficulties your facing now which discourage me from trying to train MLP I think most of the time taken for each submission is spent on data cleaning specially on dealing with missing value and normalization and categorical feature encoding
", "votes": null, "replies": [] }, { "id": 3042932, "author_name": "shiyili", "author_url": "", "post_date": "11/11/2024 23:35:58", "content": "\n\nMy R² score reaches its highest value too early, and after that, the results get worse.
\n
Seems like some overfitting is happening here. What is your model configuration?
", "votes": null, "replies": [ { "id": 3043501, "author_name": "nandodmelo", "author_url": "", "post_date": "11/12/2024 12:56:49", "content": "I tried many configurations: pure NN, AE + NN, and in all cases, the network reaches its highest value (Val_R2) too early (around epochs 4-5), and after that, the val_r2 starts to decline (while val_loss keeps decreasing).
\nI have already tried different data splits, such as using files 1-7 for training and 8-9 for validation, using files 1-9 for training, and using the last 100 days for validation. However, this results in an inconsistent relationship between val_r2 and the leaderboard (LB).
", "votes": null, "replies": [ { "id": 3043799, "author_name": "shiyili", "author_url": "", "post_date": "11/12/2024 17:58:26", "content": "what is the size of your model ? For example, how many neurons and layers does your MLP have?
", "votes": null, "replies": [ { "id": 3043888, "author_name": "nandodmelo", "author_url": "", "post_date": "11/12/2024 19:18:31", "content": "My best model so far (0.0043):
\nTotal params: 344,221 (1.31 MB)
\n Trainable params: 114,175 (446.00 KB)
\n Non-trainable params: 1,694 (6.62 KB)
\n Optimizer params: 228,352 (892.00 KB)
Total 7 Dense Layers, + dropout, BatchNormalization,
", "votes": null, "replies": [ { "id": 3043978, "author_name": "shiyili", "author_url": "", "post_date": "11/12/2024 21:45:43", "content": "try less layers & less params?
", "votes": null, "replies": [ { "id": 3044442, "author_name": "nandodmelo", "author_url": "", "post_date": "11/13/2024 12:28:38", "content": "Do you think I have too many parameters?
", "votes": null, "replies": [ { "id": 3044578, "author_name": "shiyili", "author_url": "", "post_date": "11/13/2024 14:43:41", "content": "Yes, I would try 3 layers and reduce the number of hidden neurons. The public NN model used an MLP with only 3 layers of [512, 512, 256] hidden neurons and it works. I'd rather start from a simple model and gradually grow its parameter, instead of starting directly with a complicated one.
", "votes": null, "replies": [] } ] } ] }, { "id": 3044182, "author_name": "wjjjjs", "author_url": "", "post_date": "11/13/2024 06:18:24", "content": "and your dropout rate?
", "votes": null, "replies": [ { "id": 3044447, "author_name": "nandodmelo", "author_url": "", "post_date": "11/13/2024 12:31:31", "content": "Dropout after dense layers (0.1)
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3043899, "author_name": "natanlabarrere", "author_url": "", "post_date": "11/12/2024 19:28:37", "content": "Maybe your learning rate is too high? Have you tried smaller values?
", "votes": null, "replies": [ { "id": 3044450, "author_name": "nandodmelo", "author_url": "", "post_date": "11/13/2024 12:32:13", "content": "5e-5.
\nI will try lr decay or reduce on plateau.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3044362, "author_name": "brtvandenbroeck", "author_url": "", "post_date": "11/13/2024 10:12:55", "content": "And how about your validation split, which parts did you take?
", "votes": null, "replies": [ { "id": 3044446, "author_name": "nandodmelo", "author_url": "", "post_date": "11/13/2024 12:31:11", "content": "Parquet 8 and 9 to val.
\nI've tried in the last 100 days, my results have gotten worse
I had a similar val and similar issues. Now trying different splits and overfitting seems better on local CV. But didn't yet submitted to see how the private score will act…
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3043225, "author_name": "wjjjjs", "author_url": "", "post_date": "11/12/2024 07:45:02", "content": "Keep it up! I’m working on the same thing. After a week of effort, I’ve improved my MLP model’s score from 0.0031 to 0.0045.
", "votes": null, "replies": [] }, { "id": 3043665, "author_name": "voix97", "author_url": "", "post_date": "11/12/2024 15:14:21", "content": "This notebook (MLP baseline) might be useful for Neural Networks.
", "votes": null, "replies": [ { "id": 3044357, "author_name": "carmenai", "author_url": "", "post_date": "11/13/2024 10:03:20", "content": "Hello @voix97 thank you for sharing. Can you release your training pipeline. I am not used to pytorch, I use tensorflow.
", "votes": null, "replies": [ { "id": 3044374, "author_name": "voix97", "author_url": "", "post_date": "11/13/2024 10:28:33", "content": "Hi, I'm preparing for collecting training code. Could you please upvote the inference notebook?
", "votes": null, "replies": [] }, { "id": 3044615, "author_name": "voix97", "author_url": "", "post_date": "11/13/2024 15:48:43", "content": "@carmenai This notebook has shown training pipeline.
", "votes": null, "replies": [ { "id": 3044749, "author_name": "verictan", "author_url": "", "post_date": "11/13/2024 19:11:25", "content": "@voix97 Thanks! The notebook is super helpful.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3043889, "author_name": "natanlabarrere", "author_url": "", "post_date": "11/12/2024 19:18:37", "content": "I was using LGBM and with a few attempts got 0.0050. I'm trying NN now and my best score has been 0.0046 with it. The biggest issue here with NNs seems to be dealing with the time span overfitting - i.e. the model loses accuracy the more you go forward in time. I suspect the top scores have been able to overcome that.
", "votes": null, "replies": [] }, { "id": 3046128, "author_name": "weihuanggame", "author_url": "", "post_date": "11/15/2024 06:31:38", "content": "great, i like this topic
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3042628": "Looking at the LB scores of this competition, the data seems to be very hard to model, and the variance of the target seems to be very high. I've been trying lgbm models (as the rest of the community it seems) and I'm currently thinking of creating an NN model (transformer or LSTM based), but my MLP baseline doesn't seem to be better than lgbm yet. Have you experimented with NNs on this data?", "3042741": "I have been trying neural networks for the past week, and so far, my best result is 0.0038. However, I am facing some issues:\n\nValidation R² instability: My R² score reaches its highest value too early, and after that, the results get worse.\nValidation R² discrepancy with leaderboard (LB): The validation R² score is quite different from the LB score and is not stable. For example, an increase in the validation R² does not always lead to an improvement on the LB. This is probably my biggest issue right now, as submitting takes a lot of time, making it prohibitive to test many things.", "3042843": "I'm curios and I want to try MLP however I expected some of the difficulties your facing now which discourage me from trying to train MLP I think most of the time taken for each submission is spent on data cleaning specially on dealing with missing value and normalization and categorical feature encoding", "3042932": ">My R² score reaches its highest value too early, and after that, the results get worse.\n\nSeems like some overfitting is happening here. What is your model configuration?", "3043225": "Keep it up! I’m working on the same thing. After a week of effort, I’ve improved my MLP model’s score from 0.0031 to 0.0045.", "3043501": "I tried many configurations: pure NN, AE + NN, and in all cases, the network reaches its highest value (Val_R2) too early (around epochs 4-5), and after that, the val_r2 starts to decline (while val_loss keeps decreasing).\n\nI have already tried different data splits, such as using files 1-7 for training and 8-9 for validation, using files 1-9 for training, and using the last 100 days for validation. However, this results in an inconsistent relationship between val_r2 and the leaderboard (LB).", "3043665": "This [notebook](https://www.kaggle.com/code/voix97/jane-street-rmf-nn-with-pytorch-lightning) (MLP baseline) might be useful for Neural Networks.", "3043799": "what is the size of your model ? For example, how many neurons and layers does your MLP have?", "3043888": "My best model so far (0.0043):\n\n Total params: 344,221 (1.31 MB)\n Trainable params: 114,175 (446.00 KB)\n Non-trainable params: 1,694 (6.62 KB)\n Optimizer params: 228,352 (892.00 KB)\n\nTotal 7 Dense Layers, + dropout, BatchNormalization,", "3043889": "I was using LGBM and with a few attempts got 0.0050. I'm trying NN now and my best score has been 0.0046 with it. The biggest issue here with NNs seems to be dealing with the time span overfitting - i.e. the model loses accuracy the more you go forward in time. I suspect the top scores have been able to overcome that.", "3043899": "Maybe your learning rate is too high? Have you tried smaller values?", "3043978": "try less layers & less params?", "3044182": "and your dropout rate?", "3044357": "Hello @voix97 thank you for sharing. Can you release your training pipeline. I am not used to pytorch, I use tensorflow.", "3044362": "And how about your validation split, which parts did you take?", "3044374": "Hi, I'm preparing for collecting training code. Could you please upvote the inference notebook?", "3044442": "Do you think I have too many parameters?", "3044446": "Parquet 8 and 9 to val.\nI've tried in the last 100 days, my results have gotten worse", "3044447": "Dropout after dense layers (0.1)", "3044450": "5e-5.\n\nI will try lr decay or reduce on plateau.", "3044542": "I had a similar val and similar issues. Now trying different splits and overfitting seems better on local CV. But didn't yet submitted to see how the private score will act...", "3044578": "Yes, I would try 3 layers and reduce the number of hidden neurons. The public NN model used an MLP with only 3 layers of [512, 512, 256] hidden neurons and it works. I'd rather start from a simple model and gradually grow its parameter, instead of starting directly with a complicated one.", "3044615": "carmenai This [notebook](https://www.kaggle.com/code/voix97/jane-street-rmf-training-nn) has shown training pipeline.", "3044749": "voix97 Thanks! The notebook is super helpful.", "3046128": "great, i like this topic" }, "source": "meta" }