![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16529018%2Fd4d8a857c4925bae93b53d42c249e2cb%2F2024-11-21%20130641.png?generation=1732166502269814&alt=media\")
My training notebook here : https://www.kaggle.com/code/i2nfinit3y/jane-street-tabm-ft-transformer-training/notebook
\nWhether I use simple model -- TabM with MLP or more complex model --FT-Transformer, the validation metrics （valid dataset are last 180 dates）get worse and worse:
I also use the network from public notebook:https://www.kaggle.com/code/voix97/jane-street-rmf-training-nn,
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16529018%2F95ff41855f738c5e902dd1b8983b70f7%2F2024-11-21%20132254.png?generation=1732166596324930&alt=media\")
but there is no change :
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16529018%2F18c7f62452250d058e1cc59a3e14aff4%2F2024-11-21%20132458.png?generation=1732166721040595&alt=media\")
I also tried to change my loss funcion from MSE to the right part of this competition evaluation metric:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16529018%2F637d42aea5ac2ae3f2eaf313f5d57e02%2F2024-11-21%20132537.png?generation=1732166760595946&alt=media\")
but it also didn't work.
\nHow do you train the NN model?
", "messages": [ { "id": "3051265", "postDate": "11/21/2024 05:27:58", "content": "My training notebook here : https://www.kaggle.com/code/i2nfinit3y/jane-street-tabm-ft-transformer-training/notebook
\nWhether I use simple model -- TabM with MLP or more complex model --FT-Transformer, the validation metrics （valid dataset are last 180 dates）get worse and worse:
I also use the network from public notebook:https://www.kaggle.com/code/voix97/jane-street-rmf-training-nn,
\n
but there is no change :
\n
I also tried to change my loss funcion from MSE to the right part of this competition evaluation metric:
\n
but it also didn't work.
\nHow do you train the NN model?
", "rawMarkdown": "My training notebook here : https://www.kaggle.com/code/i2nfinit3y/jane-street-tabm-ft-transformer-training/notebook\nWhether I use simple model -- TabM with MLP or more complex model --FT-Transformer, the validation metrics （valid dataset are last 180 dates）get worse and worse:\n\n\n\nI also use the network from public notebook:https://www.kaggle.com/code/voix97/jane-street-rmf-training-nn, \n\n\n\nbut there is no change :\n\n\n\nI also tried to change my loss funcion from MSE to the right part of this competition evaluation metric:\n\n\n\nbut it also didn't work.\n\nHow do you train the NN model?", "votes": null }, { "id": "3051425", "postDate": "11/21/2024 09:28:20", "content": "nn model overfits to the training dataset very quickly. If you plot your train loss and valid metric, you can see the loss is getting better but the valid metric gets worse.
\nBut there’s nothing wrong with it. The market pattern changes over time, and you can’t expect a simple MLP model trained in the past to be consistently effective in a long future.
", "rawMarkdown": "nn model overfits to the training dataset very quickly. If you plot your train loss and valid metric, you can see the loss is getting better but the valid metric gets worse.\n\nBut there’s nothing wrong with it. The market pattern changes over time, and you can’t expect a simple MLP model trained in the past to be consistently effective in a long future.", "votes": null }, { "id": "3051446", "postDate": "11/21/2024 09:53:48", "content": "So I might need to use some regularization method or don't rely too heavily on validation sets？🧐
", "rawMarkdown": "So I might need to use some regularization method or don't rely too heavily on validation sets？🧐", "votes": null }, { "id": "3051878", "postDate": "11/21/2024 18:15:04", "content": "I think everyone is facing the same challenges.
\nAdd some dropout/regularization layers, change your MSE loss to Huber loss, build a simple NN.
", "rawMarkdown": "I think everyone is facing the same challenges.\n\nAdd some dropout/regularization layers, change your MSE loss to Huber loss, build a simple NN.", "votes": null }, { "id": "3088213", "postDate": "01/04/2025 12:48:35", "content": "why the Huber loss should be better in this particular case? My understanding its main advantage is that it is less sensitive to outliers but in this particular competition, the response variable is already clipped between -5 and 5. Can you please share your thoughts? Thanks in advance.
", "rawMarkdown": "why the Huber loss should be better in this particular case? My understanding its main advantage is that it is less sensitive to outliers but in this particular competition, the response variable is already clipped between -5 and 5. Can you please share your thoughts? Thanks in advance.", "votes": null } ], "comments": [ { "id": 3051425, "author_name": "shiyili", "author_url": "", "post_date": "11/21/2024 09:28:20", "content": "nn model overfits to the training dataset very quickly. If you plot your train loss and valid metric, you can see the loss is getting better but the valid metric gets worse.
\nBut there’s nothing wrong with it. The market pattern changes over time, and you can’t expect a simple MLP model trained in the past to be consistently effective in a long future.
", "votes": null, "replies": [ { "id": 3051446, "author_name": "i2nfinit3y", "author_url": "", "post_date": "11/21/2024 09:53:48", "content": "So I might need to use some regularization method or don't rely too heavily on validation sets？🧐
", "votes": null, "replies": [] } ] }, { "id": 3051878, "author_name": "nandodmelo", "author_url": "", "post_date": "11/21/2024 18:15:04", "content": "I think everyone is facing the same challenges.
\nAdd some dropout/regularization layers, change your MSE loss to Huber loss, build a simple NN.
", "votes": null, "replies": [ { "id": 3088213, "author_name": "gregled", "author_url": "", "post_date": "01/04/2025 12:48:35", "content": "why the Huber loss should be better in this particular case? My understanding its main advantage is that it is less sensitive to outliers but in this particular competition, the response variable is already clipped between -5 and 5. Can you please share your thoughts? Thanks in advance.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3051265": "My training notebook here : https://www.kaggle.com/code/i2nfinit3y/jane-street-tabm-ft-transformer-training/notebook\nWhether I use simple model -- TabM with MLP or more complex model --FT-Transformer, the validation metrics （valid dataset are last 180 dates）get worse and worse:\n\n\n\nI also use the network from public notebook:https://www.kaggle.com/code/voix97/jane-street-rmf-training-nn, \n\n\n\nbut there is no change :\n\n\n\nI also tried to change my loss funcion from MSE to the right part of this competition evaluation metric:\n\n\n\nbut it also didn't work.\n\nHow do you train the NN model?", "3051425": "nn model overfits to the training dataset very quickly. If you plot your train loss and valid metric, you can see the loss is getting better but the valid metric gets worse.\n\nBut there’s nothing wrong with it. The market pattern changes over time, and you can’t expect a simple MLP model trained in the past to be consistently effective in a long future.", "3051446": "So I might need to use some regularization method or don't rely too heavily on validation sets？🧐", "3051878": "I think everyone is facing the same challenges.\n\nAdd some dropout/regularization layers, change your MSE loss to Huber loss, build a simple NN.", "3088213": "why the Huber loss should be better in this particular case? My understanding its main advantage is that it is less sensitive to outliers but in this particular competition, the response variable is already clipped between -5 and 5. Can you please share your thoughts? Thanks in advance." }, "source": "meta" }