Thanks to this amazing competition and the efforts of every one of my teammates ! @lechengyan @chronoscop
\nOur final solution of 0.0092 lb is ensembling NN models with online learning , GBDT offline models and a ridge model. My part is mainly for TabM model and some GBDT models, which is what I am gonna to talk about. AutoencoderMLP and online learning are @lechengyan 's part and @chronoscop is in charge of one of XGB.
I simply used the last 120 dates as my validation and it shows good correlations with LB.
\nActually, in the early time of this competition, I noticed TabM and found it is a tabular NN model with great potential. I also public a baseline notebook.
\nThe model parts I mainly adjust are feature embed layers and backbone.
\nFor category features, I use onehot encoding for every category tensor, which contains feature_09, feature_10, feature_11, symbol_id and time_id. I remap these category features with ordinal encode and every new category in the test data will be mapped to the same category. Apart from one-hot encoding, I also use embedding layers but it didn't help much. It's worth noting that taking time_id as category features improve my model al lot.
\nBut for continuous features, I have tried LinearEmbeddings, PeriodicEmbeddings and PiecewiseLinearEmbeddings, which didn't work well. So I don't use any embedding layer for continuous features.
\nThe backbone of TabM is just 3 layers MLP with the same 512 dimensions. I haven't experimented with too many dimensional combinations, but 512 seems to be the most stable.
I have used huber loss, logcosh loss, mae loss, zero-mean R2 loss and mse loss, and R2 loss and mse loss work best. Specifically, mse loss can get higher score in the cv but R2 loss is more robust in every validation epoch. So I choose R2 loss as loss function.
\nDropout : 0.25. Higher dropout rate (0.5) will result in slower convergence and lower (0.1) will be overfitting, so 0.25 seems to be a good choice for me.
\nLearning rate: 1e-3.
\nWeight decay: 8e-4.
\nBatch size : 8192
\nepoch : 5 or 6. Generally, I will submit multiple epoch to confirm the best model checkpoint and most of time 5 or 6 is the best.
\noptimizer : AdamW
\nk : 16. k is the number of ensemble in the finial output. I tried 8, 16, 24, 32, and I found 16 can takes both scores and training time into account.
Simple mean-std standardize and fill zero for nan values.
\nI use time_id, symbol_id, 78 original features (besides feature_61, because it just change with dates) and 9 responder lags 1 of last record. Additionally, I also use sin and cos of time_id with 967 periods or 483 periods and feature_61 with 20 period to catch some periodical change. Because though feature_61 only changes with the date, it also changes in about 20 dates. Also, I tried to use more lags features but it work worse in cv and lb.
\nBesides responder_6, I use responder_3 as my auxiliary target, because it has high correlations with responder_6. I tried to add more auxiliary targets in my training, but there was no obvious improvement in my model.
I add some gaussian noise in continuous features with 0.02 std.
\nI use the data with dates after 252, because there are too many Nan values in the first 252 dates. I have tried use the dates after 750 to train, which improve my cv but decrease my lb.
\nwithout online learning cv : 0.0106 lb : 0.0077
\nwith online learning cv : 0.0116 lb : 0.0083
Fill 3 for Nan values without standardization.
\nWe use time_id, symbol_id, 79 original features and responder_6 lags 1 of last. We didn't use auxilary target in the AutoencoderMLP
\nEncoder and Decoder : Both are a lieanr layer with 96 hidden dimension.
\nMLP : 5 layers with hidden dimension 96, 896, 448, 448, 256
Dropout : [0.035, 0.038, 0.424, 0.104, 0.25, 0.32, 0.271, 0.25]
\nLearning rate: 1e-3
\nBatch size : 8192
\nEpoch : about 15
\noptimizer : Adam
We use mse loss as reconstruction loss, and weighted-mse loss as prediction loss
\nwithout online learning cv : 0.0103 lb : 0.0072
\nwith online learning cv : 0.0110 lb : 0.0078
It is more simple now. For LGB, XGB and Ridge, I just use time_id, symbol_id, 79 features, responder lags 1 of last, mean, std and max. And I did not specify category features in the GBDT model.
\nThe dates after 750 for XGB and Ridge and dates after 678 for LGB
\nI didn't optimize my hyperparameter just use fixed one.
\nLGB_Params = {\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ' : '',\n ': ,\n ': ',\n }\n\nXGB_Params = {\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ',\n ' : '',\n ': ,\n ' : ':squarederror',\n}\nLGB cv: 0.0096 lb : 0.0072
\nXGB cv: 0.0102 lb : 0.0073
\nRidge cv: 0.0035 lb : 0.0044
\nAfter ensembling these 3 models, the lb is 0.0076.
Fill 0 for Nan values without standardization
\nuse symbol_id, time_id, 79 original feature and responder lags1 of last. If add more lags features, it would be easy to overfit.
\nuse dates after 917 to train the model.
\n = {\n: ,\n: ,\n: ,\n : ,\n: ,\n: ,\n: ,\n: ,\n: ,\n: \n}\ncv : 0.0102 lb : 0.0072
\nOnline learning play an important role in this competition.
\nWe collect data for each previous date and then update model in every date. For every date, we will combine the data of the last date with partial data from the previous dates (using random sampling) for online training. In the training part, instead of batch processing, we input all the data into the model and then train 5 epochs.
if previous_test is not and lags is not :\n train = previous_test.(\n lags_.([, , pl.().()]),\n on=[, ],\n how=\n )\n\n if pre_train is not None and (pre_train) > :\n pre_train = pre_train.(n=, seed=)\n if pre_train is None:\n pre_train = train\n else:\n pre_train = pl.([pre_train, train])\n X = pre_train.()[cols].().values\n y = pre_train[].()\n weights = pre_train[].()\n\n\n model.()\n optimizer = torch.optim.(model.(), lr=e-)\n criterion = nn.()\n\n\n for epoch in ():\n optimizer.()\n _, y_pred = (torch.(X).(device))\n loss = (y_pred.(), torch.(y.()).(device))\n loss.()\n optimizer.()\n (f)\n\nif previous_test is None:\n previous_test = test\nelse:\n previous_test = pl.([previous_test, test])\nWe tried to add online learning into GBDT, but it didn't work well and cost too many time.
\nI think TabM can get higher score if adjust different kind of backbones, but I have no time to experiment. I also tried GRU, LSTM or Transformer in the early time, but they all failed, so then I focus on tabular model. Hoping top teams can share more trick and I indeed learn a lot in the competition. Thanks to all of you! :)
\nPS: Our code will be released when it is sorted out.
UPDATE :
\nTabM training code https://www.kaggle.com/code/i2nfinit3y/jane-street-tabm-training?scriptVersionId=217873214
\nSingle TabM online learning : https://www.kaggle.com/code/i2nfinit3y/jane-street-tabm-online-learning
UPDATE :
\nwe’ve open-sourced both our inference and training code.
Thanks to this amazing competition and the efforts of every one of my teammates ! @lechengyan @chronoscop
\nOur final solution of 0.0092 lb is ensembling NN models with online learning , GBDT offline models and a ridge model. My part is mainly for TabM model and some GBDT models, which is what I am gonna to talk about. AutoencoderMLP and online learning are @lechengyan 's part and @chronoscop is in charge of one of XGB.
I simply used the last 120 dates as my validation and it shows good correlations with LB.
\nActually, in the early time of this competition, I noticed TabM and found it is a tabular NN model with great potential. I also public a baseline notebook.
\nThe model parts I mainly adjust are feature embed layers and backbone.
\nFor category features, I use onehot encoding for every category tensor, which contains feature_09, feature_10, feature_11, symbol_id and time_id. I remap these category features with ordinal encode and every new category in the test data will be mapped to the same category. Apart from one-hot encoding, I also use embedding layers but it didn't help much. It's worth noting that taking time_id as category features improve my model al lot.
\nBut for continuous features, I have tried LinearEmbeddings, PeriodicEmbeddings and PiecewiseLinearEmbeddings, which didn't work well. So I don't use any embedding layer for continuous features.
\nThe backbone of TabM is just 3 layers MLP with the same 512 dimensions. I haven't experimented with too many dimensional combinations, but 512 seems to be the most stable.
I have used huber loss, logcosh loss, mae loss, zero-mean R2 loss and mse loss, and R2 loss and mse loss work best. Specifically, mse loss can get higher score in the cv but R2 loss is more robust in every validation epoch. So I choose R2 loss as loss function.
\nDropout : 0.25. Higher dropout rate (0.5) will result in slower convergence and lower (0.1) will be overfitting, so 0.25 seems to be a good choice for me.
\nLearning rate: 1e-3.
\nWeight decay: 8e-4.
\nBatch size : 8192
\nepoch : 5 or 6. Generally, I will submit multiple epoch to confirm the best model checkpoint and most of time 5 or 6 is the best.
\noptimizer : AdamW
\nk : 16. k is the number of ensemble in the finial output. I tried 8, 16, 24, 32, and I found 16 can takes both scores and training time into account.
Simple mean-std standardize and fill zero for nan values.
\nI use time_id, symbol_id, 78 original features (besides feature_61, because it just change with dates) and 9 responder lags 1 of last record. Additionally, I also use sin and cos of time_id with 967 periods or 483 periods and feature_61 with 20 period to catch some periodical change. Because though feature_61 only changes with the date, it also changes in about 20 dates. Also, I tried to use more lags features but it work worse in cv and lb.
\nBesides responder_6, I use responder_3 as my auxiliary target, because it has high correlations with responder_6. I tried to add more auxiliary targets in my training, but there was no obvious improvement in my model.
I add some gaussian noise in continuous features with 0.02 std.
\nI use the data with dates after 252, because there are too many Nan values in the first 252 dates. I have tried use the dates after 750 to train, which improve my cv but decrease my lb.
\nwithout online learning cv : 0.0106 lb : 0.0077
\nwith online learning cv : 0.0116 lb : 0.0083
Fill 3 for Nan values without standardization.
\nWe use time_id, symbol_id, 79 original features and responder_6 lags 1 of last. We didn't use auxilary target in the AutoencoderMLP
\nEncoder and Decoder : Both are a lieanr layer with 96 hidden dimension.
\nMLP : 5 layers with hidden dimension 96, 896, 448, 448, 256
Dropout : [0.035, 0.038, 0.424, 0.104, 0.25, 0.32, 0.271, 0.25]
\nLearning rate: 1e-3
\nBatch size : 8192
\nEpoch : about 15
\noptimizer : Adam
We use mse loss as reconstruction loss, and weighted-mse loss as prediction loss
\nwithout online learning cv : 0.0103 lb : 0.0072
\nwith online learning cv : 0.0110 lb : 0.0078
It is more simple now. For LGB, XGB and Ridge, I just use time_id, symbol_id, 79 features, responder lags 1 of last, mean, std and max. And I did not specify category features in the GBDT model.
\nThe dates after 750 for XGB and Ridge and dates after 678 for LGB
\nI didn't optimize my hyperparameter just use fixed one.
\nLGB_Params = {\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ' : '',\n ': ,\n ': ',\n }\n\nXGB_Params = {\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ,\n ': ',\n ' : '',\n ': ,\n ' : ':squarederror',\n}\nLGB cv: 0.0096 lb : 0.0072
\nXGB cv: 0.0102 lb : 0.0073
\nRidge cv: 0.0035 lb : 0.0044
\nAfter ensembling these 3 models, the lb is 0.0076.
Fill 0 for Nan values without standardization
\nuse symbol_id, time_id, 79 original feature and responder lags1 of last. If add more lags features, it would be easy to overfit.
\nuse dates after 917 to train the model.
\n = {\n: ,\n: ,\n: ,\n : ,\n: ,\n: ,\n: ,\n: ,\n: ,\n: \n}\ncv : 0.0102 lb : 0.0072
\nOnline learning play an important role in this competition.
\nWe collect data for each previous date and then update model in every date. For every date, we will combine the data of the last date with partial data from the previous dates (using random sampling) for online training. In the training part, instead of batch processing, we input all the data into the model and then train 5 epochs.
if previous_test is not and lags is not :\n train = previous_test.(\n lags_.([, , pl.().()]),\n on=[, ],\n how=\n )\n\n if pre_train is not None and (pre_train) > :\n pre_train = pre_train.(n=, seed=)\n if pre_train is None:\n pre_train = train\n else:\n pre_train = pl.([pre_train, train])\n X = pre_train.()[cols].().values\n y = pre_train[].()\n weights = pre_train[].()\n\n\n model.()\n optimizer = torch.optim.(model.(), lr=e-)\n criterion = nn.()\n\n\n for epoch in ():\n optimizer.()\n _, y_pred = (torch.(X).(device))\n loss = (y_pred.(), torch.(y.()).(device))\n loss.()\n optimizer.()\n (f)\n\nif previous_test is None:\n previous_test = test\nelse:\n previous_test = pl.([previous_test, test])\nWe tried to add online learning into GBDT, but it didn't work well and cost too many time.
\nI think TabM can get higher score if adjust different kind of backbones, but I have no time to experiment. I also tried GRU, LSTM or Transformer in the early time, but they all failed, so then I focus on tabular model. Hoping top teams can share more trick and I indeed learn a lot in the competition. Thanks to all of you! :)
\nPS: Our code will be released when it is sorted out.
UPDATE :
\nTabM training code https://www.kaggle.com/code/i2nfinit3y/jane-street-tabm-training?scriptVersionId=217873214
\nSingle TabM online learning : https://www.kaggle.com/code/i2nfinit3y/jane-street-tabm-online-learning
UPDATE :
\nwe’ve open-sourced both our inference and training code.