We would like to thank the organizers and participants of this great competition! We learned a lot and had a lot of fun. Special thanks to ZhaounBooty for his early sharing solution, Robin Smits for sharing further improvements of the RNN idea, and Iafoss for sharing the dataloader.
\nBelow is a (preliminary) description of the two models we trained.
\nWe preprocessed the whole training dataset and saved it as a memory-mapped file. We have written an iterable dataset that allowed very fast data loading.
\nPrivate and public score 0.983
\nTraining:
\nPrivate and public score 0.987
\nTraining
\nIn both cases, the network output was a 3-dimensional unit vector. We then directly used the mean angular error as the loss.
\nInterestingly, despite very limited computer resources, the GRU model scores quite well. Unfortunately, we didn't have time to scale it further on the GTX 1080.
", "messages": [ { "id": "2228377", "postDate": "04/20/2023 13:57:06", "content": "We would like to thank the organizers and participants of this great competition! We learned a lot and had a lot of fun. Special thanks to ZhaounBooty for his early sharing solution, Robin Smits for sharing further improvements of the RNN idea, and Iafoss for sharing the dataloader.
\nBelow is a (preliminary) description of the two models we trained.
\nWe preprocessed the whole training dataset and saved it as a memory-mapped file. We have written an iterable dataset that allowed very fast data loading.
\nPrivate and public score 0.983
\nTraining:
\nPrivate and public score 0.987
\nTraining
\nIn both cases, the network output was a 3-dimensional unit vector. We then directly used the mean angular error as the loss.
\nInterestingly, despite very limited computer resources, the GRU model scores quite well. Unfortunately, we didn't have time to scale it further on the GTX 1080.
", "rawMarkdown": "We would like to thank the organizers and participants of this great competition! We learned a lot and had a lot of fun. Special thanks to [ZhaounBooty](https://www.kaggle.com/seungmoklee) for his early sharing solution, [Robin Smits](https://www.kaggle.com/rsmits) for sharing further improvements of the RNN idea, and [Iafoss](https://www.kaggle.com/iafoss) for sharing the dataloader.\n\nBelow is a (preliminary) description of the two models we trained.\n\nWe preprocessed the whole training dataset and saved it as a memory-mapped file. We have written an iterable dataset that allowed very fast data loading.\n\n\n\n### Transformer model:\n\n*Private and public score 0.983*\n\n* Size: 4.9M params\n* Batch: 2000 events\n* Hidden size: 256\n* Activation: SiLU\n* Layernorm\n* Num transformer layers: 3\n* Num heads: 4\n* Global aggregation: [sum, mean, max, std_dev]\n\nTraining:\n* LR: 0.002\n* Patience: 1, multiplicative decay: 0.9\n* Run for 77 epochs\n* 32 GPUs\n* Warmup: 2 epochs\n\n### Bidirectional GRU model:\n\n*Private and public score 0.987*\n\n* Size: 1.7 M parameters\n* Batch size: 2048\n* Input shape: (96, 6)\n* Number of layers: 3\n* hidden_size: 160\n* Hidden states of the last GRU unit are passed to a fully connected layer (size: 512)\n\nTraining\n* max LR 1e-3 with a warm up and cosine annealing ( OneCycleLR, pct_start: 0.001)\n* AdamW with weight decay 1e-4\n* Trained for 14 epochs on the full dataset on GTX 1080 GPU for 90 hours\n\nIn both cases, the network output was a 3-dimensional unit vector. We then directly used the mean angular error as the loss.\n\nInterestingly, despite very limited computer resources, the GRU model scores quite well. Unfortunately, we didn't have time to scale it further on the GTX 1080.", "votes": null }, { "id": "2228390", "postDate": "04/20/2023 14:08:09", "content": "Congratulations! Great score for single models!
\nIs any chance you will publish Transformer model script (architecture)?
Hi Remek, thanks very much!
\nSure - here is a gist for the Transformer: https://gist.github.com/timinar/264d2279f5116caa8d824c9427d800a5
\nWould love to hear your thoughts on our implementation!
", "rawMarkdown": "Hi Remek, thanks very much!\n\nSure - here is a gist for the Transformer: https://gist.github.com/timinar/264d2279f5116caa8d824c9427d800a5\n\nWould love to hear your thoughts on our implementation!", "votes": null }, { "id": "2229503", "postDate": "04/21/2023 12:29:04", "content": "Very nice solution! And great to see that the GRU model is able to score into the 0.98X region. Congratulations!
", "rawMarkdown": "Very nice solution! And great to see that the GRU model is able to score into the 0.98X region. Congratulations!", "votes": null }, { "id": "2229968", "postDate": "04/21/2023 21:41:09", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null } ], "comments": [ { "id": 2228390, "author_name": "remekkinas", "author_url": "", "post_date": "04/20/2023 14:08:09", "content": "Congratulations! Great score for single models!
\nIs any chance you will publish Transformer model script (architecture)?
Hi Remek, thanks very much!
\nSure - here is a gist for the Transformer: https://gist.github.com/timinar/264d2279f5116caa8d824c9427d800a5
\nWould love to hear your thoughts on our implementation!
", "votes": null, "replies": [] } ] }, { "id": 2229503, "author_name": "rsmits", "author_url": "", "post_date": "04/21/2023 12:29:04", "content": "Very nice solution! And great to see that the GRU model is able to score into the 0.98X region. Congratulations!
", "votes": null, "replies": [ { "id": 2229968, "author_name": "inartimiryasov", "author_url": "", "post_date": "04/21/2023 21:41:09", "content": "Thank you!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2228377": "We would like to thank the organizers and participants of this great competition! We learned a lot and had a lot of fun. Special thanks to [ZhaounBooty](https://www.kaggle.com/seungmoklee) for his early sharing solution, [Robin Smits](https://www.kaggle.com/rsmits) for sharing further improvements of the RNN idea, and [Iafoss](https://www.kaggle.com/iafoss) for sharing the dataloader.\n\nBelow is a (preliminary) description of the two models we trained.\n\nWe preprocessed the whole training dataset and saved it as a memory-mapped file. We have written an iterable dataset that allowed very fast data loading.\n\n\n\n### Transformer model:\n\n*Private and public score 0.983*\n\n* Size: 4.9M params\n* Batch: 2000 events\n* Hidden size: 256\n* Activation: SiLU\n* Layernorm\n* Num transformer layers: 3\n* Num heads: 4\n* Global aggregation: [sum, mean, max, std_dev]\n\nTraining:\n* LR: 0.002\n* Patience: 1, multiplicative decay: 0.9\n* Run for 77 epochs\n* 32 GPUs\n* Warmup: 2 epochs\n\n### Bidirectional GRU model:\n\n*Private and public score 0.987*\n\n* Size: 1.7 M parameters\n* Batch size: 2048\n* Input shape: (96, 6)\n* Number of layers: 3\n* hidden_size: 160\n* Hidden states of the last GRU unit are passed to a fully connected layer (size: 512)\n\nTraining\n* max LR 1e-3 with a warm up and cosine annealing ( OneCycleLR, pct_start: 0.001)\n* AdamW with weight decay 1e-4\n* Trained for 14 epochs on the full dataset on GTX 1080 GPU for 90 hours\n\nIn both cases, the network output was a 3-dimensional unit vector. We then directly used the mean angular error as the loss.\n\nInterestingly, despite very limited computer resources, the GRU model scores quite well. Unfortunately, we didn't have time to scale it further on the GTX 1080.", "2228390": "Congratulations! Great score for single models! \nIs any chance you will publish Transformer model script (architecture)?", "2229319": "Hi Remek, thanks very much!\n\nSure - here is a gist for the Transformer: https://gist.github.com/timinar/264d2279f5116caa8d824c9427d800a5\n\nWould love to hear your thoughts on our implementation!", "2229503": "Very nice solution! And great to see that the GRU model is able to score into the 0.98X region. Congratulations!", "2229968": "Thank you!" }, "source": "meta" }