First of all congratulations to all top scoring solutions and competitors. This was really a challenging but very interresting competition.
\nThe basic training and inference setup I kept using are mostly the same as the training and inference notebooks I published earlier in the competition.
\nI continued optimizing those. Made them run on the TPU's in combination with TFRecords. This allowed me to continue increasing the size of the GRU models and drastically increased the training time.
\nI also tried continuing the training of the public GNN model. With some mixed succes. Eventually my problem was the amount of compute resources available. I decided to stick with the GRU based models only. Also ensembling the GRU models seemed to have better performance than ensembling with GNN models.
\nMy final solution was a combination of 5 GRU models trained on data with a max of 128 pulses and 6 GRU models trained on data with a max of 160 pulses used for each event.
\nMy final submission can be found here
\nWhat did work:
\nWhat didn't work or didn't seem to work:
\nFirst of all congratulations to all top scoring solutions and competitors. This was really a challenging but very interresting competition.
\nThe basic training and inference setup I kept using are mostly the same as the training and inference notebooks I published earlier in the competition.
\nI continued optimizing those. Made them run on the TPU's in combination with TFRecords. This allowed me to continue increasing the size of the GRU models and drastically increased the training time.
\nI also tried continuing the training of the public GNN model. With some mixed succes. Eventually my problem was the amount of compute resources available. I decided to stick with the GRU based models only. Also ensembling the GRU models seemed to have better performance than ensembling with GNN models.
\nMy final solution was a combination of 5 GRU models trained on data with a max of 128 pulses and 6 GRU models trained on data with a max of 160 pulses used for each event.
\nMy final submission can be found here
\nWhat did work:
\nWhat didn't work or didn't seem to work:
\nThank you very much for publishing your versions of GRU!
\nInterestingly, we used a much smaller net: a Bidirectional GRU with 3 layers and a hidden size of 160, followed by one hidden layer of size 512 and a 3-dimensional output (which we normalized to one). We essentially used the angular distance score as a loss. This setup was sometimes unstable in the first couple of thousands of steps (we used a batch size of 2048 due to hardware limitations), so we had to restart the run with a different random seed. But after that, it trained well.
I would like to thank you @rsmits for your contribution in this competition. Part of our solution was inspired by your work.
", "rawMarkdown": "I would like to thank you @rsmits for your contribution in this competition. Part of our solution was inspired by your work.", "votes": null }, { "id": "2229450", "postDate": "04/21/2023 11:26:21", "content": "@remekkinas You're welcome :-) With an 8th place your team did great!
", "rawMarkdown": "remekkinas You're welcome :-) With an 8th place your team did great!", "votes": null }, { "id": "2229502", "postDate": "04/21/2023 12:27:29", "content": "@inartimiryasov Thanks and your welcome :-) Just went through your solution. I think that the combination of using different Loss function and the fact that you used the returned states likely make the difference for improving the GRU based model to perform into the 0.98X region. Very cool!
", "rawMarkdown": "inartimiryasov Thanks and your welcome :-) Just went through your solution. I think that the combination of using different Loss function and the fact that you used the returned states likely make the difference for improving the GRU based model to perform into the 0.98X region. Very cool!", "votes": null }, { "id": "2229800", "postDate": "04/21/2023 17:33:12", "content": "@rsmits thanks for publishing your training and inference code. I started this competition by going through your code and it was super helpful.
\nWhat was your best single model on local validation?
", "rawMarkdown": "rsmits thanks for publishing your training and inference code. I started this competition by going through your code and it was super helpful.\n\nWhat was your best single model on local validation?", "votes": null }, { "id": "2229836", "postDate": "04/21/2023 18:37:12", "content": "Hi @crodoc Thanks and you're welcome :-)
\nMy best local validation was 0.99757 on training batches 1, 2 and 3. Number of pulses used was 128.
Nice! I wasn't able to squeeze out that much with practically the same approach.
", "rawMarkdown": "Nice! I wasn't able to squeeze out that much with practically the same approach.", "votes": null }, { "id": "2229967", "postDate": "04/21/2023 21:40:20", "content": "Thank you! I read what I had written in the solution, and it wasn't very clear. We also used the hidden state of the last unit (2 * hidden_size since there are hidden states for both the forward and backward GRUs). I believe this corresponds to setting return_sequences = False in TensorFlow.
\nApart from using a different loss function, which outperformed some other regression options that we tried, we also trained the model on the entire training set.
", "rawMarkdown": "Thank you! I read what I had written in the solution, and it wasn't very clear. We also used the hidden state of the last unit (2 * hidden_size since there are hidden states for both the forward and backward GRUs). I believe this corresponds to setting return_sequences = False in TensorFlow.\n\nApart from using a different loss function, which outperformed some other regression options that we tried, we also trained the model on the entire training set.", "votes": null }, { "id": "2231052", "postDate": "04/23/2023 03:50:12", "content": "@rsmits your notebooks were very much helpful for us. Thank you.
", "rawMarkdown": "rsmits your notebooks were very much helpful for us. Thank you.", "votes": null } ], "comments": [ { "id": 2229341, "author_name": "inartimiryasov", "author_url": "", "post_date": "04/21/2023 09:30:35", "content": "Thank you very much for publishing your versions of GRU!
\nInterestingly, we used a much smaller net: a Bidirectional GRU with 3 layers and a hidden size of 160, followed by one hidden layer of size 512 and a 3-dimensional output (which we normalized to one). We essentially used the angular distance score as a loss. This setup was sometimes unstable in the first couple of thousands of steps (we used a batch size of 2048 due to hardware limitations), so we had to restart the run with a different random seed. But after that, it trained well.
@inartimiryasov Thanks and your welcome :-) Just went through your solution. I think that the combination of using different Loss function and the fact that you used the returned states likely make the difference for improving the GRU based model to perform into the 0.98X region. Very cool!
", "votes": null, "replies": [ { "id": 2229967, "author_name": "inartimiryasov", "author_url": "", "post_date": "04/21/2023 21:40:20", "content": "Thank you! I read what I had written in the solution, and it wasn't very clear. We also used the hidden state of the last unit (2 * hidden_size since there are hidden states for both the forward and backward GRUs). I believe this corresponds to setting return_sequences = False in TensorFlow.
\nApart from using a different loss function, which outperformed some other regression options that we tried, we also trained the model on the entire training set.
", "votes": null, "replies": [] } ] } ] }, { "id": 2229371, "author_name": "remekkinas", "author_url": "", "post_date": "04/21/2023 10:05:04", "content": "I would like to thank you @rsmits for your contribution in this competition. Part of our solution was inspired by your work.
", "votes": null, "replies": [ { "id": 2229450, "author_name": "rsmits", "author_url": "", "post_date": "04/21/2023 11:26:21", "content": "@remekkinas You're welcome :-) With an 8th place your team did great!
", "votes": null, "replies": [] } ] }, { "id": 2229800, "author_name": "crodoc", "author_url": "", "post_date": "04/21/2023 17:33:12", "content": "@rsmits thanks for publishing your training and inference code. I started this competition by going through your code and it was super helpful.
\nWhat was your best single model on local validation?
", "votes": null, "replies": [ { "id": 2229836, "author_name": "rsmits", "author_url": "", "post_date": "04/21/2023 18:37:12", "content": "Hi @crodoc Thanks and you're welcome :-)
\nMy best local validation was 0.99757 on training batches 1, 2 and 3. Number of pulses used was 128.
Nice! I wasn't able to squeeze out that much with practically the same approach.
", "votes": null, "replies": [] } ] } ] }, { "id": 2231052, "author_name": "junseonglee11", "author_url": "", "post_date": "04/23/2023 03:50:12", "content": "@rsmits your notebooks were very much helpful for us. Thank you.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2228865": "First of all congratulations to all top scoring solutions and competitors. This was really a challenging but very interresting competition.\n\nThe basic training and inference setup I kept using are mostly the same as the [training](https://www.kaggle.com/code/rsmits/tensorflow-lstm-model-training-tpu) and [inference](https://www.kaggle.com/code/rsmits/tensorflow-lstm-model-inference) notebooks I published earlier in the competition.\n\nI continued optimizing those. Made them run on the TPU's in combination with TFRecords. This allowed me to continue increasing the size of the GRU models and drastically increased the training time.\n\nI also tried continuing the training of the public GNN model. With some mixed succes. Eventually my problem was the amount of compute resources available. I decided to stick with the GRU based models only. Also ensembling the GRU models seemed to have better performance than ensembling with GNN models.\n\nMy final solution was a combination of 5 GRU models trained on data with a max of 128 pulses and 6 GRU models trained on data with a max of 160 pulses used for each event.\n\nMy final submission can be found [here](https://www.kaggle.com/code/rsmits/26th-place-solution-lstm-ensemble)\n\n**What did work:**\n1. I kept increasing the number of bins to use. While performance kept increasing it started to slowly level of. Eventually I settled for using 64 bins which resulted in a head layer of 4096 dense units.\n2. Moving from the .npz files to convert everything to TFRecords. This allowed me to use almost all data and train the various notebooks efficiently on both Kaggle TPU VM and Colab Pro TPU's.\n3. Increasing the number of GRU layers.\n4. Increasing the amount of cells in each GRU layer.\n5. Depending on the model and amount of data I could train 8 to 12 epochs in a Colab Pro session of 12 hours. I used a stepwise learning rate decay. First notebook trained with around 0.0004. Second notebook continue training with 0.0003, third with 0.0002 and fourth with 0.00015.\n6. I kept using a basic batch size of 4096.\n\n**What didn't work or didn't seem to work:**\n1. Predicting - again in a classification setup - zenith and azimuth seperately. Allthough performance was good and close to just using 1 classification layer.\n2. Predicting - again in a classification setup - the x, y, z coordinates.\n3. Various learning rate schedules and larger batch sizes (beyond 4096). This didn't seem to have much effect.\n4. I did some experiments with weight decay and warm-up but it didn't seem to have much effect. This could however be related to the chosen settings.", "2229341": "Thank you very much for publishing your versions of GRU! \nInterestingly, we used a much smaller net: a Bidirectional GRU with 3 layers and a hidden size of 160, followed by one hidden layer of size 512 and a 3-dimensional output (which we normalized to one). We essentially used the angular distance score as a loss. This setup was sometimes unstable in the first couple of thousands of steps (we used a batch size of 2048 due to hardware limitations), so we had to restart the run with a different random seed. But after that, it trained well.", "2229371": "I would like to thank you @rsmits for your contribution in this competition. Part of our solution was inspired by your work.", "2229450": "remekkinas You're welcome :-) With an 8th place your team did great!", "2229502": "inartimiryasov Thanks and your welcome :-) Just went through your solution. I think that the combination of using different Loss function and the fact that you used the returned states likely make the difference for improving the GRU based model to perform into the 0.98X region. Very cool!", "2229800": "rsmits thanks for publishing your training and inference code. I started this competition by going through your code and it was super helpful.\n\nWhat was your best single model on local validation?", "2229836": "Hi @crodoc Thanks and you're welcome :-)\nMy best local validation was 0.99757 on training batches 1, 2 and 3. Number of pulses used was 128.", "2229842": "Nice! I wasn't able to squeeze out that much with practically the same approach.", "2229967": "Thank you! I read what I had written in the solution, and it wasn't very clear. We also used the hidden state of the last unit (2 * hidden_size since there are hidden states for both the forward and backward GRUs). I believe this corresponds to setting return_sequences = False in TensorFlow.\n\nApart from using a different loss function, which outperformed some other regression options that we tried, we also trained the model on the entire training set.", "2231052": "rsmits your notebooks were very much helpful for us. Thank you." }, "source": "meta" }