\nThe most dramatic change I achieved was for the last bucket with 145+ pulses. Train on events with 145-2000 pulses. Rented machine with 4x3090 ~2 days. 0.983/0.982
\n
![\"Screen-Shot-2023-04-20-at-17-21-45.png\"](\"https://i.postimg.cc/K84WLyPD/Screen-Shot-2023-04-20-at-17-21-45.png\")
First of all, I'd like to thank organizers and congratulations to all the winners!
\nSpecial thanks to:
Our entire solution is available here: https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402849
\nHere is my part. I'll skip all things I tried and focus on the most significant improvements.
\nFor me it wasn't easy. But when tried to play with learning rate I achieved 1.007/1.005
\nadditional_features = ['is_abovedust', 'is_deepcore', 'is_dustlayer', 'is_underdust', 'scattering', 'absorption', 'relative_qe']
\nTrain on first 200 batches, 3x3090 rented host, 1d 6h. 0.999/0.998
I noticed that model performance is poor for events with a low amount of pulses, especially less than 50. I tuned a model for such events with increased nb_neighbours. 0.992/0.990
\nTwo parts inference:
\nSQlite approach required a lot of disk space, long processing. I tried to switch to Posgresql but the result was almost the same. CPU, disk space, memory for each connection are bottlenecks. Then @iafoss published his amazing approach with per batch iteration and shuffling batches every epoch. With little modification I make it work with multiple GPUs.
\nWhen I was about to despair, I suddenly discovered that EuclidianDistanceLoss shows much better results.
\nSo now I have a good train pipeline. I continued to train models but this time I split data into ~equal 10 buckets by number of pulses.
\nThe most dramatic change I achieved was for the last bucket with 145+ pulses. Train on events with 145-2000 pulses. Rented machine with 4x3090 ~2 days. 0.983/0.982
\n
Weights 60/40 (my part) for x,y,z coordinates + norm to 1
\nFirst day: 0.6 * 0.984 + 0.4 * 0.987 -> 0.981/0.979
\nLast day: 0.6 * 0.978 + 0.4 * 0.982 -> 0.976/0.975
Good luck and happy kaggling!
", "messages": [ { "id": "2228523", "postDate": "04/20/2023 15:47:07", "content": "First of all, I'd like to thank organizers and congratulations to all the winners!
\nSpecial thanks to:
Our entire solution is available here: https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402849
\nHere is my part. I'll skip all things I tried and focus on the most significant improvements.
\nFor me it wasn't easy. But when tried to play with learning rate I achieved 1.007/1.005
\nadditional_features = ['is_abovedust', 'is_deepcore', 'is_dustlayer', 'is_underdust', 'scattering', 'absorption', 'relative_qe']
\nTrain on first 200 batches, 3x3090 rented host, 1d 6h. 0.999/0.998
I noticed that model performance is poor for events with a low amount of pulses, especially less than 50. I tuned a model for such events with increased nb_neighbours. 0.992/0.990
\nTwo parts inference:
\nSQlite approach required a lot of disk space, long processing. I tried to switch to Posgresql but the result was almost the same. CPU, disk space, memory for each connection are bottlenecks. Then @iafoss published his amazing approach with per batch iteration and shuffling batches every epoch. With little modification I make it work with multiple GPUs.
\nWhen I was about to despair, I suddenly discovered that EuclidianDistanceLoss shows much better results.
\nSo now I have a good train pipeline. I continued to train models but this time I split data into ~equal 10 buckets by number of pulses.
\nThe most dramatic change I achieved was for the last bucket with 145+ pulses. Train on events with 145-2000 pulses. Rented machine with 4x3090 ~2 days. 0.983/0.982
\n
Weights 60/40 (my part) for x,y,z coordinates + norm to 1
\nFirst day: 0.6 * 0.984 + 0.4 * 0.987 -> 0.981/0.979
\nLast day: 0.6 * 0.978 + 0.4 * 0.982 -> 0.976/0.975
Good luck and happy kaggling!
", "rawMarkdown": "First of all, I'd like to thank organizers and congratulations to all the winners!\nSpecial thanks to:\n- my teammates: Nikita Churkin @churkinnikita, Dmitry Simakov @simakov, Dmitriy Ukrainskiy @ukrainskiydv\n- @iafoss for per batch iteration approach (https://www.kaggle.com/code/iafoss/chunk-based-data-loading-with-caching)\n- @antonsevostianov for sensors feature engineering (https://www.kaggle.com/code/antonsevostianov/icecube-sensor-efficiency-feature-engineering/notebook)\n- @rasmusrse for clean code and tidy code (https://github.com/graphnet-team/graphnet)\n\nOur entire solution is available here: https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402849 \n\nHere is my part. I'll skip all things I tried and focus on the most significant improvements.\n\n## Reproduce baseline score (1.017) from scratch \nFor me it wasn't easy. But when tried to play with learning rate I achieved **1.007/1.005**\n\n## Add more features\n`additional_features = ['is_abovedust', 'is_deepcore', 'is_dustlayer', 'is_underdust', 'scattering', 'absorption', 'relative_qe']`\nTrain on first 200 batches, 3x3090 rented host, 1d 6h. **0.999/0.998**\n\n## Two models\nI noticed that model performance is poor for events with a low amount of pulses, especially less than 50. I tuned a model for such events with increased nb_neighbours. **0.992/0.990**\n\nTwo parts inference:\n- model_1 for events with low amount of pulses\n- model_2 for the rest\n\n## Train pipeline optimization\nSQlite approach required a lot of disk space, long processing. I tried to switch to Posgresql but the result was almost the same. CPU, disk space, memory for each connection are bottlenecks. Then @iafoss published his amazing approach with per batch iteration and shuffling batches every epoch. With little modification I make it work with multiple GPUs.\n\n## EuclidianDistanceLoss\nWhen I was about to despair, I suddenly discovered that EuclidianDistanceLoss shows much better results.\n\n## Split events by n_pulses in 10 buckets\nSo now I have a good train pipeline. I continued to train models but this time I split data into ~equal 10 buckets by number of pulses. \nThe most dramatic change I achieved was for the last bucket with 145+ pulses. Train on events with 145-2000 pulses. Rented machine with 4x3090 ~2 days. **0.983/0.982**\n[](https://postimg.cc/JsVKVvds)\n\n## Team merge\nWeights 60/40 (my part) for x,y,z coordinates + norm to 1\nFirst day: 0.6 * 0.984 + 0.4 * 0.987 -> **0.981/0.979**\nLast day: 0.6 * 0.978 + 0.4 * 0.982 -> **0.976/0.975**\n\n## Lessons\n- spend a little more time on code quality, experiments logging\n- GPUs is not all you need, but it matters\n- Try well known architectures instead of trying paperswithcode graph convolutions\n\nGood luck and happy kaggling!", "votes": null }, { "id": "2230910", "postDate": "04/22/2023 21:57:09", "content": "Congratulations, Ruslan! Glad that my messy code (found a few mislabeled sensors later on) still helped you! Probably could've squeezed a few more 0.00001s if not for me being lazy bones, haha.
", "rawMarkdown": "Congratulations, Ruslan! Glad that my messy code (found a few mislabeled sensors later on) still helped you! Probably could've squeezed a few more 0.00001s if not for me being lazy bones, haha.", "votes": null } ], "comments": [ { "id": 2230910, "author_name": "antonsevostianov", "author_url": "", "post_date": "04/22/2023 21:57:09", "content": "Congratulations, Ruslan! Glad that my messy code (found a few mislabeled sensors later on) still helped you! Probably could've squeezed a few more 0.00001s if not for me being lazy bones, haha.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2228523": "First of all, I'd like to thank organizers and congratulations to all the winners!\nSpecial thanks to:\n- my teammates: Nikita Churkin @churkinnikita, Dmitry Simakov @simakov, Dmitriy Ukrainskiy @ukrainskiydv\n- @iafoss for per batch iteration approach (https://www.kaggle.com/code/iafoss/chunk-based-data-loading-with-caching)\n- @antonsevostianov for sensors feature engineering (https://www.kaggle.com/code/antonsevostianov/icecube-sensor-efficiency-feature-engineering/notebook)\n- @rasmusrse for clean code and tidy code (https://github.com/graphnet-team/graphnet)\n\nOur entire solution is available here: https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402849 \n\nHere is my part. I'll skip all things I tried and focus on the most significant improvements.\n\n## Reproduce baseline score (1.017) from scratch \nFor me it wasn't easy. But when tried to play with learning rate I achieved **1.007/1.005**\n\n## Add more features\n`additional_features = ['is_abovedust', 'is_deepcore', 'is_dustlayer', 'is_underdust', 'scattering', 'absorption', 'relative_qe']`\nTrain on first 200 batches, 3x3090 rented host, 1d 6h. **0.999/0.998**\n\n## Two models\nI noticed that model performance is poor for events with a low amount of pulses, especially less than 50. I tuned a model for such events with increased nb_neighbours. **0.992/0.990**\n\nTwo parts inference:\n- model_1 for events with low amount of pulses\n- model_2 for the rest\n\n## Train pipeline optimization\nSQlite approach required a lot of disk space, long processing. I tried to switch to Posgresql but the result was almost the same. CPU, disk space, memory for each connection are bottlenecks. Then @iafoss published his amazing approach with per batch iteration and shuffling batches every epoch. With little modification I make it work with multiple GPUs.\n\n## EuclidianDistanceLoss\nWhen I was about to despair, I suddenly discovered that EuclidianDistanceLoss shows much better results.\n\n## Split events by n_pulses in 10 buckets\nSo now I have a good train pipeline. I continued to train models but this time I split data into ~equal 10 buckets by number of pulses. \nThe most dramatic change I achieved was for the last bucket with 145+ pulses. Train on events with 145-2000 pulses. Rented machine with 4x3090 ~2 days. **0.983/0.982**\n[](https://postimg.cc/JsVKVvds)\n\n## Team merge\nWeights 60/40 (my part) for x,y,z coordinates + norm to 1\nFirst day: 0.6 * 0.984 + 0.4 * 0.987 -> **0.981/0.979**\nLast day: 0.6 * 0.978 + 0.4 * 0.982 -> **0.976/0.975**\n\n## Lessons\n- spend a little more time on code quality, experiments logging\n- GPUs is not all you need, but it matters\n- Try well known architectures instead of trying paperswithcode graph convolutions\n\nGood luck and happy kaggling!", "2230910": "Congratulations, Ruslan! Glad that my messy code (found a few mislabeled sensors later on) still helped you! Probably could've squeezed a few more 0.00001s if not for me being lazy bones, haha." }, "source": "meta" }