![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F743648%2F829df4033d8353d91b74f0559259506c%2FModels%20Zoo%20RESIZED.png?generation=1682034404975611&alt=media\")
First of all, we want to thank Kaggle and IceCube Collaboration for providing this spectacular competition!
\nOur solution is based on an MLP stack with shared weights of 6 DynEdge models + a blend of 4 GNNs inside the segments by pulse number.
\nWe took the GraphNeT model with DynEdge architecture as a baseline and modified it in different ways. Main variations and their nicknames:
\nEvery model had longrun training from scratch with von Mises-Fisher Loss on events up to 600 pulses. Then a couple of additional epochs were trained with Euclidean Distance Loss and up to 200 pulses. As @rusg77 noted below, this loss works much better than the original one for second-stage training. The details on the optimizer choice and second-stage training strategy are presented by @churkinnikita in the comment section.
\nFinally, SWA was applied to the best checkpoints of each model for better robustness. As a result, we had v1, v2, v3 models with performance < 1.000 LB and v1, v2, v3 models with performance < 0.990 LB. These 6 models were blended and stacked. Stack showed better performance ~ 0.977 LB in the very final version, @simakov added the details below.
\nAnother part of our solution consists of several GNN models for different neutrino events. The dataset was segmented by ten uniformly bins on pulse number (that represents energy levels of neutrino), and 4 new models were trained on their compositions. Eventually, the models were blended segment by segment, @rusg77 added the details on this part of the solution in the post here.
\nThe final model is a 60 / 40 blend of 6-stacked / 4-segmented GNNs and shows 0.975 LB (0.976 Private). We used a blend of stacked models to improve the overall stability of the ensemble and reduce the variance of individual models. We also hope this approach leads to better resilience of the model to possible noise in the data.
\nFor sure, Transformer-based models have great performance on the task, but a diversified ensemble of properly tuned GNNs can also show a keen score on the considered reconstruction problem.
\n
First of all, we want to thank Kaggle and IceCube Collaboration for providing this spectacular competition!
\nOur solution is based on an MLP stack with shared weights of 6 DynEdge models + a blend of 4 GNNs inside the segments by pulse number.
\nWe took the GraphNeT model with DynEdge architecture as a baseline and modified it in different ways. Main variations and their nicknames:
\nEvery model had longrun training from scratch with von Mises-Fisher Loss on events up to 600 pulses. Then a couple of additional epochs were trained with Euclidean Distance Loss and up to 200 pulses. As @rusg77 noted below, this loss works much better than the original one for second-stage training. The details on the optimizer choice and second-stage training strategy are presented by @churkinnikita in the comment section.
\nFinally, SWA was applied to the best checkpoints of each model for better robustness. As a result, we had v1, v2, v3 models with performance < 1.000 LB and v1, v2, v3 models with performance < 0.990 LB. These 6 models were blended and stacked. Stack showed better performance ~ 0.977 LB in the very final version, @simakov added the details below.
\nAnother part of our solution consists of several GNN models for different neutrino events. The dataset was segmented by ten uniformly bins on pulse number (that represents energy levels of neutrino), and 4 new models were trained on their compositions. Eventually, the models were blended segment by segment, @rusg77 added the details on this part of the solution in the post here.
\nThe final model is a 60 / 40 blend of 6-stacked / 4-segmented GNNs and shows 0.975 LB (0.976 Private). We used a blend of stacked models to improve the overall stability of the ensemble and reduce the variance of individual models. We also hope this approach leads to better resilience of the model to possible noise in the data.
\nFor sure, Transformer-based models have great performance on the task, but a diversified ensemble of properly tuned GNNs can also show a keen score on the considered reconstruction problem.
\n
Stack Model:
\n (nn.Module):\n ():\n (MLPShared, self).__init__()\n\n self.fc1 = nn.Linear(, ) \n self.fc2 = nn.Linear(+, )\n self.fc3 = nn.Linear(+, )\n\n self.act = nn.SiLU()\n\n self.bn1 = nn.BatchNorm1d()\n self.bn2 = nn.BatchNorm1d()\n\n ():\n x, meta = x[:, :-], x[:, -:]\n meta = torch.repeat_interleave(meta, , dim=)\n meta = meta.view(-, , )\n\n x = x.view(-, , )\n x = torch.cat([x, meta], dim=)\n x = torch.permute(x, (, , ))\n x1 = self.fc1(x)\n x1 = torch.permute(x1, (, , ))\n x1 = self.bn1(x1)\n x1 = torch.permute(x1, (, , ))\n x1 = self.act(x1)\n\n \n x1 = torch.cat((x, x1), dim=-) \n \n x2 = self.fc2(x1)\n x2 = torch.permute(x2, (, , ))\n x2 = self.bn2(x2)\n x2 = torch.permute(x2, (, , ))\n x2 = self.act(x2)\n \n x2 = torch.cat((x, x2), dim=-)\n \n\n x3 = self.fc3(x2)\n x3 = x3.view(-, )\n x3\nEuclideanDistanceLoss shows better performance for pretrained models
Congratulations on your gold medals! Solid purely-GNN score.
", "rawMarkdown": "Congratulations on your gold medals! Solid purely-GNN score.", "votes": null }, { "id": "2228618", "postDate": "04/20/2023 16:49:24", "content": "GNN Training Process.
\nInital loss is von Mises-Fisher 3D Loss. For initial network learning we used QHAdam, AdamW and Lion optimizers.
\nI have already touched Lion in my previous competition. Learning process often gets whimsical when I try to train using Lion: usually extremely fast convergence to low loss values (but not enough low) followed by bizzare behaviour or complete divergence. But after some tuning and tinkering (there are some tips for working with Lion) it was possible to get nice checkpoint.
\nWe continued training our models using L2 loss and different training protocol.
\nOur hypothesis on learning our GNNs: loss surface often looked like Rastrigin function, sophisticated surface with dozens of local minima.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F578229%2F7f4f0049a0ec841238a88275ead39f8f%2Frastr.png?generation=1682008443614288&alt=media\")
\nSource: https://www.sfu.ca/~ssurjano/rastr.html
\nLion optimizer led to extremely bad results on that stage; we grid searched through optimizers from pytorch optimizer lib that have the very best performance in the \"Rastrigin scenario\". In our setup AdaBound & RAdam provided splendid performance.
We used short warmup (5%), non-default betas for optimizers and lower learning rate during 2nd stage training.
\nFor my models (\"DynEmb\" variations) only 30m-1h of training time was enough to receive impressive loss drop.
Stack Model:
\n (nn.Module):\n ():\n (MLPShared, self).__init__()\n\n self.fc1 = nn.Linear(, ) \n self.fc2 = nn.Linear(+, )\n self.fc3 = nn.Linear(+, )\n\n self.act = nn.SiLU()\n\n self.bn1 = nn.BatchNorm1d()\n self.bn2 = nn.BatchNorm1d()\n\n ():\n x, meta = x[:, :-], x[:, -:]\n meta = torch.repeat_interleave(meta, , dim=)\n meta = meta.view(-, , )\n\n x = x.view(-, , )\n x = torch.cat([x, meta], dim=)\n x = torch.permute(x, (, , ))\n x1 = self.fc1(x)\n x1 = torch.permute(x1, (, , ))\n x1 = self.bn1(x1)\n x1 = torch.permute(x1, (, , ))\n x1 = self.act(x1)\n\n \n x1 = torch.cat((x, x1), dim=-) \n \n x2 = self.fc2(x1)\n x2 = torch.permute(x2, (, , ))\n x2 = self.bn2(x2)\n x2 = torch.permute(x2, (, , ))\n x2 = self.act(x2)\n \n x2 = torch.cat((x, x2), dim=-)\n \n\n x3 = self.fc3(x2)\n x3 = x3.view(-, )\n x3\nEuclideanDistanceLoss shows better performance for pretrained models
Congratulations on your gold medals! Solid purely-GNN score.
", "votes": null, "replies": [] }, { "id": 2228618, "author_name": "churkinnikita", "author_url": "", "post_date": "04/20/2023 16:49:24", "content": "GNN Training Process.
\nInital loss is von Mises-Fisher 3D Loss. For initial network learning we used QHAdam, AdamW and Lion optimizers.
\nI have already touched Lion in my previous competition. Learning process often gets whimsical when I try to train using Lion: usually extremely fast convergence to low loss values (but not enough low) followed by bizzare behaviour or complete divergence. But after some tuning and tinkering (there are some tips for working with Lion) it was possible to get nice checkpoint.
\nWe continued training our models using L2 loss and different training protocol.
\nOur hypothesis on learning our GNNs: loss surface often looked like Rastrigin function, sophisticated surface with dozens of local minima.
\n
\nSource: https://www.sfu.ca/~ssurjano/rastr.html
\nLion optimizer led to extremely bad results on that stage; we grid searched through optimizers from pytorch optimizer lib that have the very best performance in the \"Rastrigin scenario\". In our setup AdaBound & RAdam provided splendid performance.
We used short warmup (5%), non-default betas for optimizers and lower learning rate during 2nd stage training.
\nFor my models (\"DynEmb\" variations) only 30m-1h of training time was enough to receive impressive loss drop.