First of all, as a former HEPer, I am really excited to see HEP-related challenges on Kaggle. Thank you, IceCube and Kaggle, for hosting this competition!
\nIt's impossible to achieve great results without the sharing of ideas and code within the community. As a newcomer, I have learned a lot. In particular, I would like to express my gratitude to @rasmusrse for sharing the GraphNet example and to @amoshuangyc for the rewriting, which made it even more accessible.
\nThe notebook that I submitted can be found here.
\nThe DynEdge architecture and the loss function seem to be pretty solid and optimal for the problem, although further information may be required for real physics analysis to make it more powerful. This information may include the position of interaction vertices, the type of events, etc. In fact, the GraphNet is capable of doing all that! Meanwhile, I'm really curious to see other architectures/loss functions that have been developed during the competition!
\nTo learn the architecture, I studied the published paper and code on GitHub. When I came across the pre-trained model shared at here, I decided to start from there. With the help of the aforementioned re-write of the GraphNet DynEdge model and some minor modifications, I was able to load the pre-trained model. My fine-tuning scheme is as follows:
\n[Step 1] I used batch 101 to 600 (61 to 80 for validation) and tuned the model with a piecewise-linear learning rate. In the first half epoch, the learning rate was increased from 1e-5 to 1e-3 (warming up), and then until the 10th epoch, it was decreased from 1e-3 to 1e-6. This resulted in an error of around 0.996.
\nIt's worth mentioning that I used a maximum of 200 pulses in each event partially due to computational resource constraints. These pulses were not randomly selected, but priority was given to those that were not labelled as auxiliary. Here is my implementation of pulse down-sampling:
\nif len(df) > self.max_pulses:\n df_pass = df[~df.auxiliary]\n df_fail = df[df.auxiliary]\n if len(df_pass) >= self.max_pulses:\n df = df_pass.sample(self.max_pulses)\n else:\n df_fail = df_fail.sample(self.max_pulses - len(df_pass))\n df = pd.concat([df_fail, df_pass])\nIn a training with a smaller amount of data, I found that this technique helps the GraphNet to converge a bit faster.
\n[Step 2] I used batch 101, 103, …, 599 (a total of 250 batches due to starting quite late) and added additional noise to the times and charges to help with the training. In PyTorch, the implementation looks something like this:
\nfrom torch.distributions import Gamma, Normal\n...\ndist_normal = Normal(0, 1.2)\ndist_gamma = Gamma(c * 10, 10)\nt += dist_normal.sample()\nc = dist_gamma.sample()\nFor the additional noise, I chose a Gaussian distribution for the time parameter, considering a time resolution of 1.2 ns as stated in some literature. I used a Gamma distribution for the charge parameter, taking into account the Poisson nature of the photoelectrics. The choice of parameters for the Gamma distribution was somewhat random, as I was trying to control the width of the distribution. I acknowledge that this could potentially be incorrect, but it resulted in a further improvement of 0.002.
\nI trained this step on top of the model trained in step (1) with a learning rate of 1e-5, which linearly decreased to 1e-8 after 5 epochs.
At the same time, I wanted to explore a simple solution or obtain additional information while the training was running. I noticed that the pulse points could be used to fit a plane in 3-D by solving a least squares problem. I found that the average angle between the plane and the true direction of the neutrino was relatively small, around 16 degrees. This led me to consider projecting the DynEdge prediction onto the plane as a possible way to improve the results.
\nAbout half of the projected predictions were better than the DynEdge-only predictions. If we could determine which prediction was better, we could achieve a final error of approximately 0.91. Even if we could only determine this with an accuracy of 80%, we could still achieve an error below 0.95. However, since we cannot determine this accurately, I trained a boosted decision tree (BDT) to distinguish for each event whether the projected or the original prediction was better. The input variables for the BDT are defined in block (11) of my notebook, with zenith, kappa, dt, ez, and xe providing the greatest discrimination power. The final accuracy of the BDT to determine which cases the plane-fit projection was better was only about 61%, which is not great. By tuning the cut threshold of the BDT output score, the final error was improved by approximately 0.001.
\nOverall, I think the solution is relatively straightforward and cost-effective, comprising only a GNN, a simple least square fit, and a BDT. Eventually, I have some additional thoughts to share:
\nThe second stage training with noise data can be further optimised. The distribution can be tuned and the batches and epoches for training may not be enough.
One could categorize events based on the GraphNet output kappa. For instance, one might consider dividing events into those with low (kappa<0.5) and high (kappa>=0.5) values, but further optimization would be needed to determine the impact of this categorization. From what I understand, kappa represents the difficulty of predicting a given event. The distribution of kappa is intriguing, with three peaks: one below 0.5, one near 1, and one near 2. Events with kappa values below 0.5 tend to have errors around 1.48 (close to pi/2), which is almost the worst one can get, while events with higher kappa values have pretty low errors. I noticed that events with kappa values below 0.5 tend to have a center of charge inside the dust layer and above, where there is no DeepCore, quantum efficiency is worse and ice absorption and scattering are larger. Therefore, I surmise that the three peaks might correspond to:
\na. poor detection or cluster/double-bang events
\nb. good detection or poor detection + muon track
\nc. excellent detection or good detection + muon track
The better the DynEdge prediction is, the less the plane fit contributes. It could be interesting to explore how much the plane fit helps with low kappa values. However, the plane fit + BDT contribution have only a marginal impact on the final error.
Cheers,
\nBlue Zenith
First of all, as a former HEPer, I am really excited to see HEP-related challenges on Kaggle. Thank you, IceCube and Kaggle, for hosting this competition!
\nIt's impossible to achieve great results without the sharing of ideas and code within the community. As a newcomer, I have learned a lot. In particular, I would like to express my gratitude to @rasmusrse for sharing the GraphNet example and to @amoshuangyc for the rewriting, which made it even more accessible.
\nThe notebook that I submitted can be found here.
\nThe DynEdge architecture and the loss function seem to be pretty solid and optimal for the problem, although further information may be required for real physics analysis to make it more powerful. This information may include the position of interaction vertices, the type of events, etc. In fact, the GraphNet is capable of doing all that! Meanwhile, I'm really curious to see other architectures/loss functions that have been developed during the competition!
\nTo learn the architecture, I studied the published paper and code on GitHub. When I came across the pre-trained model shared at here, I decided to start from there. With the help of the aforementioned re-write of the GraphNet DynEdge model and some minor modifications, I was able to load the pre-trained model. My fine-tuning scheme is as follows:
\n[Step 1] I used batch 101 to 600 (61 to 80 for validation) and tuned the model with a piecewise-linear learning rate. In the first half epoch, the learning rate was increased from 1e-5 to 1e-3 (warming up), and then until the 10th epoch, it was decreased from 1e-3 to 1e-6. This resulted in an error of around 0.996.
\nIt's worth mentioning that I used a maximum of 200 pulses in each event partially due to computational resource constraints. These pulses were not randomly selected, but priority was given to those that were not labelled as auxiliary. Here is my implementation of pulse down-sampling:
\nif len(df) > self.max_pulses:\n df_pass = df[~df.auxiliary]\n df_fail = df[df.auxiliary]\n if len(df_pass) >= self.max_pulses:\n df = df_pass.sample(self.max_pulses)\n else:\n df_fail = df_fail.sample(self.max_pulses - len(df_pass))\n df = pd.concat([df_fail, df_pass])\nIn a training with a smaller amount of data, I found that this technique helps the GraphNet to converge a bit faster.
\n[Step 2] I used batch 101, 103, …, 599 (a total of 250 batches due to starting quite late) and added additional noise to the times and charges to help with the training. In PyTorch, the implementation looks something like this:
\nfrom torch.distributions import Gamma, Normal\n...\ndist_normal = Normal(0, 1.2)\ndist_gamma = Gamma(c * 10, 10)\nt += dist_normal.sample()\nc = dist_gamma.sample()\nFor the additional noise, I chose a Gaussian distribution for the time parameter, considering a time resolution of 1.2 ns as stated in some literature. I used a Gamma distribution for the charge parameter, taking into account the Poisson nature of the photoelectrics. The choice of parameters for the Gamma distribution was somewhat random, as I was trying to control the width of the distribution. I acknowledge that this could potentially be incorrect, but it resulted in a further improvement of 0.002.
\nI trained this step on top of the model trained in step (1) with a learning rate of 1e-5, which linearly decreased to 1e-8 after 5 epochs.
At the same time, I wanted to explore a simple solution or obtain additional information while the training was running. I noticed that the pulse points could be used to fit a plane in 3-D by solving a least squares problem. I found that the average angle between the plane and the true direction of the neutrino was relatively small, around 16 degrees. This led me to consider projecting the DynEdge prediction onto the plane as a possible way to improve the results.
\nAbout half of the projected predictions were better than the DynEdge-only predictions. If we could determine which prediction was better, we could achieve a final error of approximately 0.91. Even if we could only determine this with an accuracy of 80%, we could still achieve an error below 0.95. However, since we cannot determine this accurately, I trained a boosted decision tree (BDT) to distinguish for each event whether the projected or the original prediction was better. The input variables for the BDT are defined in block (11) of my notebook, with zenith, kappa, dt, ez, and xe providing the greatest discrimination power. The final accuracy of the BDT to determine which cases the plane-fit projection was better was only about 61%, which is not great. By tuning the cut threshold of the BDT output score, the final error was improved by approximately 0.001.
\nOverall, I think the solution is relatively straightforward and cost-effective, comprising only a GNN, a simple least square fit, and a BDT. Eventually, I have some additional thoughts to share:
\nThe second stage training with noise data can be further optimised. The distribution can be tuned and the batches and epoches for training may not be enough.
One could categorize events based on the GraphNet output kappa. For instance, one might consider dividing events into those with low (kappa<0.5) and high (kappa>=0.5) values, but further optimization would be needed to determine the impact of this categorization. From what I understand, kappa represents the difficulty of predicting a given event. The distribution of kappa is intriguing, with three peaks: one below 0.5, one near 1, and one near 2. Events with kappa values below 0.5 tend to have errors around 1.48 (close to pi/2), which is almost the worst one can get, while events with higher kappa values have pretty low errors. I noticed that events with kappa values below 0.5 tend to have a center of charge inside the dust layer and above, where there is no DeepCore, quantum efficiency is worse and ice absorption and scattering are larger. Therefore, I surmise that the three peaks might correspond to:
\na. poor detection or cluster/double-bang events
\nb. good detection or poor detection + muon track
\nc. excellent detection or good detection + muon track
The better the DynEdge prediction is, the less the plane fit contributes. It could be interesting to explore how much the plane fit helps with low kappa values. However, the plane fit + BDT contribution have only a marginal impact on the final error.
Cheers,
\nBlue Zenith