\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10068397%2F1930323decb4f4beea57e1424faffc25%2Fgraphnet.png?generation=1675848475278340&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10068397%2F33ce55857bc0e44184cc1769837a135d%2Fgraphnet_zoom.png?generation=1675848524327156&alt=media\")
I may have missed this in the competition description. If someone were to score the current state-of-the-art (but very slow) reconstructions with the same metrics used to score this competition, what would that score be? Are we allowed to know this? It'd be nice to benchmark the top scores on the leaderboard against the current state-of-the-art.
", "messages": [ { "id": 2132674, "postDate": "2023-02-07T01:34:22.747Z", "content": "I may have missed this in the competition description. If someone were to score the current state-of-the-art (but very slow) reconstructions with the same metrics used to score this competition, what would that score be? Are we allowed to know this? It'd be nice to benchmark the top scores on the leaderboard against the current state-of-the-art.
", "rawMarkdown": "I may have missed this in the competition description. If someone were to score the current state-of-the-art (but very slow) reconstructions with the same metrics used to score this competition, what would that score be? Are we allowed to know this? It'd be nice to benchmark the top scores on the leaderboard against the current state-of-the-art.", "votes": 15 }, { "id": 2134852, "postDate": "2023-02-08T09:37:19.473Z", "content": "I agree with what others have already commented, it is difficult to put a number on this because our best reconstructions target certain event topologies, like tracks or cascades, separately.
\nThat being said, we know that our most basic reconstruction \"LineFit\" gets a score of around 1.2ish on the kaggle dataset, and this is quite well in line with what people here got using similar geometric line fitting approaches.
\nThen for the ML approaches: the Graphnet example that Rasmus has provided is currently one of our most promising ML approaches that we have tried within the IceCube collaboration, **BUT ** the example provided was optimized for low-energy events (< 100 GeV) and was applied almost out-of-the-box to the high-energy (> 100 GeV) sample here, without much optimization. So there certainly is room for improvement.
\nThe two plots below show the Graphnet performance on the kaggle sample (the second plot is a zoomed-in version of the first plot):
\n
\n
What can be observed is that there are some events that are very well reconstructable (sharp peak), those are tracks. This distribution in the plot is peaking at around 2 degree opening angle. Our best track reconstructions are able to get that peak to below 1 degree.
\nThen there is some in-between distribution of cascades. Our algorithms get cascades to around 10ish degree (ballpark).
\nAnd then there is some unreconstructable background that will remain as a relatively wide distribution peaking at 90 degrees (pi/2 radian).
", "rawMarkdown": "I agree with what others have already commented, it is difficult to put a number on this because our best reconstructions target certain event topologies, like tracks or cascades, separately.\n\nThat being said, we know that our most basic reconstruction \"LineFit\" gets a score of around 1.2ish on the kaggle dataset, and this is quite well in line with what people here got using similar geometric line fitting approaches.\n\nThen for the ML approaches: the Graphnet example that Rasmus has provided is currently one of our most promising ML approaches that we have tried within the IceCube collaboration, **BUT ** the example provided was optimized for low-energy events (< 100 GeV) and was applied almost out-of-the-box to the high-energy (> 100 GeV) sample here, without much optimization. So there certainly is room for improvement.\n\nThe two plots below show the Graphnet performance on the kaggle sample (the second plot is a zoomed-in version of the first plot):\n \n\n\nWhat can be observed is that there are some events that are very well reconstructable (sharp peak), those are tracks. This distribution in the plot is peaking at around 2 degree opening angle. Our best track reconstructions are able to get that peak to below 1 degree.\n\nThen there is some in-between distribution of cascades. Our algorithms get cascades to around 10ish degree (ballpark).\n\nAnd then there is some unreconstructable background that will remain as a relatively wide distribution peaking at 90 degrees (pi/2 radian).", "votes": 14, "replies": [ { "id": 2147719, "postDate": "2023-02-16T20:21:40.203Z", "content": "New here and still learning, but little confused. You say \"provided was optimized for low-energy events (< 100 GeV) and was applied almost out-of-the-box to the high-energy (> 100 GeV) sample here\".
\nI don't see the energy for a particular eventID in the data or perhaps you are adding up the total charge for each hit to come up with this number? Thank you.
This information is not included in the data provided. But the data for this competition is high energy, >100 GeV. In simulation we know the energy, and in data we have ways to estimate it. The total charge per event as you say is a good and simple proxy.
", "rawMarkdown": "This information is not included in the data provided. But the data for this competition is high energy, >100 GeV. In simulation we know the energy, and in data we have ways to estimate it. The total charge per event as you say is a good and simple proxy.", "votes": 1 } ] }, { "id": 2188779, "postDate": "2023-03-20T01:25:35.450Z", "content": "what 1.2ish stands for? thanks
", "rawMarkdown": "what 1.2ish stands for? thanks", "replies": [ { "id": 2188945, "postDate": "2023-03-20T05:33:48.257Z", "content": "It means that the mean angular error is around 1.2, which would be around 70° deg. The best in the competition so far is 0.979, which translates to 56° deg.
", "rawMarkdown": "It means that the mean angular error is around 1.2, which would be around 70° deg. The best in the competition so far is 0.979, which translates to 56° deg.", "votes": 2 } ] } ] }, { "id": 2137154, "postDate": "2023-02-09T18:59:46.137Z", "content": "Here is one other example of a neural net performance on neutrino direction reconstruction, from a paper I've browsed. It is specifically for the cascade event topology, that is for the harder subset of events. The energies range here seems to correspond well to the competition dataset (not sure if it distributed in the same way, though).
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1581556%2Fb4436f818c3d95c8943e15faae778f5d%2Ficecube.png?generation=1675969152586295&alt=media\")
\nIf I'm reading the figure correctly, that cnn performance is roughly around the 10-15 degrees, that is 0.2-0.25 radian. Hopefully, our dataset also contains a significant part of track events, which are more easily reconstructed and - with a bit of luck and hard work - we can achieve even better score.
Could you cite the paper for further reading?
", "rawMarkdown": "Could you cite the paper for further reading?", "replies": [ { "id": 2137884, "postDate": "2023-02-10T11:53:34.487Z", "content": "The figure is taken from arXiv: 2101.11589 and is for a CNN. This is more recent than the GNN paper, but I don’t know if it performs better or worse (I don’t think they compare the two architectures).
\nAlso, keep in mind that this CNN uses more fine-grained features than what was made available for the present competition (see the caption of fig. 4 from the CNN paper: \"The pulses are therefore reduced to nine input parameters (𝑐total, 𝑐500ns, 𝑐100ns, 𝑡first, 𝑡last, 𝑡20%, 𝑡50%, 𝑡mean, 𝑡std) which aim to summarize the pulse distribution.\").
", "rawMarkdown": "The figure is taken from [arXiv: 2101.11589](http://arxiv.org/abs/2101.11589) and is for a CNN. This is more recent than the [GNN paper](http://arxiv.org/abs/1809.06166), but I don’t know if it performs better or worse (I don’t think they compare the two architectures).\n\nAlso, keep in mind that this CNN uses more fine-grained features than what was made available for the present competition (see the caption of fig. 4 from the CNN paper: \"The pulses are therefore reduced to nine input parameters (𝑐total, 𝑐500ns, 𝑐100ns, 𝑡first, 𝑡last, 𝑡20%, 𝑡50%, 𝑡mean, 𝑡std) which aim to summarize the pulse distribution.\").", "votes": 2 }, { "id": 2137937, "postDate": "2023-02-10T12:51:01.643Z", "content": "I read this as they integrate all the pulses from one DOM into these parameters, rather then they have the detailed internal waveform for every pulse. I may be mistaken, of course, and did not find a definite description in the paper.
", "rawMarkdown": "I read this as they integrate all the pulses from one DOM into these parameters, rather then they have the detailed internal waveform for every pulse. I may be mistaken, of course, and did not find a definite description in the paper." } ] } ] }, { "id": 2134303, "postDate": "2023-02-07T22:03:01.517Z", "content": "I am guessing your question stems from the context section on the competition main page, \"Researchers have developed multiple approaches over the past ten years to reconstruct neutrino events. However, problems arise as existing solutions are far from perfect. They're either fast but inaccurate or more accurate at the price of huge computational costs.\"
\nFrom the limited research I have done so far it seems like we can use some really sophisticated mathematics and statistical methods to run simulations of a neutrino event. Those methods can narrow down the Azimuth and Zenith to high levels of precision. However, to do this requires a boat load of calculations, think hours on super computers.
\nAlternatively the machine learned methods can be significantly faster, orders of magnitude faster. My guess is that these machine learning methods are prone to errors, but on average they are correct enough. Think of the old joke about a statistician with his head in the ice box and feet in the oven… on average he is quite comfortable.
\nIf you're looking for the methods that are slow but super accurate, I'd suggest skimming thru some of the references in the papers listed on the main page. Hopefully that helps a little.
", "rawMarkdown": "I am guessing your question stems from the context section on the competition main page, \"Researchers have developed multiple approaches over the past ten years to reconstruct neutrino events. However, problems arise as existing solutions are far from perfect. They're either fast but inaccurate or more accurate at the price of huge computational costs.\" \n\nFrom the limited research I have done so far it seems like we can use some really sophisticated mathematics and statistical methods to run simulations of a neutrino event. Those methods can narrow down the Azimuth and Zenith to high levels of precision. However, to do this requires a boat load of calculations, think hours on super computers. \n\nAlternatively the machine learned methods can be significantly faster, orders of magnitude faster. My guess is that these machine learning methods are prone to errors, but on average they are correct enough. Think of the old joke about a statistician with his head in the ice box and feet in the oven... on average he is quite comfortable.\n\nIf you're looking for the methods that are slow but super accurate, I'd suggest skimming thru some of the references in the papers listed on the main page. Hopefully that helps a little. ", "votes": 4 }, { "id": 2134047, "postDate": "2023-02-07T18:13:55.657Z", "content": "If you read the various papers on CNN or GNN reconstruction methods, they are compared to the \"standard\" methods. Sometimes they are better, sometimes worse. So I am not sure any one knows what the best state-of-the-art is.
\nFor example, from Mirco Hennefeld, in Deep Learning In Physics exemplified by the Reconstruction of Muon-Neutrino Events in IceCube, a CNN method:
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F651278%2Fb08b224aba0db3eade07a8994b9171ce%2FScreenshot%20from%202023-02-07%2013-05-23.png?generation=1675793153156609&alt=media\")
Or from R. Abbasi 2022 JINST 17 P1 1003 Graph Neural Networks for low-energy event classification and reconstruction in IceCube. Retro method is a standard program:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F651278%2F175277888c8cd34fd128ba0f05f205d6%2FScreenshot%20from%202023-02-07%2013-07-42.png?generation=1675793276084002&alt=media\")
To make it more complicated, a lot of this depends on:
\nThe \"standard\" methods, which are based on a maximum liklihood approach, could take 40 sec per event. Since there are millions of events in our dataset, I am pretty sure no one has run the standard reconstruction on all the events
", "rawMarkdown": "If you read the various papers on CNN or GNN reconstruction methods, they are compared to the \"standard\" methods. Sometimes they are better, sometimes worse. So I am not sure any one knows what the best state-of-the-art is.\nFor example, from Mirco Hennefeld, in *Deep Learning In Physics exemplified by the Reconstruction of Muon-Neutrino Events in IceCube*, a CNN method:\n\n\n\nOr from R. Abbasi 2022 JINST 17 P1 1003 *Graph Neural Networks for low-energy event classification and reconstruction in IceCube*. Retro method is a standard program: \n\n\nTo make it more complicated, a lot of this depends on:\n1. Is this a Muon Track event or a Cascade event? A Muon track event is what we think of as a straight line of signals passing through the detector. This a the best case for determining an accurate direction. Papers say accuracies on the order of a few degrees are possible. Cascade events are more of a spherical light distrubution, and are much more difficult to estimate a direction from.\n2. What is the energy of the neutrino? Papers variously talk about low-energy or high-energy events, which have different issues.\n\nThe \"standard\" methods, which are based on a maximum liklihood approach, could take 40 sec per event. Since there are millions of events in our dataset, I am pretty sure no one has run the standard reconstruction on all the events ", "votes": 1, "replies": [ { "id": 2134168, "postDate": "2023-02-07T19:58:33.347Z", "content": "Well, they don't have to run it on millions of examples to get a rough estimate of the accuracy. A few thousands would probably suffice. Anyway, I'm sure they already have a pretty good estimation of the SOTA methods accuracy.
", "rawMarkdown": "Well, they don't have to run it on millions of examples to get a rough estimate of the accuracy. A few thousands would probably suffice. Anyway, I'm sure they already have a pretty good estimation of the SOTA methods accuracy. " } ] }, { "id": 2132822, "postDate": "2023-02-07T04:01:42.123Z", "content": "I got the impression that the GraphNeT 1.014 from the organizers is the state of the art? I could very well be wrong.
", "rawMarkdown": "I got the impression that the GraphNeT 1.014 from the organizers is the state of the art? I could very well be wrong. ", "votes": 2, "replies": [ { "id": 2133006, "postDate": "2023-02-07T07:42:30.820Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2133597, "postDate": "2023-02-07T14:07:49.780Z", "content": "So I'm confused - if that's the best reconstruction, then how are the GT values generated and how accurate are they? Are they simulated as the reverse of the competition workflow (assume a trajectory, compute the detector's response), or forwards (assume a response, work out the trajectory)?
", "rawMarkdown": "So I'm confused - if that's the best reconstruction, then how are the GT values generated and how accurate are they? Are they simulated as the reverse of the competition workflow (assume a trajectory, compute the detector's response), or forwards (assume a response, work out the trajectory)?", "replies": [ { "id": 2133645, "postDate": "2023-02-07T14:37:55.200Z", "content": "My understanding is yes. The data we have is the result of putting specified neutrino events into the production simulation software they used for designing the experiment. I would hope they have results of running their really slow but very accurate algorithms over simulations. I'd love to know ballpark what that accuracy is.
", "rawMarkdown": "My understanding is yes. The data we have is the result of putting specified neutrino events into the production simulation software they used for designing the experiment. I would hope they have results of running their really slow but very accurate algorithms over simulations. I'd love to know ballpark what that accuracy is.", "votes": 1 } ] }, { "id": 2133668, "postDate": "2023-02-07T14:47:18.877Z", "content": "GraphNet if a fast solution. I think there are slower, more precise solutions. I would also like to know what is the 'slow' benchmark.
", "rawMarkdown": "GraphNet if a fast solution. I think there are slower, more precise solutions. I would also like to know what is the 'slow' benchmark. ", "votes": 1, "replies": [ { "id": 2189396, "postDate": "2023-03-20T13:11:42.973Z", "content": "See for example this article about one of our latest recos: https://icecube.wisc.edu/news/research/2021/04/new-algorithm-improves-icecubes-pointing-accuracy/
\nIt shows that for select events (high quality tracks only, so does not apply to all events provided here), the median angular error is as low as 0.2 - 0.3 degree! Other events have larger errors. This reconstruction, however, is comparably slow and costly.
\nFrom what I have seen so far in this competition, tracks are reconstructed to a precision of at best around 1 degree….so I think there is still a little bit room for improvement.
", "rawMarkdown": "See for example this article about one of our latest recos: https://icecube.wisc.edu/news/research/2021/04/new-algorithm-improves-icecubes-pointing-accuracy/\n\nIt shows that for select events (high quality tracks only, so does not apply to all events provided here), the median angular error is as low as 0.2 - 0.3 degree! Other events have larger errors. This reconstruction, however, is comparably slow and costly.\n\nFrom what I have seen so far in this competition, tracks are reconstructed to a precision of at best around 1 degree....so I think there is still a little bit room for improvement.", "votes": 3 }, { "id": 2189502, "postDate": "2023-03-20T14:55:43.987Z", "content": "How can we accurately ascertain whether a reconstruction is fast or slow, and understand the trade-offs involved? In the context of the Kaggle inference environment, which has limited resources and an allocated time of 9 hours, would you regard this as a fast reconstruction process?
", "rawMarkdown": "How can we accurately ascertain whether a reconstruction is fast or slow, and understand the trade-offs involved? In the context of the Kaggle inference environment, which has limited resources and an allocated time of 9 hours, would you regard this as a fast reconstruction process?" }, { "id": 2189523, "postDate": "2023-03-20T15:13:00.197Z", "content": "Yes, we chose the size of the test set (roughly 1e6 events) in such a way that any valid submission within the 9h compute limit is fast enough for IceCube.
\nFor some context: the IceCube global trigger rate is around 2.7 kHz, and ideally we would like to be able to process all these events with limited resources (not more than a few servers/GPUs).
\nSlow reconstructions, on the other hand, are usually only applied to a small subset of select events, and take >> 1 second per event to complete, meaning up to minutes or even hours for a single event.
", "rawMarkdown": "Yes, we chose the size of the test set (roughly 1e6 events) in such a way that any valid submission within the 9h compute limit is fast enough for IceCube.\n\nFor some context: the IceCube global trigger rate is around 2.7 kHz, and ideally we would like to be able to process all these events with limited resources (not more than a few servers/GPUs).\n\nSlow reconstructions, on the other hand, are usually only applied to a small subset of select events, and take >> 1 second per event to complete, meaning up to minutes or even hours for a single event.", "votes": 1 } ] } ] }, { "id": 2133871, "postDate": "2023-02-07T16:44:58.333Z", "content": "@pellerphys This is mainly just a curiosity question, but I think you may be the most likely person to know the answer to this.
", "rawMarkdown": "@pellerphys This is mainly just a curiosity question, but I think you may be the most likely person to know the answer to this." } ], "comments": [ { "id": 2134852, "author_name": "Philipp Eller", "author_url": "", "post_date": "2023-02-08T09:37:19.473000", "content": "I agree with what others have already commented, it is difficult to put a number on this because our best reconstructions target certain event topologies, like tracks or cascades, separately.
\nThat being said, we know that our most basic reconstruction \"LineFit\" gets a score of around 1.2ish on the kaggle dataset, and this is quite well in line with what people here got using similar geometric line fitting approaches.
\nThen for the ML approaches: the Graphnet example that Rasmus has provided is currently one of our most promising ML approaches that we have tried within the IceCube collaboration, **BUT ** the example provided was optimized for low-energy events (< 100 GeV) and was applied almost out-of-the-box to the high-energy (> 100 GeV) sample here, without much optimization. So there certainly is room for improvement.
\nThe two plots below show the Graphnet performance on the kaggle sample (the second plot is a zoomed-in version of the first plot):
\n
\n
What can be observed is that there are some events that are very well reconstructable (sharp peak), those are tracks. This distribution in the plot is peaking at around 2 degree opening angle. Our best track reconstructions are able to get that peak to below 1 degree.
\nThen there is some in-between distribution of cascades. Our algorithms get cascades to around 10ish degree (ballpark).
\nAnd then there is some unreconstructable background that will remain as a relatively wide distribution peaking at 90 degrees (pi/2 radian).
", "votes": 14, "replies": [ { "id": 2147719, "author_name": "edguy99", "author_url": "", "post_date": "2023-02-16T20:21:40.203000", "content": "New here and still learning, but little confused. You say \"provided was optimized for low-energy events (< 100 GeV) and was applied almost out-of-the-box to the high-energy (> 100 GeV) sample here\".
\nI don't see the energy for a particular eventID in the data or perhaps you are adding up the total charge for each hit to come up with this number? Thank you.
This information is not included in the data provided. But the data for this competition is high energy, >100 GeV. In simulation we know the energy, and in data we have ways to estimate it. The total charge per event as you say is a good and simple proxy.
", "votes": 1, "replies": [] } ] }, { "id": 2188779, "author_name": "Sophie", "author_url": "", "post_date": "2023-03-20T01:25:35.450000", "content": "what 1.2ish stands for? thanks
", "votes": 0, "replies": [ { "id": 2188945, "author_name": "Taq Seorangpun", "author_url": "", "post_date": "2023-03-20T05:33:48.257000", "content": "It means that the mean angular error is around 1.2, which would be around 70° deg. The best in the competition so far is 0.979, which translates to 56° deg.
", "votes": 2, "replies": [] } ] } ] }, { "id": 2137154, "author_name": "Alex Z", "author_url": "", "post_date": "2023-02-09T18:59:46.137000", "content": "Here is one other example of a neural net performance on neutrino direction reconstruction, from a paper I've browsed. It is specifically for the cascade event topology, that is for the harder subset of events. The energies range here seems to correspond well to the competition dataset (not sure if it distributed in the same way, though).
\n
\nIf I'm reading the figure correctly, that cnn performance is roughly around the 10-15 degrees, that is 0.2-0.25 radian. Hopefully, our dataset also contains a significant part of track events, which are more easily reconstructed and - with a bit of luck and hard work - we can achieve even better score.
Could you cite the paper for further reading?
", "votes": 0, "replies": [ { "id": 2137884, "author_name": "Jean-Loup Tastet", "author_url": "", "post_date": "2023-02-10T11:53:34.487000", "content": "The figure is taken from arXiv: 2101.11589 and is for a CNN. This is more recent than the GNN paper, but I don’t know if it performs better or worse (I don’t think they compare the two architectures).
\nAlso, keep in mind that this CNN uses more fine-grained features than what was made available for the present competition (see the caption of fig. 4 from the CNN paper: \"The pulses are therefore reduced to nine input parameters (𝑐total, 𝑐500ns, 𝑐100ns, 𝑡first, 𝑡last, 𝑡20%, 𝑡50%, 𝑡mean, 𝑡std) which aim to summarize the pulse distribution.\").
", "votes": 2, "replies": [] }, { "id": 2137937, "author_name": "Alex Z", "author_url": "", "post_date": "2023-02-10T12:51:01.643000", "content": "I read this as they integrate all the pulses from one DOM into these parameters, rather then they have the detailed internal waveform for every pulse. I may be mistaken, of course, and did not find a definite description in the paper.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2134303, "author_name": "Jared V", "author_url": "", "post_date": "2023-02-07T22:03:01.517000", "content": "I am guessing your question stems from the context section on the competition main page, \"Researchers have developed multiple approaches over the past ten years to reconstruct neutrino events. However, problems arise as existing solutions are far from perfect. They're either fast but inaccurate or more accurate at the price of huge computational costs.\"
\nFrom the limited research I have done so far it seems like we can use some really sophisticated mathematics and statistical methods to run simulations of a neutrino event. Those methods can narrow down the Azimuth and Zenith to high levels of precision. However, to do this requires a boat load of calculations, think hours on super computers.
\nAlternatively the machine learned methods can be significantly faster, orders of magnitude faster. My guess is that these machine learning methods are prone to errors, but on average they are correct enough. Think of the old joke about a statistician with his head in the ice box and feet in the oven… on average he is quite comfortable.
\nIf you're looking for the methods that are slow but super accurate, I'd suggest skimming thru some of the references in the papers listed on the main page. Hopefully that helps a little.
", "votes": 4, "replies": [] }, { "id": 2134047, "author_name": "SolverWorld", "author_url": "", "post_date": "2023-02-07T18:13:55.657000", "content": "If you read the various papers on CNN or GNN reconstruction methods, they are compared to the \"standard\" methods. Sometimes they are better, sometimes worse. So I am not sure any one knows what the best state-of-the-art is.
\nFor example, from Mirco Hennefeld, in Deep Learning In Physics exemplified by the Reconstruction of Muon-Neutrino Events in IceCube, a CNN method:
Or from R. Abbasi 2022 JINST 17 P1 1003 Graph Neural Networks for low-energy event classification and reconstruction in IceCube. Retro method is a standard program:
\n
To make it more complicated, a lot of this depends on:
\nThe \"standard\" methods, which are based on a maximum liklihood approach, could take 40 sec per event. Since there are millions of events in our dataset, I am pretty sure no one has run the standard reconstruction on all the events
", "votes": 1, "replies": [ { "id": 2134168, "author_name": "greySnow", "author_url": "", "post_date": "2023-02-07T19:58:33.347000", "content": "Well, they don't have to run it on millions of examples to get a rough estimate of the accuracy. A few thousands would probably suffice. Anyway, I'm sure they already have a pretty good estimation of the SOTA methods accuracy.
", "votes": 0, "replies": [] } ] }, { "id": 2132822, "author_name": "Robert Hatch", "author_url": "", "post_date": "2023-02-07T04:01:42.123000", "content": "I got the impression that the GraphNeT 1.014 from the organizers is the state of the art? I could very well be wrong.
", "votes": 2, "replies": [ { "id": 2133006, "author_name": "", "author_url": "", "post_date": "2023-02-07T07:42:30.820000", "content": "", "votes": 0, "replies": [] }, { "id": 2133597, "author_name": "John Mitchell", "author_url": "", "post_date": "2023-02-07T14:07:49.780000", "content": "So I'm confused - if that's the best reconstruction, then how are the GT values generated and how accurate are they? Are they simulated as the reverse of the competition workflow (assume a trajectory, compute the detector's response), or forwards (assume a response, work out the trajectory)?
", "votes": 0, "replies": [ { "id": 2133645, "author_name": "sophron", "author_url": "", "post_date": "2023-02-07T14:37:55.200000", "content": "My understanding is yes. The data we have is the result of putting specified neutrino events into the production simulation software they used for designing the experiment. I would hope they have results of running their really slow but very accurate algorithms over simulations. I'd love to know ballpark what that accuracy is.
", "votes": 1, "replies": [] } ] }, { "id": 2133668, "author_name": "greySnow", "author_url": "", "post_date": "2023-02-07T14:47:18.877000", "content": "GraphNet if a fast solution. I think there are slower, more precise solutions. I would also like to know what is the 'slow' benchmark.
", "votes": 1, "replies": [ { "id": 2189396, "author_name": "Philipp Eller", "author_url": "", "post_date": "2023-03-20T13:11:42.973000", "content": "See for example this article about one of our latest recos: https://icecube.wisc.edu/news/research/2021/04/new-algorithm-improves-icecubes-pointing-accuracy/
\nIt shows that for select events (high quality tracks only, so does not apply to all events provided here), the median angular error is as low as 0.2 - 0.3 degree! Other events have larger errors. This reconstruction, however, is comparably slow and costly.
\nFrom what I have seen so far in this competition, tracks are reconstructed to a precision of at best around 1 degree….so I think there is still a little bit room for improvement.
", "votes": 3, "replies": [] }, { "id": 2189502, "author_name": "DrHB", "author_url": "", "post_date": "2023-03-20T14:55:43.987000", "content": "How can we accurately ascertain whether a reconstruction is fast or slow, and understand the trade-offs involved? In the context of the Kaggle inference environment, which has limited resources and an allocated time of 9 hours, would you regard this as a fast reconstruction process?
", "votes": 0, "replies": [] }, { "id": 2189523, "author_name": "Philipp Eller", "author_url": "", "post_date": "2023-03-20T15:13:00.197000", "content": "Yes, we chose the size of the test set (roughly 1e6 events) in such a way that any valid submission within the 9h compute limit is fast enough for IceCube.
\nFor some context: the IceCube global trigger rate is around 2.7 kHz, and ideally we would like to be able to process all these events with limited resources (not more than a few servers/GPUs).
\nSlow reconstructions, on the other hand, are usually only applied to a small subset of select events, and take >> 1 second per event to complete, meaning up to minutes or even hours for a single event.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2133871, "author_name": "sophron", "author_url": "", "post_date": "2023-02-07T16:44:58.333000", "content": "@pellerphys This is mainly just a curiosity question, but I think you may be the most likely person to know the answer to this.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2132674": "I may have missed this in the competition description. If someone were to score the current state-of-the-art (but very slow) reconstructions with the same metrics used to score this competition, what would that score be? Are we allowed to know this? It'd be nice to benchmark the top scores on the leaderboard against the current state-of-the-art.", "2134852": "I agree with what others have already commented, it is difficult to put a number on this because our best reconstructions target certain event topologies, like tracks or cascades, separately.\n\nThat being said, we know that our most basic reconstruction \"LineFit\" gets a score of around 1.2ish on the kaggle dataset, and this is quite well in line with what people here got using similar geometric line fitting approaches.\n\nThen for the ML approaches: the Graphnet example that Rasmus has provided is currently one of our most promising ML approaches that we have tried within the IceCube collaboration, **BUT ** the example provided was optimized for low-energy events (< 100 GeV) and was applied almost out-of-the-box to the high-energy (> 100 GeV) sample here, without much optimization. So there certainly is room for improvement.\n\nThe two plots below show the Graphnet performance on the kaggle sample (the second plot is a zoomed-in version of the first plot):\n \n\n\nWhat can be observed is that there are some events that are very well reconstructable (sharp peak), those are tracks. This distribution in the plot is peaking at around 2 degree opening angle. Our best track reconstructions are able to get that peak to below 1 degree.\n\nThen there is some in-between distribution of cascades. Our algorithms get cascades to around 10ish degree (ballpark).\n\nAnd then there is some unreconstructable background that will remain as a relatively wide distribution peaking at 90 degrees (pi/2 radian).", "2137154": "Here is one other example of a neural net performance on neutrino direction reconstruction, from a paper I've browsed. It is specifically for the cascade event topology, that is for the harder subset of events. The energies range here seems to correspond well to the competition dataset (not sure if it distributed in the same way, though). \n\nIf I'm reading the figure correctly, that cnn performance is roughly around the 10-15 degrees, that is 0.2-0.25 radian. Hopefully, our dataset also contains a significant part of track events, which are more easily reconstructed and - with a bit of luck and hard work - we can achieve even better score. ", "2134303": "I am guessing your question stems from the context section on the competition main page, \"Researchers have developed multiple approaches over the past ten years to reconstruct neutrino events. However, problems arise as existing solutions are far from perfect. They're either fast but inaccurate or more accurate at the price of huge computational costs.\" \n\nFrom the limited research I have done so far it seems like we can use some really sophisticated mathematics and statistical methods to run simulations of a neutrino event. Those methods can narrow down the Azimuth and Zenith to high levels of precision. However, to do this requires a boat load of calculations, think hours on super computers. \n\nAlternatively the machine learned methods can be significantly faster, orders of magnitude faster. My guess is that these machine learning methods are prone to errors, but on average they are correct enough. Think of the old joke about a statistician with his head in the ice box and feet in the oven... on average he is quite comfortable.\n\nIf you're looking for the methods that are slow but super accurate, I'd suggest skimming thru some of the references in the papers listed on the main page. Hopefully that helps a little. ", "2134047": "If you read the various papers on CNN or GNN reconstruction methods, they are compared to the \"standard\" methods. Sometimes they are better, sometimes worse. So I am not sure any one knows what the best state-of-the-art is.\nFor example, from Mirco Hennefeld, in *Deep Learning In Physics exemplified by the Reconstruction of Muon-Neutrino Events in IceCube*, a CNN method:\n\n\n\nOr from R. Abbasi 2022 JINST 17 P1 1003 *Graph Neural Networks for low-energy event classification and reconstruction in IceCube*. Retro method is a standard program: \n\n\nTo make it more complicated, a lot of this depends on:\n1. Is this a Muon Track event or a Cascade event? A Muon track event is what we think of as a straight line of signals passing through the detector. This a the best case for determining an accurate direction. Papers say accuracies on the order of a few degrees are possible. Cascade events are more of a spherical light distrubution, and are much more difficult to estimate a direction from.\n2. What is the energy of the neutrino? Papers variously talk about low-energy or high-energy events, which have different issues.\n\nThe \"standard\" methods, which are based on a maximum liklihood approach, could take 40 sec per event. Since there are millions of events in our dataset, I am pretty sure no one has run the standard reconstruction on all the events ", "2132822": "I got the impression that the GraphNeT 1.014 from the organizers is the state of the art? I could very well be wrong. ", "2133871": "@pellerphys This is mainly just a curiosity question, but I think you may be the most likely person to know the answer to this." } }