Dear Kagglers,
\nWelcome to the IceCube neutrino challenge!
\nNeutrinos allow us to gain deep insights into the fundamental structure of matter, and to research our universe uncovering cataclysmic phenomena far away from our Earth. At the South Pole, in the middle of the Antarctica ice, we have constructed the world's largest neutrino detector and are taking data for more than 10 years. Interpreting this data, and in particular figuring out where exactly neutrinos came from, especially in a timely manner for real-time observations, is a daunting task. This is where we need your help! Your task is to estimate the direction of neutrinos that interacted in IceCube and provide fast and accurate predictions for the zenith and azimuth angles per recorded event.
\nWe hope that you take up the challenge and that maybe together with your ideas and algorithms we will be able to unravel some scientific mysteries in the future!
\nPlease enjoy, have fun, thank you, and the best of luck!
\nPhilipp Eller on behalf of the organizing team
The webinar recording is now available here and a pdf of the slides is attached to this post.
", "messages": [ { "id": 1958559, "postDate": "2022-09-27T13:59:48.833Z", "content": "Dear Kagglers,
\nWelcome to the IceCube neutrino challenge!
\nNeutrinos allow us to gain deep insights into the fundamental structure of matter, and to research our universe uncovering cataclysmic phenomena far away from our Earth. At the South Pole, in the middle of the Antarctica ice, we have constructed the world's largest neutrino detector and are taking data for more than 10 years. Interpreting this data, and in particular figuring out where exactly neutrinos came from, especially in a timely manner for real-time observations, is a daunting task. This is where we need your help! Your task is to estimate the direction of neutrinos that interacted in IceCube and provide fast and accurate predictions for the zenith and azimuth angles per recorded event.
\nWe hope that you take up the challenge and that maybe together with your ideas and algorithms we will be able to unravel some scientific mysteries in the future!
\nPlease enjoy, have fun, thank you, and the best of luck!
\nPhilipp Eller on behalf of the organizing team
The webinar recording is now available here and a pdf of the slides is attached to this post.
", "rawMarkdown": "Dear Kagglers,\n\nWelcome to the IceCube neutrino challenge!\n\nNeutrinos allow us to gain deep insights into the fundamental structure of matter, and to research our universe uncovering cataclysmic phenomena far away from our Earth. At the South Pole, in the middle of the Antarctica ice, we have constructed the world's largest neutrino detector and are taking data for more than 10 years. Interpreting this data, and in particular figuring out where exactly neutrinos came from, especially in a timely manner for real-time observations, is a daunting task. This is where we need your help! Your task is to estimate the direction of neutrinos that interacted in IceCube and provide fast and accurate predictions for the zenith and azimuth angles per recorded event.\n\nWe hope that you take up the challenge and that maybe together with your ideas and algorithms we will be able to unravel some scientific mysteries in the future!\n\nPlease enjoy, have fun, thank you, and the best of luck!\nPhilipp Eller on behalf of the organizing team\n\n**The webinar recording is now available [here](https://www.youtube.com/watch?v=_G4Nx20BVPg) and a pdf of the slides is attached to this post.**", "votes": 75 }, { "id": 2107690, "postDate": "2023-01-20T00:59:26.283Z", "content": "Thanks for the exciting competition. I have a question for both train and test is this real data or simulated or a mix of both?
Hi, this is all simulated data, generated by the official state-of-the-art IceCube simulation chain.
", "rawMarkdown": "Hi, this is all simulated data, generated by the official state-of-the-art IceCube simulation chain.", "votes": 14, "replies": [ { "id": 2108613, "postDate": "2023-01-20T17:05:24.660Z", "content": "@pellerphys Can you share link to any more info on that?
", "rawMarkdown": "@pellerphys Can you share link to any more info on that?", "votes": 2 }, { "id": 2110585, "postDate": "2023-01-22T09:32:31.377Z", "content": "That's what I've found so far, but I haven't looked too much into it,
\nso I hope that @pellerphys can share some more useful resources about the hows of the simulation.
How will I join the competition?
", "rawMarkdown": "How will I join the competition?" } ] }, { "id": 2114364, "postDate": "2023-01-24T23:02:17.870Z", "content": "what kind or error do you get in your current algorithm? I assume you tested it against this type of data. This would give us something to aim for.
", "rawMarkdown": "what kind or error do you get in your current algorithm? I assume you tested it against this type of data. This would give us something to aim for.", "votes": 8 }, { "id": 2110463, "postDate": "2023-01-22T08:03:59.393Z", "content": "Thank you for organizing this interesting competition! Three questions,
\nHi,
\nThe neutrinos are above 100 GeV in Energy, and are simulated with a spectrum that decreases towards higher energies. But you will also find neutrinos up to PeV in the dataset.
All our software that is open source can be found here: https://github.com/icecube (However, not all software is available as open source)
Yes, indeed there can also be other signals! In particular, there can be so-called coincident muons from cosmic ray air showers. These muons will manifest themselves as long tracks that start outside the detector and come at angles roughly between 0 and 90 degrees in zenith. Furthermore, there are hits caused by what we call noise. for instance radioactive decays in the sensor glass housing.
Thanks a lot for the details. Can you please comment more on point 3?
\nIn the description of the data you say: \"[azimuth/zenith] (float32): the [azimuth/zenith] angle in radians of the neutrino\". I read this as: all events are caused by neutrinos. Is this correct?
Of course in the realistic case there will be muons from cosmic rays. I believe most of those should be vetoed by IceTop. I guess it is much harder to veto muons from showers using the competition data alone. So, another question, is it correct to assume that auxiliary == False pulses are caused by neutrinos?
Yes, all events in the dataset contain a simulated neutrino! But it does not necessarily mean that it is creating much light in the detector in every case and, say, a muon could be the dominant signal in an event.
\nThere is no IceTop veto applied here, and in general, IceTop does not help that much to reject muons as one might think.
\nAnd no, you cannot make the assumption that auxiliary == False pulses are exclusively caused by neutrinos. They can be caused by anything, really, because the detctor cannot distinguish from where the light that is registered originates from.
", "rawMarkdown": "Yes, all events in the dataset contain a simulated neutrino! But it does not necessarily mean that it is creating much light in the detector in every case and, say, a muon could be the dominant signal in an event.\n\nThere is no IceTop veto applied here, and in general, IceTop does not help that much to reject muons as one might think.\n\nAnd no, you cannot make the assumption that auxiliary == False pulses are exclusively caused by neutrinos. They can be caused by anything, really, because the detctor cannot distinguish from where the light that is registered originates from.", "votes": 5 }, { "id": 2110948, "postDate": "2023-01-22T14:47:23.303Z", "content": "I see. Thanks a lot for clarifications!
", "rawMarkdown": "I see. Thanks a lot for clarifications!" }, { "id": 2110955, "postDate": "2023-01-22T14:51:14.623Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2110968, "postDate": "2023-01-22T14:59:54.173Z", "content": "Thanks for the reply. The information really helps!
\nMy first guess was that auxiliary=false means that the pulse was originated from neutrinos, but it seems that that is not the case. Then, am I correct to say that when creating the simulated data, each pulse originally does not have information about \"auxiliary\", and that auxiliary=true only means that the pulse was used for the trigger decision?
\nAlso, I couldn't understand what a \"fully digitized\" pulse was; could you please give more information about this?
\nIf False, then this pulse was contributed to the trigger decision and the pulse was fully digitized.
See this post by @roberthatch https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/379677
", "rawMarkdown": "See this post by @roberthatch https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/379677", "votes": 1 }, { "id": 2112755, "postDate": "2023-01-23T20:02:11.647Z", "content": "@pellerphys I do still have questions, myself. See my update today in that thread :)
", "rawMarkdown": "@pellerphys I do still have questions, myself. See my update today in that thread :)" }, { "id": 2112902, "postDate": "2023-01-23T23:42:11.410Z", "content": "Thanks for the information!
", "rawMarkdown": "Thanks for the information!" } ] } ] }, { "id": 2116853, "postDate": "2023-01-26T19:23:37.480Z", "content": "Thanks for hosting a great webinar today. I am looking forward to the viewing the recording, do you know when it will be available ?
", "rawMarkdown": "Thanks for hosting a great webinar today. I am looking forward to the viewing the recording, do you know when it will be available ?", "votes": 4 }, { "id": 2119293, "postDate": "2023-01-28T16:50:52.687Z", "content": "why so much data? It appears that results are very stable across batches, so CV over 1 batch would be almost the same as CV over all batches. So why give us > 100 Gg? Most competitors would not be able to use all of it anyway, due to RAM limits. So, what was the point?
", "rawMarkdown": "why so much data? It appears that results are very stable across batches, so CV over 1 batch would be almost the same as CV over all batches. So why give us > 100 Gg? Most competitors would not be able to use all of it anyway, due to RAM limits. So, what was the point?", "votes": 2, "replies": [ { "id": 2158595, "postDate": "2023-02-25T02:10:50.767Z", "content": "Still just getting started myself - but I see a very decent improvement training my tensorflow model with 3 batches rather than 1. Getting close to the point were I work on code revisions so I can train with a lot more batches. So I think the point is that Icecube has noticed in the efforts they have made to use AI that the models do need to see events at most of the sensor locations. With the right coding the issue becomes time to train rather than RAM as you suggested.
\nUpdate: Have gone from 1 training batch, to 3, than 10, just finishing 60 and will be firing up 660 today. Each increment up improves result.
", "rawMarkdown": "Still just getting started myself - but I see a very decent improvement training my tensorflow model with 3 batches rather than 1. Getting close to the point were I work on code revisions so I can train with a lot more batches. So I think the point is that Icecube has noticed in the efforts they have made to use AI that the models do need to see events at most of the sensor locations. With the right coding the issue becomes time to train rather than RAM as you suggested.\n\nUpdate: Have gone from 1 training batch, to 3, than 10, just finishing 60 and will be firing up 660 today. Each increment up improves result.", "votes": 1 } ] }, { "id": 2184471, "postDate": "2023-03-16T12:03:44.313Z", "content": "This small PDF should be made bigger and should be given a first priority over all the listed resources about neutrinos. As it talks about the approach to solve the problem. And thanks for sharing.
\nWhat's the surface area for the neutrinos to enter into the ice cube? Is it from all directions or just the head part of the ice cube?
\nIs ice cube half empty, just the bottom part contains the sensors? [1450m-2540m]? Why half n half?
\nLet's talk about simplicity as it beats complexity, for a given line what will be the same at start point and at the last point? time? as its faster comparable to speed of light?
", "rawMarkdown": "This small PDF should be made bigger and should be given a first priority over all the listed resources about neutrinos. As it talks about the approach to solve the problem. And thanks for sharing.\n\nWhat's the surface area for the neutrinos to enter into the ice cube? Is it from all directions or just the head part of the ice cube?\n\nIs ice cube half empty, just the bottom part contains the sensors? [1450m-2540m]? Why half n half?\n\nLet's talk about simplicity as it beats complexity, for a given line what will be the same at start point and at the last point? time? as its faster comparable to speed of light?", "votes": 1 }, { "id": 2116457, "postDate": "2023-01-26T14:04:21.227Z", "content": "Will the webinar be recorded? Unfortunately, I’m unable to attend live due to my “day job”.
", "rawMarkdown": "Will the webinar be recorded? Unfortunately, I’m unable to attend live due to my “day job”.", "votes": 1, "replies": [ { "id": 2117161, "postDate": "2023-01-27T04:30:32.673Z", "content": "Same ,have you receive or found the webinar?
", "rawMarkdown": "Same ,have you receive or found the webinar?" } ] }, { "id": 2117120, "postDate": "2023-01-27T03:04:53.533Z", "content": "It was a wonderful session with great insights and has lot of scope to learn amazing topics. It would be very helpful if we get recording.
", "rawMarkdown": "It was a wonderful session with great insights and has lot of scope to learn amazing topics. It would be very helpful if we get recording.", "votes": 2, "replies": [ { "id": 2118794, "postDate": "2023-01-28T09:36:01.670Z", "content": "I just posted the recordings above!
", "rawMarkdown": "I just posted the recordings above!", "votes": 2, "replies": [ { "id": 2119162, "postDate": "2023-01-28T14:24:02.930Z", "content": "Thank u so much.
", "rawMarkdown": "Thank u so much.\n" } ] } ] }, { "id": 2116805, "postDate": "2023-01-26T18:33:25.427Z", "content": "Thank you, for the wonderful seminar!!
\nI had to drop halfway, and am 100% looking forward to reviewing the recording.
\nI can’t wait to dive deeper and see what I can bring to this competition. Thanks again!
", "rawMarkdown": "Thank you, for the wonderful seminar!!\n\nI had to drop halfway, and am 100% looking forward to reviewing the recording.\n\nI can’t wait to dive deeper and see what I can bring to this competition. Thanks again! ", "votes": 2, "replies": [ { "id": 2117290, "postDate": "2023-01-27T07:06:55.160Z", "content": "Thank you! The recordings will hopefully be online later today, and we will add the link to the top of this thread.
", "rawMarkdown": "Thank you! The recordings will hopefully be online later today, and we will add the link to the top of this thread.", "votes": 1 } ] }, { "id": 2108807, "postDate": "2023-01-20T20:27:41.617Z", "content": "Hi. mind to provide the event id / code used for the exemple image in the data tab ? most of the event do not seem to have such correlations between observed (auxiliary = False) and the target path.
", "rawMarkdown": "Hi. mind to provide the event id / code used for the exemple image in the data tab ? most of the event do not seem to have such correlations between observed (auxiliary = False) and the target path. ", "votes": 2, "replies": [ { "id": 2108840, "postDate": "2023-01-20T21:31:35.123Z", "content": "Oh, okay. Glad it's not just me…
", "rawMarkdown": "Oh, okay. Glad it's not just me..." }, { "id": 2109462, "postDate": "2023-01-21T12:15:19.747Z", "content": "I cannot give you that picture's exact event_id since the dataset has been scrambled by kaggle.
But it is true that the displayed event is a particularly pretty one. Some events consist predominantly of noise, and others have few to many pulses from neutrinos. The dataset is a mixed bag of everything; not all events are as easy to reconstruct as the displayed one.
", "rawMarkdown": "I cannot give you that picture's exact `event_id` since the dataset has been scrambled by kaggle.\n\nBut it is true that the displayed event is a particularly pretty one. Some events consist predominantly of noise, and others have few to many pulses from neutrinos. The dataset is a mixed bag of everything; not all events are as easy to reconstruct as the displayed one.", "votes": 1 } ] }, { "id": 2216950, "postDate": "2023-04-10T13:33:53.563Z", "content": "Hello
\nI have been trying to submit my notebook for a long time, but every time my notebook gets submitted without error, I get a Submission scoring error. I checked my submission format 10 times but still got samesame error. I did index = False, have an equal number of rows demanded, and my values are in the accepted range. I am not getting where my submission is wrong. If you can help with that, it will be of great help.
\nLink to one of my submission: https://www.kaggle.com/code/rohitmalawat/neutrinoicecube?scriptVersionId=118860061
If I somehow manage to get in the top ranks and write a paper I will vent on this topic for multiple paragraphs.
\nThe notion that machine learning can make sense of junk is pretty common in today's world. The data set seems to be built with that idea.
\nThere is a minimum of 20% junk events. I am pretty sure it's junk. I am pretty sure with your current filter's it never leaves the South Pole. Two or three lines of Python sufficient to determine it's a junk event and discard it. A pretty decent number of my models parameters are attempting to make sense of it but they never will - they can only eventually arrive at the junk average.
\nI can't of course discard it and improve accuracy of the events to keep because my LB score will suck.
\nThat being said - it's a very interesting challenge.
", "rawMarkdown": "If I somehow manage to get in the top ranks and write a paper I will vent on this topic for multiple paragraphs.\n\nThe notion that machine learning can make sense of junk is pretty common in today's world. The data set seems to be built with that idea.\n\nThere is a minimum of 20% junk events. I am pretty sure it's junk. I am pretty sure with your current filter's it never leaves the South Pole. Two or three lines of Python sufficient to determine it's a junk event and discard it. A pretty decent number of my models parameters are attempting to make sense of it but they never will - they can only eventually arrive at the junk average. \n\nI can't of course discard it and improve accuracy of the events to keep because my LB score will suck. \n\nThat being said - it's a very interesting challenge. ", "replies": [ { "id": 2189831, "postDate": "2023-03-20T19:42:24.450Z", "content": "I would rather look at this from a different angle. As I understand the aim of the competition, it is to build a fast pipeline that is able to analyze the events in real time, probably right after the hardware filters in DOMs. They have something like a few kHz trigger events, and quite probably the most of these events are junk in real world. So, for the solution to be useful, it has to handle those junk events as well, as there will be no filtering before it.
\nAll in all, as far as physics is concerned, it seems to me that the dataset is quite accurate simulation of what you might see if you went to the South Pole and run a real experiment.
", "rawMarkdown": "I would rather look at this from a different angle. As I understand the aim of the competition, it is to build a fast pipeline that is able to analyze the events in real time, probably right after the hardware filters in DOMs. They have something like a few kHz trigger events, and quite probably the most of these events are junk in real world. So, for the solution to be useful, it has to handle those junk events as well, as there will be no filtering before it. \n\nAll in all, as far as physics is concerned, it seems to me that the dataset is quite accurate simulation of what you might see if you went to the South Pole and run a real experiment. ", "votes": 1 }, { "id": 2190527, "postDate": "2023-03-21T10:03:11.463Z", "content": "I understand your frustration! However, this challenge is designed to be really pushing the current state-of-the-art and see what is possible. For that reason, we wanted to supply as unfiltered data as possible.
\nWe don't expect you to be able to reconstruct \"un-reconstructable\" events, but your model must be able to deal with those. There are several ways to achieve that, the von Mises Fisher loss being but one example (The distribution has an additional parameter kappa that encodes the uncertainty of the reco, which the NN can predict to be very large for un-reconstrutable events). Another option would be first to clean the dataset before training, such that your NN can really home in on the good events. Then when applying it to junk it will spit out more or less random numbers, which is ok.
\nThe test set is quite large with 1e6 events, such that the performance on the junk data will average out and really the performance on the good data will drive your score.
\nThe comment made by @alexz0 is also correct, we would like to be able to run the reco as early as possible in the processing chain. The rate of events like those in the training set + even more \"junk\" is around 2.7 kHz, which is what is actually read out from the detector and being processed. ideally, we would apply the reco to all those events.
", "rawMarkdown": "I understand your frustration! However, this challenge is designed to be really pushing the current state-of-the-art and see what is possible. For that reason, we wanted to supply as unfiltered data as possible.\n\nWe don't expect you to be able to reconstruct \"un-reconstructable\" events, but your model must be able to deal with those. There are several ways to achieve that, the von Mises Fisher loss being but one example (The distribution has an additional parameter kappa that encodes the uncertainty of the reco, which the NN can predict to be very large for un-reconstrutable events). Another option would be first to clean the dataset before training, such that your NN can really home in on the good events. Then when applying it to junk it will spit out more or less random numbers, which is ok.\n\nThe test set is quite large with 1e6 events, such that the performance on the junk data will average out and really the performance on the good data will drive your score.\n\nThe comment made by @alexz0 is also correct, we would like to be able to run the reco as early as possible in the processing chain. The rate of events like those in the training set + even more \"junk\" is around 2.7 kHz, which is what is actually read out from the detector and being processed. ideally, we would apply the reco to all those events.", "votes": 1 } ] }, { "id": 2171880, "postDate": "2023-03-07T06:36:49.490Z", "content": "Thanks for the exciting competition! In the recording above, you said that the data is simulated and the flavour of neutrino that caused these events is mixed. So, the flavour of neutrino that caused these events completely random?
", "rawMarkdown": "Thanks for the exciting competition! In the recording above, you said that the data is simulated and the flavour of neutrino that caused these events is mixed. So, the flavour of neutrino that caused these events completely random?" }, { "id": 2158076, "postDate": "2023-02-24T15:52:38.897Z", "content": "Hello,
\nThank you for the organisation of this challenge.
\nThe link to \"Writeup Scoring Rubric\" in the A.3. part about the \"MOST INTERESTING SOLUTION WRITEUP PRIZES.\" is broken.
\nCan you make it work, please?
Hello @pellerphys, would it be feasible to incorporate a feature that allows us to view the recorded time for running the hidden test dataset after every submission? This feature would be beneficial in estimating the total running time of the code for the private dataset and can be used to design inference accordingly to prevent any cut-offs due to time constraints for final submissions.
", "rawMarkdown": "Hello @pellerphys, would it be feasible to incorporate a feature that allows us to view the recorded time for running the hidden test dataset after every submission? This feature would be beneficial in estimating the total running time of the code for the private dataset and can be used to design inference accordingly to prevent any cut-offs due to time constraints for final submissions.", "replies": [ { "id": 2155849, "postDate": "2023-02-23T00:24:30.340Z", "content": "We don't support providing precise runtimes for hidden test sets. This is a Kaggle-wide policy.
", "rawMarkdown": "We don't support providing precise runtimes for hidden test sets. This is a Kaggle-wide policy.", "votes": 2, "replies": [ { "id": 2156230, "postDate": "2023-02-23T07:18:56.577Z", "content": "Thank you for responding @sohier. I'd like to confirm if the time it takes to run the code and obtain the public score is the same as the time it takes to obtain the private score. In other words, when Kaggle calculates the score, does it process the entire test dataset and then measure the score based on only 43% of the output, or does the current running time reflect the processing of only 43% of the data?
", "rawMarkdown": "Thank you for responding @sohier. I'd like to confirm if the time it takes to run the code and obtain the public score is the same as the time it takes to obtain the private score. In other words, when Kaggle calculates the score, does it process the entire test dataset and then measure the score based on only 43% of the output, or does the current running time reflect the processing of only 43% of the data?" }, { "id": 2157157, "postDate": "2023-02-23T18:28:02.833Z", "content": "We process the entire test dataset. If your submission got a public score we also have already saved your private score.
", "rawMarkdown": "We process the entire test dataset. If your submission got a public score we also have already saved your private score.", "votes": 1 } ] } ] }, { "id": 2122043, "postDate": "2023-01-30T17:12:25.640Z", "content": "@pellerphys Thank you for sharing, just downloaded the pdf, and will watch the webinar.
", "rawMarkdown": "@pellerphys Thank you for sharing, just downloaded the pdf, and will watch the webinar." }, { "id": 2121434, "postDate": "2023-01-30T10:06:26.427Z", "content": "soooooo excited! looking forward to tackle this.
", "rawMarkdown": "soooooo excited! looking forward to tackle this." }, { "id": 2121042, "postDate": "2023-01-30T02:42:15.383Z", "content": "i have a question, you provide a true data , why in your ppt , you write the following words?
\n'Every event contains a simulated neutrino
\n+Noise
what does you mean ' a simulated neutrino' ! it is not coming from true data? thank you!
", "rawMarkdown": "i have a question, you provide a true data , why in your ppt , you write the following words?\n'Every event contains a simulated neutrino\n+Noise\n+ potentially other particles, i.e.“downgoingmuons\" '\n\nwhat does you mean ' a simulated neutrino' ! it is not coming from true data? thank you!", "replies": [ { "id": 2122031, "postDate": "2023-01-30T17:07:22.337Z", "content": "The data that they have given us is not from using their sensors to measure real neutrinos. Instead, it is data generated by their simulator.
\nMost of the work they do actually involves analysing the simulated data. They then compare their simulated data against the actual data to ensure that the simulation is accurate and reliable. They do it this way because the simulated data is able to provide a much higher resolution than the real-world sensors are capable of producing. You can read more about it here.
", "rawMarkdown": "The data that they have given us is not from using their sensors to measure *real* neutrinos. Instead, it is data generated by their *simulator*. \n\nMost of the work they do actually involves analysing the *simulated* data. They then compare their simulated data against the actual data to ensure that the simulation is accurate and reliable. They do it this way because the simulated data is able to provide a much higher resolution than the real-world sensors are capable of producing. You can read more about it [here](https://sciencenode.org/feature/simulating-icecube-data-using-gpus.php).", "votes": 2, "replies": [ { "id": 2123033, "postDate": "2023-01-31T09:24:07.757Z", "content": "Hi, that is correct, we do most of our work on simulated data. This is also best practice in science to develop everything first on simulated (synthetic/fake) data and only in the very end apply our methods to the real data to make measurements. This is to avoid biased results (c.f. blind analysis/study).
", "rawMarkdown": "Hi, that is correct, we do most of our work on simulated data. This is also best practice in science to develop everything first on simulated (synthetic/fake) data and only in the very end apply our methods to the real data to make measurements. This is to avoid biased results (c.f. blind analysis/study).", "votes": 1 }, { "id": 2124578, "postDate": "2023-02-01T04:49:13.450Z", "content": "dankwel,viel dank!this is a real challenging and interesting and significant task! the more people join, the better results can it achieve and serve for our descendants and our human beings. here is a knowledge ocean, it is very important to participate! happy!
", "rawMarkdown": "dankwel,viel dank!this is a real challenging and interesting and significant task! the more people join, the better results can it achieve and serve for our descendants and our human beings. here is a knowledge ocean, it is very important to participate! happy!" }, { "id": 2124579, "postDate": "2023-02-01T04:50:07.467Z", "content": "thanks your reply happy and i hope that i can learn more from you !
", "rawMarkdown": "thanks your reply happy and i hope that i can learn more from you !" } ] } ] }, { "id": 2121026, "postDate": "2023-01-30T02:05:36.660Z", "content": "sir, what does your 'official experiment simulation' mean? is it a model? or a simulated data? i am not quite understand! thanks!
", "rawMarkdown": "sir, what does your 'official experiment simulation' mean? is it a model? or a simulated data? i am not quite understand! thanks! " }, { "id": 2117340, "postDate": "2023-01-27T07:47:45.047Z", "content": "Please post the recording links. I missed the live session.
", "rawMarkdown": "Please post the recording links. I missed the live session." }, { "id": 2117042, "postDate": "2023-01-26T23:19:38.297Z", "content": "i need a team to help me i know some statistical software like stata,r,spss,kobo toolbox,
", "rawMarkdown": "i need a team to help me i know some statistical software like stata,r,spss,kobo toolbox,", "replies": [ { "id": 2117048, "postDate": "2023-01-26T23:36:59.803Z", "content": "I also need a team
", "rawMarkdown": "I also need a team " } ] }, { "id": 2117041, "postDate": "2023-01-26T23:14:47.590Z", "content": "how can i join webinar?
", "rawMarkdown": "how can i join webinar?" }, { "id": 2117038, "postDate": "2023-01-26T23:10:52.077Z", "content": "hello every one am very proud to be here unfortunately am not good in programming but now days am student in university in applied statistics so i hope to gathers we can, I just need access on data set and some software set up even if my laptop is not enough power full but never give up
", "rawMarkdown": "hello every one am very proud to be here unfortunately am not good in programming but now days am student in university in applied statistics so i hope to gathers we can, I just need access on data set and some software set up even if my laptop is not enough power full but never give up" }, { "id": 2116195, "postDate": "2023-01-26T09:48:20.790Z", "content": "How can I join Webinar ?
", "rawMarkdown": "How can I join Webinar ?" }, { "id": 2116143, "postDate": "2023-01-26T08:45:31.590Z", "content": "How can I join Webinar ?
", "rawMarkdown": "How can I join Webinar ?", "replies": [ { "id": 2116199, "postDate": "2023-01-26T09:52:13.427Z", "content": "please see above for the connection info
", "rawMarkdown": "please see above for the connection info" } ] }, { "id": 2115933, "postDate": "2023-01-26T04:31:37.747Z", "content": "what is the beijing time on January 26, thanks a lot ! it is fun and learn in the processing of competition! what is the link to your live broadcast!
", "rawMarkdown": "what is the beijing time on January 26, thanks a lot ! it is fun and learn in the processing of competition! what is the link to your live broadcast!", "replies": [ { "id": 2116349, "postDate": "2023-01-26T12:40:27.637Z", "content": "If I counted correctly, it is 1:00 AM (1:00) Beijing the next day (January 27)? but just in case check)
", "rawMarkdown": "If I counted correctly, it is 1:00 AM (1:00) Beijing the next day (January 27)? but just in case check)" } ] }, { "id": 2115911, "postDate": "2023-01-26T04:16:33.720Z", "content": "Hi Philipp Eller, how can I join the live webinar? I already have a team.
", "rawMarkdown": "Hi Philipp Eller, how can I join the live webinar? I already have a team.", "replies": [ { "id": 2116202, "postDate": "2023-01-26T09:52:56.423Z", "content": "Hi, I just posted the connection info for the webinar above at the beginning of the thread
", "rawMarkdown": "Hi, I just posted the connection info for the webinar above at the beginning of the thread" } ] }, { "id": 2115886, "postDate": "2023-01-26T03:45:25.073Z", "content": "Its interesting to learnt about these kind of topics
", "rawMarkdown": "Its interesting to learnt about these kind of topics" }, { "id": 2115760, "postDate": "2023-01-26T00:14:34.387Z", "content": "Such an interesting challenge!
", "rawMarkdown": "Such an interesting challenge!" }, { "id": 2115694, "postDate": "2023-01-25T23:01:26.427Z", "content": "Thank you Teresa Jy-Yang,
\nthis is very entrainment.one hour practice session, enjoy it very much. at the same time, we learned a little how to code for the 3rd dimension regression? as a introduction in the group, really appreciated.
", "rawMarkdown": "Thank you Teresa Jy-Yang, \n\nthis is very entrainment.one hour practice session, enjoy it very much. at the same time, we learned a little how to code for the 3rd dimension regression? as a introduction in the group, really appreciated. " }, { "id": 2115678, "postDate": "2023-01-25T22:48:27.820Z", "content": "How can i join Webinar??
", "rawMarkdown": "How can i join Webinar??", "replies": [ { "id": 2115698, "postDate": "2023-01-25T23:06:56.007Z", "content": "Hi, Dilare, I am so sorry you have not catch the time. Please DM me on the Slack . so I able to invite you into our team first. please attachment with your linkin and email address. there are more to come. Thanks.
", "rawMarkdown": "Hi, Dilare, I am so sorry you have not catch the time. Please DM me on the Slack . so I able to invite you into our team first. please attachment with your linkin and email address. there are more to come. Thanks. " } ] }, { "id": 2115564, "postDate": "2023-01-25T20:57:33.763Z", "content": "How can I join a team ? I have been studied space technology .
", "rawMarkdown": "How can I join a team ? I have been studied space technology .", "replies": [ { "id": 2115700, "postDate": "2023-01-25T23:08:50.953Z", "content": "Hi, Noha, welcome, Sorry you have missed the time. If you are interested to join our team. please log in the Slack and DM me, I will invite you into the team first. Thanks.
", "rawMarkdown": "Hi, Noha, welcome, Sorry you have missed the time. If you are interested to join our team. please log in the Slack and DM me, I will invite you into the team first. Thanks. ", "replies": [ { "id": 2115753, "postDate": "2023-01-26T00:04:29.297Z", "content": "Ok , Thanks alot
", "rawMarkdown": "Ok , Thanks alot " }, { "id": 2115755, "postDate": "2023-01-26T00:06:11.843Z", "content": "I have already logged in slack
", "rawMarkdown": "I have already logged in slack " } ] } ] }, { "id": 2115514, "postDate": "2023-01-25T20:14:54.013Z", "content": "This is a very interesting competition. Can link to the article be provided for more insight.
\nAnd where will the competition be submitted
Hi, Salome, Sorry did not catch the time with you. Kaggle forward your massage to me. Please go to the Slack in the building Team channel, DM me, so I will invite you first. and please post your linkin and your email. Thanks.
", "rawMarkdown": "Hi, Salome, Sorry did not catch the time with you. Kaggle forward your massage to me. Please go to the Slack in the building Team channel, DM me, so I will invite you first. and please post your linkin and your email. Thanks. ", "replies": [ { "id": 2115921, "postDate": "2023-01-26T04:26:45.723Z", "content": "hey i have also missed that.
", "rawMarkdown": "hey i have also missed that.\n" }, { "id": 2116081, "postDate": "2023-01-26T07:25:39.440Z", "content": "http://www.linkedin.com/in/salome-bassey
\nsallybassey19@gmail.com
\nPlease kindly drop slack link with me. Thanks
Hi, I wonder whether the SANTA algorithm is available as code anywhere. I looked for it in a number of theses available online and did not find it. I am aware that you may not want us to look into that direction because it is pretty much tried on your side but I think it can make for an interesting pre-processing for the data that can be then fed into a model. It would be helpful, or at least it would save us some time to rewrite it.
", "rawMarkdown": "Hi, I wonder whether the SANTA algorithm is available as code anywhere. I looked for it in a number of theses available online and did not find it. I am aware that you may not want us to look into that direction because it is pretty much tried on your side but I think it can make for an interesting pre-processing for the data that can be then fed into a model. It would be helpful, or at least it would save us some time to rewrite it. " }, { "id": 2108636, "postDate": "2023-01-20T17:22:41.917Z", "content": "Can you tell us how many events are in the hidden test set?
", "rawMarkdown": "Can you tell us how many events are in the hidden test set?", "replies": [ { "id": 2108682, "postDate": "2023-01-20T18:17:14.377Z", "content": "From the data tab: Expect to see roughly one million events in the hidden test set.
@sohier will there be multiple batches in the hidden test set or whole one million events will be part only one batch, i.e of batch_661 ?
", "rawMarkdown": "@sohier will there be multiple batches in the hidden test set or whole one million events will be part only one batch, i.e of batch_661 ?", "votes": 2 }, { "id": 2111278, "postDate": "2023-01-22T18:44:22.297Z", "content": "I would expect the test dataset to be split into batches (just like the train dataset). Those batch numbers will be in the test metadata parquet file. Exactly the same as the train metadata. The current test metadata and sample_submission files will be replaced by different ones when you submit to the competition.
\nThe host can confirm though.
", "rawMarkdown": "I would expect the test dataset to be split into batches (just like the train dataset). Those batch numbers will be in the test metadata parquet file. Exactly the same as the train metadata. The current test metadata and sample_submission files will be replaced by different ones when you submit to the competition.\n\nThe host can confirm though.", "votes": 1 }, { "id": 2112638, "postDate": "2023-01-23T18:17:44.603Z", "content": "There are multiple batches. I've also updated the data description to clarify that.
", "rawMarkdown": "There are multiple batches. I've also updated the data description to clarify that." } ] } ] }, { "id": 2115497, "postDate": "2023-01-25T19:56:45.793Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2109827, "postDate": "2023-01-21T17:44:34.823Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2108630, "postDate": "2023-01-20T17:19:32.830Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing\n", "votes": 2 } ], "comments": [ { "id": 2107690, "author_name": "DrHB", "author_url": "", "post_date": "2023-01-20T00:59:26.283000", "content": "Thanks for the exciting competition. I have a question for both train and test is this real data or simulated or a mix of both?
Hi, this is all simulated data, generated by the official state-of-the-art IceCube simulation chain.
", "votes": 14, "replies": [ { "id": 2108613, "author_name": "Robert Hatch", "author_url": "", "post_date": "2023-01-20T17:05:24.660000", "content": "@pellerphys Can you share link to any more info on that?
", "votes": 2, "replies": [] }, { "id": 2110585, "author_name": "nofreewill42", "author_url": "", "post_date": "2023-01-22T09:32:31.377000", "content": "That's what I've found so far, but I haven't looked too much into it,
\nso I hope that @pellerphys can share some more useful resources about the hows of the simulation.
How will I join the competition?
", "votes": 0, "replies": [] } ] }, { "id": 2114364, "author_name": "Youri Matiounine", "author_url": "", "post_date": "2023-01-24T23:02:17.870000", "content": "what kind or error do you get in your current algorithm? I assume you tested it against this type of data. This would give us something to aim for.
", "votes": 8, "replies": [] }, { "id": 2110463, "author_name": "c-number", "author_url": "", "post_date": "2023-01-22T08:03:59.393000", "content": "Thank you for organizing this interesting competition! Three questions,
\nHi,
\nThe neutrinos are above 100 GeV in Energy, and are simulated with a spectrum that decreases towards higher energies. But you will also find neutrinos up to PeV in the dataset.
All our software that is open source can be found here: https://github.com/icecube (However, not all software is available as open source)
Yes, indeed there can also be other signals! In particular, there can be so-called coincident muons from cosmic ray air showers. These muons will manifest themselves as long tracks that start outside the detector and come at angles roughly between 0 and 90 degrees in zenith. Furthermore, there are hits caused by what we call noise. for instance radioactive decays in the sensor glass housing.
Thanks a lot for the details. Can you please comment more on point 3?
\nIn the description of the data you say: \"[azimuth/zenith] (float32): the [azimuth/zenith] angle in radians of the neutrino\". I read this as: all events are caused by neutrinos. Is this correct?
Of course in the realistic case there will be muons from cosmic rays. I believe most of those should be vetoed by IceTop. I guess it is much harder to veto muons from showers using the competition data alone. So, another question, is it correct to assume that auxiliary == False pulses are caused by neutrinos?
Yes, all events in the dataset contain a simulated neutrino! But it does not necessarily mean that it is creating much light in the detector in every case and, say, a muon could be the dominant signal in an event.
\nThere is no IceTop veto applied here, and in general, IceTop does not help that much to reject muons as one might think.
\nAnd no, you cannot make the assumption that auxiliary == False pulses are exclusively caused by neutrinos. They can be caused by anything, really, because the detctor cannot distinguish from where the light that is registered originates from.
", "votes": 5, "replies": [] }, { "id": 2110948, "author_name": "Inar Timiryasov", "author_url": "", "post_date": "2023-01-22T14:47:23.303000", "content": "I see. Thanks a lot for clarifications!
", "votes": 0, "replies": [] }, { "id": 2110955, "author_name": "", "author_url": "", "post_date": "2023-01-22T14:51:14.623000", "content": "", "votes": 0, "replies": [] }, { "id": 2110968, "author_name": "c-number", "author_url": "", "post_date": "2023-01-22T14:59:54.173000", "content": "Thanks for the reply. The information really helps!
\nMy first guess was that auxiliary=false means that the pulse was originated from neutrinos, but it seems that that is not the case. Then, am I correct to say that when creating the simulated data, each pulse originally does not have information about \"auxiliary\", and that auxiliary=true only means that the pulse was used for the trigger decision?
\nAlso, I couldn't understand what a \"fully digitized\" pulse was; could you please give more information about this?
\nIf False, then this pulse was contributed to the trigger decision and the pulse was fully digitized.
See this post by @roberthatch https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/379677
", "votes": 1, "replies": [] }, { "id": 2112755, "author_name": "Robert Hatch", "author_url": "", "post_date": "2023-01-23T20:02:11.647000", "content": "@pellerphys I do still have questions, myself. See my update today in that thread :)
", "votes": 0, "replies": [] }, { "id": 2112902, "author_name": "c-number", "author_url": "", "post_date": "2023-01-23T23:42:11.410000", "content": "Thanks for the information!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2116853, "author_name": "Steve Jenkins", "author_url": "", "post_date": "2023-01-26T19:23:37.480000", "content": "Thanks for hosting a great webinar today. I am looking forward to the viewing the recording, do you know when it will be available ?
", "votes": 4, "replies": [] }, { "id": 2119293, "author_name": "Youri Matiounine", "author_url": "", "post_date": "2023-01-28T16:50:52.687000", "content": "why so much data? It appears that results are very stable across batches, so CV over 1 batch would be almost the same as CV over all batches. So why give us > 100 Gg? Most competitors would not be able to use all of it anyway, due to RAM limits. So, what was the point?
", "votes": 2, "replies": [ { "id": 2158595, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2023-02-25T02:10:50.767000", "content": "Still just getting started myself - but I see a very decent improvement training my tensorflow model with 3 batches rather than 1. Getting close to the point were I work on code revisions so I can train with a lot more batches. So I think the point is that Icecube has noticed in the efforts they have made to use AI that the models do need to see events at most of the sensor locations. With the right coding the issue becomes time to train rather than RAM as you suggested.
\nUpdate: Have gone from 1 training batch, to 3, than 10, just finishing 60 and will be firing up 660 today. Each increment up improves result.
", "votes": 1, "replies": [] } ] }, { "id": 2184471, "author_name": "Saalim Zafar", "author_url": "", "post_date": "2023-03-16T12:03:44.313000", "content": "This small PDF should be made bigger and should be given a first priority over all the listed resources about neutrinos. As it talks about the approach to solve the problem. And thanks for sharing.
\nWhat's the surface area for the neutrinos to enter into the ice cube? Is it from all directions or just the head part of the ice cube?
\nIs ice cube half empty, just the bottom part contains the sensors? [1450m-2540m]? Why half n half?
\nLet's talk about simplicity as it beats complexity, for a given line what will be the same at start point and at the last point? time? as its faster comparable to speed of light?
", "votes": 1, "replies": [] }, { "id": 2116457, "author_name": "Austin Pantall", "author_url": "", "post_date": "2023-01-26T14:04:21.227000", "content": "Will the webinar be recorded? Unfortunately, I’m unable to attend live due to my “day job”.
", "votes": 1, "replies": [ { "id": 2117161, "author_name": "Abdul Manan", "author_url": "", "post_date": "2023-01-27T04:30:32.673000", "content": "Same ,have you receive or found the webinar?
", "votes": 0, "replies": [] } ] }, { "id": 2117120, "author_name": "Uj_K", "author_url": "", "post_date": "2023-01-27T03:04:53.533000", "content": "It was a wonderful session with great insights and has lot of scope to learn amazing topics. It would be very helpful if we get recording.
", "votes": 2, "replies": [ { "id": 2118794, "author_name": "Philipp Eller", "author_url": "", "post_date": "2023-01-28T09:36:01.670000", "content": "I just posted the recordings above!
", "votes": 2, "replies": [ { "id": 2119162, "author_name": "Uj_K", "author_url": "", "post_date": "2023-01-28T14:24:02.930000", "content": "Thank u so much.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2116805, "author_name": "Darien Schettler", "author_url": "", "post_date": "2023-01-26T18:33:25.427000", "content": "Thank you, for the wonderful seminar!!
\nI had to drop halfway, and am 100% looking forward to reviewing the recording.
\nI can’t wait to dive deeper and see what I can bring to this competition. Thanks again!
", "votes": 2, "replies": [ { "id": 2117290, "author_name": "Philipp Eller", "author_url": "", "post_date": "2023-01-27T07:06:55.160000", "content": "Thank you! The recordings will hopefully be online later today, and we will add the link to the top of this thread.
", "votes": 1, "replies": [] } ] }, { "id": 2108807, "author_name": "Lucas Morin", "author_url": "", "post_date": "2023-01-20T20:27:41.617000", "content": "Hi. mind to provide the event id / code used for the exemple image in the data tab ? most of the event do not seem to have such correlations between observed (auxiliary = False) and the target path.
", "votes": 2, "replies": [ { "id": 2108840, "author_name": "Robert Hatch", "author_url": "", "post_date": "2023-01-20T21:31:35.123000", "content": "Oh, okay. Glad it's not just me…
", "votes": 0, "replies": [] }, { "id": 2109462, "author_name": "Philipp Eller", "author_url": "", "post_date": "2023-01-21T12:15:19.747000", "content": "I cannot give you that picture's exact event_id since the dataset has been scrambled by kaggle.
But it is true that the displayed event is a particularly pretty one. Some events consist predominantly of noise, and others have few to many pulses from neutrinos. The dataset is a mixed bag of everything; not all events are as easy to reconstruct as the displayed one.
", "votes": 1, "replies": [] } ] }, { "id": 2216950, "author_name": "ROHIT MALAWAT", "author_url": "", "post_date": "2023-04-10T13:33:53.563000", "content": "Hello
\nI have been trying to submit my notebook for a long time, but every time my notebook gets submitted without error, I get a Submission scoring error. I checked my submission format 10 times but still got samesame error. I did index = False, have an equal number of rows demanded, and my values are in the accepted range. I am not getting where my submission is wrong. If you can help with that, it will be of great help.
\nLink to one of my submission: https://www.kaggle.com/code/rohitmalawat/neutrinoicecube?scriptVersionId=118860061
If I somehow manage to get in the top ranks and write a paper I will vent on this topic for multiple paragraphs.
\nThe notion that machine learning can make sense of junk is pretty common in today's world. The data set seems to be built with that idea.
\nThere is a minimum of 20% junk events. I am pretty sure it's junk. I am pretty sure with your current filter's it never leaves the South Pole. Two or three lines of Python sufficient to determine it's a junk event and discard it. A pretty decent number of my models parameters are attempting to make sense of it but they never will - they can only eventually arrive at the junk average.
\nI can't of course discard it and improve accuracy of the events to keep because my LB score will suck.
\nThat being said - it's a very interesting challenge.
", "votes": 0, "replies": [ { "id": 2189831, "author_name": "Alex Z", "author_url": "", "post_date": "2023-03-20T19:42:24.450000", "content": "I would rather look at this from a different angle. As I understand the aim of the competition, it is to build a fast pipeline that is able to analyze the events in real time, probably right after the hardware filters in DOMs. They have something like a few kHz trigger events, and quite probably the most of these events are junk in real world. So, for the solution to be useful, it has to handle those junk events as well, as there will be no filtering before it.
\nAll in all, as far as physics is concerned, it seems to me that the dataset is quite accurate simulation of what you might see if you went to the South Pole and run a real experiment.
", "votes": 1, "replies": [] }, { "id": 2190527, "author_name": "Philipp Eller", "author_url": "", "post_date": "2023-03-21T10:03:11.463000", "content": "I understand your frustration! However, this challenge is designed to be really pushing the current state-of-the-art and see what is possible. For that reason, we wanted to supply as unfiltered data as possible.
\nWe don't expect you to be able to reconstruct \"un-reconstructable\" events, but your model must be able to deal with those. There are several ways to achieve that, the von Mises Fisher loss being but one example (The distribution has an additional parameter kappa that encodes the uncertainty of the reco, which the NN can predict to be very large for un-reconstrutable events). Another option would be first to clean the dataset before training, such that your NN can really home in on the good events. Then when applying it to junk it will spit out more or less random numbers, which is ok.
\nThe test set is quite large with 1e6 events, such that the performance on the junk data will average out and really the performance on the good data will drive your score.
\nThe comment made by @alexz0 is also correct, we would like to be able to run the reco as early as possible in the processing chain. The rate of events like those in the training set + even more \"junk\" is around 2.7 kHz, which is what is actually read out from the detector and being processed. ideally, we would apply the reco to all those events.
", "votes": 1, "replies": [] } ] }, { "id": 2171880, "author_name": "Kazuki HIROTA", "author_url": "", "post_date": "2023-03-07T06:36:49.490000", "content": "Thanks for the exciting competition! In the recording above, you said that the data is simulated and the flavour of neutrino that caused these events is mixed. So, the flavour of neutrino that caused these events completely random?
", "votes": 0, "replies": [] }, { "id": 2158076, "author_name": "Corentin Houpert", "author_url": "", "post_date": "2023-02-24T15:52:38.897000", "content": "Hello,
\nThank you for the organisation of this challenge.
\nThe link to \"Writeup Scoring Rubric\" in the A.3. part about the \"MOST INTERESTING SOLUTION WRITEUP PRIZES.\" is broken.
\nCan you make it work, please?
Hello @pellerphys, would it be feasible to incorporate a feature that allows us to view the recorded time for running the hidden test dataset after every submission? This feature would be beneficial in estimating the total running time of the code for the private dataset and can be used to design inference accordingly to prevent any cut-offs due to time constraints for final submissions.
", "votes": 0, "replies": [ { "id": 2155849, "author_name": "Sohier Dane", "author_url": "", "post_date": "2023-02-23T00:24:30.340000", "content": "We don't support providing precise runtimes for hidden test sets. This is a Kaggle-wide policy.
", "votes": 2, "replies": [ { "id": 2156230, "author_name": "1110Ra", "author_url": "", "post_date": "2023-02-23T07:18:56.577000", "content": "Thank you for responding @sohier. I'd like to confirm if the time it takes to run the code and obtain the public score is the same as the time it takes to obtain the private score. In other words, when Kaggle calculates the score, does it process the entire test dataset and then measure the score based on only 43% of the output, or does the current running time reflect the processing of only 43% of the data?
", "votes": 0, "replies": [] }, { "id": 2157157, "author_name": "Sohier Dane", "author_url": "", "post_date": "2023-02-23T18:28:02.833000", "content": "We process the entire test dataset. If your submission got a public score we also have already saved your private score.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2122043, "author_name": "Microtus", "author_url": "", "post_date": "2023-01-30T17:12:25.640000", "content": "@pellerphys Thank you for sharing, just downloaded the pdf, and will watch the webinar.
", "votes": 0, "replies": [] }, { "id": 2121434, "author_name": "Daisuke Ishii", "author_url": "", "post_date": "2023-01-30T10:06:26.427000", "content": "soooooo excited! looking forward to tackle this.
", "votes": 0, "replies": [] }, { "id": 2121042, "author_name": "Sophie", "author_url": "", "post_date": "2023-01-30T02:42:15.383000", "content": "i have a question, you provide a true data , why in your ppt , you write the following words?
\n'Every event contains a simulated neutrino
\n+Noise
what does you mean ' a simulated neutrino' ! it is not coming from true data? thank you!
", "votes": 0, "replies": [ { "id": 2122031, "author_name": "Michael Barrett", "author_url": "", "post_date": "2023-01-30T17:07:22.337000", "content": "The data that they have given us is not from using their sensors to measure real neutrinos. Instead, it is data generated by their simulator.
\nMost of the work they do actually involves analysing the simulated data. They then compare their simulated data against the actual data to ensure that the simulation is accurate and reliable. They do it this way because the simulated data is able to provide a much higher resolution than the real-world sensors are capable of producing. You can read more about it here.
", "votes": 2, "replies": [ { "id": 2123033, "author_name": "Philipp Eller", "author_url": "", "post_date": "2023-01-31T09:24:07.757000", "content": "Hi, that is correct, we do most of our work on simulated data. This is also best practice in science to develop everything first on simulated (synthetic/fake) data and only in the very end apply our methods to the real data to make measurements. This is to avoid biased results (c.f. blind analysis/study).
", "votes": 1, "replies": [] }, { "id": 2124578, "author_name": "Sophie", "author_url": "", "post_date": "2023-02-01T04:49:13.450000", "content": "dankwel,viel dank!this is a real challenging and interesting and significant task! the more people join, the better results can it achieve and serve for our descendants and our human beings. here is a knowledge ocean, it is very important to participate! happy!
", "votes": 0, "replies": [] }, { "id": 2124579, "author_name": "Sophie", "author_url": "", "post_date": "2023-02-01T04:50:07.467000", "content": "thanks your reply happy and i hope that i can learn more from you !
", "votes": 0, "replies": [] } ] } ] }, { "id": 2121026, "author_name": "Sophie", "author_url": "", "post_date": "2023-01-30T02:05:36.660000", "content": "sir, what does your 'official experiment simulation' mean? is it a model? or a simulated data? i am not quite understand! thanks!
", "votes": 0, "replies": [] }, { "id": 2117340, "author_name": "Pavan Attili", "author_url": "", "post_date": "2023-01-27T07:47:45.047000", "content": "Please post the recording links. I missed the live session.
", "votes": 0, "replies": [] }, { "id": 2117042, "author_name": "Djuma Manirakiza", "author_url": "", "post_date": "2023-01-26T23:19:38.297000", "content": "i need a team to help me i know some statistical software like stata,r,spss,kobo toolbox,
", "votes": 0, "replies": [ { "id": 2117048, "author_name": "Noha Zaid", "author_url": "", "post_date": "2023-01-26T23:36:59.803000", "content": "I also need a team
", "votes": 0, "replies": [] } ] }, { "id": 2117041, "author_name": "Djuma Manirakiza", "author_url": "", "post_date": "2023-01-26T23:14:47.590000", "content": "how can i join webinar?
", "votes": 0, "replies": [] }, { "id": 2117038, "author_name": "Djuma Manirakiza", "author_url": "", "post_date": "2023-01-26T23:10:52.077000", "content": "hello every one am very proud to be here unfortunately am not good in programming but now days am student in university in applied statistics so i hope to gathers we can, I just need access on data set and some software set up even if my laptop is not enough power full but never give up
", "votes": 0, "replies": [] }, { "id": 2116195, "author_name": "adamramciss@gmail.com", "author_url": "", "post_date": "2023-01-26T09:48:20.790000", "content": "How can I join Webinar ?
", "votes": 0, "replies": [] }, { "id": 2116143, "author_name": "KHALIL BOUKTHIR", "author_url": "", "post_date": "2023-01-26T08:45:31.590000", "content": "How can I join Webinar ?
", "votes": 0, "replies": [ { "id": 2116199, "author_name": "Philipp Eller", "author_url": "", "post_date": "2023-01-26T09:52:13.427000", "content": "please see above for the connection info
", "votes": 0, "replies": [] } ] }, { "id": 2115933, "author_name": "Sophie", "author_url": "", "post_date": "2023-01-26T04:31:37.747000", "content": "what is the beijing time on January 26, thanks a lot ! it is fun and learn in the processing of competition! what is the link to your live broadcast!
", "votes": 0, "replies": [ { "id": 2116349, "author_name": "Pavel Orlov", "author_url": "", "post_date": "2023-01-26T12:40:27.637000", "content": "If I counted correctly, it is 1:00 AM (1:00) Beijing the next day (January 27)? but just in case check)
", "votes": 0, "replies": [] } ] }, { "id": 2115911, "author_name": "Amador Zuñiga", "author_url": "", "post_date": "2023-01-26T04:16:33.720000", "content": "Hi Philipp Eller, how can I join the live webinar? I already have a team.
", "votes": 0, "replies": [ { "id": 2116202, "author_name": "Philipp Eller", "author_url": "", "post_date": "2023-01-26T09:52:56.423000", "content": "Hi, I just posted the connection info for the webinar above at the beginning of the thread
", "votes": 0, "replies": [] } ] }, { "id": 2115886, "author_name": "Tharu Jack", "author_url": "", "post_date": "2023-01-26T03:45:25.073000", "content": "Its interesting to learnt about these kind of topics
", "votes": 0, "replies": [] }, { "id": 2115760, "author_name": "Claudee", "author_url": "", "post_date": "2023-01-26T00:14:34.387000", "content": "Such an interesting challenge!
", "votes": 0, "replies": [] }, { "id": 2115694, "author_name": "", "author_url": "", "post_date": "2023-01-25T23:01:26.427000", "content": "", "votes": 0, "replies": [] }, { "id": 2115678, "author_name": "", "author_url": "", "post_date": "2023-01-25T22:48:27.820000", "content": "", "votes": 0, "replies": [ { "id": 2115698, "author_name": "", "author_url": "", "post_date": "2023-01-25T23:06:56.007000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2115564, "author_name": "", "author_url": "", "post_date": "2023-01-25T20:57:33.763000", "content": "", "votes": 0, "replies": [ { "id": 2115700, "author_name": "", "author_url": "", "post_date": "2023-01-25T23:08:50.953000", "content": "", "votes": 0, "replies": [ { "id": 2115753, "author_name": "", "author_url": "", "post_date": "2023-01-26T00:04:29.297000", "content": "", "votes": 0, "replies": [] }, { "id": 2115755, "author_name": "", "author_url": "", "post_date": "2023-01-26T00:06:11.843000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2115514, "author_name": "", "author_url": "", "post_date": "2023-01-25T20:14:54.013000", "content": "", "votes": 0, "replies": [ { "id": 2115702, "author_name": "", "author_url": "", "post_date": "2023-01-25T23:11:25.930000", "content": "", "votes": 0, "replies": [ { "id": 2115921, "author_name": "", "author_url": "", "post_date": "2023-01-26T04:26:45.723000", "content": "", "votes": 0, "replies": [] }, { "id": 2116081, "author_name": "", "author_url": "", "post_date": "2023-01-26T07:25:39.440000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2114288, "author_name": "", "author_url": "", "post_date": "2023-01-24T20:47:56.280000", "content": "", "votes": 0, "replies": [] }, { "id": 2108636, "author_name": "", "author_url": "", "post_date": "2023-01-20T17:22:41.917000", "content": "", "votes": 0, "replies": [ { "id": 2108682, "author_name": "", "author_url": "", "post_date": "2023-01-20T18:17:14.377000", "content": "", "votes": 4, "replies": [ { "id": 2109376, "author_name": "", "author_url": "", "post_date": "2023-01-21T11:25:18.490000", "content": "", "votes": 2, "replies": [] }, { "id": 2111278, "author_name": "", "author_url": "", "post_date": "2023-01-22T18:44:22.297000", "content": "", "votes": 1, "replies": [] }, { "id": 2112638, "author_name": "", "author_url": "", "post_date": "2023-01-23T18:17:44.603000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2115497, "author_name": "", "author_url": "", "post_date": "2023-01-25T19:56:45.793000", "content": "", "votes": 0, "replies": [] }, { "id": 2109827, "author_name": "", "author_url": "", "post_date": "2023-01-21T17:44:34.823000", "content": "", "votes": 0, "replies": [] }, { "id": 2108630, "author_name": "", "author_url": "", "post_date": "2023-01-20T17:19:32.830000", "content": "", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "1958559": "Dear Kagglers,\n\nWelcome to the IceCube neutrino challenge!\n\nNeutrinos allow us to gain deep insights into the fundamental structure of matter, and to research our universe uncovering cataclysmic phenomena far away from our Earth. At the South Pole, in the middle of the Antarctica ice, we have constructed the world's largest neutrino detector and are taking data for more than 10 years. Interpreting this data, and in particular figuring out where exactly neutrinos came from, especially in a timely manner for real-time observations, is a daunting task. This is where we need your help! Your task is to estimate the direction of neutrinos that interacted in IceCube and provide fast and accurate predictions for the zenith and azimuth angles per recorded event.\n\nWe hope that you take up the challenge and that maybe together with your ideas and algorithms we will be able to unravel some scientific mysteries in the future!\n\nPlease enjoy, have fun, thank you, and the best of luck!\nPhilipp Eller on behalf of the organizing team\n\n**The webinar recording is now available [here](https://www.youtube.com/watch?v=_G4Nx20BVPg) and a pdf of the slides is attached to this post.**", "2107690": "Thanks for the exciting competition. I have a question for both `train` and `test` is this real data or simulated or a mix of both? ", "2114364": "what kind or error do you get in your current algorithm? I assume you tested it against this type of data. This would give us something to aim for.", "2110463": "Thank you for organizing this interesting competition! Three questions,\n1. what is the energy distribution of the neutrinos in the data? When searching for papers about neutrino observation, I noticed that neutrinos interact differently with matter depending on their energy, and I am assuming that this information would help us a lot.\n2. would you mind to provide us more information about the IceCube simulator? A link to the document would be grateful.\n3. Are there any events in which more than one neutrino is associated, or in which signals from particles other than neutrinos appear?", "2116853": "Thanks for hosting a great webinar today. I am looking forward to the viewing the recording, do you know when it will be available ?", "2119293": "why so much data? It appears that results are very stable across batches, so CV over 1 batch would be almost the same as CV over all batches. So why give us > 100 Gg? Most competitors would not be able to use all of it anyway, due to RAM limits. So, what was the point?", "2184471": "This small PDF should be made bigger and should be given a first priority over all the listed resources about neutrinos. As it talks about the approach to solve the problem. And thanks for sharing.\n\nWhat's the surface area for the neutrinos to enter into the ice cube? Is it from all directions or just the head part of the ice cube?\n\nIs ice cube half empty, just the bottom part contains the sensors? [1450m-2540m]? Why half n half?\n\nLet's talk about simplicity as it beats complexity, for a given line what will be the same at start point and at the last point? time? as its faster comparable to speed of light?", "2116457": "Will the webinar be recorded? Unfortunately, I’m unable to attend live due to my “day job”.", "2117120": "It was a wonderful session with great insights and has lot of scope to learn amazing topics. It would be very helpful if we get recording.", "2116805": "Thank you, for the wonderful seminar!!\n\nI had to drop halfway, and am 100% looking forward to reviewing the recording.\n\nI can’t wait to dive deeper and see what I can bring to this competition. Thanks again! ", "2108807": "Hi. mind to provide the event id / code used for the exemple image in the data tab ? most of the event do not seem to have such correlations between observed (auxiliary = False) and the target path. ", "2216950": "Hello \nI have been trying to submit my notebook for a long time, but every time my notebook gets submitted without error, I get a Submission scoring error. I checked my submission format 10 times but still got samesame error. I did index = False, have an equal number of rows demanded, and my values are in the accepted range. I am not getting where my submission is wrong. If you can help with that, it will be of great help. \nLink to one of my submission: https://www.kaggle.com/code/rohitmalawat/neutrinoicecube?scriptVersionId=118860061", "2189298": "If I somehow manage to get in the top ranks and write a paper I will vent on this topic for multiple paragraphs.\n\nThe notion that machine learning can make sense of junk is pretty common in today's world. The data set seems to be built with that idea.\n\nThere is a minimum of 20% junk events. I am pretty sure it's junk. I am pretty sure with your current filter's it never leaves the South Pole. Two or three lines of Python sufficient to determine it's a junk event and discard it. A pretty decent number of my models parameters are attempting to make sense of it but they never will - they can only eventually arrive at the junk average. \n\nI can't of course discard it and improve accuracy of the events to keep because my LB score will suck. \n\nThat being said - it's a very interesting challenge. ", "2171880": "Thanks for the exciting competition! In the recording above, you said that the data is simulated and the flavour of neutrino that caused these events is mixed. So, the flavour of neutrino that caused these events completely random?", "2158076": "Hello,\n\nThank you for the organisation of this challenge.\n\nThe link to \"Writeup Scoring Rubric\" in the A.3. part about the \"MOST INTERESTING SOLUTION WRITEUP PRIZES.\" is broken.\nCan you make it work, please?", "2155838": "Hello @pellerphys, would it be feasible to incorporate a feature that allows us to view the recorded time for running the hidden test dataset after every submission? This feature would be beneficial in estimating the total running time of the code for the private dataset and can be used to design inference accordingly to prevent any cut-offs due to time constraints for final submissions.", "2122043": "@pellerphys Thank you for sharing, just downloaded the pdf, and will watch the webinar.", "2121434": "soooooo excited! looking forward to tackle this.", "2121042": "i have a question, you provide a true data , why in your ppt , you write the following words?\n'Every event contains a simulated neutrino\n+Noise\n+ potentially other particles, i.e.“downgoingmuons\" '\n\nwhat does you mean ' a simulated neutrino' ! it is not coming from true data? thank you!", "2121026": "sir, what does your 'official experiment simulation' mean? is it a model? or a simulated data? i am not quite understand! thanks! ", "2117340": "Please post the recording links. I missed the live session.", "2117042": "i need a team to help me i know some statistical software like stata,r,spss,kobo toolbox,", "2117041": "how can i join webinar?", "2117038": "hello every one am very proud to be here unfortunately am not good in programming but now days am student in university in applied statistics so i hope to gathers we can, I just need access on data set and some software set up even if my laptop is not enough power full but never give up", "2116195": "How can I join Webinar ?", "2116143": "How can I join Webinar ?", "2115933": "what is the beijing time on January 26, thanks a lot ! it is fun and learn in the processing of competition! what is the link to your live broadcast!", "2115911": "Hi Philipp Eller, how can I join the live webinar? I already have a team.", "2115886": "Its interesting to learnt about these kind of topics", "2115760": "Such an interesting challenge!", "2115694": "Thank you Teresa Jy-Yang, \n\nthis is very entrainment.one hour practice session, enjoy it very much. at the same time, we learned a little how to code for the 3rd dimension regression? as a introduction in the group, really appreciated. ", "2115678": "How can i join Webinar??", "2115564": "How can I join a team ? I have been studied space technology .", "2115514": "This is a very interesting competition. Can link to the article be provided for more insight. \nAnd where will the competition be submitted ", "2114288": "Hi, I wonder whether the SANTA algorithm is available as code anywhere. I looked for it in a number of theses available online and did not find it. I am aware that you may not want us to look into that direction because it is pretty much tried on your side but I think it can make for an interesting pre-processing for the data that can be then fed into a model. It would be helpful, or at least it would save us some time to rewrite it. ", "2108636": "Can you tell us how many events are in the hidden test set?", "2115497": "", "2109827": "", "2108630": "Thanks for sharing\n" } }