\n
![\"\"](\"https://i.imgur.com/Gz8Dvqd.png\")
I'd like to initiate a post to estimate the best score we could get from using only the sensors tagged with auxiliary = False.
\nSo I ran a little experiment on the first 30 000 events from the first batch. If you just do a simple line fitting, and evaluate the error of your forecast vs the true value, you should find something similar to this distribution:
\n
We can see clearly a mixture of two distributions.
\nThe way I interpreted it:
If we follow the reasoning until the end, by improving the models using only auxiliary = False we can asymptote to a best-case scenario where all the errors from the green distribution are 0 and where the errors from the red distribution are on average 1.5.
\nA quick estimation for my 30k samples tells me there are ~70% of points that belong to the random distribution (I really did it quickly, so it might not be totally accurate).
\nSo in the best-case scenario where the error of the green distribution is always 0, we should tend to a \"best score\" somewhere around 1.1 (0.7 x pi/2)
", "messages": [ { "id": "2122231", "postDate": "01/30/2023 19:18:33", "content": "I'd like to initiate a post to estimate the best score we could get from using only the sensors tagged with auxiliary = False.
\nSo I ran a little experiment on the first 30 000 events from the first batch. If you just do a simple line fitting, and evaluate the error of your forecast vs the true value, you should find something similar to this distribution:
\n
We can see clearly a mixture of two distributions.
\nThe way I interpreted it:
If we follow the reasoning until the end, by improving the models using only auxiliary = False we can asymptote to a best-case scenario where all the errors from the green distribution are 0 and where the errors from the red distribution are on average 1.5.
\nA quick estimation for my 30k samples tells me there are ~70% of points that belong to the random distribution (I really did it quickly, so it might not be totally accurate).
\nSo in the best-case scenario where the error of the green distribution is always 0, we should tend to a \"best score\" somewhere around 1.1 (0.7 x pi/2)
", "rawMarkdown": "I'd like to initiate a post to estimate the best score we could get from using only the sensors tagged with auxiliary = False. \n\nSo I ran a little experiment on the first 30 000 events from the first batch. If you just do a simple line fitting, and evaluate the error of your forecast vs the true value, you should find something similar to this distribution:\n\n\nWe can see clearly a mixture of two distributions. \nThe way I interpreted it: \n- The red distribution is looking like a purely random distribution. It is centered on pi/2 and is symmetric, it looks typically like the result of a random fit.\n- The green distribution, on the other hand, looks like the part in which the line fitted is imperfect, but meaningful. It is typically the case when a few outliers might change a bit our forecast. \n\nIf we follow the reasoning until the end, by improving the models using only auxiliary = False we can asymptote to a best-case scenario where all the errors from the green distribution are 0 and where the errors from the red distribution are on average 1.5.\n\nA quick estimation for my 30k samples tells me there are ~70% of points that belong to the random distribution (I really did it quickly, so it might not be totally accurate).\n\nSo in the best-case scenario where the error of the green distribution is always 0, we should tend to a \"best score\" somewhere around 1.1 (0.7 x pi/2)", "votes": null }, { "id": "2122250", "postDate": "01/30/2023 19:47:17", "content": "The green distribution is likely the \"track\" category of events, which will be predicted pretty accurately using line fitting. I think the red distribution will likely be a combination of the other types of events (e.g. cascade & double-bang)
\n![\"\"](\"https://i.imgur.com/7Q88JH3.png\")
Interesting how the cascade events have a 10 degree angular resolution. This gives us a lower bound on the cascade accuracy.
\n", "rawMarkdown": "The green distribution is likely the \"track\" category of events, which will be predicted pretty accurately using line fitting. I think the red distribution will likely be a combination of the other types of events (e.g. cascade & double-bang)\n\n\n\nInteresting how the cascade events have a 10 degree angular resolution. This gives us a lower bound on the cascade accuracy.\n\n[From page 6 of this slidepack](https://indico.cern.ch/event/472838/contributions/1150248/attachments/1296103/1932596/CAP_2016_IceCube.pdf)", "votes": null }, { "id": "2122263", "postDate": "01/30/2023 19:58:49", "content": "Thanks for sharing, this is in fact very interesting! I join you on your conclusions regarding green/red distributions, and I guess other methodes should be investigated for cascades and double bangs other than a simple line fitting…
", "rawMarkdown": "Thanks for sharing, this is in fact very interesting! I join you on your conclusions regarding green/red distributions, and I guess other methodes should be investigated for cascades and double bangs other than a simple line fitting...", "votes": null }, { "id": "2123588", "postDate": "01/31/2023 15:39:28", "content": "My predictions also showed the same distribution as yours.
\nAnd I found so many predictions are pointing up; particles are estimated to be going down, while the ground truth is omni directional.
\nI think there are many down particles as noise events (, which is named \"Atmospheric neutrino\", maybe?)
![\"histogram](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2026558%2F979b32b7cd0f73327a81b00db96f06ad%2Fhist.png?generation=1675179897149169&alt=media\")
![\"histogram](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2026558%2Fbff56a7bc894b80d7667b0c238224b40%2Fhist2.jpg?generation=1675179942082220&alt=media\")
Following on this topic, and thanks to datasaurus comment, I started to work on a classifier that would help detect the \"easy\" samples for which a line fit will be sufficient. Notebook Here.
\nI used only basic statistics, for now, but I managed to get already promising results. See the roc curve below:
\n![\"\"](\"https://i.imgur.com/GK8kJ5P.png\")
Used with a threshold low enough, such an approach can help maki a pre-segmentation of the samples. The illustration below shows the distribution of samples after a simple boosting algo forecast.
\n![\"\"](\"https://i.imgur.com/h76ZZ65.png\")
I had a look at your interesting notebook @bowaka.
\nI think there is a delicate trade-off between accuracy and speed here.
\nA richer set of input variables to the muon/else classifier leads to a better AUC, but it is also(typically) slower to generate.
\nIn the version on your notebook it takes about 3min to generate output for 5k events -> 2h for 1 batch.
\nIf we need to pre-classify all test events this might be an issue.
\nIn general finding a fast and accurate muon/else classifier seems to me the next major challenge.
The green distribution is likely the \"track\" category of events, which will be predicted pretty accurately using line fitting. I think the red distribution will likely be a combination of the other types of events (e.g. cascade & double-bang)
\n
Interesting how the cascade events have a 10 degree angular resolution. This gives us a lower bound on the cascade accuracy.
\n", "votes": null, "replies": [ { "id": 2122263, "author_name": "bowaka", "author_url": "", "post_date": "01/30/2023 19:58:49", "content": "Thanks for sharing, this is in fact very interesting! I join you on your conclusions regarding green/red distributions, and I guess other methodes should be investigated for cascades and double bangs other than a simple line fitting…
", "votes": null, "replies": [] } ] }, { "id": 2123588, "author_name": "yosshi999", "author_url": "", "post_date": "01/31/2023 15:39:28", "content": "My predictions also showed the same distribution as yours.
\nAnd I found so many predictions are pointing up; particles are estimated to be going down, while the ground truth is omni directional.
\nI think there are many down particles as noise events (, which is named \"Atmospheric neutrino\", maybe?)
Following on this topic, and thanks to datasaurus comment, I started to work on a classifier that would help detect the \"easy\" samples for which a line fit will be sufficient. Notebook Here.
\nI used only basic statistics, for now, but I managed to get already promising results. See the roc curve below:
\n
Used with a threshold low enough, such an approach can help maki a pre-segmentation of the samples. The illustration below shows the distribution of samples after a simple boosting algo forecast.
\n
I had a look at your interesting notebook @bowaka.
\nI think there is a delicate trade-off between accuracy and speed here.
\nA richer set of input variables to the muon/else classifier leads to a better AUC, but it is also(typically) slower to generate.
\nIn the version on your notebook it takes about 3min to generate output for 5k events -> 2h for 1 batch.
\nIf we need to pre-classify all test events this might be an issue.
\nIn general finding a fast and accurate muon/else classifier seems to me the next major challenge.