![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8459123%2F5e5230fc08fc70959f16586ec643fb7f%2F1.png?generation=1674913792499616&alt=media\")
Upon conducting an analysis of the coordinates, it has been determined that one of the strings located at approximately 443.43, -194.3 coordinates deviates from the ideal of a perfectly vertical line.
\nCheck this code for further analysis.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8459123%2F415dc480a0e494374c61d3ca26d457e2%2F2.png?generation=1674913799675571&alt=media\")
Upon conducting an analysis of the coordinates, it has been determined that one of the strings located at approximately 443.43, -194.3 coordinates deviates from the ideal of a perfectly vertical line.
\nCheck this code for further analysis.
I also found this string here: https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/379847
\nMy theory is that this was the only borehole that had a gyro survey, and the other boreholes have not had their x-y locations verified by measurement and assumed vertical. This is just speculation based on my experience with borehole data
", "rawMarkdown": "I also found this string here: https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/379847\n\nMy theory is that this was the only borehole that had a gyro survey, and the other boreholes have not had their x-y locations verified by measurement and assumed vertical. This is just speculation based on my experience with borehole data", "votes": null }, { "id": "2119136", "postDate": "01/28/2023 14:08:37", "content": "Thanks for sharing your work. Your theory makes total sense.
\nI am going to replace these coordinates (red) with average values (green).
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8459123%2F7923ba27c457a332843cae56a08b5642%2F3.png?generation=1674914894024994&alt=media\")
Keep in mind that we are working with synthetic data. The sensor locations are wherever the computer says they are. (Of course, the guys who wrote the simulation may have added in some sensor miscalibration just to vex us.)
\nedit: I suppose, I can't rule out the possibility that they have included some real noise data from the actual sensor field.
", "rawMarkdown": "Keep in mind that we are working with synthetic data. The sensor locations are wherever the computer says they are. (Of course, the guys who wrote the simulation may have added in some sensor miscalibration just to vex us.)\n\nedit: I suppose, I can't rule out the possibility that they have included some real noise data from the actual sensor field.", "votes": null }, { "id": "2119304", "postDate": "01/28/2023 16:57:30", "content": "It's possible, but I'm worried the CNN might have a hard time figuring out all the different noises in the x, y, and z axis. So, we should try to make them consistent.
", "rawMarkdown": "It's possible, but I'm worried the CNN might have a hard time figuring out all the different noises in the x, y, and z axis. So, we should try to make them consistent.", "votes": null }, { "id": "2119408", "postDate": "01/28/2023 18:40:40", "content": "Maybe it goes without saying, but I think whether to use the exact location or 'straighten' them depends entirely on the details of the both the model and the (engineered) features being used?
\nIt could even make sense to have it both ways, some features one way, other features the other way, depending on the design of the feature.
\nFor my early work, I am using 'string_id' as a quick way of knowing which sensors are on the same vertical string, and that much is as simple as string_id = sensor_id // 60 for the sensor dataset.
Similarly, there's also considerable noise for all sensors when comparing heights. So for some like a CNN, it might be better to use depth_id = sensor_id % 60 for height rather than the precise z value. Except then you have to exclude and deal with the very important difference for sensor_id >= 78*60, the 8 'deep_ice' strings start much deeper and are closer together.
A generation ago, I was shanghaied onto one of those panels of experts that the US government likes to assemble. And I was having lunch with a couple of guys for whom that was their day job - every week a shiny new problem and a new panel of fresh-faced experts opining about it … I'll cut to the chase: the way you know that you're finally hearing the answer from someone who actually knows the answer is when they start with \"it depends.\"
", "rawMarkdown": "A generation ago, I was shanghaied onto one of those panels of experts that the US government likes to assemble. And I was having lunch with a couple of guys for whom that was their day job - every week a shiny new problem and a new panel of fresh-faced experts opining about it ... I'll cut to the chase: the way you know that you're finally hearing the answer from someone who actually knows the answer is when they start with *\"it depends.\"*", "votes": null }, { "id": "2119925", "postDate": "01/29/2023 08:08:19", "content": "@roberthatch Thanks Robert! I appreciate your help! :)
", "rawMarkdown": "roberthatch Thanks Robert! I appreciate your help! :)", "votes": null }, { "id": "2120283", "postDate": "01/29/2023 13:40:03", "content": "@glazed - I think it’s worth clarifying that the sensor positions are not synthetic. Simply the events are. Sensor positions (and other non event meta) given in this competition are the same as those used on real events.
\nThat’s my understanding anyway.
\nOne additional point:
\nThe simulation in this competition is described by a forward model that uses a combination of physics, domain information, historical events (and more) given an azimuth and zenith and some other info. It’s our job to create a model that can reverse that.
Maybe this was already clear but hopefully it helps!
", "rawMarkdown": "glazed - I think it’s worth clarifying that the sensor positions are not synthetic. Simply the events are. Sensor positions (and other non event meta) given in this competition are the same as those used on real events.\n\nThat’s my understanding anyway.\n\n\nOne additional point:\nThe simulation in this competition is described by a forward model that uses a combination of physics, domain information, historical events (and more) given an azimuth and zenith and some other info. It’s our job to create a model that can reverse that.\n\nMaybe this was already clear but hopefully it helps!", "votes": null }, { "id": "2120595", "postDate": "01/29/2023 17:43:53", "content": "@dschettler8845 I'll say this another way. Suppose you flew down to the South Pole with an ice penetrating sensor, surveyed the Icecube field yourself and wound up knowing the locations of the real, physical DOMs much better than the organizers themselves do. Even if the differences were significant, I claim that you would gain no benefit in this contest from using that extra knowledge. (That's not to say it wouldn't help on real data.) The forward model uses the DOM's estimated locations, right or wrong. (The people who needed that explained might be an audience of zero.)
\nThere is a somewhat-related issue of whether using more evenly-spaced approximations of the DOM locations might be helpful in processing.
From a quick read of the DOM deployment process, it looks as the actual displacement of the DOMs from the vertical line is within 1 meter (as measured after they were put in place). So, I guess that this one string is \"correct\" and for the other strings the \"ideal\" coordinates are given as if the strings were aligned with vertical.
", "rawMarkdown": "From a quick read of the DOM deployment process, it looks as the actual displacement of the DOMs from the vertical line is within 1 meter (as measured after they were put in place). So, I guess that this one string is \"correct\" and for the other strings the \"ideal\" coordinates are given as if the strings were aligned with vertical.", "votes": null } ], "comments": [ { "id": 2119130, "author_name": "anjum48", "author_url": "", "post_date": "01/28/2023 14:03:18", "content": "I also found this string here: https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/379847
\nMy theory is that this was the only borehole that had a gyro survey, and the other boreholes have not had their x-y locations verified by measurement and assumed vertical. This is just speculation based on my experience with borehole data
", "votes": null, "replies": [ { "id": 2119136, "author_name": "mohammadrahmati", "author_url": "", "post_date": "01/28/2023 14:08:37", "content": "Thanks for sharing your work. Your theory makes total sense.
\nI am going to replace these coordinates (red) with average values (green).
Keep in mind that we are working with synthetic data. The sensor locations are wherever the computer says they are. (Of course, the guys who wrote the simulation may have added in some sensor miscalibration just to vex us.)
\nedit: I suppose, I can't rule out the possibility that they have included some real noise data from the actual sensor field.
", "votes": null, "replies": [ { "id": 2119304, "author_name": "mohammadrahmati", "author_url": "", "post_date": "01/28/2023 16:57:30", "content": "It's possible, but I'm worried the CNN might have a hard time figuring out all the different noises in the x, y, and z axis. So, we should try to make them consistent.
", "votes": null, "replies": [ { "id": 2119408, "author_name": "roberthatch", "author_url": "", "post_date": "01/28/2023 18:40:40", "content": "Maybe it goes without saying, but I think whether to use the exact location or 'straighten' them depends entirely on the details of the both the model and the (engineered) features being used?
\nIt could even make sense to have it both ways, some features one way, other features the other way, depending on the design of the feature.
\nFor my early work, I am using 'string_id' as a quick way of knowing which sensors are on the same vertical string, and that much is as simple as string_id = sensor_id // 60 for the sensor dataset.
Similarly, there's also considerable noise for all sensors when comparing heights. So for some like a CNN, it might be better to use depth_id = sensor_id % 60 for height rather than the precise z value. Except then you have to exclude and deal with the very important difference for sensor_id >= 78*60, the 8 'deep_ice' strings start much deeper and are closer together.
A generation ago, I was shanghaied onto one of those panels of experts that the US government likes to assemble. And I was having lunch with a couple of guys for whom that was their day job - every week a shiny new problem and a new panel of fresh-faced experts opining about it … I'll cut to the chase: the way you know that you're finally hearing the answer from someone who actually knows the answer is when they start with \"it depends.\"
", "votes": null, "replies": [ { "id": 2119925, "author_name": "mohammadrahmati", "author_url": "", "post_date": "01/29/2023 08:08:19", "content": "@roberthatch Thanks Robert! I appreciate your help! :)
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2120283, "author_name": "dschettler8845", "author_url": "", "post_date": "01/29/2023 13:40:03", "content": "@glazed - I think it’s worth clarifying that the sensor positions are not synthetic. Simply the events are. Sensor positions (and other non event meta) given in this competition are the same as those used on real events.
\nThat’s my understanding anyway.
\nOne additional point:
\nThe simulation in this competition is described by a forward model that uses a combination of physics, domain information, historical events (and more) given an azimuth and zenith and some other info. It’s our job to create a model that can reverse that.
Maybe this was already clear but hopefully it helps!
", "votes": null, "replies": [ { "id": 2120595, "author_name": "glazed", "author_url": "", "post_date": "01/29/2023 17:43:53", "content": "@dschettler8845 I'll say this another way. Suppose you flew down to the South Pole with an ice penetrating sensor, surveyed the Icecube field yourself and wound up knowing the locations of the real, physical DOMs much better than the organizers themselves do. Even if the differences were significant, I claim that you would gain no benefit in this contest from using that extra knowledge. (That's not to say it wouldn't help on real data.) The forward model uses the DOM's estimated locations, right or wrong. (The people who needed that explained might be an audience of zero.)
\nThere is a somewhat-related issue of whether using more evenly-spaced approximations of the DOM locations might be helpful in processing.
From a quick read of the DOM deployment process, it looks as the actual displacement of the DOMs from the vertical line is within 1 meter (as measured after they were put in place). So, I guess that this one string is \"correct\" and for the other strings the \"ideal\" coordinates are given as if the strings were aligned with vertical.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2119098": "Upon conducting an analysis of the coordinates, it has been determined that one of the strings located at approximately 443.43, -194.3 coordinates deviates from the ideal of a perfectly vertical line.\nCheck this [code](https://www.kaggle.com/code/mohammadrahmati/one-bad-string-in-dataset) for further analysis. \n\n\n\n\n\n\n\n", "2119130": "I also found this string here: https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/379847\n\nMy theory is that this was the only borehole that had a gyro survey, and the other boreholes have not had their x-y locations verified by measurement and assumed vertical. This is just speculation based on my experience with borehole data", "2119136": "Thanks for sharing your work. Your theory makes total sense. \nI am going to replace these coordinates (red) with average values (green).\n\n", "2119228": "Keep in mind that we are working with synthetic data. The sensor locations are wherever the computer says they are. (Of course, the guys who wrote the simulation may have added in some sensor miscalibration just to vex us.)\n\nedit: I suppose, I can't rule out the possibility that they have included some real noise data from the actual sensor field.", "2119304": "It's possible, but I'm worried the CNN might have a hard time figuring out all the different noises in the x, y, and z axis. So, we should try to make them consistent.", "2119408": "Maybe it goes without saying, but I think whether to use the exact location or 'straighten' them depends entirely on the details of the both the model and the (engineered) features being used?\n\nIt could even make sense to have it both ways, some features one way, other features the other way, depending on the design of the feature.\n\nFor my early work, I am using 'string_id' as a quick way of knowing which sensors are on the same vertical string, and that much is as simple as `string_id = sensor_id // 60` for the sensor dataset.\n\nSimilarly, there's also considerable noise for all sensors when comparing heights. So for some like a CNN, it might be better to use `depth_id = sensor_id % 60` for height rather than the precise z value. Except then you have to exclude and deal with the very important difference for sensor_id >= 78*60, the 8 'deep_ice' strings start much deeper and are closer together.", "2119642": "A generation ago, I was shanghaied onto one of those panels of experts that the US government likes to assemble. And I was having lunch with a couple of guys for whom that was their day job - every week a shiny new problem and a new panel of fresh-faced experts opining about it ... I'll cut to the chase: the way you know that you're finally hearing the answer from someone who actually knows the answer is when they start with *\"it depends.\"*", "2119925": "roberthatch Thanks Robert! I appreciate your help! :)", "2120283": "glazed - I think it’s worth clarifying that the sensor positions are not synthetic. Simply the events are. Sensor positions (and other non event meta) given in this competition are the same as those used on real events.\n\nThat’s my understanding anyway.\n\n\nOne additional point:\nThe simulation in this competition is described by a forward model that uses a combination of physics, domain information, historical events (and more) given an azimuth and zenith and some other info. It’s our job to create a model that can reverse that.\n\nMaybe this was already clear but hopefully it helps!", "2120595": "dschettler8845 I'll say this another way. Suppose you flew down to the South Pole with an ice penetrating sensor, surveyed the Icecube field yourself and wound up knowing the locations of the real, physical DOMs much better than the organizers themselves do. Even if the differences were significant, I claim that you would gain no benefit *in this contest* from using that extra knowledge. (That's not to say it wouldn't help on real data.) The forward model uses the DOM's estimated locations, right or wrong. (The people who needed that explained might be an audience of zero.)\n\nThere is a somewhat-related issue of whether using more evenly-spaced approximations of the DOM locations might be helpful in processing.", "2120757": "From a quick read of the DOM deployment process, it looks as the actual displacement of the DOMs from the vertical line is within 1 meter (as measured after they were put in place). So, I guess that this one string is \"correct\" and for the other strings the \"ideal\" coordinates are given as if the strings were aligned with vertical." }, "source": "meta" }