{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# A Survey of IceCube\n\nIceCube is a neutrino detector located at the South Pole. It is a large array of optical sensors embedded in the ice, which detect the [Cherenkov radiation](https://arxiv.org/pdf/0807.0034.pdf) emitted by charged particles produced when a neutrino interacts with the ice. The detector has a total volume of one cubic kilometer and contains over 5,000 sensors, making it one of the largest neutrino detectors in the world.\n\nIceCube is used to study a wide range of neutrino-related phenomena, including the search for high-energy cosmic neutrinos, the study of neutrino oscillations, and the search for dark matter. The detector is also used to study other types of particles, such as gamma rays and cosmic rays.\n\nIceCube is operated by an international collaboration of scientists from over 40 institutions in 12 countries. The data collected by the detector is used by scientists around the world to study some of the most intriguing and mysterious phenomena in the universe.\n\n## Cherenkov Radiation \nhttps://www.sciencedirect.com/topics/physics-and-astronomy/cherenkov-radiation#:~:text=Cherenkov%20radiation%20is%20composed%20of,region%20of%20the%20electromagnetic%20spectrum.&text=where%20%CE%B2%3Dv%2Fc%20is,velocity%20of%20light%20in%20vacuum.\n\n## The Angle of a Neutrino in radians\n\nIn Practice of IceCube, the theory was implemented by more accessible methodology,\n\nThe angle of a neutrino in radians is not a fixed value, it will depend on the direction that the neutrino is traveling and the direction that the detector is observing from.\n\nIn the IceCube Neutrino Observatory, the neutrino direction is reconstructed using the timing and amplitude information from the DOMs (Digital Optical Modules) that detect the Cherenkov radiation from the secondary particles produced by the neutrino interaction(muon). The detector is designed to have a large field of view, so it can observe a wide range of angles.\n\nThe reconstructed angle of a neutrino event is typically given in terms of the right ascension and declination coordinates, which are similar to the longitude and latitude coordinates used for describing the position of objects on the Earth's surface. The reconstructed angle can also be given in terms of the polar angle and azimuthal angle, which are commonly used in particle physics.\n\nIn general, the value of the angle in radians of a neutrino would depend on the particular event, and the exact value would need to be determined through analysis of the data from the IceCube observatory.\n\nIn terms of the azimuthal angle and zenith angle, a neutrino event can be represented as:\n\n\n\n**Zenith angle: $\\theta$ (in radians),\nAzimuthal angle: $\\varphi$ (in radians)**\n\n**The zenith angle is the angle between the incoming neutrino direction and the vertical axis of the detector (perpendicular to the detector plane) while the azimuthal angle is the angle between the projection of the incoming neutrino direction onto the detector plane and the detector's(DOM's) local x-axis.**\n\n## DOM\n\nThe Digital Optical Module (DOM) is a key component of the IceCube Neutrino Observatory. It is a sensor device that is used to detect the Cherenkov radiation emitted by charged particles produced when a neutrino interacts with the ice.\n\nA DOM consists of a photomultiplier tube (PMT) which is a device that converts light into an electrical signal, and a data acquisition system (DAQ) which is used to digitize and process the electrical signal from the PMT.\n\nEach DOM is a cylindrical device that is about the size of a basketball and is deployed in the ice at depths of between 1.4 and 2.4 km. The DOMs are connected to a cable that runs up to the surface and transmits the data to the IceCube data acquisition system for further analysis.\n\nThe DOMs are designed to detect the Cherenkov radiation over a wide range of wavelengths, from about 300 to 600 nanometers. They are also able to detect the Cherenkov radiation from a wide range of angles, allowing them to reconstruct the direction of the incoming neutrino with high accuracy.\n\nThe DOMs are operated at very low temperatures, around -40 degree Celsius, to ensure that they are not affected by thermal noise and are sensitive to the Cherenkov light. They are also designed to be highly reliable and able to operate for many years without maintenance.\n\n## Event\n\nIn the IceCube Neutrino Observatory, an event refers to the detection of a neutrino interaction within the detector. When a neutrino interacts with the ice or another material in the detector, it produces a charged particle, such as a muon or a tau, which then produces Cherenkov radiation. This radiation is detected by the DOMs and is used to reconstruct the neutrino's energy and direction.\n\nEach event is characterized by a set of data, including the time and location of the interaction, the energy and direction of the produced charged particle, and the pattern of Cherenkov radiation detected by the DOMs. These data are used to identify the type of neutrino interaction and to study the properties of the neutrino.\n\nIceCube observatory detect and measure neutrinos in a wide range of energies, from a few tens of GeV to more than a PeV. These events are classified based on the type of neutrino interaction and the energy of the produced charged particle.\n\nHigh-energy neutrino events, which have energies greater than 100 TeV, are particularly interesting because they are thought to be produced by some of the most powerful and distant astronomical sources in the universe, such as supermassive black holes, gamma-ray bursts and active galactic nuclei.\n\n### Some Samples of Events\n\n#### An example event detected by the IceCube Neutrino Observatory is the discovery of a high-energy neutrino event known as \"IceCube-170922A\".\n\nOn September 22, 2017, the IceCube detector recorded a neutrino interaction in the southern hemisphere with an estimated energy of about 290 tera-electronvolts (TeV). The neutrino's arrival direction was determined to be consistent with the direction of a known blazar, a type of active galaxy with a supermassive black hole at its center, located in the constellation Orion.\n\nThe data from this event was analyzed to determine the properties of the neutrino, including its energy and direction, as well as the properties of the charged particle produced in the interaction. This information was used to reconstruct the event, and the results were published in a scientific paper.\n\nThe data for this event would have included the following information:\n\nTime of the event: September 22, 2017 at 03:57:14 UTC\nPosition of the event: Within the IceCube detector array\nEnergy of the produced charged particle: 290 TeV\nDirection of the produced charged particle: Arrival direction consistent with the known blazar TXS 0506+056\nPattern of Cherenkov radiation detected by the DOMs: Digitized signals from each DOM in a binary data format.\nThis event was significant because it was the first high-energy neutrino event that was observed to be coming from a known astronomical source. It confirmed the hypothesis that high-energy neutrinos are produced by powerful astronomical sources, such as blazars.\n\n#### Another example event detected by the IceCube Neutrino Observatory is the \"IceCube-160731A\" event, which was recorded on July 31, 2016.\n\nThis event was a high-energy neutrino interaction that occurred in the northern hemisphere with an estimated energy of about 2 PeV (2*10^15 eV). The neutrino's arrival direction was determined to be consistent with the direction of the known blazar TXS 0506+056.\n\nThe data from this event was analyzed to determine the properties of the neutrino, including its energy and direction, as well as the properties of the charged particle produced in the interaction. This information was used to reconstruct the event, and the results were published in a scientific paper.\n\nThe data for this event would have included the following information:\n\nTime of the event: July 31, 2016 at 11:47:08 UTC\nPosition of the event: Within the IceCube detector array\nEnergy of the produced charged particle: 2 PeV\nDirection of the produced charged particle: Arrival direction consistent with the known blazar TXS 0506+056\nPattern of Cherenkov radiation detected by the DOMs: Digitized signals from each DOM in a binary data format.\nThis event was significant because it was one of the highest energy neutrinos ever detected, and it is still one of the most energetic neutrino events observed to date by IceCube.\n\n#### Event number \"IceCube-221224A\". This event was detected on December 2022, GRB Coordinates Network, Circular Service, No. 33118\n\nIceCube has performed a search [1] for additional track-like muon neutrino events arriving from the direction of IceCube-221224A (https://gcn.gsfc.nasa.gov/gcn3/33097.gcn3) in a time range of 1000 seconds centered on the alert event time (2022-12-24 00:46:49.291 UTC to 2022-12-24 01:03:29.291 UTC) during which IceCube was collecting good quality data. Excluding the event that prompted the alert, zero track-like events are found within the 90% containment region of IceCube-221224A. The IceCube sensitivity to neutrino point sources with an E^-2.5 spectrum (E^2 dN/dE at 1 TeV) within the locations spanned by the 90% spatial containment region of IceCube-221224A is 1.4e-01 GeV cm^-2 in a 1000 second time window. 90% of events IceCube would detect from a source at this declination with an E^-2.5 spectrum have energies in the approximate energy range between 3e+02 GeV and 2e+05 GeV. A subsequent search was performed including 2 days of data centered on the alert event time (2022-12-23 00:55:09.291 UTC to 2022-12-25 00:55:09.291 UTC). In this case, we report a p-value of 0.02, consistent with no significant excess of track events. The IceCube sensitivity to neutrino point sources with an E^-2.5 spectrum (E^2 dN/dE at 1 TeV) within the locations spanned by the 90% spatial containment region of IceCube-221224A is 1.6e-01 GeV cm^-2 in a 2 day time window. The IceCube Neutrino Observatory is a cubic-kilometer neutrino detector operating at the geographic South Pole, Antarctica. The IceCube realtime alert point of contact can be reached at roc@icecube.wisc.edu. [1] IceCube Collaboration, R. Abbasi et al., ApJ 910 4 (2021)\n\n\n\n## Challenges for IceCube\n\nThere are several challenges associated with the IceCube Neutrino Observatory:\n\n1. Extreme environment: The IceCube detector is located at the South Pole, where temperatures can drop below -60 degrees Celsius and winds can reach up to 200 km/h. This makes it difficult to deploy and maintain the equipment, as well as to ensure that the data is transmitted and stored reliably.\n\n2. Depth: The DOMs are deployed at depths of between 1.4 and 2.4 km in the ice. This makes it difficult to retrieve the DOMs for maintenance or repair, and also makes it difficult to transmit data from the DOMs to the surface.\n\n3. Noise: The detector is sensitive to all kinds of radiation and noise, including cosmic rays, natural radioactivity and temperature fluctuation. This makes it difficult to separate the signal from the background noise and to accurately determine the energy and direction of the neutrino.\n\n4. Data analysis: The detector generates a large amount of data, which needs to be analyzed to identify neutrino events. This is a complex task that requires sophisticated algorithms and powerful computing resources.\n\n5. Cost: Building and maintaining such a large detector is a significant financial undertaking. The cost of deploying and maintaining the detector, as well as the cost of data analysis and storage, are all significant challenges.\n\n","metadata":{}},{"cell_type":"markdown","source":"# A first glance at Data Exploring \n\nThe Target of this Datathon by IceCube is to predict **(azimuth, zenith) in three sensors**.There are reasons that could affect the measurement of the azimuth and zenith angles:\n\n1. Uncertainties caused by the timing and amplitude measurements of the Cherenkov radiation\n2. Background noise from other sources (e.g. cosmic rays)\n3. Uncertainties caused by the position and orientation of the DOMs\n4. Uncertainties caused by the properties of the ice surrounding the DOMs\n5. Issues with the reconstruction algorithms used to determine the neutrino direction\n\nNote that The Dataset provided by IceCube does not include other Quantities like neutrino energy and interaction type and kinematics, the most usable feature is of `sensor_geometry.csv` i.e. **the coordinate information of the DOMs' Array**. The simplest Hyperthesis which might make sense would be the **The Cartesian Coordinates of sensors determines their (azimuth, zenith) in each batch Linearly** according to the observation of [JIRKA BOROVEC](https://www.kaggle.com/code/jirkaborovec/icecube-neutrino-fitting-3d-points-lb-1-25) which implemented the 3rd above.\n\nThere are 660 parquet files in train directory each of which is corresponding a batch of the neutrinos event.","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport glob\nimport gc, time\nimport math\nimport warnings\nwarnings.filterwarnings('ignore')\n\nPATH_DATASET = \"../input/icecube-neutrinos-in-deep-ice/\"","metadata":{"execution":{"iopub.status.busy":"2023-01-29T18:44:12.342344Z","iopub.execute_input":"2023-01-29T18:44:12.343266Z","iopub.status.idle":"2023-01-29T18:44:12.379139Z","shell.execute_reply.started":"2023-01-29T18:44:12.343168Z","shell.execute_reply":"2023-01-29T18:44:12.377822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"PATH_DATASET = \"../input/icecube-neutrinos-in-deep-ice/\"\n\ndf_train_meta = pd.read_parquet(os.path.join(PATH_DATASET, \"train_meta.parquet\"))\ngeometry = pd.read_csv(os.path.join(PATH_DATASET, \"sensor_geometry.csv\"))","metadata":{"execution":{"iopub.status.busy":"2023-01-29T18:44:12.381656Z","iopub.execute_input":"2023-01-29T18:44:12.382206Z","iopub.status.idle":"2023-01-29T18:45:00.631251Z","shell.execute_reply.started":"2023-01-29T18:44:12.382156Z","shell.execute_reply":"2023-01-29T18:45:00.629741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"for _, _, files in os.walk(os.path.join(PATH_DATASET, 'train')):\n train_files_name = files\n for file in files:\n print(file)\nprint(f\"There are {len(train_files_name)} parquet files in train directory\")","metadata":{}},{"cell_type":"code","source":"#train_1 = pd.read_parquet(os.path.join(PATH_DATASET, 'train', \"batch_1.parquet\"))\n\ntrain_2 = pd.read_parquet(os.path.join(PATH_DATASET, 'train', \"batch_2.parquet\"))\n\n#train_15 = pd.read_parquet(os.path.join(PATH_DATASET, 'train', \"batch_15.parquet\"))","metadata":{"execution":{"iopub.status.busy":"2023-01-29T18:45:00.633161Z","iopub.execute_input":"2023-01-29T18:45:00.633618Z","iopub.status.idle":"2023-01-29T18:45:04.935324Z","shell.execute_reply.started":"2023-01-29T18:45:00.633577Z","shell.execute_reply":"2023-01-29T18:45:04.934190Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Array of sensors(DOM)","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nfrom mpl_toolkits.mplot3d import Axes3D\n\nfig = plt.figure(figsize=(18,12))\nax = fig.add_subplot(111, projection='3d')\nscatter = ax.scatter(geometry.x, geometry.y, geometry.z, c=geometry.sensor_id, cmap='jet', s=1)\n\ncbar = plt.colorbar(scatter)\ncbar.set_label('sensor_id')\n\nax.set_xlabel('x')\nax.set_ylabel('y')\nax.set_zlabel('z')\nax.set_title('4D Data Visualization of IceCube Array')\n\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-01-29T18:45:04.940180Z","iopub.execute_input":"2023-01-29T18:45:04.941063Z","iopub.status.idle":"2023-01-29T18:45:05.523977Z","shell.execute_reply.started":"2023-01-29T18:45:04.941018Z","shell.execute_reply":"2023-01-29T18:45:05.522735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Event Reconstruction by IceCube Array\nThe Neutrion Event is a angular radians pair **(azimuth, zenith)** computed from the data detected by DOMs' Array. The \n`train_meta.parquet` is the well processed and corrected dataset from them. The raw data in size of **100+GB** are the **660 bathes** of historical data about **sensor_id, time, charge** detected by DOMs with **coordiante record in geometry**. \n\nBy viewing **(azimuth, zenith)** in `train_meta.parquet` as correct label in train set, we'd better reconstruct it from scratch that is **Recontructing Event by IceCube Array**. This is enlightened and implemented by [JIRKA BOROVEC](https://www.kaggle.com/code/jirkaborovec/icecube-neutrino-fitting-3d-points-lb-1-25) orginally in this Datathon, Thanks for his dedication. To illustrated it, we select `batch_2.parquet` to rescontruct the Event.","metadata":{}},{"cell_type":"code","source":"train = train_2\n\nevent_ids = list(set(train.index))\n#event_id = 46528394\nevent_id = event_ids[0]\n\ndata = train\nsensors = geometry\nmetadata = df_train_meta\nmeta = dict(metadata[metadata[\"event_id\"]==event_id].iloc[0])","metadata":{"execution":{"iopub.status.busy":"2023-01-29T18:45:05.525931Z","iopub.execute_input":"2023-01-29T18:45:05.527036Z","iopub.status.idle":"2023-01-29T18:45:10.060680Z","shell.execute_reply.started":"2023-01-29T18:45:05.526983Z","shell.execute_reply":"2023-01-29T18:45:10.059243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta","metadata":{"execution":{"iopub.status.busy":"2023-01-29T18:45:10.062571Z","iopub.execute_input":"2023-01-29T18:45:10.064037Z","iopub.status.idle":"2023-01-29T18:45:10.073908Z","shell.execute_reply.started":"2023-01-29T18:45:10.063992Z","shell.execute_reply":"2023-01-29T18:45:10.072475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### meta data from `train_meta.parquet`\nWe select the some event_id in the set of event_ids in batch_2.parquet, the radians pair comes from `train_meta.parquet` then it is correct by definition. The reconstruction introduced by [JIRKA BOROVEC](https://www.kaggle.com/code/jirkaborovec/icecube-neutrino-fitting-3d-points-lb-1-25) is intuitive. **Make linear fitting over the cartesian coordinates of those sensors which detected this event_id. Then extract the direction vector of it and convert it into radians pair.** I modified the coordinate his transformation and add some comments.\nNotice that at this step, only the **direction vector extracted from sensors(DOMs) array** is non-trivial.","metadata":{}},{"cell_type":"code","source":"class coordinateT:\n def cartesian_to_sphere(x, y, z):\n x2y2 = x**2 + y**2\n r = math.sqrt(x2y2 + z**2)\n azimuth = math.acos(x / math.sqrt(x2y2)) * np.sign(y)\n zenith = math.acos(z / r)\n # adjust here directly:\n return coordinateT.adjust_sphere(azimuth, zenith)\n\n def adjust_sphere(azimuth, zenith):\n '''represent the direction vector by direction vector pointed to upper-half sphere and with radians arange in [0:2*pi].\n Take anti-polar point if zenith < 0 which means it pointed lower-half sphere by addition a math.pi to both of them\n for azimuth angular, just consider the case negative, the case > 2*pi excluded by data provider.\n '''\n \n if zenith < 0:\n zenith += math.pi\n azimuth += math.pi\n if azimuth < 0:\n azimuth += math.pi * 2\n azimuth = azimuth % (2 * math.pi)\n return azimuth, zenith\n def sphere_to_cartesian(azimuth, zenith):\n '''\n mapping sphere coordinate to an unit vector representing a direction.\n '''\n x = math.cos(azimuth) * math.sin(zenith)\n y = math.sin(azimuth) * math.sin(zenith)\n z = math.cos(zenith)\n return x, y, z","metadata":{"execution":{"iopub.status.busy":"2023-01-29T18:45:10.075822Z","iopub.execute_input":"2023-01-29T18:45:10.076776Z","iopub.status.idle":"2023-01-29T18:45:10.094192Z","shell.execute_reply.started":"2023-01-29T18:45:10.076717Z","shell.execute_reply":"2023-01-29T18:45:10.092375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### I had computed the direction vector and the origin of the line that fits the array ahead.","metadata":{}},{"cell_type":"code","source":"v_direction = np.array([0.,0.,-1.])\np_origin = np.array([224.58 , 432.35 , 421.41555556])","metadata":{"execution":{"iopub.status.busy":"2023-01-29T18:45:10.095674Z","iopub.execute_input":"2023-01-29T18:45:10.096115Z","iopub.status.idle":"2023-01-29T18:45:10.112485Z","shell.execute_reply.started":"2023-01-29T18:45:10.096072Z","shell.execute_reply":"2023-01-29T18:45:10.110973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nevent = data[data.index==event_id]\nevent = pd.merge(event, sensors, left_on='sensor_id', right_index=True)\n\n\nevent['charge'] /= max(event['charge']) \nevent = event.sort_values('time')\n\n\n\nevt_ = event[~event['auxiliary']]\n\npoints = evt_[[\"x\", \"y\", \"z\"]].values\nazimuth_, zenith_ = coordinateT.cartesian_to_sphere(*(v_direction)) \n\nauxiliaries = [False, True]\n\n\n\nx_, y_, z_, = coordinateT.sphere_to_cartesian(meta['azimuth'], meta['zenith']) \n","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2023-01-29T18:45:10.114427Z","iopub.execute_input":"2023-01-29T18:45:10.115009Z","iopub.status.idle":"2023-01-29T18:45:10.183530Z","shell.execute_reply.started":"2023-01-29T18:45:10.114948Z","shell.execute_reply":"2023-01-29T18:45:10.181649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(nrows=len(auxiliaries), ncols=1, figsize=(22, 22), subplot_kw={'projection': '3d'})\nscale = 512\n\nfor i, aux in enumerate(auxiliaries):\n data_set = event[event['auxiliary'] == aux]\n ax = axes[i]\n source = data_set.iloc[0]\n ax.scatter(geometry.x, geometry.y, geometry.z, s=1, c='green')\n ax.scatter(data_set.x, data_set.y, data_set.z, c = 'red')\n ax.quiver(source.x, source.y, source.z, scale*v_direction[0], scale*v_direction[1], scale*v_direction[2], color='blue', label='fit direction')\n ax.quiver(source.x, source.y, source.z, scale*x_, scale*y_, scale*z_, color='purple', label='fit direction')\n ax.set_title(f\"auxiliary={str(aux)}\")\n ","metadata":{"execution":{"iopub.status.busy":"2023-01-29T18:45:10.187261Z","iopub.execute_input":"2023-01-29T18:45:10.187726Z","iopub.status.idle":"2023-01-29T18:45:10.856286Z","shell.execute_reply.started":"2023-01-29T18:45:10.187683Z","shell.execute_reply":"2023-01-29T18:45:10.855435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Conclusion at this step\n1. As you can see above, those sensors with `auxiliary=False` formed much better cluster to represent direction vector, and `auxiliary=True`......\n2. Line fitting with Cartesian coordinates in Array can not reconstruct the event very well, the other two columns provided by raw data must be keystone to improve the results.\n3. Geometrically Observation: A Virtual Prediction would adjust the **direction(blue arrow)** provided by pure coordinate of DOMs **to the correct direction(purple arrow) based on the other Physical Quantities like charge and time.**","metadata":{}}]}