{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Goal of the Competition\n","metadata":{}},{"cell_type":"markdown","source":"### TL;DR\n## **The goal of this competition is to predict a neutrino particle’s direction.**\n\n### More details\n## **The goal of this competition is to identify which direction neutrinos detected by the IceCube neutrino observatory came from. When detection events can be localized quickly enough, traditional telescopes are recruited to investigate short-lived neutrino sources such as supernovae or gamma ray bursts. Because the sky is huge better localization will not only associate neutrinos with sources but also to help partner observatories limit their search space. With an average of three thousand events per second to process, it's difficult to keep up with the stream of data using traditional methods. Your challenge in this competition is to quickly and accurately process a large number of events.**","metadata":{}},{"cell_type":"markdown","source":"# Evaluation\n","metadata":{}},{"cell_type":"markdown","source":"## Submissions are evaluated using the mean angular error between the predicted and true event origins. \n\n","metadata":{}},{"cell_type":"code","source":"import numpy as np\n\n\ndef angular_dist_score(az_true, zen_true, az_pred, zen_pred):\n    '''\n    calculate the MAE of the angular distance between two directions.\n    The two vectors are first converted to cartesian unit vectors,\n    and then their scalar product is computed, which is equal to\n    the cosine of the angle between the two vectors. The inverse \n    cosine (arccos) thereof is then the angle between the two input vectors\n    \n    Parameters:\n    -----------\n    \n    az_true : float (or array thereof)\n        true azimuth value(s) in radian\n    zen_true : float (or array thereof)\n        true zenith value(s) in radian\n    az_pred : float (or array thereof)\n        predicted azimuth value(s) in radian\n    zen_pred : float (or array thereof)\n        predicted zenith value(s) in radian\n    \n    Returns:\n    --------\n    \n    dist : float\n        mean over the angular distance(s) in radian\n    '''\n    \n    if not (np.all(np.isfinite(az_true)) and\n            np.all(np.isfinite(zen_true)) and\n            np.all(np.isfinite(az_pred)) and\n            np.all(np.isfinite(zen_pred))):\n        raise ValueError(\"All arguments must be finite\")\n    \n    # pre-compute all sine and cosine values\n    sa1 = np.sin(az_true)\n    ca1 = np.cos(az_true)\n    sz1 = np.sin(zen_true)\n    cz1 = np.cos(zen_true)\n    \n    sa2 = np.sin(az_pred)\n    ca2 = np.cos(az_pred)\n    sz2 = np.sin(zen_pred)\n    cz2 = np.cos(zen_pred)\n    \n    # scalar product of the two cartesian vectors (x = sz*ca, y = sz*sa, z = cz)\n    scalar_prod = sz1*sz2*(ca1*ca2 + sa1*sa2) + (cz1*cz2)\n    \n    # scalar product of two unit vectors is always between -1 and 1, this is against nummerical instability\n    # that might otherwise occure from the finite precision of the sine and cosine functions\n    scalar_prod =  np.clip(scalar_prod, -1, 1)\n    \n    # convert back to an angle (in radian)\n    return np.average(np.abs(np.arccos(scalar_prod)))\n","metadata":{"execution":{"iopub.status.busy":"2023-01-20T14:58:42.813094Z","iopub.execute_input":"2023-01-20T14:58:42.813486Z","iopub.status.idle":"2023-01-20T14:58:42.826990Z","shell.execute_reply.started":"2023-01-20T14:58:42.813455Z","shell.execute_reply":"2023-01-20T14:58:42.825537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### source [angular error calculating](https://www.kaggle.com/code/sohier/mean-angular-error)","metadata":{}},{"cell_type":"markdown","source":"### Simply, given a two vectors **a** and **b**, angular error is just: \n<pre>\narc_cos = arccos(scalar_product(a, b)))\nangular_error = mean(abs(arc_cos))\n<pre>\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np","metadata":{"execution":{"iopub.status.busy":"2023-01-20T15:33:34.223294Z","iopub.execute_input":"2023-01-20T15:33:34.223680Z","iopub.status.idle":"2023-01-20T15:33:34.228743Z","shell.execute_reply.started":"2023-01-20T15:33:34.223650Z","shell.execute_reply":"2023-01-20T15:33:34.227857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Files\n","metadata":{}},{"cell_type":"markdown","source":"**[train/test]_meta.parquet**\n\n1. **batch_id (int)**: the ID of the batch the event was placed into.\n1. **event_id (int)**: the event ID.\n1. **[first/last]_pulse_index (int)**: index of the first/last row in the features dataframe belonging to this event.\n1. **[azimuth/zenith] (float32)**: the [azimuth/zenith] angle in radians of the neutrino. A value between 0 and 2*pi for the azimuth and 0 and pi for zenith. The target columns. Not provided for the test set. The direction vector represented by zenith and azimuth points to where the neutrino came from.\n\n**[train/test]/batch_[n].parquet Each batch contains tens of thousands of events. Each event may contain thousands of pulses, each of which is the digitized output from a photomultiplier tube and occupies one row.**\n\n1. **event_id (int)**: the event ID. Saved as the index column in parquet.\n1. **time (int)**: the time of the pulse in nanoseconds in the current event time window. The absolute time of a pulse has no relevance, and only the relative time with respect to other pulses within an event is of relevance.\n1. **sensor_id (int)**: the ID of which of the 5160 IceCube photomultiplier sensors recorded this pulse.\n1. **charge (float32)**: An estimate of the amount of light in the pulse, in units of photoelectrons (p.e.). A physical photon does not exactly result in a measurement of 1 p.e. but rather can take values spread around 1 p.e. As an example, a pulse with charge 2.7 p.e. could quite likely be the result of two or three photons hitting the photomultiplier tube around the same time. **This data has float16 precision but is stored as float32 due to limitations of the version of pyarrow the data was prepared with.**\n1. **auxiliary (bool)**: If True, the pulse was not fully digitized, is of lower quality, and was more likely to originate from noise. If False, then this pulse was contributed to the trigger decision and the pulse was fully digitized\n\n**sample_submission.parquet**\nAn example submission with the correct columns and properly ordered event IDs. The sample submission is provided in the parquet format so it can be read quickly but your final submission must be a csv.\n\n**sensor_geometry.csv**\nThe x, y, and z positions for each of the 5160 IceCube sensors. The row index corresponds to the sensor_idx feature of pulses. The x, y, and z coordinates are in units of meters, with the origin at the center of the IceCube detector. The coordinate system is right-handed, and the z-axis points upwards when standing at the South Pole. You can convert from these coordinates to azimuth and zenith with the following formulas (here the vector (x,y,z) is normalized):\n<pre>\nx = cos(azimuth) * sin(zenith)\ny = sin(azimuth) * sin(zenith)\nz = cos(zenith)\n<pre>","metadata":{}},{"cell_type":"code","source":"train = pd.read_parquet('/kaggle/input/icecube-neutrinos-in-deep-ice/train/batch_1.parquet')\ntest = pd.read_parquet('/kaggle/input/icecube-neutrinos-in-deep-ice/test/batch_661.parquet') # only one example test, hidden test in this competition\nsample_submission = pd.read_parquet('/kaggle/input/icecube-neutrinos-in-deep-ice/sample_submission.parquet')\nsensor_geometry = pd.read_csv('/kaggle/input/icecube-neutrinos-in-deep-ice/sensor_geometry.csv')\ntest_meta = pd.read_parquet('/kaggle/input/icecube-neutrinos-in-deep-ice/test_meta.parquet')\ntrain_meta = pd.read_parquet('/kaggle/input/icecube-neutrinos-in-deep-ice/train_meta.parquet')","metadata":{"execution":{"iopub.status.busy":"2023-01-20T15:33:40.612476Z","iopub.execute_input":"2023-01-20T15:33:40.612935Z","iopub.status.idle":"2023-01-20T15:33:43.068752Z","shell.execute_reply.started":"2023-01-20T15:33:40.612902Z","shell.execute_reply":"2023-01-20T15:33:43.067726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Our goal is predict **azimuth** and **zenith**","metadata":{}},{"cell_type":"code","source":"train","metadata":{"execution":{"iopub.status.busy":"2023-01-20T15:33:45.821718Z","iopub.execute_input":"2023-01-20T15:33:45.822164Z","iopub.status.idle":"2023-01-20T15:33:45.839630Z","shell.execute_reply.started":"2023-01-20T15:33:45.822132Z","shell.execute_reply":"2023-01-20T15:33:45.838712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\n# Approximately 21e9 rows of train data, amaizing🤯\nlen(os.listdir('/kaggle/input/icecube-neutrinos-in-deep-ice/train')) * train.shape[0]","metadata":{"execution":{"iopub.status.busy":"2023-01-20T15:33:56.126443Z","iopub.execute_input":"2023-01-20T15:33:56.127638Z","iopub.status.idle":"2023-01-20T15:33:56.141263Z","shell.execute_reply.started":"2023-01-20T15:33:56.127569Z","shell.execute_reply":"2023-01-20T15:33:56.139828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reference","metadata":{}},{"cell_type":"markdown","source":"## Useful [post](https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/379546) from [ravi20076](https://www.kaggle.com/ravi20076) to break into Domain information - neutrino, anti-neutrino and spins\n## [Arxiv paper](https://arxiv.org/pdf/2205.15872.pdf) **\"Deep-learning-based reconstruction of the neutrino direction and energy for in-ice radio detectors\"**, 21.09.2022!\n","metadata":{}}]}