{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Let's start with Import libraries","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-02-27T17:23:33.127480Z","iopub.execute_input":"2023-02-27T17:23:33.128069Z","iopub.status.idle":"2023-02-27T17:23:33.163177Z","shell.execute_reply.started":"2023-02-27T17:23:33.127946Z","shell.execute_reply":"2023-02-27T17:23:33.161629Z"}}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport os\n\n# Magic function that will make your plot outputs appear and be stored within the notebook\n%matplotlib inline\n\n# Function used to to render higher resolution images\n%config InlineBackend.figure_format = 'retina'\n\n# Ignore all warnings\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n# Data manipulation\n#import pandas as pd\n#import numpy as np\n#import os # its a important module for programming language that manages computer's memory, all software and hardware,\n# also it is used to script your files while sharing documnets.\n\npd.set_option('display.max_columns', None)\npd.set_option('display.expand_frame_repr', False)\n\n# Data visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly\nplotly.offline.init_notebook_mode()\nimport plotly.express as px\nimport plotly.graph_objects as go\n\n# Standardizing the style for the visualizations \nsns.set_theme()\nsns.set(font_scale=1.2)\nsns.set_palette(\"pastel\")\nplt.style.use('seaborn-whitegrid')\n\n# Singular Vector Decomposition\nfrom sklearn.decomposition import TruncatedSVD\n\n#! pip install -q scikit-spatial --no-index -f /kaggle/input/icecube-neutrino-eda-3d-interactive-viewer/frozen-packages/\n\n# Mathematical Calculation \nimport math","metadata":{"execution":{"iopub.status.busy":"2023-03-03T14:59:28.464820Z","iopub.execute_input":"2023-03-03T14:59:28.465509Z","iopub.status.idle":"2023-03-03T14:59:28.574336Z","shell.execute_reply.started":"2023-03-03T14:59:28.465454Z","shell.execute_reply":"2023-03-03T14:59:28.571331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Firstly let's summarize what we actually going to work","metadata":{}},{"cell_type":"markdown","source":"**I am a student of Data Science and I have participated in this competition to increase my knowledge in Machine Learning**, \n\n**For the Refrence, I am using **: https://www.kaggle.com/code/solverworld/icecube-neutrino-path-least-squares-1-214\nand this : https://www.kaggle.com/code/rasmusrse/graphnet-baseline-submission/notebook\n","metadata":{}},{"cell_type":"markdown","source":"**Lets understand what is Neutrino particle, and Why it is most important for ICE CUBE NEUTRINO OBSERVATORY ?** \n\n**NEUTRINO:** \n* It is a sub-atomic particle similar to the electron but having no charge and having very less mass which might be even zero.\n* Neutrinos are the most abundant particle in the universe. Because they have very little interaction with matter, however, they are incredibly difficult to detect.\n* When two or more big bodies are intract to each other, they emits large amount of energy in the form of photons, electrons and electron-neutrinos. \n\n\n**Neutrons came out from the interaction of two or more celestial bodies solve lots of our questions:**\n1. What are the masses of the various neutrinos? \n2. How do they affect Big Bang cosmology? \n3. Do neutrinos oscillate? Or can neutrinos of one type change into another type as they travel through matter and space? \n4. Are neutrinos fundamentally distinct from their anti-particles? \n5. How do stars collapse and form supernovae? \n6. What is the role of the neutrino in cosmology?\n\n**What actually we need to find in this case studyt?**\n* we need to find out azimuth, zenith values by use of given data_set.\n\n**AZIMUTH ANGLE:** \n* The compass direction from which the sunlight is coming. \n\n**ZENITH ANGLE:**\n* The zenith angle is the angle between the sun and the vertical. The zenith angle is similar to the elevation angle but it is measured from the vertical rather than from the horizontal, thus making the zenith angle = 90° - elevation.\n\n\n\n\n> In Ice Cube Neutrino Observatory lab, Neutrinos are detected by use of many Digital Optical Module (DOM) that are placed within a square kilometre of Antarctic ice is an array of 3600 Digital Optical Modules. \nThe DOMs are held on \"strings\" of sixty modules each at depths ranging from 1,450 to 2,450 meters, inserted into holes melted in the ice by a hot water drill.\n\n#### Note: Neutrinos are similar to photons having no electric charge but photon of light unable to travel in any object or body, whereas, Neutrinos can able to travel from any objects or body upto 2000 to 3000 meter approx.\n\n> So, Neutrinos travel through a very pure and transparent ‘lens’ of Antarctic ice, in which there are no other signals to confuse the detected radiation. The radiation is detected by photomultiplier tubes within the DOMs and the signal sent by cable to the surface for analysis.\n\nFor Refrence:\n* https://collection.sciencemuseumgroup.org.uk/objects/co8234420/digital-optical-module-dom-from-the-icecube-neutrino-detector-digital-optical-module\n* https://www.pveducation.org/pvcdrom/properties-of-sunlight/elevation-angle#:~:text=The%20zenith%20angle%20is%20the,angle%20%3D%2090%C2%B0%20%2D%20elevation.\n* https://www.scientificamerican.com/article/what-is-a-neutrino/\n* https://www.kaggle.com/code/diegoasuarezg/icecube-neutrino-trajectory-3d-projection","metadata":{}},{"cell_type":"markdown","source":"# Let's import and understand dataset","metadata":{"execution":{"iopub.status.busy":"2023-02-07T12:53:57.083481Z","iopub.execute_input":"2023-02-07T12:53:57.083915Z","iopub.status.idle":"2023-02-07T12:53:57.089266Z","shell.execute_reply.started":"2023-02-07T12:53:57.083882Z","shell.execute_reply":"2023-02-07T12:53:57.088313Z"}}},{"cell_type":"code","source":"# Principal Path\nPATH_DATASET = \"/kaggle/input/icecube-neutrinos-in-deep-ice\"\nDATA_DIRECTORY = PATH_DATASET","metadata":{"execution":{"iopub.status.busy":"2023-03-03T14:59:28.577676Z","iopub.execute_input":"2023-03-03T14:59:28.578117Z","iopub.status.idle":"2023-03-03T14:59:28.583844Z","shell.execute_reply.started":"2023-03-03T14:59:28.578080Z","shell.execute_reply":"2023-03-03T14:59:28.582589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Meta Training Data","metadata":{}},{"cell_type":"code","source":"# Meta train dataset path\ntrain_meta_p=\"train_meta.parquet\"","metadata":{"execution":{"iopub.status.busy":"2023-03-03T14:59:28.585562Z","iopub.execute_input":"2023-03-03T14:59:28.585975Z","iopub.status.idle":"2023-03-03T14:59:28.599354Z","shell.execute_reply.started":"2023-03-03T14:59:28.585937Z","shell.execute_reply":"2023-03-03T14:59:28.597898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The meta data contains the following features:\n\n[train/test]_meta.parquet\n\n* batch_id (int): the ID of the batch the event was placed into.\n\n* event_id (int): the event ID.\n\n* [first/last]_pulse_index (int): index of the first/last row in the features dataframe belonging to this event.\n\n* [azimuth/zenith] (float32): the [azimuth/zenith] angle in radians of the neutrino. A value between 0 and 2*pi for the azimuth and 0 and pi for zenith. The target columns. Not provided for the test set. The direction vector represented by zenith and azimuth points to where the neutrino came from.","metadata":{}},{"cell_type":"markdown","source":"We proceed to read the training meta parquet data file.","metadata":{}},{"cell_type":"code","source":"# Reading the meta parquet file \ntrain_meta=pd.read_parquet(os.path.join(DATA_DIRECTORY,train_meta_p))","metadata":{"execution":{"iopub.status.busy":"2023-03-03T14:59:28.601095Z","iopub.execute_input":"2023-03-03T14:59:28.601555Z","iopub.status.idle":"2023-03-03T15:00:15.068723Z","shell.execute_reply.started":"2023-03-03T14:59:28.601520Z","shell.execute_reply":"2023-03-03T15:00:15.066339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets take a look into train meta data","metadata":{}},{"cell_type":"code","source":"# Train meta dataset sneak peek\ntrain_meta.head(3).append(train_meta.tail(3))","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:15.073276Z","iopub.execute_input":"2023-03-03T15:00:15.074217Z","iopub.status.idle":"2023-03-03T15:00:15.112546Z","shell.execute_reply.started":"2023-03-03T15:00:15.074144Z","shell.execute_reply":"2023-03-03T15:00:15.110607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It appears that we have total 131953923 instances in train meta data, lets confirm it by len() function.","metadata":{}},{"cell_type":"code","source":"len(train_meta)","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:15.114779Z","iopub.execute_input":"2023-03-03T15:00:15.115417Z","iopub.status.idle":"2023-03-03T15:00:15.125437Z","shell.execute_reply.started":"2023-03-03T15:00:15.115359Z","shell.execute_reply":"2023-03-03T15:00:15.124371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Batch Data","metadata":{}},{"cell_type":"code","source":"# Train batch data paths \ntrain_batch1_p = \"train/batch_1.parquet\"","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:15.127115Z","iopub.execute_input":"2023-03-03T15:00:15.127629Z","iopub.status.idle":"2023-03-03T15:00:15.145664Z","shell.execute_reply.started":"2023-03-03T15:00:15.127584Z","shell.execute_reply":"2023-03-03T15:00:15.143604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's take a look at what features are inside the batch_[n] parquet data file:\n\n\n**[train/test]/batch_[n].parquet** - Each batch contains tens of thousands of events. Each event may contain thousands of pulses, each of which is the digitized output from a photomultiplier tube and occupies one row.\n\n* **event_id (int):** the event ID. Saved as the index column in parquet.\n\n* **time (int):** the time of the pulse in nanoseconds in the current event time window. The absolute time of a pulse has no relevance, and only the relative time with respect to other pulses within an event is of relevance.\n\n* **sensor_id (int):** the ID of which of the 5160 IceCube photomultiplier sensors recorded this pulse.\n\n* **charge (float32):** An estimate of the amount of light in the pulse, in units of photoelectrons (p.e.). A physical photon does not exactly result in a measurement of 1 p.e. but rather can take values spread around 1 p.e. As an example, a pulse with charge 2.7 p.e. could quite likely be the result of two or three photons hitting the photomultiplier tube around the same time. This data has float16 precision but is stored as float32 due to limitations of the version of pyarrow the data was prepared with.\n\n* **auxiliary (bool):** If True, the pulse was not fully digitized, is of lower quality, and was more likely to originate from noise. If False, then this pulse was contributed to the trigger decision and the pulse was fully digitized.\n\n\n\nAccording to the meta data we have 660 batch files to choose from. Let's choose number a random number like perhabs batch #1, #10 , #100.\n\n\n\nLet's now read the batch file.","metadata":{}},{"cell_type":"code","source":"# Reading batch file\ntrain_batch1 = pd.read_parquet(os.path.join(DATA_DIRECTORY,train_batch1_p))","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:15.148909Z","iopub.execute_input":"2023-03-03T15:00:15.149719Z","iopub.status.idle":"2023-03-03T15:00:18.695333Z","shell.execute_reply.started":"2023-03-03T15:00:15.149660Z","shell.execute_reply":"2023-03-03T15:00:18.693756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's inspect batch data.","metadata":{}},{"cell_type":"code","source":"# Showing first and last instances of batch #1\ntrain_batch1.head(3).append(train_batch1.tail(3))","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:18.698027Z","iopub.execute_input":"2023-03-03T15:00:18.698676Z","iopub.status.idle":"2023-03-03T15:00:18.725150Z","shell.execute_reply.started":"2023-03-03T15:00:18.698621Z","shell.execute_reply":"2023-03-03T15:00:18.723550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's import the sensor geometry csv file.\n\n**sensor_geometry.csv** - The x, y, and z positions for each of the 5160 IceCube sensors. The row index corresponds to the sensor_idx feature of pulses. The x, y, and z coordinates are in units of meters, with the origin at the center of the IceCube detector. The coordinate system is right-handed, and the z-axis points upwards when standing at the South Pole. You can convert from these coordinates to azimuth and zenith with the following formulas (here the vector (x,y,z) is normalized):\n\nx = cos(azimuth) sin(zenith) y = sin(azimuth) sin(zenith) z = cos(zenith)\n\nWe start by reading the geometry data file","metadata":{}},{"cell_type":"code","source":"#Read dataset\ngeometry = pd.read_csv(os.path.join(DATA_DIRECTORY, \"sensor_geometry.csv\"))\ngeometry.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:18.727478Z","iopub.execute_input":"2023-03-03T15:00:18.728023Z","iopub.status.idle":"2023-03-03T15:00:18.768112Z","shell.execute_reply.started":"2023-03-03T15:00:18.727970Z","shell.execute_reply":"2023-03-03T15:00:18.765126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# need to set index \ngeometry.set_index('sensor_id', inplace = True)","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:18.770181Z","iopub.execute_input":"2023-03-03T15:00:18.771729Z","iopub.status.idle":"2023-03-03T15:00:18.783410Z","shell.execute_reply.started":"2023-03-03T15:00:18.771657Z","shell.execute_reply":"2023-03-03T15:00:18.781456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before doing a projection of the 3D space reflecting the DOM's position, let's have a look at the first and last instances of the dataset.","metadata":{}},{"cell_type":"code","source":"geometry.head().append(geometry.tail(3))","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:18.786116Z","iopub.execute_input":"2023-03-03T15:00:18.787259Z","iopub.status.idle":"2023-03-03T15:00:18.813664Z","shell.execute_reply.started":"2023-03-03T15:00:18.787171Z","shell.execute_reply":"2023-03-03T15:00:18.811619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"geometry.info()","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:18.816749Z","iopub.execute_input":"2023-03-03T15:00:18.817945Z","iopub.status.idle":"2023-03-03T15:00:18.843286Z","shell.execute_reply.started":"2023-03-03T15:00:18.817876Z","shell.execute_reply":"2023-03-03T15:00:18.841377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here I create a dataframe containing the limit of the projection space.","metadata":{}},{"cell_type":"code","source":"geometry.reset_index(inplace = True)","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:18.849512Z","iopub.execute_input":"2023-03-03T15:00:18.850004Z","iopub.status.idle":"2023-03-03T15:00:18.861240Z","shell.execute_reply.started":"2023-03-03T15:00:18.849965Z","shell.execute_reply":"2023-03-03T15:00:18.859128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"geometry.head().append(geometry.tail(3))","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:18.863525Z","iopub.execute_input":"2023-03-03T15:00:18.864047Z","iopub.status.idle":"2023-03-03T15:00:18.889776Z","shell.execute_reply.started":"2023-03-03T15:00:18.864006Z","shell.execute_reply":"2023-03-03T15:00:18.888316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Projection space limits\nindex=[\"x-min\",\"x-max\",\"y-min\",\"y-max\",\"z-min\",\"z-max\"]\n\ndata= {\n    'Limits': [geometry.x.min(),geometry.x.max(),\n              geometry.y.min(),geometry.y.max(),\n              geometry.z.min(),geometry.z.max()]\n}\n\ngeometry_limits=pd.DataFrame(data=data,index=index)\n\ngeometry_limits.T","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:18.891995Z","iopub.execute_input":"2023-03-03T15:00:18.892588Z","iopub.status.idle":"2023-03-03T15:00:18.917441Z","shell.execute_reply.started":"2023-03-03T15:00:18.892527Z","shell.execute_reply":"2023-03-03T15:00:18.916303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Plotting a scatter plot that show top view layout of Digital Optical Module (DOM) Sensors location.","metadata":{}},{"cell_type":"code","source":"# 2D view of the DOM Sensor distribution\n\nfig, ax =plt.subplots(figsize=(8,8))\nsns.despine()\nax.tick_params(axis='x', labelrotation=0)\nax.tick_params(axis='y', labelrotation=0)\nax.set(xlabel='x axis', ylabel='y axis')\nax.set_title('DOM Sensor Location Geometry in 2D', size=20)\nsns.scatterplot(x='x',y='y',hue=\"sensor_id\",data=geometry,marker='s',s=100,palette='plasma',legend=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:18.919626Z","iopub.execute_input":"2023-03-03T15:00:18.920454Z","iopub.status.idle":"2023-03-03T15:00:19.407831Z","shell.execute_reply.started":"2023-03-03T15:00:18.920400Z","shell.execute_reply":"2023-03-03T15:00:19.406344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"3D projection of the Sensors Layout.","metadata":{}},{"cell_type":"code","source":"# 3D view of the DOM Sensor distribution\n\nfig = px.scatter_3d(geometry, x='x', y='y', z='z', opacity=0.6, color=\"sensor_id\",title=\"DOM Sensor Location Geometry in 3D\")\nfig.update_traces(marker_size=2)\nfig.update_layout(height=800, width=800)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:19.409052Z","iopub.execute_input":"2023-03-03T15:00:19.409447Z","iopub.status.idle":"2023-03-03T15:00:21.025448Z","shell.execute_reply.started":"2023-03-03T15:00:19.409409Z","shell.execute_reply":"2023-03-03T15:00:21.023997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now lets plot Azimuth and Zenith histogram and box plot to see the better arrangement of datas.","metadata":{}},{"cell_type":"markdown","source":"**Note: Points given in the x axis and y axis are in Radian. \n(conversion: 1 Radian = 57.2958 Degree)**","metadata":{}},{"cell_type":"code","source":"def get_pulse_coor(event_num):\n    \n    # get event index in the meta dataset\n    event_idx=int(\"\".join(str(i) for i in(train_meta[train_meta.event_id==event_num].index.tolist())))\n    \n    # slice training batch dataset to include only the instances from the selected event\n    event_pulses = train_batch1.iloc[train_meta.iloc[event_idx].first_pulse_index.astype(int) : \n                                         train_meta.iloc[event_idx].last_pulse_index.astype(int)+1].copy()\n    \n    # reset back indexes to dataframe \n    event_pulses = event_pulses.reset_index()\n    \n    # map sensor location (x, y, z) to corresponding sensor id \n    event_pulses[['x', 'y', 'z']] = geometry.loc[event_pulses.sensor_id].loc[:, ['x', 'y', 'z']].values\n\n    \n    #centering the data points for SVD\n    event_pulses[['x', 'y', 'z']] -= event_pulses[['x', 'y', 'z']].mean()\n\n    return event_pulses","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:21.027298Z","iopub.execute_input":"2023-03-03T15:00:21.027755Z","iopub.status.idle":"2023-03-03T15:00:21.037881Z","shell.execute_reply.started":"2023-03-03T15:00:21.027715Z","shell.execute_reply":"2023-03-03T15:00:21.036380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Function to get target vector coordinates.","metadata":{}},{"cell_type":"code","source":"def target_vector(event_num):\n    \n    # get event index in the meta dataset\n    event_idx=event_idx=int(\"\".join(str(i) for i in(train_meta[train_meta.event_id==event_num].index.tolist())))\n    \n    # get neutrino true trajectory from the meta dataset\n    zenith_target = train_meta.loc[event_idx, \"zenith\"]\n    azimuth_target = train_meta.loc[event_idx, \"azimuth\"]\n    \n    # transform azimuth and zenith in to coordinates\n    vector_target = [np.cos(azimuth_target) * np.sin(zenith_target), \n                     np.sin(azimuth_target) * np.sin(zenith_target),\n                     np.cos(zenith_target)]\n    \n    # multiply target vector by base vector (maginutude 500 and -500) for visualization purpose\n    vector_base = np.array([-500, 500])\n    x = vector_base*vector_target[0]\n    y = vector_base*vector_target[1]\n    z = vector_base*vector_target[2]\n    \n    # Covert the target vector to dataframe \n    vector_target_df = pd.DataFrame({'x': x, 'y': y, 'z': z})\n    \n    return vector_target_df","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:21.040373Z","iopub.execute_input":"2023-03-03T15:00:21.041072Z","iopub.status.idle":"2023-03-03T15:00:21.061580Z","shell.execute_reply.started":"2023-03-03T15:00:21.041021Z","shell.execute_reply":"2023-03-03T15:00:21.059369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Function to fit pulses coordinates to SVD and get the predicted vector dataframe. This function will be mainly used to visualize the fitted vector in the 3D scatter plot.**","metadata":{}},{"cell_type":"code","source":"def fit_vector(event_pulses):\n    \n    # Instantiate SVD and fit with respective \n    svd = TruncatedSVD(n_components=1).fit(event_pulses[[\"x\",\"y\",\"z\"]])\n    \n    # Decompose SVD components \n    vector= svd.components_[0]\n    \n    # multiply the fit vector by base vector (maginutude 500 and -500) for visualization purpose\n    vector_base = np.array([-500, 500])\n    x = vector_base*vector[0]\n    y = vector_base*vector[1]\n    z = vector_base*vector[2]\n    \n    # Covert the fit vector to dataframe \n    vector_df = pd.DataFrame({'x': x, 'y': y, 'z': z})\n    \n    return vector_df","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:21.063566Z","iopub.execute_input":"2023-03-03T15:00:21.064207Z","iopub.status.idle":"2023-03-03T15:00:21.077866Z","shell.execute_reply.started":"2023-03-03T15:00:21.064156Z","shell.execute_reply":"2023-03-03T15:00:21.075489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The next function allows us to get the best fitted line(vector in this case) regarding the x,y,z coordinates and then convert it to Azimuth and Zenith so we can calculate the error between the target and fitted vector.**","metadata":{}},{"cell_type":"code","source":"def plot_event(event_pulses,vector,target_vector,auxiliary):\n    \n    # 3D scaterplot of auxiliary pulses \n    fig_auxiliary = px.scatter_3d(event_pulses.loc[event_pulses.auxiliary],\n                              x='x', y='y', z='z', opacity=0.5, color_discrete_sequence=['red'],size='charge')\n    \n    # 3D scaterplot of non-auxiliary pulses \n    fig_non_auxiliary = px.scatter_3d(event_pulses.loc[~event_pulses.auxiliary],\n                          x='x', y='y', z='z', opacity=0.5, color_discrete_sequence=['blue'],size='charge')\n    \n    # Target neutrino trajectory\n    fig_line_target = px.line_3d(target_vector, x=\"x\", y=\"y\", z=\"z\")\n    \n    # Predicted neutrino trajectory (best fit auxiliary = True)\n    fig_line_svd= px.line_3d(vector, x=\"x\", y=\"y\", z=\"z\",color_discrete_sequence=['magenta'])\n    \n    # Plot auxiliary pulses\n    if auxiliary == True:\n    \n        fig = go.Figure(data = fig_auxiliary.data + fig_non_auxiliary.data + fig_line_svd.data +  fig_line_target.data)\n        \n    \n        return fig \n    \n    # Remove auxiliary pulses \n    else:\n        \n        fig = go.Figure(data = fig_non_auxiliary.data + fig_line_svd.data +  fig_line_target.data)\n        \n        return fig ","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:21.079683Z","iopub.execute_input":"2023-03-03T15:00:21.080162Z","iopub.status.idle":"2023-03-03T15:00:21.100078Z","shell.execute_reply.started":"2023-03-03T15:00:21.080123Z","shell.execute_reply":"2023-03-03T15:00:21.098386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Claculations part","metadata":{}},{"cell_type":"code","source":"# Calculating target vector for event #3266196 from train_batch1\ntv =target_vector(3266196)\ntv","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:21.102032Z","iopub.execute_input":"2023-03-03T15:00:21.103052Z","iopub.status.idle":"2023-03-03T15:00:22.056625Z","shell.execute_reply.started":"2023-03-03T15:00:21.102975Z","shell.execute_reply":"2023-03-03T15:00:22.055188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Obtaining pulse coordinates \nfit_coor = get_pulse_coor(3266196)\nfit_coor","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:22.058355Z","iopub.execute_input":"2023-03-03T15:00:22.058829Z","iopub.status.idle":"2023-03-03T15:00:22.371396Z","shell.execute_reply.started":"2023-03-03T15:00:22.058791Z","shell.execute_reply":"2023-03-03T15:00:22.370064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculating best fit line\nv_aux=fit_vector(fit_coor)\nv_aux","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:22.373682Z","iopub.execute_input":"2023-03-03T15:00:22.375777Z","iopub.status.idle":"2023-03-03T15:00:22.414527Z","shell.execute_reply.started":"2023-03-03T15:00:22.375712Z","shell.execute_reply":"2023-03-03T15:00:22.412978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plotting event # 100867570 auxiliary = True \n#Magenta line is predicted trajectory / Blue line is true trajectory\n\nplot_event(fit_coor,v_aux,tv,True)","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:22.416797Z","iopub.execute_input":"2023-03-03T15:00:22.418435Z","iopub.status.idle":"2023-03-03T15:00:22.721412Z","shell.execute_reply.started":"2023-03-03T15:00:22.418369Z","shell.execute_reply":"2023-03-03T15:00:22.720086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Pulse coordinates auxiliary \nno_aux=fit_coor.loc[~fit_coor.auxiliary]\nno_aux","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:22.723035Z","iopub.execute_input":"2023-03-03T15:00:22.723456Z","iopub.status.idle":"2023-03-03T15:00:22.752329Z","shell.execute_reply.started":"2023-03-03T15:00:22.723417Z","shell.execute_reply":"2023-03-03T15:00:22.750706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculating best fit line\nv_non_aux=fit_vector(no_aux)\nv_non_aux","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:22.754149Z","iopub.execute_input":"2023-03-03T15:00:22.754628Z","iopub.status.idle":"2023-03-03T15:00:22.796516Z","shell.execute_reply.started":"2023-03-03T15:00:22.754588Z","shell.execute_reply":"2023-03-03T15:00:22.793798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plotting event # 100867570 auxiliary = False\n#Magenta line is predicted trajectory / Blue line is true trajectory\n\nplot_event(fit_coor,v_non_aux,tv,False)","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:22.801897Z","iopub.execute_input":"2023-03-03T15:00:22.803533Z","iopub.status.idle":"2023-03-03T15:00:23.114618Z","shell.execute_reply.started":"2023-03-03T15:00:22.803441Z","shell.execute_reply":"2023-03-03T15:00:23.113269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Final Calculation with Test Data**","metadata":{}},{"cell_type":"code","source":"def cartesian_to_sphere_A(f):\n    x = f[0]\n    y = f[1]\n    z = f[2]\n    x2y2 = x**2 + y**2\n    azimuth = math.acos(x / math.sqrt(x2y2)) * np.sign(y)\n    return azimuth\n\ndef cartesian_to_sphere_Z(f):\n    x = f[0]\n    y = f[1]\n    z = f[2]\n    x2y2 = x**2 + y**2\n    r = math.sqrt(x2y2 + z**2)\n    zenith = math.acos(z / r)\n    return zenith\n\ndef adjust_sphere_A(a):\n    azimuth = a[0]\n    zenith = a[1]\n    if zenith < 0:\n        azimuth += math.pi\n    if azimuth < 0:\n        azimuth += math.pi * 2\n    azimuth = azimuth % (2 * math.pi)\n    return azimuth\n\ndef adjust_sphere_Z(a):\n    azimuth = a[0]\n    zenith = a[1]\n    if zenith < 0:\n        zenith += math.pi\n    return zenith","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:23.116399Z","iopub.execute_input":"2023-03-03T15:00:23.117565Z","iopub.status.idle":"2023-03-03T15:00:23.128479Z","shell.execute_reply.started":"2023-03-03T15:00:23.117511Z","shell.execute_reply":"2023-03-03T15:00:23.127368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Instead of applying functions to the entire train batch files, we are directly applying function to test batch file, because batch file size are too large to compute.**","metadata":{}},{"cell_type":"code","source":"test_d = pd.read_parquet(os.path.join(DATA_DIRECTORY, \"/kaggle/input/icecube-neutrinos-in-deep-ice/test/batch_661.parquet\"))\ntest_d.reset_index(inplace = True)\ntest_d.head(3).append(test_d.tail(3))","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:23.130248Z","iopub.execute_input":"2023-03-03T15:00:23.131342Z","iopub.status.idle":"2023-03-03T15:00:23.171928Z","shell.execute_reply.started":"2023-03-03T15:00:23.131294Z","shell.execute_reply":"2023-03-03T15:00:23.170432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for test_data in test_d.groupby(\"event_id\"):\n    test_data = test_d[~test_d['auxiliary']]\n    test_data = test_data.merge(geometry, left_on=\"sensor_id\", right_index=True)\n\ntest_data['Cs_Azimuth']=test_data[['x','y','z']].apply(lambda f: cartesian_to_sphere_A(f),axis = 1)\ntest_data['Cs_Zenith']=test_data[['x','y','z']].apply(lambda f: cartesian_to_sphere_Z(f),axis = 1)\ntest_data['Azimuth']=test_data[['Cs_Azimuth','Cs_Zenith']].apply(lambda a: adjust_sphere_A(a),axis = 1)\ntest_data['Zenith']=test_data[['Cs_Azimuth','Cs_Zenith']].apply(lambda a: adjust_sphere_Z(a),axis = 1)\ntest_data.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:23.176335Z","iopub.execute_input":"2023-03-03T15:00:23.176797Z","iopub.status.idle":"2023-03-03T15:00:23.244646Z","shell.execute_reply.started":"2023-03-03T15:00:23.176758Z","shell.execute_reply":"2023-03-03T15:00:23.243461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"we need only (event_id, Azimuth, Zenith) Columns in the submission file.","metadata":{}},{"cell_type":"code","source":"test_data.columns","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:00:23.245982Z","iopub.execute_input":"2023-03-03T15:00:23.246357Z","iopub.status.idle":"2023-03-03T15:00:23.255100Z","shell.execute_reply.started":"2023-03-03T15:00:23.246323Z","shell.execute_reply":"2023-03-03T15:00:23.253949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_pre = test_data.drop(['sensor_id', 'sensor_id_x','time','charge','auxiliary','sensor_id_y','x','y','z','Cs_Azimuth','Cs_Zenith'], axis=1)","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:02:25.821698Z","iopub.execute_input":"2023-03-03T15:02:25.824578Z","iopub.status.idle":"2023-03-03T15:02:25.837810Z","shell.execute_reply.started":"2023-03-03T15:02:25.824514Z","shell.execute_reply":"2023-03-03T15:02:25.835779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_pre.event_id.unique()","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:02:33.324609Z","iopub.execute_input":"2023-03-03T15:02:33.325652Z","iopub.status.idle":"2023-03-03T15:02:33.337027Z","shell.execute_reply.started":"2023-03-03T15:02:33.325590Z","shell.execute_reply":"2023-03-03T15:02:33.335466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[2092, 7344, 9482] these are the unique event_id in the dataframe.","metadata":{}},{"cell_type":"code","source":"submission_pre","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:02:44.645490Z","iopub.execute_input":"2023-03-03T15:02:44.645958Z","iopub.status.idle":"2023-03-03T15:02:44.664940Z","shell.execute_reply.started":"2023-03-03T15:02:44.645922Z","shell.execute_reply":"2023-03-03T15:02:44.663348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Final Submission","metadata":{}},{"cell_type":"code","source":"submission_pre.fillna(0).to_csv('submission.csv', index=True)\n\n!head submission.csv","metadata":{"execution":{"iopub.status.busy":"2023-03-03T15:03:26.164991Z","iopub.execute_input":"2023-03-03T15:03:26.165502Z","iopub.status.idle":"2023-03-03T15:03:27.500713Z","shell.execute_reply.started":"2023-03-03T15:03:26.165465Z","shell.execute_reply":"2023-03-03T15:03:27.499014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Afterword\n\nEven if we drop the auxiliary pulses for this particular event the best fitted 3D line had quite noticeable discrepancy from the target trajectory. We could try changing the type of model to a more complex one plus input some constraints on the data we feed to the model.\n\nObviously to evaluate the true potential of this method we will have to train it on all the batches and calculate the respective error with the meta data.\n\nFeel free to use this notebook and try with other events and see how SVD performs.","metadata":{"execution":{"iopub.status.busy":"2023-02-26T20:06:56.937291Z","iopub.execute_input":"2023-02-26T20:06:56.937811Z","iopub.status.idle":"2023-02-26T20:06:56.946379Z","shell.execute_reply.started":"2023-02-26T20:06:56.937775Z","shell.execute_reply":"2023-02-26T20:06:56.944841Z"}}}]}