{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n \"\"

\n \"\"

\n
","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"\n

Table of Content

\n\n* [1. BASIC CONCEPTS](#1)\n\n* [2. IMPORTING LIBRARIES](#2)\n\n* [3. CONFIG](#3) \n\n* [4. LOADING DATASET](#4)","metadata":{}},{"cell_type":"markdown","source":"##
Objectives of Notebook 📌
","metadata":{}},{"cell_type":"markdown","source":"
\n
\n - 🔍Analyze the basic concepts\n
\n - 📊Explore every feature in the dataset;\n
\n - 🗺️Practice using libraries to visualize data; \n
\n - 🐝Analyze the relationship between attributes;\n
\n
","metadata":{}},{"cell_type":"markdown","source":"\n#

Basic concepts

\n\n\n- - -\n
\n \n Base Terms\n \n
\n\n- - - \n\n> __Neutrinos__ are tiny particles that are similar to electrons, but have no electric charge and very little mass. They are produced in the Sun and other stars, as well as in nuclear reactions like those that occur in nuclear power plants.\n\n> __IceCube__ is a scientific observatory located at the South Pole. It is designed to detect neutrinos by using a large detector made of ice. The detector is buried deep in the ice, and it is able to detect the faint flashes of light that are produced when a neutrino interacts with the atoms in the ice.\n\n*So in simple terms, Neutrino is a tiny particle that is produced in the stars and IceCube is a scientific observatory located in Antarctica, which detect the Neutrinos using large detector made of ice.*\n\n_ _ _\n
\n \"\"

\n
\n\n- - -\n
\n \n Brief Definition\n \n
\n\n- - - \n\n> __Neutrino__ A nearly massless and electrically neutral particle that is abundant in the universe and difficult to detect.\n\n\n> __IceCube detector__ A cubic kilometer-sized detector located at the South Pole, designed to search for neutrinos.\n\n\n> __Particle physics__ The branch of physics that studies subatomic particles and their interactions.\n\n\n> __Computational costs__ The resources (e.g. time, energy, and computational power) required to perform a computation.\n\n> __Cosmic neutrino sources__ The origin of neutrinos in the universe.\n\n> __Reconstruction__ The process of estimating the direction of neutrino events.\n\n> __Transient phenomena__ Short-lived events that occur in the universe.\n\n> __IceCube Collaboration__ An international group of scientists responsible for the scientific research at the IceCube Neutrino Observatory.\n\n> __Black holes__ Regions in space where the gravitational pull is so strong that nothing, not even light, can escape.\n\n> __Neutron stars__ Dense, compact stars that are the remnants of supernovae.\n\n> __Gamma-ray bursts__ High-energy explosions that release intense bursts of gamma rays.\n\n> __Real-time analysis__ The process of analyzing data as it is being generated, without any delay.\n\n> __Massless__ Without mass, having a negligible amount of mass.\n\n> __Electrically neutral__ Without an electric charge.\n\n> __Fundamental properties__ The basic and inherent characteristics that define a particle's behavior.\n\n> __Network of telescopes__ A group of telescopes that are interconnected and used for observing celestial objects.\n\n\n\n_ _ _\n
\n \"\"

\n
","metadata":{}},{"cell_type":"markdown","source":"- - -\n
\n \n Goal of the Competition\n \n
\n\n- - - \n\nThe goal of the competition is to predict the direction of neutrino particles, using data from the IceCube detector located at the South Pole. The focus is on developing a model that is both fast and accurate, which will help scientists better understand the universe and improve the reconstruction of neutrinos. The competition aims to find solutions that will allow for more neutrino events to be analyzed, potentially even in real-time, and increase the chance to identify cosmic neutrino sources, leading to a clearer image of the universe.\n\n*In simple terms, it's like a game where people use their knowledge to try to guess where a tiny particle is heading.*","metadata":{}},{"cell_type":"markdown","source":"\n#

Importing Libraries

","metadata":{}},{"cell_type":"code","source":"! pip install plotly==5.11.0","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:22:45.539574Z","iopub.execute_input":"2023-01-30T22:22:45.540059Z","iopub.status.idle":"2023-01-30T22:23:39.645346Z","shell.execute_reply.started":"2023-01-30T22:22:45.539964Z","shell.execute_reply":"2023-01-30T22:23:39.644251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport seaborn as sns\nimport plotly.express as px\nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\nfrom mpl_toolkits.mplot3d import Axes3D\nfrom plotly.offline import init_notebook_mode, iplot","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:23:39.647635Z","iopub.execute_input":"2023-01-30T22:23:39.648038Z","iopub.status.idle":"2023-01-30T22:23:41.464413Z","shell.execute_reply.started":"2023-01-30T22:23:39.648000Z","shell.execute_reply":"2023-01-30T22:23:41.463172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#

CONFIG

","metadata":{}},{"cell_type":"code","source":"class CFG:\n class data:\n path_to_train_folder=\"/kaggle/input/icecube-neutrinos-in-deep-ice/train\"\n path_to_train_meta=\"/kaggle/input/icecube-neutrinos-in-deep-ice/train_meta.parquet\"\n path_to_sensor_geometry=\"/kaggle/input/icecube-neutrinos-in-deep-ice/sensor_geometry.csv\"","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:23:41.466004Z","iopub.execute_input":"2023-01-30T22:23:41.466446Z","iopub.status.idle":"2023-01-30T22:23:41.472986Z","shell.execute_reply.started":"2023-01-30T22:23:41.466412Z","shell.execute_reply":"2023-01-30T22:23:41.471553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#

LOADING DATASET

","metadata":{}},{"cell_type":"markdown","source":"_ _ _\n
\n \n [train/test]_meta.parquet\n \n
\n\n_ _ _\n- batch_id (int)): the ID of the batch the event was placed into.\n- event_id (int): the event ID.\n- first/last_pulse_index (int): index of the first/last row in the features dataframe belonging to this event.\n- azimuth/zenith(float32): the [azimuth/zenith] angle in radians of the neutrino. A value between 0 and 2 x pi for the azimuth and 0 and pi for zenith. \n\n*The target columns. Not provided for the test set. The direction vector represented by zenith and azimuth points to where the neutrino came from.*Azimuth and Zenith are two astronomical terms used to describe the direction of an object in the sky. Azimuth is the angle between an object and the observer's meridian plane, measured clockwise from the observer's North (0°) to East (90°). Zenith, on the other hand, is the angle between the object and the observer's line of sight, measured from the observer's line of sight to the overhead point (90°). These two angles determine the position of an object in the sky relative to an observer's location.","metadata":{}},{"cell_type":"markdown","source":"Azimuth and Zenith are two astronomical terms used to describe the direction of an object in the sky. Azimuth is the angle between an object and the observer's meridian plane, measured clockwise from the observer's North (0°) to East (90°). Zenith, on the other hand, is the angle between the object and the observer's line of sight, measured from the observer's line of sight to the overhead point (90°). These two angles determine the position of an object in the sky relative to an observer's location.","metadata":{}},{"cell_type":"code","source":"train_meta = pd.read_parquet(CFG.data.path_to_train_meta)\n\nprint(f\"train_meta.shape={train_meta.shape}\")\ntrain_meta.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:23:41.476534Z","iopub.execute_input":"2023-01-30T22:23:41.477339Z","iopub.status.idle":"2023-01-30T22:24:35.204769Z","shell.execute_reply.started":"2023-01-30T22:23:41.477297Z","shell.execute_reply":"2023-01-30T22:24:35.203676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n \"\"

\n
\n\n- - - - \n*Azimuth and Zenith are two astronomical terms used to describe the direction of an object in the sky. Azimuth is the angle between an object and the observer's meridian plane, measured clockwise from the observer's North (0°) to East (90°). Zenith, on the other hand, is the angle between the object and the observer's line of sight, measured from the observer's line of sight to the overhead point (90°). These two angles determine the position of an object in the sky relative to an observer's location.*\n_ _ _","metadata":{}},{"cell_type":"code","source":"fig, axs = plt.subplots(1, 2, figsize=(20, 7))\n\naxs[0].hist(train_meta[\"azimuth\"], bins=100)\naxs[0].set_title('Distribution of azimuth values')\n\naxs[1].hist(train_meta[\"zenith\"], bins=100)\naxs[1].set_title('Distribution of zenith values')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:24:35.206066Z","iopub.execute_input":"2023-01-30T22:24:35.206392Z","iopub.status.idle":"2023-01-30T22:24:41.227618Z","shell.execute_reply.started":"2023-01-30T22:24:35.206361Z","shell.execute_reply":"2023-01-30T22:24:41.226474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axs = plt.subplots(1, 2, figsize=(20, 10))\n\naxs[0].hist(train_meta[\"first_pulse_index\"], bins=50, alpha=0.5)\naxs[0].set_xlabel('first_pulse_index')\naxs[0].set_ylabel('Frequency')\naxs[0].set_title('Histogram of first_pulse_index')\n\naxs[1].hist(train_meta[\"last_pulse_index\"], bins=50, alpha=0.5)\naxs[1].set_xlabel('last_pulse_index')\naxs[1].set_ylabel('Frequency')\naxs[1].set_title('Histogram of last_pulse_index')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:24:41.229216Z","iopub.execute_input":"2023-01-30T22:24:41.229540Z","iopub.status.idle":"2023-01-30T22:24:50.825689Z","shell.execute_reply.started":"2023-01-30T22:24:41.229512Z","shell.execute_reply":"2023-01-30T22:24:50.821566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_meta = train_meta.head(1000)\n\nsns.jointplot(x='azimuth', y='event_id', data=train_meta, kind='scatter')\nsns.jointplot(x='azimuth', y='zenith', data=train_meta, kind='hex', color='b')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:24:50.831307Z","iopub.execute_input":"2023-01-30T22:24:50.832513Z","iopub.status.idle":"2023-01-30T22:24:54.009251Z","shell.execute_reply.started":"2023-01-30T22:24:50.832463Z","shell.execute_reply":"2023-01-30T22:24:54.004852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.kdeplot(data=train_meta, x='azimuth', y='zenith')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:24:54.015820Z","iopub.execute_input":"2023-01-30T22:24:54.017877Z","iopub.status.idle":"2023-01-30T22:24:55.752329Z","shell.execute_reply.started":"2023-01-30T22:24:54.017584Z","shell.execute_reply":"2023-01-30T22:24:55.751082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"_ _ _\n
\n \n [train/test]/batch_[n].parquet\n \n
\n\n_ _ _\nEach batch contains tens of thousands of events. Each event may contain thousands of pulses, each of which is the digitized output from a photomultiplier tube and occupies one row.\n\n- event_id (int): the event ID. Saved as the index column in parquet.\n- time (int): the time of the pulse in nanoseconds in the current event time window. The absolute time of a pulse has no relevance, and only the relative time with respect to other pulses within an event is of relevance.\n- sensor_id (int): the ID of which of the 5160 IceCube photomultiplier sensors recorded this pulse.\n- charge (float32): An estimate of the amount of light in the pulse, in units of photoelectrons (p.e.). A physical photon does not exactly result in a measurement of 1 p.e. but rather can take values spread around 1 p.e. As an example, a pulse with charge 2.7 p.e. could quite likely be the result of two or three photons hitting the photomultiplier tube around the same time. This data has float16 precision but is stored as float32 due to limitations of the version of pyarrow the data was prepared with.\n- auxiliary (bool): If True, the pulse was not fully digitized, is of lower quality, and was more likely to originate from noise. If False, then this pulse was contributed to the trigger decision and the pulse was fully digitized.","metadata":{}},{"cell_type":"markdown","source":"_ _ _\n
\n \n What is Parquet?\n \n
\n\n_ _ _\nParquet is an open source file format built to handle flat columnar storage data formats. Parquet operates well with complex data in large volumes.It is known for its both performant data compression and its ability to handle a wide variety of encoding types.\n\nParquet is an open source file format available to any project in the Hadoop ecosystem. Apache Parquet is designed for efficient as well as performant flat columnar storage format of data compared to row based files like CSV or TSV files.\n\nParquet uses the record shredding and assembly algorithm which is superior to simple flattening of nested namespaces. Parquet is optimized to work with complex data in bulk and features different ways for efficient data compression and encoding types. This approach is best especially for those queries that need to read certain columns from a large table. Parquet can only read the needed columns therefore greatly minimizing the IO.","metadata":{}},{"cell_type":"code","source":"train_batch_1 = pd.read_parquet(f\"{CFG.data.path_to_train_folder}/batch_1.parquet\").reset_index()\n\nprint(f\"train_batch_1.shape={train_batch_1.shape}\")\ntrain_batch_1.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:24:55.754271Z","iopub.execute_input":"2023-01-30T22:24:55.754632Z","iopub.status.idle":"2023-01-30T22:25:05.791968Z","shell.execute_reply.started":"2023-01-30T22:24:55.754599Z","shell.execute_reply":"2023-01-30T22:25:05.790694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sensor_counts = train_batch_1['sensor_id'].value_counts()\n\nsns.barplot(x=sensor_counts.index, y=sensor_counts.values)\nplt.xlabel('Sensor ID')\nplt.ylabel('Frequency of Pulses')\nplt.title('Bar plot of the frequency of pulses recorded by each sensor')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:25:05.795380Z","iopub.execute_input":"2023-01-30T22:25:05.795810Z","iopub.status.idle":"2023-01-30T22:26:08.665551Z","shell.execute_reply.started":"2023-01-30T22:25:05.795778Z","shell.execute_reply":"2023-01-30T22:26:08.664674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.scatter(train_batch_1['time'], train_batch_1['charge'])\n\nplt.xlabel('Time (ns)')\nplt.ylabel('Charge (p.e.)')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:26:08.666627Z","iopub.execute_input":"2023-01-30T22:26:08.667158Z","iopub.status.idle":"2023-01-30T22:26:53.547602Z","shell.execute_reply.started":"2023-01-30T22:26:08.667127Z","shell.execute_reply":"2023-01-30T22:26:53.546372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f, ax = plt.subplots(figsize=(11, 9))\n\ncorr = train_batch_1[['time', 'charge', 'sensor_id']].corr()\nsns.heatmap(corr, vmax=.3, center=0,\n square=True, linewidths=.5, cbar_kws={\"shrink\": .5}, annot=True);","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:26:53.549542Z","iopub.execute_input":"2023-01-30T22:26:53.549996Z","iopub.status.idle":"2023-01-30T22:26:57.551314Z","shell.execute_reply.started":"2023-01-30T22:26:53.549953Z","shell.execute_reply":"2023-01-30T22:26:57.550009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist(train_batch_1['charge'], bins=50, edgecolor='k')\nplt.xlabel(\"Charge (p.e.)\")\nplt.ylabel(\"Count\")\nplt.title(\"Histogram of Charge\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:26:57.552846Z","iopub.execute_input":"2023-01-30T22:26:57.553313Z","iopub.status.idle":"2023-01-30T22:26:58.514756Z","shell.execute_reply.started":"2023-01-30T22:26:57.553280Z","shell.execute_reply":"2023-01-30T22:26:58.513730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.scatter(train_batch_1['time'], train_batch_1['charge'], s=1)\nplt.xlabel(\"Time (ns)\")\nplt.ylabel(\"Charge (p.e.)\")\nplt.title(\"Scatter Plot of Time vs Charge\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:26:58.516305Z","iopub.execute_input":"2023-01-30T22:26:58.516861Z","iopub.status.idle":"2023-01-30T22:27:11.266937Z","shell.execute_reply.started":"2023-01-30T22:26:58.516817Z","shell.execute_reply":"2023-01-30T22:27:11.265708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"_ _ _\n
\n \n Browse geometry\n \n
\n\n_ _ _\nThe x, y, and z positions for each of the 5160 IceCube sensors. The row index corresponds to the sensor_idx feature of pulses. The x, y, and z coordinates are in units of meters, with the origin at the center of the IceCube detector. The coordinate system is right-handed, and the z-axis points upwards when standing at the South Pole.","metadata":{}},{"cell_type":"code","source":"sensor_geometry = pd.read_csv(CFG.data.path_to_sensor_geometry)\n\nprint(f\"sensor_geometry.shape={sensor_geometry.shape}\")\nsensor_geometry.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:27:11.268729Z","iopub.execute_input":"2023-01-30T22:27:11.269273Z","iopub.status.idle":"2023-01-30T22:27:11.299782Z","shell.execute_reply.started":"2023-01-30T22:27:11.269226Z","shell.execute_reply":"2023-01-30T22:27:11.298912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.scatter_3d(sensor_geometry, x='x', y='y', z='z', color='z', opacity=0.5)\nfig.update_traces(marker = dict(size = 2, symbol = \"diamond-open\"))\nfig.update_coloraxes(showscale = False)\nfig.update_layout(template = \"plotly_dark\", font = dict(family = \"PT Sans\", size = 12, color = \"#97FFFF\"))\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:27:53.938248Z","iopub.execute_input":"2023-01-30T22:27:53.938691Z","iopub.status.idle":"2023-01-30T22:27:54.040599Z","shell.execute_reply.started":"2023-01-30T22:27:53.938639Z","shell.execute_reply":"2023-01-30T22:27:54.039405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure()\nax = fig.add_subplot(111, projection='3d')\nax.scatter(sensor_geometry['x'], sensor_geometry['y'], sensor_geometry['z'], s=1)\nax.set_xlabel(\"X\")\nax.set_ylabel(\"Y\")\nax.set_zlabel(\"Z\")\nax.set_title(\"3D Scatter Plot of Sensor Positions\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-30T22:27:11.651939Z","iopub.execute_input":"2023-01-30T22:27:11.653076Z","iopub.status.idle":"2023-01-30T22:27:11.904599Z","shell.execute_reply.started":"2023-01-30T22:27:11.653027Z","shell.execute_reply":"2023-01-30T22:27:11.903496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#

Thank you for watching! 🙏

\n\n![](https://pinscreen.com/static/Terry_grey-e9c9adf34700234c4bed2e3da207b1b2.jpg)\n\n
\n \n![](https://pinscreen.com/static/Terry_grey-e9c9adf34700234c4bed2e3da207b1b2.jpg)\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}