{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

❄️IceCube Neutrinos - Domain & EDA for DS folks

","metadata":{}},{"cell_type":"markdown","source":"\n

Table of Contents

\n\n* [1. Introduction](#1)\n \n * [1.1 What is this notebook about and why do I do that](#1.1)\n \n * [1.2 Sources of information and inspiration](#1.2)\n\n* [2. Domain information](#2)\n\n * [2.1 What is neutrino](#2.1)\n \n * [2.2 What is IceCube laboratory](#2.2)\n\n* [3. Data exploration](#3)\n \n * [3.1 Explore sensors](#3.1)\n \n * [3.2 Explore train data](#3.2)\n \n * [3.3 Explore metadata](#3.3)\n\n* [4. Break down an event](#4)","metadata":{}},{"cell_type":"markdown","source":"\n##

1. Introduction

","metadata":{}},{"cell_type":"markdown","source":"\n##

1.1 What is this notebook about and why do I do that

","metadata":{}},{"cell_type":"markdown","source":"First of all, I do this EDA for myself, trying to understand the domain and the task of this competition. I'm really hopeful, that my exploration of this data would be useful for you too.\n\nI'm a data scientist, and I don't have a degree in physics. This topic is new for me, and here I just try to put things together and understand the domain of this competition. So, if you will notice some mistakes in my work, feel free to comment!","metadata":{}},{"cell_type":"markdown","source":"Also, I want to join the Kaggle family and practice my EDA-building skills on new datasets. So I hope, that my findings will be helpful to someone😊","metadata":{}},{"cell_type":"markdown","source":"\n##

1.2 Sources of information and inspiration

","metadata":{}},{"cell_type":"markdown","source":"I used these sources to make a better understanding of this domain:\n\n- [Chasing the Ghost Particle: From the South Pole to the Edge of the Universe](https://www.youtube.com/watch?v=xuyTgAlPOGY) - really cool video from IceCube team","metadata":{}},{"cell_type":"markdown","source":"Also, I used these notebooks as a source of inspiration. Check them out, they are also cool\n\n- https://www.kaggle.com/code/dschettler8845/ndi-let-s-learn-together-eli5-and-eda\n- https://www.kaggle.com/code/jirkaborovec/icecube-neutrino-fitting-3d-point-cloud\n- https://www.kaggle.com/code/asimple/eda-neutrinos/notebook","metadata":{}},{"cell_type":"markdown","source":"\n##

2. Domain information

","metadata":{}},{"cell_type":"markdown","source":"\n##

2.1 What is neutrino

","metadata":{}},{"cell_type":"markdown","source":"Neutrinos are elementary particles like protons or electrons that fill our world. Unlike electrons, they have no electric charge. They are also very light and small (much smaller than electrons). They move at a speed close to the speed of light, but they are invisible to our eyes. Because neutrinos are tiny, light, and do not carry an electric charge, they are able to pass through large objects such as planets, however, they are tough to catch.","metadata":{}},{"cell_type":"markdown","source":"\"neutrino\n\nSource - https://neutrinos.fnal.gov/whats-a-neutrino/","metadata":{}},{"cell_type":"markdown","source":"Neutrinos occur during nuclear reactions, such as decay or fusion. During these reactions, energy arises, and a neutrino takes part of this energy with it and flies away.","metadata":{}},{"cell_type":"markdown","source":"\"neutrino","metadata":{}},{"cell_type":"markdown","source":"Because nuclear reactions of various scales occur everywhere from the fusion of hydrogen atoms in the Sun to a banana, which also has nuclear reactions and neutrinos. Therefore, neutrinos fly past our bodies all the time. Their number is even greater than the number of atoms in the universe. However, scientists are interested in neutrinos with more energy, which have arisen in connection with more extreme cases, such as the explosion of a supernova or radiation from a black hole.","metadata":{}},{"cell_type":"markdown","source":"Neutrinos move really with near-to-light speed and almost never interact with another particle, due to their weight and size. So, it's really hard to spot them. And it's a place, where IceCube laboratory comes into the game.","metadata":{}},{"cell_type":"markdown","source":"\n##

2.2 What is IceCube laboratory

","metadata":{}},{"cell_type":"markdown","source":"IceCube is an laboratory in Antarctica. At a depth of 1,450 to 2,450 m, deep in the ice, 5,160 digital optical modules (DOM) were placed on threads one above the other. Optical modules catch the blue glow, which is the result of Cherenkov radiation, which occurs in connection with the passage of neutrinos through the layer of ice.","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"However, why was it necessary to build a laboratory in Antarctica? Antarctic ice at this depth is extremely clean, and the blue glow propagates here much better than in other substances, so the sensors are better able to catch these flashes.","metadata":{}},{"cell_type":"markdown","source":"\n##

3. Data Exploration

","metadata":{}},{"cell_type":"code","source":"import os\nimport random\nimport math\nfrom pathlib import Path\nfrom collections import Counter\n\nfrom tqdm import tqdm\nimport pandas as pd\nimport numpy as np\nimport pyarrow.parquet as pq\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\nsns.set_style(\"darkgrid\")","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:44:44.371680Z","iopub.execute_input":"2023-02-01T10:44:44.372345Z","iopub.status.idle":"2023-02-01T10:44:47.131603Z","shell.execute_reply.started":"2023-02-01T10:44:44.372217Z","shell.execute_reply":"2023-02-01T10:44:47.129184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_path = Path(\"/kaggle/input/icecube-neutrinos-in-deep-ice/\")","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:44:47.134269Z","iopub.execute_input":"2023-02-01T10:44:47.134812Z","iopub.status.idle":"2023-02-01T10:44:47.141533Z","shell.execute_reply.started":"2023-02-01T10:44:47.134757Z","shell.execute_reply":"2023-02-01T10:44:47.140133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##

3.1 Explore sensors

","metadata":{}},{"cell_type":"markdown","source":"First of all, let's check out sensors information, located in sensors_geometry.csv file","metadata":{}},{"cell_type":"code","source":"sensor_geometry = pd.read_csv(data_path / \"sensor_geometry.csv\")\nprint(f\"Shape: {sensor_geometry.shape}\")\nsensor_geometry.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:44:47.143261Z","iopub.execute_input":"2023-02-01T10:44:47.143649Z","iopub.status.idle":"2023-02-01T10:44:47.199209Z","shell.execute_reply.started":"2023-02-01T10:44:47.143614Z","shell.execute_reply":"2023-02-01T10:44:47.197755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we see, there are really 5160 DOMs and each DOM has its own id. Each DOM has a location by x, y, and z in units of meters, with the origin at the center of the IceCube detector. The coordinate system is right-handed, and the z-axis points upwards when standing at the South Pole.","metadata":{}},{"cell_type":"markdown","source":"Let's look at this sensors on 3D plot","metadata":{}},{"cell_type":"code","source":"fig = px.scatter_3d(sensor_geometry, x='x', y='y', z='z', color=\"z\", opacity=0.75)\nfig.update_traces(marker_size=3)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:44:47.201562Z","iopub.execute_input":"2023-02-01T10:44:47.201984Z","iopub.status.idle":"2023-02-01T10:44:48.737339Z","shell.execute_reply.started":"2023-02-01T10:44:47.201947Z","shell.execute_reply":"2023-02-01T10:44:48.736127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are strings, where sensors go one by one by the z-axis. Interestingly, there are 8 strings of sensors, which looks different. They have lower distances between sensors but are located at some particular parts of the z-axis. As we see, the coordinates are normalized, and in reality position of these sensors is between 1450-2450 meters by the z-axis.","metadata":{}},{"cell_type":"markdown","source":"Also, we can check out ranges, where these sensors are located","metadata":{}},{"cell_type":"code","source":"print(f'X axis: top {sensor_geometry[\"x\"].max()} bottom {sensor_geometry[\"x\"].min()}')\nprint(f'Y axis: top {sensor_geometry[\"y\"].max()} bottom {sensor_geometry[\"y\"].min()}')\nprint(f'Z axis: top {sensor_geometry[\"z\"].max()} bottom {sensor_geometry[\"z\"].min()}')","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:44:48.738723Z","iopub.execute_input":"2023-02-01T10:44:48.739401Z","iopub.status.idle":"2023-02-01T10:44:48.747199Z","shell.execute_reply.started":"2023-02-01T10:44:48.739366Z","shell.execute_reply":"2023-02-01T10:44:48.746079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##

3.2 Explore train data

","metadata":{}},{"cell_type":"markdown","source":"Train data is stored in the train folder, where data is separated into batches. First of all, let's count the number of batches","metadata":{}},{"cell_type":"code","source":"def count_batches(path):\n counter = 0\n for item in path.glob('*'):\n if item.is_file():\n counter += 1\n return counter\n\nprint(f'Batches in train folder: {count_batches(data_path / \"train\")}')\nprint(f'Batches in test folder: {count_batches(data_path / \"test\")}')","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:44:48.748829Z","iopub.execute_input":"2023-02-01T10:44:48.749537Z","iopub.status.idle":"2023-02-01T10:44:48.828869Z","shell.execute_reply.started":"2023-02-01T10:44:48.749502Z","shell.execute_reply":"2023-02-01T10:44:48.827208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's examine the structure of one batch","metadata":{}},{"cell_type":"code","source":"train_batch = pd.read_parquet(data_path / \"train\" / \"batch_1.parquet\")\nprint(f\"Shape: {train_batch.shape}\")\ntrain_batch.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:44:48.831189Z","iopub.execute_input":"2023-02-01T10:44:48.831585Z","iopub.status.idle":"2023-02-01T10:44:52.645765Z","shell.execute_reply.started":"2023-02-01T10:44:48.831548Z","shell.execute_reply":"2023-02-01T10:44:52.644773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_batch = pd.read_parquet(data_path / \"test\" / \"batch_661.parquet\")\nprint(f\"Shape: {test_batch.shape}\")\ntest_batch.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:44:52.647122Z","iopub.execute_input":"2023-02-01T10:44:52.647708Z","iopub.status.idle":"2023-02-01T10:44:52.675479Z","shell.execute_reply.started":"2023-02-01T10:44:52.647654Z","shell.execute_reply":"2023-02-01T10:44:52.674098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The batch consists of many events, and every event consists of pulses. Each pulse has sensor_id, which catches the pulse. The time column indicates the time of the pulse in nanoseconds in the current event time window. The charge is an estimated amount of light in units of photoelectrons. Auxiliary is a boolean column, where true means, that the pulse is more likely to be noisy.\n\nLet's count events and pulses in the first batch.","metadata":{}},{"cell_type":"code","source":"print(f\"Events in first batch: {train_batch.index.nunique()}\")","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:44:52.677587Z","iopub.execute_input":"2023-02-01T10:44:52.678164Z","iopub.status.idle":"2023-02-01T10:44:53.280450Z","shell.execute_reply.started":"2023-02-01T10:44:52.678114Z","shell.execute_reply":"2023-02-01T10:44:53.279101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pulses_in_batch = pd.DataFrame(train_batch.groupby('event_id').size(), columns=[\"n_pulses\"])","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:44:53.284859Z","iopub.execute_input":"2023-02-01T10:44:53.285338Z","iopub.status.idle":"2023-02-01T10:44:54.160633Z","shell.execute_reply.started":"2023-02-01T10:44:53.285286Z","shell.execute_reply":"2023-02-01T10:44:54.159639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(\n pulses_in_batch,\n log_y=True,\n title=\"Number of pulses in events (log scale)\"\n)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:44:54.161966Z","iopub.execute_input":"2023-02-01T10:44:54.163154Z","iopub.status.idle":"2023-02-01T10:44:54.351836Z","shell.execute_reply.started":"2023-02-01T10:44:54.163104Z","shell.execute_reply":"2023-02-01T10:44:54.350404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As seen from the histogram, the most of events have from 1 to 1000 pulses, but there are events with more than 100,000 pulses. I think that condition, that each event may have a much different number of pulses will cause the model architecture to solve this problem.","metadata":{}},{"cell_type":"code","source":"event_time_lengthes = pd.DataFrame(train_batch.groupby('event_id')['time'].agg(np.ptp))","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:44:54.354076Z","iopub.execute_input":"2023-02-01T10:44:54.354537Z","iopub.status.idle":"2023-02-01T10:45:15.008106Z","shell.execute_reply.started":"2023-02-01T10:44:54.354498Z","shell.execute_reply":"2023-02-01T10:45:15.006578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(\n event_time_lengthes,\n x=\"time\",\n title=\"Length of events (in nanoseconds)\"\n)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:45:15.009602Z","iopub.execute_input":"2023-02-01T10:45:15.010125Z","iopub.status.idle":"2023-02-01T10:45:15.120063Z","shell.execute_reply.started":"2023-02-01T10:45:15.010076Z","shell.execute_reply":"2023-02-01T10:45:15.118277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As seen from hist, most of the events last for around 10,000 nanoseconds (0.00001 seconds). This graph looks different from the previous one, so it looks like there is no correlation between the length of the event and the number of pulses in the event. Let's check that out","metadata":{}},{"cell_type":"code","source":"event_time_lengthes.corrwith(pulses_in_batch[\"n_pulses\"], axis=0)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:45:15.121725Z","iopub.execute_input":"2023-02-01T10:45:15.122845Z","iopub.status.idle":"2023-02-01T10:45:15.143993Z","shell.execute_reply.started":"2023-02-01T10:45:15.122775Z","shell.execute_reply":"2023-02-01T10:45:15.142838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Yes, there is not a big correlation between the number of pulses and the length of the event","metadata":{}},{"cell_type":"markdown","source":"\n##

3.3 Explore metadata

","metadata":{}},{"cell_type":"markdown","source":"There are two metadata files, for train and test sets. Let's examine them","metadata":{}},{"cell_type":"code","source":"train_meta = pd.read_parquet(data_path / \"train_meta.parquet\")\nprint(f\"Shape: {train_meta.shape}\")\ntrain_meta.head(5)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:45:15.145461Z","iopub.execute_input":"2023-02-01T10:45:15.146146Z","iopub.status.idle":"2023-02-01T10:45:59.017748Z","shell.execute_reply.started":"2023-02-01T10:45:15.146102Z","shell.execute_reply":"2023-02-01T10:45:59.016374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_meta = pd.read_parquet(data_path / \"test_meta.parquet\")\nprint(f\"Shape: {test_meta.shape}\")\ntest_meta.head(5)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:45:59.019756Z","iopub.execute_input":"2023-02-01T10:45:59.020592Z","iopub.status.idle":"2023-02-01T10:45:59.046357Z","shell.execute_reply.started":"2023-02-01T10:45:59.020540Z","shell.execute_reply":"2023-02-01T10:45:59.045005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In metadata files stored information about every event. For every event, we have batch_id, which determines, in which batch this event pulses are stored. Also, it has the first and last pulse index, related to this event.\n\nFinally, it has azimuth and zenith values. This value describes, where the neutrino came from. Azimuth is the angle of the direction of the sun measured clockwise north from the horizon. Zenith angle measured from the local zenith and the line of sight of the sun. In our dataset, these values are given in radians. Azimuth values are between 0 and $2*pi$ and zenith values are between 0 and pi (because azimuth varies from 0° to 360° and zenith from 0° to 180°) ","metadata":{}},{"cell_type":"markdown","source":"\"neutrino","metadata":{}},{"cell_type":"markdown","source":"We may plot the distribution of azimuth and zenith angles in the dataset","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(14, 4))\nax.hist(train_meta[\"azimuth\"], bins=100, color=\"lightblue\")\nax.set_title(\"Distribution of azimuth values\");","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:45:59.048054Z","iopub.execute_input":"2023-02-01T10:45:59.048886Z","iopub.status.idle":"2023-02-01T10:46:02.413045Z","shell.execute_reply.started":"2023-02-01T10:45:59.048843Z","shell.execute_reply":"2023-02-01T10:46:02.411542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(14, 4))\nax.hist(train_meta[\"zenith\"], bins=100, color=\"lightblue\")\nax.set_title(\"Distribution of zenith values\");","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:46:02.415301Z","iopub.execute_input":"2023-02-01T10:46:02.416282Z","iopub.status.idle":"2023-02-01T10:46:05.674613Z","shell.execute_reply.started":"2023-02-01T10:46:02.416228Z","shell.execute_reply":"2023-02-01T10:46:05.673200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.scatter(train_meta.sample(1000), x=\"azimuth\", y=\"zenith\", marginal_x=\"violin\", marginal_y=\"violin\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:46:05.676348Z","iopub.execute_input":"2023-02-01T10:46:05.676777Z","iopub.status.idle":"2023-02-01T10:46:13.407559Z","shell.execute_reply.started":"2023-02-01T10:46:05.676741Z","shell.execute_reply":"2023-02-01T10:46:13.405741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we see, azimuth values are between 0 and $2*pi$ and zenith values are between 0 and pi.\n\nThe distribution of azimuth values is uniform, which means, there are equal numbers of events with any azimuth angles. But the distribution of zenith values has a more normal-like distribution. It means, that most of the neutrinos, detected by IceCube, go perpendicularly to the strings of DOMs. ","metadata":{}},{"cell_type":"markdown","source":"\n##

4. Break down an event

","metadata":{}},{"cell_type":"markdown","source":"Let's analyze some events to understand better, what these events look like.\n\nFirst, we have to take some event id for our analysis","metadata":{}},{"cell_type":"code","source":"analyze_event_id = 24","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:46:13.409427Z","iopub.execute_input":"2023-02-01T10:46:13.409942Z","iopub.status.idle":"2023-02-01T10:46:13.416123Z","shell.execute_reply.started":"2023-02-01T10:46:13.409891Z","shell.execute_reply":"2023-02-01T10:46:13.414758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_of_event = train_meta[train_meta[\"event_id\"] == analyze_event_id]\nmeta_of_event","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:46:13.417909Z","iopub.execute_input":"2023-02-01T10:46:13.419105Z","iopub.status.idle":"2023-02-01T10:46:13.694079Z","shell.execute_reply.started":"2023-02-01T10:46:13.419046Z","shell.execute_reply":"2023-02-01T10:46:13.692663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"event_pulses = train_batch[train_batch.index == analyze_event_id]\nevent_pulses","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:46:13.695791Z","iopub.execute_input":"2023-02-01T10:46:13.696246Z","iopub.status.idle":"2023-02-01T10:46:13.747958Z","shell.execute_reply.started":"2023-02-01T10:46:13.696209Z","shell.execute_reply":"2023-02-01T10:46:13.746732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 61 pulses in this event. It could be worse, like 100,000 pulses😅","metadata":{}},{"cell_type":"markdown","source":"Now I want to create a function for events visualization","metadata":{}},{"cell_type":"code","source":"def visualize_event(pulse_id: str, sensor_geometry: pd.DataFrame, pulses_data: pd.DataFrame, meta_data: pd.DataFrame):\n event_pulses = pulses_data[pulses_data.index == pulse_id]\n meta_of_event = train_meta[train_meta[\"event_id\"] == pulse_id]\n \n fig = make_subplots(\n rows=1, cols=2,\n specs=[[{\"type\": \"scatter3d\"}, {\"type\": \"scatter3d\"}]],\n subplot_titles=(\"All events\", \"Not auxiliary events\")\n )\n \n aux_pulses_data = event_pulses[event_pulses[\"auxiliary\"] == True]\n not_aux_pulses_data = event_pulses[event_pulses[\"auxiliary\"] == False]\n aux_df = aux_pulses_data.merge(sensor_geometry, left_on='sensor_id', right_on='sensor_id')[[\"x\", \"y\", \"z\", \"charge\", \"time\"]]\n not_aux_df = not_aux_pulses_data.merge(sensor_geometry, left_on='sensor_id', right_on='sensor_id')[[\"x\", \"y\", \"z\", \"charge\", \"time\"]]\n\n fig.add_trace(\n go.Scatter3d(x=aux_df[\"x\"], y=aux_df[\"y\"], z=aux_df[\"z\"], opacity=0.75, mode='markers', marker_size=aux_df[\"charge\"]*10, text=aux_df[\"charge\"], marker=dict(color=aux_df[\"time\"], cmin=0, cmax=aux_df.iloc[-1][\"time\"])), row=1, col=1\n )\n fig.add_trace(\n go.Scatter3d(x=not_aux_df[\"x\"], y=not_aux_df[\"y\"], z=not_aux_df[\"z\"], opacity=0.75, mode='markers', marker_size=not_aux_df[\"charge\"]*10, text=aux_df[\"charge\"], marker=dict(color=not_aux_df[\"time\"], cmin=0, cmax=not_aux_df.iloc[-1][\"time\"])), row=1, col=2\n )\n fig.add_trace(\n go.Scatter3d(x=sensor_geometry[\"x\"], y=sensor_geometry[\"y\"], z=sensor_geometry[\"z\"], mode='markers', opacity=0.3, marker=dict(size=1, color=\"gray\")), row=1, col=1\n )\n fig.add_trace(\n go.Scatter3d(x=sensor_geometry[\"x\"], y=sensor_geometry[\"y\"], z=sensor_geometry[\"z\"], mode='markers', opacity=0.3, marker=dict(size=1, color=\"gray\")), row=1, col=2\n )\n\n azimuth, zenith = meta_of_event[\"azimuth\"].values[0], meta_of_event[\"zenith\"].values[0]\n true_x = math.cos(azimuth) * math.sin(zenith)\n true_y = math.sin(azimuth) * math.sin(zenith)\n true_z = math.cos(zenith)\n \n fig.add_trace(\n go.Scatter3d(\n x=[-true_x * 500, true_x * 500], y=[-true_y * 500, true_y * 500], z=[-true_z * 500, true_z * 500],\n opacity=0.8, mode='lines', line=dict(color='red', width=5)\n ),\n row=1, col=1\n )\n fig.add_trace(\n go.Scatter3d(\n x=[-true_x * 500, true_x * 500], y=[-true_y * 500, true_y * 500], z=[-true_z * 500, true_z * 500],\n opacity=0.8, mode='lines', line=dict(color='red', width=5)\n ),\n row=1, col=2\n )\n \n fig.update_layout(title_text=f\"Event {pulse_id}\")\n \n fig.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:47:42.883621Z","iopub.execute_input":"2023-02-01T10:47:42.884908Z","iopub.status.idle":"2023-02-01T10:47:42.908491Z","shell.execute_reply.started":"2023-02-01T10:47:42.884859Z","shell.execute_reply":"2023-02-01T10:47:42.906765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And now I can visualize the pulses and path of the neutrino particle of a few events","metadata":{}},{"cell_type":"code","source":"visualize_event(24, sensor_geometry, train_batch, train_meta)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:47:44.139181Z","iopub.execute_input":"2023-02-01T10:47:44.139721Z","iopub.status.idle":"2023-02-01T10:47:44.629797Z","shell.execute_reply.started":"2023-02-01T10:47:44.139671Z","shell.execute_reply":"2023-02-01T10:47:44.627446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, there are only 13 not auxiliary pulses from 61 pulses in total in this event. Also, it seems like the path of neutrino does not go through sensors, which detected pulse. So, now this task looks harder😉","metadata":{}},{"cell_type":"code","source":"visualize_event(41, sensor_geometry, train_batch, train_meta)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:48:01.898544Z","iopub.execute_input":"2023-02-01T10:48:01.900185Z","iopub.status.idle":"2023-02-01T10:48:02.357576Z","shell.execute_reply.started":"2023-02-01T10:48:01.900128Z","shell.execute_reply":"2023-02-01T10:48:02.356247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_event(59, sensor_geometry, train_batch, train_meta)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:47:57.119875Z","iopub.execute_input":"2023-02-01T10:47:57.120835Z","iopub.status.idle":"2023-02-01T10:47:57.591628Z","shell.execute_reply.started":"2023-02-01T10:47:57.120787Z","shell.execute_reply":"2023-02-01T10:47:57.590626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_event(67, sensor_geometry, train_batch, train_meta)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:47:57.593455Z","iopub.execute_input":"2023-02-01T10:47:57.593892Z","iopub.status.idle":"2023-02-01T10:47:58.068954Z","shell.execute_reply.started":"2023-02-01T10:47:57.593854Z","shell.execute_reply":"2023-02-01T10:47:58.067925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_event(72, sensor_geometry, train_batch, train_meta)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T10:48:14.506791Z","iopub.execute_input":"2023-02-01T10:48:14.507610Z","iopub.status.idle":"2023-02-01T10:48:14.947606Z","shell.execute_reply.started":"2023-02-01T10:48:14.507551Z","shell.execute_reply":"2023-02-01T10:48:14.946491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I'm working on updates to this notebook. Thanks for your attention. More things soon😊","metadata":{}}]}