{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**Why do we care about Neutrinos?**\n\nNeutrinos, “invented” to balance a physics equation, have grown to fascinate astrophysicists, galactic voyeurs seeking signals from astonishingly energetic structures and events in the deep universe. The direction and energy of neutrinos from each source should offer clues about the origin\n\n**Gamma Ray Burst**: In a couple of dozen seconds, these gargantuan gamma-ray sources can send out as much energy as our sun will during its entire life. The bursts, billions of light years distant, may result from the collapse of a massive star, but a paper from the IceCube group will soon question whether they are major neutrino sources [ref]\n\n**Active Galactic Nucleus**: This stormy region around a black hole emits huge amounts of energy but is shrouded by gas and dust. Active galactic nuclei are astonishingly bright source of microwave, infrared, visible, ultraviolet and gamma radiation, and likely neutrinos as well.\n\n**Supernova**: The explosion of a dying star occurs when gravity overwhelms the outward pressure from nuclear fusion. The last nearby supernova, in 1987, energized astronomers and caused a 10-second burst of neutrinos that lent credibility to neutrino science.\n\n**Neutron Star**: This relic of a supernova is composed of pure neutrons, which don’t repel each other. Therefore, neutron stars are rather dense: a teaspoonful probably weighs several billion tons. Neutron stars start life at about 10 11° C to 10 12° C, but quickly radiate away energy via an intense blast of neutrinos and electromagnetic radiation.","metadata":{}},{"cell_type":"markdown","source":"**If Neutrino's are 'Ghost Particles' and they don't interact with anything, how can we possibly 'detect' them?**\n\n- The tiny percentage of neutrinos that interact with atomic nuclei in the ice produce muons\n- These muons create Cherenkov Radiation/Light when they interact with matter.\n- The neutrino cross section is a measure of how likely the neutrino is to be stopped by regular matter. The higher energy a neutrino has, the more likely it is to interact.","metadata":{}},{"cell_type":"code","source":"%matplotlib inline\n\nimport os\nimport glob\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nPATH_DATASET = \"/kaggle/input/icecube-neutrinos-in-deep-ice\"","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:32:02.750005Z","iopub.execute_input":"2023-01-26T10:32:02.750530Z","iopub.status.idle":"2023-01-26T10:32:02.769716Z","shell.execute_reply.started":"2023-01-26T10:32:02.750469Z","shell.execute_reply":"2023-01-26T10:32:02.768109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_test = pd.read_parquet(os.path.join(PATH_DATASET, \"test_meta.parquet\"))\nprint(f\"length: {len(meta_test)}\")\nmeta_test.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:32:17.652254Z","iopub.execute_input":"2023-01-26T10:32:17.652648Z","iopub.status.idle":"2023-01-26T10:32:17.696271Z","shell.execute_reply.started":"2023-01-26T10:32:17.652621Z","shell.execute_reply":"2023-01-26T10:32:17.695422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_train = pd.read_parquet(os.path.join(PATH_DATASET, \"train_meta.parquet\"))\nprint(f\"length: {len(meta_train)}\")\ndisplay(meta_train.head())\ndisplay(meta_train.info())","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:32:38.759365Z","iopub.execute_input":"2023-01-26T10:32:38.760362Z","iopub.status.idle":"2023-01-26T10:33:01.409006Z","shell.execute_reply.started":"2023-01-26T10:32:38.760332Z","shell.execute_reply":"2023-01-26T10:33:01.405083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(6, 2))\nmeta_train.groupby(\"batch_id\").size().plot.hist(bins=50)\nplt.xlabel(\"nb. events in batch\"), plt.ylabel(\"nb. cases\")\nplt.grid()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:35:01.775064Z","iopub.execute_input":"2023-01-26T10:35:01.775505Z","iopub.status.idle":"2023-01-26T10:35:05.500704Z","shell.execute_reply.started":"2023-01-26T10:35:01.775477Z","shell.execute_reply":"2023-01-26T10:35:05.499316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_train.groupby(\"event_id\").size().max()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:35:17.552831Z","iopub.execute_input":"2023-01-26T10:35:17.553190Z","iopub.status.idle":"2023-01-26T10:37:40.410225Z","shell.execute_reply.started":"2023-01-26T10:35:17.553165Z","shell.execute_reply":"2023-01-26T10:37:40.408578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_train[[\"first_pulse_index\", \"last_pulse_index\"]].plot.hist(bins=100, alpha=0.5, figsize=(8, 3))\nplt.xlabel('time')","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:37:40.412107Z","iopub.execute_input":"2023-01-26T10:37:40.412523Z","iopub.status.idle":"2023-01-26T10:37:56.898923Z","shell.execute_reply.started":"2023-01-26T10:37:40.412483Z","shell.execute_reply":"2023-01-26T10:37:56.897804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_train[\"delay_pulse_index\"] = meta_train[\"last_pulse_index\"] - meta_train[\"first_pulse_index\"]\n_ = plt.hist(meta_train[\"delay_pulse_index\"], bins=100, log=True)\nplt.ylabel('count cases'), plt.xlabel('duration')\nplt.grid()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:37:56.900238Z","iopub.execute_input":"2023-01-26T10:37:56.902386Z","iopub.status.idle":"2023-01-26T10:38:00.251053Z","shell.execute_reply.started":"2023-01-26T10:37:56.902338Z","shell.execute_reply":"2023-01-26T10:38:00.249917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_train[[\"azimuth\", \"zenith\"]].plot.hist(bins=50, alpha=0.5, figsize=(8, 3))\nplt.xlabel('azimuth/zenith')\nplt.grid()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:38:00.253214Z","iopub.execute_input":"2023-01-26T10:38:00.254012Z","iopub.status.idle":"2023-01-26T10:38:42.910146Z","shell.execute_reply.started":"2023-01-26T10:38:00.253982Z","shell.execute_reply":"2023-01-26T10:38:42.909115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist2d(meta_train[\"azimuth\"], meta_train[\"zenith\"], bins=(50, 50), cmap=plt.cm.jet)\nplt.xlabel('azimuth'), plt.ylabel('zenith')\nplt.colorbar()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:38:42.911776Z","iopub.execute_input":"2023-01-26T10:38:42.912104Z","iopub.status.idle":"2023-01-26T10:38:57.453468Z","shell.execute_reply.started":"2023-01-26T10:38:42.912077Z","shell.execute_reply":"2023-01-26T10:38:57.450785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"geometry = pd.read_csv(os.path.join(PATH_DATASET, \"sensor_geometry.csv\"))\nprint(f\"length: {len(geometry)}\")\ngeometry.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:38:57.455463Z","iopub.execute_input":"2023-01-26T10:38:57.455976Z","iopub.status.idle":"2023-01-26T10:38:57.481522Z","shell.execute_reply.started":"2023-01-26T10:38:57.455949Z","shell.execute_reply":"2023-01-26T10:38:57.480122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.express as px\n\nfig = px.scatter_3d(geometry, x='x', y='y', z='z', opacity=0.6, color=\"sensor_id\")\nfig.update_traces(marker_size=2)\nfig.update_layout(height=600, width=600)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:38:57.483753Z","iopub.execute_input":"2023-01-26T10:38:57.484192Z","iopub.status.idle":"2023-01-26T10:38:57.563329Z","shell.execute_reply.started":"2023-01-26T10:38:57.484161Z","shell.execute_reply":"2023-01-26T10:38:57.561708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = pd.read_parquet(os.path.join(PATH_DATASET, \"test/batch_661.parquet\"))\nprint(f\"length: {len(test)}\")\nprint(f\"events: {len(test.index.unique())}\")\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:38:57.565300Z","iopub.execute_input":"2023-01-26T10:38:57.565673Z","iopub.status.idle":"2023-01-26T10:38:57.601843Z","shell.execute_reply.started":"2023-01-26T10:38:57.565644Z","shell.execute_reply":"2023-01-26T10:38:57.600877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_parquet(os.path.join(PATH_DATASET, \"train/batch_15.parquet\"))\nprint(f\"length: {len(train)}\")\nprint(f\"events: {len(train.index.unique())}\")\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:38:57.603182Z","iopub.execute_input":"2023-01-26T10:38:57.604490Z","iopub.status.idle":"2023-01-26T10:39:00.892682Z","shell.execute_reply.started":"2023-01-26T10:38:57.604446Z","shell.execute_reply":"2023-01-26T10:39:00.891029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 4))\n_ = plt.hist(train.groupby(level=0).size(), bins=50, log=True)\nplt.xlabel(\"nb. ALL measument per event\"), plt.ylabel(\"nb. of event\")\nplt.grid()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:39:00.896792Z","iopub.execute_input":"2023-01-26T10:39:00.897209Z","iopub.status.idle":"2023-01-26T10:39:02.045997Z","shell.execute_reply.started":"2023-01-26T10:39:00.897182Z","shell.execute_reply":"2023-01-26T10:39:02.043374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 4))\n_ = plt.hist(train[~train['auxiliary']].groupby(level=0).size(), bins=50, log=True)\nplt.xlabel(\"nb. (aux==False) measument per event\"), plt.ylabel(\"nb. of event\")\nplt.grid()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:39:02.047900Z","iopub.execute_input":"2023-01-26T10:39:02.048217Z","iopub.status.idle":"2023-01-26T10:39:03.775662Z","shell.execute_reply.started":"2023-01-26T10:39:02.048191Z","shell.execute_reply":"2023-01-26T10:39:03.773893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12, 4))\n_ = plt.hist(train[~train['auxiliary']]['charge'], bins=50, log=True, label=\"False\")\n_ = plt.hist(train[train['auxiliary']]['charge'], bins=50, log=True, label=\"True\")\nplt.ylabel('count cases'), plt.xlabel('charge')\nplt.grid(), plt.legend()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:39:03.777080Z","iopub.execute_input":"2023-01-26T10:39:03.777364Z","iopub.status.idle":"2023-01-26T10:39:06.632351Z","shell.execute_reply.started":"2023-01-26T10:39:03.777340Z","shell.execute_reply":"2023-01-26T10:39:06.630756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_train[meta_train['event_id'].isin([46528394, 2135637939, 2084362251])]","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:39:06.633455Z","iopub.execute_input":"2023-01-26T10:39:06.633791Z","iopub.status.idle":"2023-01-26T10:39:07.476291Z","shell.execute_reply.started":"2023-01-26T10:39:06.633764Z","shell.execute_reply":"2023-01-26T10:39:07.474657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import math\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\ndef show_event(event_id=46528394, data=train, sensors=geometry, metadata=meta_train):\n    event = data[data.index == event_id]\n    event = event.merge(sensors, on=\"sensor_id\")\n    meta = dict(metadata[metadata[\"event_id\"]==event_id].iloc[0])\n    display(meta)\n    # event.head()\n    azimuth = meta[\"azimuth\"]\n    zenith = meta[\"zenith\"]\n    x_ = math.cos(azimuth) * math.sin(zenith)\n    y_ = math.sin(azimuth) * math.sin(zenith)\n    z_ = math.cos(zenith)\n    \n    auxiliaries = [False, True]\n    fig = make_subplots(\n        rows=2, specs=[[{'type': 'scene'}], [{'type': 'scene'}]],\n        subplot_titles=[f\"auxiliary={aux}\" for aux in auxiliaries],\n        vertical_spacing=0.05,\n    )\n    for i, aux in enumerate(auxiliaries):\n        evt_ = event[event['auxiliary'] == aux]\n        # sensors as background\n        fig.add_trace(\n            go.Scatter3d(\n                x=sensors['x'], y=sensors['y'], z=sensors['z'], \n                mode='markers', marker=dict(size=1, color=0), opacity=0.2\n            ), row=(i+1), col=1)\n        # sensors reading\n        fig.add_trace(\n            go.Scatter3d(\n                x=evt_['x'], y=evt_['y'], z=evt_['z'], opacity=0.8,\n                mode='markers', marker=dict(size=evt_['charge'] * 10, color=evt_['time'], colorscale='Viridis')\n            ), row=(i+1), col=1)\n        # direction from metad data\n        fig.add_trace(\n            go.Scatter3d(\n                x=[-x_ * 500, x_ * 500], y=[-y_ * 500, y_ * 500], z=[-z_ * 500, z_ * 500],\n                opacity=0.8, mode='lines', line=dict(color='red', width=3)\n            ), row=(i+1), col=1)\n    fig.update_layout(\n        height=800, width=600, showlegend=False,\n        title_text=f\"Event #{event_id} / azimuth={azimuth:0.3}; zenith={zenith:0.3}\\n\"\n        f\"-> x={x_:0.2}; y={y_:0.2}; z={z_:0.2}\",\n    )\n    # fig = px.scatter_3d(event, x='x', y='y', z='z', size=\"charge\", color=\"auxiliary\",\n    #     opacity=0.8, title=f\"Event #{event_id}\\n -> x={x_}; y={y_}; z={z_}\")\n    return fig\n\nshow_event().show()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:42:53.354959Z","iopub.execute_input":"2023-01-26T10:42:53.355954Z","iopub.status.idle":"2023-01-26T10:42:53.973852Z","shell.execute_reply.started":"2023-01-26T10:42:53.355869Z","shell.execute_reply":"2023-01-26T10:42:53.972226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from ipywidgets import interact, IntSlider\n\ndef interactive_show(events):\n    interact(\n        lambda i: show_event(events[i]).show(),\n        i=IntSlider(min=0, max=len(events), step=1, value=len(events) // 2),\n    )\n\nevents = train.index.unique().tolist()\ninteractive_show(events)","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:43:01.797433Z","iopub.execute_input":"2023-01-26T10:43:01.797952Z","iopub.status.idle":"2023-01-26T10:43:02.295805Z","shell.execute_reply.started":"2023-01-26T10:43:01.797920Z","shell.execute_reply":"2023-01-26T10:43:02.295006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ssub = pd.read_parquet(os.path.join(PATH_DATASET, \"sample_submission.parquet\"))\nprint(f\"length: {len(ssub)}\")\nssub.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:43:06.422240Z","iopub.execute_input":"2023-01-26T10:43:06.423133Z","iopub.status.idle":"2023-01-26T10:43:06.449237Z","shell.execute_reply.started":"2023-01-26T10:43:06.423083Z","shell.execute_reply":"2023-01-26T10:43:06.447412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ssub['zenith'] = meta_train['zenith'].median()\nssub['azimuth'] = meta_train['azimuth'].median()\nssub.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:43:08.705053Z","iopub.execute_input":"2023-01-26T10:43:08.705578Z","iopub.status.idle":"2023-01-26T10:43:20.468325Z","shell.execute_reply.started":"2023-01-26T10:43:08.705523Z","shell.execute_reply":"2023-01-26T10:43:20.465160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!head submission.csv","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:43:20.476862Z","iopub.execute_input":"2023-01-26T10:43:20.479513Z","iopub.status.idle":"2023-01-26T10:43:21.459746Z","shell.execute_reply.started":"2023-01-26T10:43:20.479430Z","shell.execute_reply":"2023-01-26T10:43:21.455649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Why the South Pole?**\n\n- Cosmic rays are deflected at the North Pole but they are detectable at the South Pole\n- The super clear ice thing!! (light propgates almost 200m vs. 2m in distilled water)","metadata":{}},{"cell_type":"markdown","source":"*Predicting a neutrino particle’s direction. I will develop a model based on data from the \"IceCube\" detector, which observes the cosmos from deep within the South Pole ice.*\n\n*The IceCube Neutrino Observatory is the first detector of its kind, encompassing a cubic kilometer of ice and designed to search for the nearly massless neutrinos. An international group of scientists is responsible for the scientific research that makes up the IceCube Collaboration.*","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}}]}