{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n\n
\n\n
\n\n

\n \n LET'S \n LEARN \n TOGETHER !\n


\n\n EXPLAIN IT \n LIKE \n I'M\n 5 ––\n (ELI5)\n
+
\n EXPLORATORY\n DATA\n ANALYSIS ––\n (EDA)\n

\n\n

CREATED BY: DARIEN SCHETTLER

\n\n
\n\n🎨 Font colour choices were inspired by the southern lights! (#40996f & #1bd19d) 🎨
If the colours cause issues when viewing, please let me know and I can edit it to make it more accessible. Thanks!\n
\n\n---\n\n
\n\n
\n ⚠️   NOTE   ⚠️

This notebook's primary purpose is to educate through exploration of this competition's data, the background information, and other notebooks and discussion posts that help in building a complete understanding.

Kaggle user's Notebooks and Discussions that I include will be listed below!
Please upvote the original authors/contributors.

\n \n\n\n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n\n
         Contributor                   Notebook          
placeholder 1notebook 1
placeholder 2notebook 2
placeholder 3notebook 3
\n \n
\n \n
\n\n\n\n
\n 🛑   WARNING:

THIS IS A WORK IN PROGRESS
\n
\n\n\n
\n 👏   IF YOU FORK THIS OR FIND THIS HELPFUL   👏

PLEASE UPVOTE!

This was a lot of work for me and it makes me feel appreciated when others like my work. 😅\n
\n\n\n","metadata":{}},{"cell_type":"markdown","source":"

\n\n

\n\n

TABLE OF CONTENTS

\n\n---\n\n

1    INTRODUCTION & JUSTIFICATION

\n\n---\n\n

2    COMPETITION BACKGROUND INFORMATION

\n\n---\n\n

3    BACKGROUND INFORMATION GLOSSARY

\n\n---\n\n

4    IMPORTS

\n\n---\n\n

5    SETUP & HELPER FUNCTIONS

\n\n---\n\n

6    DATA EXPLORATION

\n\n---\n\n

7    TBD

\n\n---\n\n

8    NEXT STEPS

\n\n---\n\n

9    BASELINE

\n\n---","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

1  INTRODUCTION & JUSTIFICATION    ⤒

\n\n
\n\nThis notebook aims to provide an educational walkthrough following my personal journey of understanding. This will hopefully bring me from completely ignorant of the background and science for this competition to a place where I feel ready to tackle the challenge in earnest!
I hope you can join me and we can learn together!\n\nUnlike my previous notebooks that were primarily Exploratory Data Analyses (EDAs) or Educational (ELI5), this notebook will merge those two educational pursuits into a single unified resource.","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.1 WHAT IS THIS?

\n\n---\n\nThis notebook will attempt to do two (maybe three) things:\n1. \"Explain It Like I'm Five\". \n * Where \"IT\" refers to everything you need to know to understand this competition (data, background information, etc)\n * Note that while I aim to honour the five-year old part of this statement, it mostly used because it is a catchy term, and I will just be explaining things as simply as possible\n2. \"Exploratory Data Analysis\"\n * I will perform a traditional Exploratory Data Analysis (EDA), which is a critical early step in tackling any data science problem!\n * I see this as a secondary and complimentary piece to the ELI5 piece, and as such, this part of the notebook comes AFTER the ELI5 part.\n3.Baseline (optional)\n * If I think I have time and if it's appropriate, I will finish this notebook out with a final section that performs a baseline submission resulting in a valid leaderboard submission.\n \nREMINDER: If I have cited other authors, please don't forget to upvote them. I'll reiterate their works below but it can be found above too!\n\n\n\n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n\n
         Contributor                   Notebook          
Peter (@pestipeti)Animated Events With Matplotlib
Robert Hatch⚡🧊⚡[LB 1.183] Lightning Fast Baseline with Polars
placeholder 3notebook 3
","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.2 WHY IS THIS?

\n\n---\n\nI wanted to share my learning journey with others as this type of work has resonated well in the past. \n* Note that I make these type of notebooks primarily for myself, however, I am finding more and more value in being able to share them with others and hope to continue to explore this.\n* By upvoting/sharing this work you are supporting me and encouraging me to spend more time on similar endeavours in the future!","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.3 WHO IS THIS FOR?

\n\n---\n\nThe primary purpose of this notebook is to educate MYSELF, however, my review/learning might be beneficial to others. Hence this notebook.\n\nPlease understand that not everyone needs nor wants to read a notebook like this. That being said the following are some good reasons why you may wish to read on:\n* If you want to learn with me!\n* If you want to learn more about the background information (ELI5)\n* If you want to learn more about the competition specifics (ELI5)\n* If you want to learn more about the competition data (EDA)\n* If you have a good understanding of the competition specifics but want to doublecheck against my understanding or reinforce your own understanding (ELI5)\n* If you have a good understanding of the background information but want to doublecheck against my understanding or reinforce your own understanding (ELI5)\n* If you have a good understanding of the competition data but want to doublecheck against my understanding or reinforce your own understanding (EDA)","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.4 HOW WILL THIS WORK?

\n\n---\n\nI'm going to assemble some markdown cells (like this one) at the beginning of the notebook to go over the competition details first, followed by general background information mostly expressed as a glossary with lots of visuals. Following this, I will perform a reasonably in-depth Exploratory Data Analysis and potentially construct a baseline submission.","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

2  COMPETITION BACKGROUND INFORMATION    ⤒

\n\n
\n\n**NOTE***\n* The first sections are dedicated to exploring the basic resources provided by the host.\n* The final sections are dedicated to my understanding of the competition","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.1 PRIMARY TASK DESCRIPTION

\n\n---\n\nThe goal of this competition is to predict a neutrino particle’s direction. \n* You will develop a model based on data from the \"IceCube\" detector, which observes the cosmos from deep within the South Pole ice.\n* This model will be required to output two float values (***regression***) for each **`event_id`** in the test set.\n * **azimuth** – An angular measurement. *[A float]*\n * **zenith** – An angular measurement. *[A float]*","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.2 BASIC HOST PROVIDED BACKGROUND INFO AND CONTEXT

\n\n---\n\n
This section is kept intentionally brief and only reflects the host provided information. The concepts that help better define the context will be explored in the GLOSSARY BACKGROUND INFORMATION section of this notebook. Some terms are linked to their glossary entries for convenience.

\n\n
DATA DESCRIPTION\n \nThe goal of this competition is to identify which direction neutrinos detected by the IceCube neutrino observatory came from. When detection events can be localized quickly enough, traditional telescopes are recruited to investigate short-lived neutrino sources such as supernovae or gamma ray bursts. Because the sky is huge, better localization will not only associate neutrinos with sources but also to help partner observatories limit their search space. With an average of three thousand events per second to process, it's difficult to keep up with the stream of data using traditional methods. Your challenge in this competition is to quickly and accurately process a large number of events.\n\n
CONTEXT DESCRIPTION\n\nOne of the most abundant particles in the universe is the neutrino. While similar to an electron, the nearly massless and electrically neutral neutrinos have fundamental properties that make them difficult to detect. Yet, to gather enough information to probe the most violent astrophysical sources, scientists must estimate the direction of neutrino events. If algorithms could be made considerably faster and more accurate, it would allow for more neutrino events to be analyzed, possibly even in real-time and dramatically increase the chance to identify cosmic neutrino sources. Rapid detection could enable networks of telescopes worldwide to search for more transient phenomena.\n\nResearchers have developed multiple approaches over the past ten years to reconstruct neutrino events. However, problems arise as existing solutions are far from perfect. They're either fast but inaccurate or more accurate at the price of huge computational costs.\n\nThe IceCube Neutrino Observatory is the first detector of its kind, encompassing a cubic kilometer of ice and designed to search for the nearly massless neutrinos. An international group of scientists is responsible for the scientific research that makes up the IceCube Collaboration.\n\n
\n\n\n\n
\n\n
EXAMPLE DATA – AN \"EVENT\"\n\nThe following image shows a visual representation of the features of an IceCube \"event\" in the dataset. \n* The **colorful dots** represent **sensors** that logged at least one **pulse** in the event. \n* The **size of the dots** corresponds to the **total charge of all pulses** while **the color** indicates **the time of the first pulse**.\n* The **small, gray points** indicate the **positions of all 5160 IceCube sensors**. \n* The **red arrow** shows the **true neutrino direction of that event, i.e. the regression target.**\n\n
\n\n\n\n
\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.3 EVALUATION INFORMATION

\n\n---\n\n
GENERAL EVALUATION INFORMATION\n\nSubmissions are evaluated using the Mean (Absolute) Angular Error between the predicted and true event origins.\n\n
HOST PROVIDED METRIC IMPLEMENTATION IN PYTHON (NUMPY)\n\n```python\nimport numpy as np\n\ndef angular_dist_score(az_true, zen_true, az_pred, zen_pred):\n '''\n calculate the MAE of the angular distance between two directions.\n The two vectors are first converted to cartesian unit vectors,\n and then their scalar product is computed, which is equal to\n the cosine of the angle between the two vectors. The inverse \n cosine (arccos) thereof is then the angle between the two input vectors\n \n Parameters:\n -----------\n \n az_true : float (or array thereof)\n true azimuth value(s) in radian\n zen_true : float (or array thereof)\n true zenith value(s) in radian\n az_pred : float (or array thereof)\n predicted azimuth value(s) in radian\n zen_pred : float (or array thereof)\n predicted zenith value(s) in radian\n \n Returns:\n --------\n \n dist : float\n mean over the angular distance(s) in radian\n '''\n \n if not (np.all(np.isfinite(az_true)) and\n np.all(np.isfinite(zen_true)) and\n np.all(np.isfinite(az_pred)) and\n np.all(np.isfinite(zen_pred))):\n raise ValueError(\"All arguments must be finite\")\n \n # pre-compute all sine and cosine values\n sa1 = np.sin(az_true)\n ca1 = np.cos(az_true)\n sz1 = np.sin(zen_true)\n cz1 = np.cos(zen_true)\n \n sa2 = np.sin(az_pred)\n ca2 = np.cos(az_pred)\n sz2 = np.sin(zen_pred)\n cz2 = np.cos(zen_pred)\n \n # scalar product of the two cartesian vectors (x = sz*ca, y = sz*sa, z = cz)\n scalar_prod = sz1*sz2*(ca1*ca2 + sa1*sa2) + (cz1*cz2)\n \n # scalar product of two unit vectors is always between -1 and 1, this is against nummerical instability\n # that might otherwise occure from the finite precision of the sine and cosine functions\n scalar_prod = np.clip(scalar_prod, -1, 1)\n \n # convert back to an angle (in radian)\n return np.average(np.abs(np.arccos(scalar_prod)))\n```\n\n
METRIC DESCRIPTION\n\nI'll expand on this in the following weeks... but my understanding of the basics are that there is some ground truth and predicted (by us) values for the **azimuth** angle and the **zenith** angle.\n* The azimuth and zenith angles (in radians) describe the path/angle of the neutrino. \n * For the **azimuth** a value between **0 to 2𝛑**\n * For the **zenith** a value between **0 to 𝛑** \n* The direction vector represented by zenith and azimuth points to where the neutrino came from.\n* The metric we are calculating compares the predicted and groundtruth values for the **azimuth/zenith** by calculating the mean absolute angular distance between them respectively. \n* The two calculated metrics are then averaged so instead of 2 values (one for the **azimuth angle** and one for the **zenith angle**) we simply get one value which represents the **mean angular error**.\n\n
SUBMISSION FILE INFORMATION\n\nFor each **event_id** in the test set, you must predict the **azimuth and zenith**. \n* The **event_id** refers to the **observation of a single neutrino**\n* For a given event (neutrino) we have to **determine where it came from**... we do this by providing the **neutrino azimuth and zenith angles**\n * > \"The direction vector represented by zenith and azimuth points to where the neutrino came from.\"\n\nThe submission file should contain a header and have the following format:\n\n```\nevent_id,azimuth,zenith\n730,1,1\n769,1,1\n774,1,1\netc.\n```\n\n
IS THIS A CODE COMPETITION ??\n\nYES\n\n> \"This competition uses a hidden test set. When your submitted notebook is scored the actual test data (including a full length sample submission) will be made available to your notebook...\"","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.4 DATASET INFORMATION

\n\n---\n\n
HIGH LEVEL DATA SUMMARY\n\nTBD\n\n
DATA FILE DESCRIPTIONS\n\n[train|test]_meta.parquet columns:\n* batch_id (int)\n * The ID of the batch the event was placed into\n* event_id (int)\n * The event ID\n* [first|last]_pulse_index (int)\n * The index of the [first|last] row in the features dataframe belonging to this event\n* [azimuth|zenith] (float32)\n * The **[azimuth|zenith]** angle in **radians** of the neutrino. \n * The direction vector represented by zenith and azimuth points to where the neutrino came from.\n * *Not provided for the test set as these are the **labels/ground-truth** values*\n\n[train|test]/batch_[n].parquet description \n* Each batch contains tens of thousands of events. \n* Each event may contain thousands of pulses\n* Each pulse is the digitized output from a photomultiplier tube and occupies one row\n\n[train|test]/batch_[n].parquet columns: \n* event_id (int)\n * The event ID. \n * Saved as the index column in parquet.\n* time (int)\n * The time of the pulse in nanoseconds in the current event time window. \n * The absolute time of a pulse has no relevance, and only the relative time with respect to other pulses within an event is of relevance.\n* sensor_id (int)\n * The ID of which of the **5160** IceCube photomultiplier sensors recorded this pulse\n* charge (float32)\n * An estimate of the amount of light in the pulse, in units of photoelectrons (p.e.). \n * A physical photon does not exactly result in a measurement of 1 p.e. but rather can take values spread around 1 p.e. \n * As an example, a pulse with charge 2.7 p.e. could quite likely be the result of two or three photons hitting the photomultiplier tube around the same time. \n * This data has float16 precision but is stored as float32 due to limitations of the version of pyarrow the data was prepared with.\n* auxiliary (bool)\n * If **`True`**: the pulse was not fully digitized, is of lower quality, and was more likely to originate from noise. \n * If **`False`**: then this pulse was contributed to the trigger decision and the pulse was fully digitized.\n\nsample_submission.parquet description\n* An example submission with the correct columns and properly ordered event IDs. \n* The sample submission is provided in the parquet format so it can be read quickly but your final submission must be a csv.\n\nsensor_geometry.csv description\n* The **x**, **y**, and **z positions** for each of the **5160** IceCube sensors. \n* The **row index corresponds to the `sensor_idx` feature of pulses**. \n* The **x**, **y**, and **z coordinates** are in **units of meters**, with the **origin at the center of the IceCube detector**. \n* The coordinate system is right-handed, and the z-axis points upwards when standing at the South Pole. \n* **You can convert from these coordinates to azimuth and zenith with the following formulas** (here the vector (x,y,z) is normalized):\n\n> ```python\n> x = cos(azimuth) * sin(zenith)\n> y = sin(azimuth) * sin(zenith)\n> z = cos(zenith)\n> ```\n\n
POST-EDA DATA OBSERVATIONS\n\nTBD","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.5 COMPETITION IMPACT INFORMATION

\n\n---\n\nNeutrinos allow us to gain deep insights into the fundamental structure of matter, and to research our universe uncovering cataclysmic phenomena far away from our Earth. At the South Pole, in the middle of the Antarctica ice, we have constructed the world's largest neutrino detector and are taking data for more than 10 years. Interpreting this data, and in particular figuring out where exactly neutrinos came from, especially in a timely manner for real-time observations, is a daunting task. This is where we need your help! Your task is to estimate the direction of neutrinos that interacted in IceCube and provide fast and accurate predictions for the zenith and azimuth angles per recorded event.\n\nBecause the sky is huge better localization will not only associate neutrinos with sources but also to help partner observatories limit their search space. With an average of three thousand events per second to process, it's difficult to keep up with the stream of data using traditional methods. Your challenge in this competition is to quickly and accurately process a large number of events.\n\nYour work could help scientists better understand exploding stars, gamma-ray bursts, and cataclysmic phenomena involving black holes, neutron stars and the fundamental properties of the neutrino itself. By making the process faster and more precise, you'll help improve the reconstruction of neutrinos. As a result, we could gain a clearer image of our universe.","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.6 COMPETITION HOST INFORMATION

\n\n---\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
The IceCube Neutrino Observatory is the world's largest neutrino detector and is providing the data for this challenge as well as helping with the organization.
The Technical University of Munich, or short TUM, is one of the top universities in Europe, a member of the IceCube collaboration, and the home of the competition organizers.
The Munich Data Science Institute is TUM's central data science infrastructure and competence center. The MDSI is helping with the organization of this kaggle challenge, and supporting its realization.
The Collaborative Research Center SFB 1258 is a reserach collaboration focused on neutrinos and dark matter in astro- and particle physics, and a partner supporting this kaggle challenge.
The Excellence Cluster ORIGINS investigates the development of the Universe from the Big Bang to the emergence of life. The cluster and in particular its data science laboratory ODSL support this ML challenge.
PUNCH4NFDI is a large German research consortium under the NFDI infrastructure, focused on particle, astro-, astroparticle, hadron and nuclear physics, and is supporting this ML challenge.
","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.7 MY UNDERSTANDING OF THIS COMPETITION

\n\n---\n\n
\n\n\n\n
\n\n\n\n
\n\n* Neutrino's are a particle that exist in our universe (Similar in some ways to Electrons, Photons, etc.)\n* Neutrino's don't interact with much.... but this means they can reach us even if they originate at the far reaches of the universe.\n * They have an incredibly, imeasureably small mass (but they aren't massless like Photons or Muons )\n * As they have no electric charge, they are not impacted by magnetic fields\n * They do not interact with matter ('ghost particle')... so they pass right through stars, planets, and even you.\n * Because of their small mass they are not impacted by Gravity\n* Some Neutrino's are ...\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.8 MY INTERNAL Q&A

\n\n---\n\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
QUESTIONSANSWERS
Why do we care about Neutrinos?... To Be Expanded (placeholder content) ...
    \n
  • Neutrinos, “invented” to balance a physics equation, have grown to fascinate astrophysicists, galactic voyeurs seeking signals from astonishingly energetic structures and events in the deep universe. The direction and energy of neutrinos from each source should offer clues about the origin
      \n
    • Gamma Ray Burst: In a couple of dozen seconds, these gargantuan gamma-ray sources can send out as much energy as our sun will during its entire life. The bursts, billions of light years distant, may result from the collapse of a massive star, but a paper from the IceCube group will soon question whether they are major neutrino sources [ref]
      • \n
    • Active Galactic Nucleus: This stormy region around a black hole emits huge amounts of energy but is shrouded by gas and dust. Active galactic nuclei are astonishingly bright source of microwave, infrared, visible, ultraviolet and gamma radiation, and likely neutrinos as well.
      • \n
    • Supernova: The explosion of a dying star occurs when gravity overwhelms the outward pressure from nuclear fusion. The last nearby supernova, in 1987, energized astronomers and caused a 10-second burst of neutrinos that lent credibility to neutrino science.
      • \n
    • Neutron Star: This relic of a supernova is composed of pure neutrons, which don’t repel each other. Therefore, neutron stars are rather dense: a teaspoonful probably weighs several billion tons. Neutron stars start life at about 10 11° C to 10 12° C, but quickly radiate away energy via an intense blast of neutrinos and electromagnetic radiation.
      • \n
    • \n
\n
If Neutrino's are 'Ghost Particles' and they don't interact with anything, how can we possibly 'detect' them?... To Be Expanded (placeholder content) ...
    \n
  • The tiny percentage of neutrinos that interact with atomic nuclei in the ice produce muons
    • \n
  • These muons create Cherenkov Radiation/Light when they interact with matter.\n
  • The neutrino cross section is a measure of how likely the neutrino is to be stopped by regular matter. The higher energy a neutrino has, the more likely it is to interact.
    • \n
  • HOST: A neutrino interaction (deep inelastic scattering) will usually create a number of (charged) particles
    • \n
  • HOST: Cherenkov radiation – When charged particles through travel a medium faster light, they emit radiation than Cherenkov. This UV/blue light is the same as can be seen in nuclear reactors
    • \n
\n
Why the South Pole?... To Be Expanded (placeholder content) ...
    \n
  • Cosmic rays are deflected at the North Pole but they are detectable at the South Pole
    • \n
  • The super clear ice thing!! (light propgates almost 200m vs. 2m in distilled water)
    • \n
\n
What does it mean when we say a Neutrino has energy?
    \n
  • the total energy of the particle as: $E^2 = p^2c^2 + m^2c^4$
    • \n
  • $p$ in the above equation is the relativistic expression for the momentum: $p = \\frac{mv}{\\sqrt{1 - v^2/c^2}}$
    • \n
  • Neutrinos have a non-zero rest mass (the $m$ in the above equation) but this mass is very small and we don't know its values for the three types of neutrino. At the moment we only have upper limits.
    • \n
  • The non-zero rest mass means there will be a minimum energy the neutrinos can have of $mc^2$.
    • \n
  • Because mass of a neutrino appears constant within a slim margin of error, the energy a neutrino has is controlled by the originating event that created that neutrino. A small event (beta decay) will produce lower energy neutrinos, while a larger event (supernova) will produce high energy events.
    • \n
  • This image shows the approximations for neutrino energies associated with various cosmological events. Note that the neutrino cross section (on the y axis with $mb$ units) is a measure of how likely the neutrino is to be stopped by regular matter ($mb$ stands for millibarn and is equal to an area that is $10^{-31}m^2$). i.e. The higher energy a neutrino has, the more likely it is to interact:

    • \n
\n
\n What are the different types of \"event\" energy signatures that are possible?\n \n Neutrinos produce one of two topologically distinct signatures: tracks and cascades\n \n
    \n
  • Tracks occur when a neutrino collides with matter in or near IceCube, resulting in a high-energy muon that travels a long distance, leaving an elongated “track” of signals in its wake. This is what's known as a Charged Current (CC) interaction and for it to leave a track like this it is imperative that the Neutrino be a Muon type Neutrino.
    • \n
  • Cascades are produced with all Neutral Current (NC) or non-Muon Neutrino CC interactions. These types of interactions yield hadronic and electromagnetic showers that typically range less than 20m (Aartsen et al. 2014a), with 90% of the light emitted within 4m of the shower maximum (Radel & Wiebusch 2013)—a short distance compared to the scattering and absorption lengths of light in the ice (Aartsen et al. 2013b) as well as the spacing of the PMTs. These showers produce a nearly spherically symmetric cascade signature in light.
    • \n
  • \n Cascades are more difficult to reconstruct than tracks, which are usually used in searches for astrophysical neutrino sources, but they have their own advantages, including providing a better measurement of neutrino energy.
    • \n
  • One final note about Cascades is that they can also appear in pairs connected sometimes by a track... this is known as a DOUBLE BANG. This happens if the incident neutrino is energetic enough, the heavy neutrino may travel some distance before decaying.
    • \n
  • Other names for more common patterns of tracks and cascades exist, but will be ignored for now
    • \n
\n
What are the different flavours of Neutrinos and do we care?TBD – Not checked or formatted...–– Neutrinos have a peculiar property of changing their “flavour” while traveling. For example, an initial electron neutrino (𝜈 neutrino ( 𝜈e ) can become a muon 𝜇 ) or a tau neutrino ( after a travelling a distance)
QA
QA
QA
QA
","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.9 HELPFUL TIPS FROM THE HOST PROVIDED IN THE WEBINAR

\n\n---\n\nWebinar Recording Link\n\n* Read publications! \n* Possibly split the events into separate classes. Improvements in any class will improve the overall score\n * Events that are easy to reconstruct \n * Events that are difficult to reconstruct and run separate algorithms on those. \n * ... etc ...\n* Some preprocessing or embedding of the data might be necessary for an ML approach \n* Typically, **the most important feature of the pulses** for directional reconstruction is not their position, but **their relative time**! \n* **Accounting for the scattering of photons in the ice is crucial **\n* **Auxiliary pulses are much more noisy, but there is still information in them** \n* Photons (Cherenkov Light) in ice travels much slower than charged particles (index/Cherenkov-Angle)\n\n
","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

3  BACKGROUND INFORMATION GLOSSARY    ⤒

\n\n!!!VERY WIP... THIS IS TAKING A WHILE... APOLOGIES... STAY TUNED!!!\n

\n\n---\n\n
\n\nNEUTRINO\n\n\nTBD\n\n\n
","metadata":{}},{"cell_type":"markdown","source":"

\n\n

3.1 NEUTRINO

\n\n---\n\n\n

\n\nPOINTFORM DEFINITION\n* WIP – More coming\n* A neutrino is a particle\n* A neutrino is one of the so-called fundamental/elementary particles\n * This means it isn’t made of any smaller pieces (at least that we know of)\n* Neutrinos are members of the same group as another, more commonly known, fundamental particle: **the electron**\n* Neutrinos have **no charge** unlike the **negative charge** found on **electrons**\n* Neutrinos are also incredibly small and light. They are the lightest of all the subatomic particles that have mass.\n * Note that Photons and Gluons are the only two confirmed examples of massless particles.\n* Neutrinos are extremely common\n * In fact, neutrinos are the most abundant massive (to reiterate – massive in this context just means 'to have mass' i.e. not a massless) particle in the universe. \n* Neutrinos come from all kinds of different sources\n\n* Neutrinos are tricky to study. \n * The only ways they interact is through gravity and the weak force, which is, well, weak. \n * This weak force is important only at very short distances, which means tiny neutrinos can skirt through the atoms of massive objects without interacting. Most neutrinos will pass through Earth without interacting at all. To increase the odds of seeing them, scientists build huge detectors and create intense sources of neutrinos.\n\nPhysicist Enrico Fermi popularized the name “neutrino”, which is Italian for “little neutral one.” Neutrinos are denoted by the Greek symbol ν, or nu (pronounced “new”). But not all neutrinos are the same. They come in different types and can be thought of in terms of flavors, masses, and energies. Some are antimatter versions. There may even be some yet undiscovered kinds of neutrinos!\n
\n\n**REFERENCES**\n* What is a Neutrino (Fermilab)\n* Neutrino (Wikipedia)\n\n

\nELI5 COMPETITION DEFINITION\n* Understanding neutrino's is vital to this competition\n* Every event in our dataset represents information about a single neutrino\n* For each event we should predict where it came from using azimuth and zenith angles.\n* TBD\n\n

\n\nEXPLAIN IT WITH PICTURES\n\n60 Second Explainer Videos on What Neutrinos Are [REF]\n\nVideo Embed Is Not Working... TO BE FIXED... See the REF\n\n\n
\n\nImage Showing What Neutrinos Are [REF]\n\n\n\n
","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

4  IMPORTS    ⤒

","metadata":{}},{"cell_type":"code","source":"# print(\"\\n... PIP INSTALLS STARTING ...\\n\")\n# print(\"\\n... PIP INSTALLS COMPLETE ...\\n\")\n\nprint(\"\\n... IMPORTS STARTING ...\\n\")\nprint(\"\\n\\tVERSION INFORMATION\")\n\n# Physics Specific Imports\n### TBD\n### TBD\n\n# Machine Learning and Data Science Imports (basics)\nimport tensorflow as tf; print(f\"\\t\\t– TENSORFLOW VERSION: {tf.__version__}\");\nimport pandas as pd; pd.options.mode.chained_assignment = None; pd.set_option('display.max_columns', None);\nimport numpy as np; print(f\"\\t\\t– NUMPY VERSION: {np.__version__}\");\nimport sklearn; print(f\"\\t\\t– SKLEARN VERSION: {sklearn.__version__}\");\n\n# Built-In Imports (mostly don't worry about these)\nfrom kaggle_datasets import KaggleDatasets\nfrom collections import Counter\nfrom datetime import datetime\nfrom zipfile import ZipFile\nfrom glob import glob\nimport Levenshtein\nimport subprocess\nimport warnings\nimport requests\nimport hashlib\nimport imageio\nimport IPython\nimport sklearn\nimport urllib\nimport zipfile\nimport pickle\nimport random\nimport shutil\nimport string\nimport json\nimport math\nimport time\nimport gzip\nimport ast\nimport sys\nimport io\nimport os\nimport gc\nimport re\n\n# Visualization Imports (overkill)\nfrom matplotlib.colors import ListedColormap, LinearSegmentedColormap\nfrom matplotlib.patches import Rectangle\nimport matplotlib.patches as patches\nimport plotly.graph_objects as go\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import tqdm; tqdm.pandas();\nimport plotly.express as px\nimport tifffile as tif\nimport seaborn as sns\nfrom PIL import Image, ImageEnhance; Image.MAX_IMAGE_PIXELS = 5_000_000_000;\nimport matplotlib; print(f\"\\t\\t– MATPLOTLIB VERSION: {matplotlib.__version__}\");\nfrom matplotlib import animation, rc; rc('animation', html='jshtml')\nimport plotly\nimport PIL\nimport cv2\n\nimport plotly.io as pio\nprint(pio.renderers)\n\ndef check_nvidia_gpu():\n try:\n subprocess.check_output('nvidia-smi')\n print('\\n... NVIDIA GPU DETECTED ...\\n')\n return True\n except Exception: # this command not being found can raise quite a few different errors depending on the configuration\n print('\\n... NO NVIDIA GPU DETECTED ...\\n')\n return False\n\ndef seed_it_all(seed=7):\n \"\"\" Attempt to be Reproducible \"\"\"\n os.environ['PYTHONHASHSEED'] = str(seed)\n random.seed(seed)\n np.random.seed(seed)\n tf.random.set_seed(seed)\n\nseed_it_all()\n\n# Check if GPU attached and imports RAPIDS if so.\nHAS_GPU = check_nvidia_gpu()\nif HAS_GPU: import cudf, cuml, cupy\n\nprint(\"\\n\\n... IMPORTS COMPLETE ...\\n\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-01-30T01:15:01.821707Z","iopub.execute_input":"2023-01-30T01:15:01.822236Z","iopub.status.idle":"2023-01-30T01:15:08.851525Z","shell.execute_reply.started":"2023-01-30T01:15:01.822125Z","shell.execute_reply":"2023-01-30T01:15:08.849716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

5  SETUP AND HELPER FUNCTIONS    ⤒

","metadata":{}},{"cell_type":"markdown","source":"
\n\n

5.1 HELPER FUNCTIONS

\n\n---\n\n
","metadata":{}},{"cell_type":"code","source":"def angular_dist_score(az_true, zen_true, az_pred, zen_pred):\n '''\n calculate the MAE of the angular distance between two directions.\n The two vectors are first converted to cartesian unit vectors,\n and then their scalar product is computed, which is equal to\n the cosine of the angle between the two vectors. The inverse \n cosine (arccos) thereof is then the angle between the two input vectors\n \n Parameters:\n -----------\n \n az_true : float (or array thereof)\n true azimuth value(s) in radian\n zen_true : float (or array thereof)\n true zenith value(s) in radian\n az_pred : float (or array thereof)\n predicted azimuth value(s) in radian\n zen_pred : float (or array thereof)\n predicted zenith value(s) in radian\n \n Returns:\n --------\n \n dist : float\n mean over the angular distance(s) in radian\n '''\n \n if not (np.all(np.isfinite(az_true)) and\n np.all(np.isfinite(zen_true)) and\n np.all(np.isfinite(az_pred)) and\n np.all(np.isfinite(zen_pred))):\n raise ValueError(\"All arguments must be finite\")\n \n # pre-compute all sine and cosine values\n sa1 = np.sin(az_true)\n ca1 = np.cos(az_true)\n sz1 = np.sin(zen_true)\n cz1 = np.cos(zen_true)\n \n sa2 = np.sin(az_pred)\n ca2 = np.cos(az_pred)\n sz2 = np.sin(zen_pred)\n cz2 = np.cos(zen_pred)\n \n # scalar product of the two cartesian vectors (x = sz*ca, y = sz*sa, z = cz)\n scalar_prod = sz1*sz2*(ca1*ca2 + sa1*sa2) + (cz1*cz2)\n \n # scalar product of two unit vectors is always between -1 and 1, this is against nummerical instability\n # that might otherwise occure from the finite precision of the sine and cosine functions\n scalar_prod = np.clip(scalar_prod, -1, 1)\n \n # convert back to an angle (in radian)\n return np.average(np.abs(np.arccos(scalar_prod)))\n\ndef flatten_l_o_l(nested_list):\n \"\"\" Flatten a list of lists \"\"\"\n return [item for sublist in nested_list for item in sublist]\n\ndef print_ln(symbol=\"-\", line_len=110):\n print(symbol*line_len)","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-01-30T01:15:08.854885Z","iopub.execute_input":"2023-01-30T01:15:08.855408Z","iopub.status.idle":"2023-01-30T01:15:08.867702Z","shell.execute_reply.started":"2023-01-30T01:15:08.855359Z","shell.execute_reply":"2023-01-30T01:15:08.866835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

5.2 LOAD THE DATA

\n\n---\n\nMuch of the data is stored in **`.parquet`** files. We will want to open all of this in pandas for easier viewing.","metadata":{}},{"cell_type":"code","source":"# Define the path to the root data directory\nUSE_CUDF = False\nDATA_DIR = \"/kaggle/input/icecube-neutrinos-in-deep-ice\"\n\n# Define the paths to the batches of train and test data respectively\nTRAIN_BATCH_DIR = os.path.join(DATA_DIR, \"train\")\nTEST_BATCH_DIR = os.path.join(DATA_DIR, \"test\")\n\nprint(\"\\n... BASIC DATA SETUP STARTING ...\\n\")\nprint(\"\\n\\n... LOAD TRAIN META DATAFRAME FROM PARQUET FILE ...\\n\")\nif (USE_CUDF and HAS_GPU):\n train_meta_df = cudf.read_parquet(os.path.join(DATA_DIR, \"train_meta.parquet\"))\nelse:\n train_meta_df = pd.read_parquet(os.path.join(DATA_DIR, \"train_meta.parquet\"))\ntrain_meta_df[\"n_time_steps\"] = train_meta_df[\"last_pulse_index\"]-train_meta_df[\"first_pulse_index\"]\ndisplay(train_meta_df)\n\nprint(\"\\n\\n... LOAD TEST META DATAFRAME FROM PARQUET FILE ...\\n\")\nif (USE_CUDF and HAS_GPU):\n test_meta_df = cudf.read_parquet(os.path.join(DATA_DIR, \"test_meta.parquet\"))\nelse:\n test_meta_df = pd.read_parquet(os.path.join(DATA_DIR, \"test_meta.parquet\"))\ntest_meta_df[\"n_time_steps\"] = test_meta_df[\"last_pulse_index\"]-test_meta_df[\"first_pulse_index\"]\ndisplay(test_meta_df)\n\nprint(\"\\n\\n... LOAD SAMPLE SUBMISSION DATAFRAME FROM PARQUET FILE ...\\n\")\nss_df = pd.read_parquet(os.path.join(DATA_DIR, \"sample_submission.parquet\"))\ndisplay(ss_df)\n\nprint(\"\\n\\n... LOAD SENSOR GEOMETRY DATAFRAME FROM CSV FILE ...\\n\")\ns_geo_df = pd.read_csv(os.path.join(DATA_DIR, \"sensor_geometry.csv\"))\ndisplay(s_geo_df)\n\nprint(\"\\n\\n\\n... BASIC DATA SETUP FINISHED ...\\n\\n\")","metadata":{"execution":{"iopub.status.busy":"2023-01-30T01:15:23.843353Z","iopub.execute_input":"2023-01-30T01:15:23.844102Z","iopub.status.idle":"2023-01-30T01:16:14.470356Z","shell.execute_reply.started":"2023-01-30T01:15:23.844063Z","shell.execute_reply":"2023-01-30T01:16:14.469126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\n... TRAIN METADATA DESCRIPTION ...\\n\")\ndisplay(train_meta_df.describe())\n\nprint(\"\\n... TRAIN METADATA DATAFRAME COLUMN INFO ...\\n\")\ntrain_meta_df.info()","metadata":{"execution":{"iopub.status.busy":"2023-01-29T21:10:38.592729Z","iopub.execute_input":"2023-01-29T21:10:38.593192Z","iopub.status.idle":"2023-01-29T21:10:42.247793Z","shell.execute_reply.started":"2023-01-29T21:10:38.593154Z","shell.execute_reply":"2023-01-29T21:10:42.246723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\n... TRAIN BATCH 1 DATAFRAME ...\")\ndisplay(pd.read_parquet(os.path.join(TRAIN_BATCH_DIR, \"batch_1.parquet\")))\n\nprint(\"\\n\\n... TEST BATCH 661 DATAFRAME ...\")\ndisplay(pd.read_parquet(os.path.join(TEST_BATCH_DIR, \"batch_661.parquet\")))","metadata":{"execution":{"iopub.status.busy":"2023-01-29T21:06:07.396186Z","iopub.execute_input":"2023-01-29T21:06:07.396548Z","iopub.status.idle":"2023-01-29T21:06:10.938525Z","shell.execute_reply.started":"2023-01-29T21:06:07.396515Z","shell.execute_reply":"2023-01-29T21:06:10.937554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

5.3 SHOW OFF THE SENSOR ARRAY AS A 3D SCATTER PLOT

\n\n---\n\nWe can easily plot the sensor array as we are provided with the coordinates for each sensor as (x,y,z) points. Note that, the **azimuth and zenith** angles are determined with respect to a spherical coordinate system centered within the sensor array.","metadata":{}},{"cell_type":"code","source":"# https://plotly.com/python-api-reference/generated/plotly.express.scatter_3d.html\nfig = px.scatter_3d(s_geo_df, x='x', y='y', z='z', color=\"z\",\n opacity=0.75, width=700, height=650, \n color_continuous_scale='dense',\n title=\"Geometry and Plot of IceCube's 5160 Sensors\")\nfig.update_traces(marker_size=3)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-27T16:12:50.208321Z","iopub.execute_input":"2023-01-27T16:12:50.208938Z","iopub.status.idle":"2023-01-27T16:12:51.546063Z","shell.execute_reply.started":"2023-01-27T16:12:50.208902Z","shell.execute_reply":"2023-01-27T16:12:51.545145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

5.4 CHECK DATA ACCESS BY EXAMINING THE PROVIDED ILLUSTRATION

\n\n---\n\nLet's find and plot this particular example:\n\n\n\n
","metadata":{}},{"cell_type":"code","source":"def find_nearest_indices(azi_arr, zen_arr, azi_val, zen_val, azi_wt=1.0, zen_wt=1.0, thresh=0.001, k=25, verbose=0):\n \"\"\" Find nearest element in search set\"\"\"\n if verbose:\n print(\"\\n... VALUES WE ARE SEARCHING FOR:\")\n print(f\"\\tAZIMUTH --> {azi_val}\\n\\tZENITH --> {zen_val}\")\n print(f\"\\n... SHAPE OF ARRAY'S TO SEARCH:\")\n print(f\"\\tAZIMUTH --> {azi_arr.shape}\\n\\tZENITH --> {zen_arr.shape}\")\n print(f\"\\n...THRESHOLD TO BE CONSIDERED 'NEAR' IS --> {thresh}\\n\")\n print(\"\\n\\t\\t... BEGINNING SEARCH ...\\n\")\n \n # calculate the difference array\n diff_azi_arr, diff_zen_arr = np.abs(azi_arr-azi_val), np.abs(zen_arr-zen_val)\n diff_arr = (diff_azi_arr*azi_wt) + (diff_zen_arr*zen_wt)\n \n if diff_arr.min()>thresh:\n if verbose: print(\"\\n... NO SIMILAR EVENT FOUND!! ...\\n\")\n return None, None\n \n # find the indices of closest events from the array (based on combination of azimuth and zenith)\n closest_ids = np.argpartition(diff_arr, k)[:k]\n \n # find the actual value that is closest\n closest_azi_vals, closest_zen_vals = azi_arr[closest_ids], zen_arr[closest_ids]\n \n # find the difference value\n diff_azi_vals, diff_zen_vals = diff_azi_arr[closest_ids], diff_zen_arr[closest_ids]\n \n # find the magnitude of each difference for azimuth and\n if verbose:\n for i, (_idx, _azi_val, _zen_val, _diff_azi_val, _diff_zen_val) in enumerate(zip(closest_ids, closest_azi_vals, closest_zen_vals, diff_azi_vals, diff_zen_vals)):\n print(f\"\\n... #{i+1} MOST SIMILAR EVENT FOUND!\")\n print(f\"\\tINDEX OF EVENT --> {_idx}\")\n print(f\"\\tSIMILAR AZIMUTH VALUE --> {_azi_val} (DELTA IS {_diff_azi_val})\")\n print(f\"\\tSIMILAR ZENITH VALUE --> {_zen_val} (DELTA IS {_diff_zen_val})\")\n\n return closest_ids\n\n# Display image shows azimuth and zenith so we call those out and do a lookup!\ni_demo_azi, i_demo_zen = 4.86, 1.96\ntrain_azi_np = train_meta_df[\"azimuth\"].to_numpy()\ntrain_zen_np = train_meta_df[\"zenith\"].to_numpy()\nk_nearest_idxs = find_nearest_indices(train_azi_np, train_zen_np, i_demo_azi, i_demo_zen, verbose=True)\n\n# # Display relevant train dataframe information\n# print(f\"\\n\\n... HERE IS THE SIMILAR EVENT (IDX={i_demo_idx}) FOUND IN OUR TRAINING DATA!!!\\n\")\n# display(train_meta_df.iloc[i_demo_idx].to_frame().T)\n\n# valid_idxs = []\n# for i in tqdm(range(10_000)):\n# _vals = (train_meta_df.iloc[k_nearest_idxs[i]][[\"azimuth\", \"zenith\"]].values)\n# if ((f\"{_vals[0]:.2f}\"==\"4.86\") and (f\"{_vals[1]:.2f}\"==\"1.96\")): \n# valid_idxs.append(k_nearest_idxs[i])\n \n# print(len(valid_idxs))","metadata":{"execution":{"iopub.status.busy":"2023-01-27T16:13:40.251544Z","iopub.execute_input":"2023-01-27T16:13:40.252229Z","iopub.status.idle":"2023-01-27T16:13:45.548652Z","shell.execute_reply.started":"2023-01-27T16:13:40.252159Z","shell.execute_reply":"2023-01-27T16:13:45.547115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_event_details(event_idx, meta_df=train_meta_df, feature_batch_dir=TRAIN_BATCH_DIR, geo_df=s_geo_df):\n event_meta = meta_df.iloc[event_idx]\n _feat_df = pd.read_parquet(os.path.join(feature_batch_dir, f\"batch_{int(event_meta['batch_id'])}.parquet\"))\n event_features = _feat_df.iloc[int(event_meta[\"first_pulse_index\"]) : int(event_meta[\"last_pulse_index\"])+1]\n event_geo_position = geo_df.iloc[event_features[\"sensor_id\"]]\n return event_meta, event_features, event_geo_position\n \ndef do_event__plot(event_meta, event_features, event_geo_position, geo_df=s_geo_df, draw_vector=True, _figsize=(19,12)):\n # if len(demo_event_features[demo_event_features[\"auxiliary\"]==False])<40: return\n \n # Setup the plot\n fig = plt.figure(figsize=_figsize)\n ax1 = plt.subplot(1, 2, 1, projection='3d')\n ax2 = plt.subplot(1, 2, 2, projection='3d')\n plt.tight_layout()\n plt.subplots_adjust(wspace=0.1, hspace=0.0)\n cmap = ListedColormap(np.array(plt.cm.turbo.colors)[np.concatenate((np.zeros(150), np.arange(256), np.ones(200)*255)).astype(int)])\n \n # Create the auxillary event feature map\n mask = ~event_features[\"auxiliary\"]\n \n # Create line details - only used if required\n line_x = math.cos(event_meta[\"azimuth\"]) * math.sin(event_meta[\"zenith\"])\n line_x_details = (-line_x*500, line_x*500)\n line_y = math.sin(event_meta[\"azimuth\"]) * math.sin(event_meta[\"zenith\"])\n line_y_details = (-line_y*500, line_y*500)\n line_z = math.cos(event_meta[\"zenith\"])\n line_z_details = (-line_z*500, line_z*500)\n line_color, line_style, line_width = \"r\", \"-\", 2\n \n for ax, _mask in zip([ax1, ax2], [mask, ~mask]):\n ax.set_xlabel('x', fontweight=\"bold\")\n ax.set_ylabel('y', fontweight=\"bold\")\n ax.set_zlabel('z', fontweight=\"bold\")\n \n ax.view_init(azim=-30, elev=30)\n ax.scatter(geo_df[\"x\"], geo_df[\"y\"], geo_df[\"z\"], \n s=0.3, color='k', alpha=0.2)\n ax.grid(False)\n\n # Primary pulses are mask and auxillary pulses are ~mask\n im = ax.scatter(event_geo_position[\"x\"].to_numpy()[_mask],\n event_geo_position[\"y\"].to_numpy()[_mask],\n event_geo_position[\"z\"].to_numpy()[_mask],\n s=event_features[\"charge\"][_mask]*100,\n c=event_features[\"time\"][_mask],\n cmap=cmap, alpha=0.7,\n vmin=np.min(event_features[\"time\"]),\n vmax=np.max(event_features[\"time\"]))\n \n # Plot if requested\n if draw_vector:\n ax.plot3D(xs=line_x_details, ys=line_y_details, zs=line_z_details,\n color=line_color, linestyle=line_style, linewidth=line_width)\n \n fig.colorbar(im, ax=[ax1, ax2], shrink=0.5, label='time',)\n ax1.set_title('auxiliary == False', y=1.05, fontweight=\"bold\")\n ax2.set_title('auxiliary == True', y=1.05, fontweight=\"bold\")\n fig.suptitle(f\"Example event from the dataset:\\n(azimuth = {event_meta['azimuth']:.2f} rad, zenith = {event_meta['zenith']:.2f} rad)\", fontweight=\"bold\")\n plt.show()\n \nfor i, _idx in enumerate(k_nearest_idxs):\n print(f\"\\n\\n\\n#{i+1} MOST SIMILAR EVENT ...\\n\\n\")\n demo_event_meta, demo_event_features, demo_event_geo_position = get_event_details(_idx)\n do_event__plot(demo_event_meta, demo_event_features, demo_event_geo_position, _figsize=(16,9))","metadata":{"execution":{"iopub.status.busy":"2023-01-27T16:14:40.770778Z","iopub.execute_input":"2023-01-27T16:14:40.771692Z","iopub.status.idle":"2023-01-27T16:16:09.410717Z","shell.execute_reply.started":"2023-01-27T16:14:40.771655Z","shell.execute_reply":"2023-01-27T16:16:09.409430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**What if we frame this as 3D CNN regression and build a colour mapping in 3D space with the given meta??**","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}