{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Selecting DeepCore Sensors","metadata":{}},{"cell_type":"markdown","source":"The array of DOMs that make up the IceCube detector are largely evenly spaced over the detector volume. However, a subset of sensors make up the 'Deep Core', a subdetector deep in the ice that is expected to have higher sensitivity, due to a combination of factors:\n\n* The ice at the depth of the DeepCore is exceptionally clear.\n* The DeepCore sensors are placed at a higher density relative to the rest of the sensor array.\n* Some of the sensors use higher quality photomultiplier tubes that may be more sensitive to low energy events.\n\nThe DeepCore is made up of 15 strings, 7 of these are repurposed strings from the original IceCube array, 6 are newly placed with 60 higher quality sensors (HQEs), and 2 are newly placed with a mixture of HQEs and original sensors. A description of DeepCore from the IceCube team is available here: https://arxiv.org/pdf/1109.6096.pdf.\n\nFor events that pass through the DeepCore subdetector, it may be the case that this higher density of sensors / higher quality sensors allow for more precise angular resolution of the neutrino's path, and so a separate ML model focused on DeepCore sensors may be a valuable addition to any ensemble.\n\nThe goal of this notebook is to identify the sensors that make up the DeepCore array to facilitate this downstream modelling task. We label sensors with the following columns:\n\nsensor_type:\n* 0: Non-DeepCore sensor\n* 1: DeepCore sensor: Strings repurposed for DeepCore; sensors placed uniformly along the string.\n* 2: DeepCore sensor: Strings placed specifically for DeepCore; sensors only placed in DeepCore region at high (vertical) densities.\n\ndeepcore_region: \n* 0: Non-DeepCore sensor\n* 1: DeepCore sensor, above dust layer\n* 2: DeepCore sensor, below dust layer\n\nThese labels are assigned by the function below.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-02-02T04:42:19.300137Z","iopub.execute_input":"2023-02-02T04:42:19.300556Z","iopub.status.idle":"2023-02-02T04:42:19.306536Z","shell.execute_reply.started":"2023-02-02T04:42:19.300523Z","shell.execute_reply":"2023-02-02T04:42:19.305341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_deepcore_flags(sensor_geometry):\n    '''\n    Given the sensor geometry as a pandas dataframe, identifies sensors that are within the DeepCore sub-detector and specifies the region of DeepCore they reside in.\n    \n    The following flags are determined for each sensor.\n    \n    sensor_type:\n        0: Non-DeepCore sensor\n        1: DeepCore sensor: Strings repurposed for DeepCore; sensors placed uniformly along the string.\n        2: 2: DeepCore sensor: Strings placed specifically for DeepCore; sensors only placed in DeepCore region at higher vertical densities.\n\n    deepcore_region:\n        0: Non-DeepCore sensor\n        1: DeepCore sensor, above dust layer\n        2: DeepCore sensor, below dust layer\n    \n    Parameters:\n    -----------\n    sensor_geomtry: Pandas DataFrame\n        Contains sensor_id's and correspond x,y,z coordinates.\n    \n    Returns:\n    --------\n    df: Pandas DataFrame\n        Contains sensor_id's and columns 'sensor_type' and 'deepcore_region' as described above.\n    '''\n    \n    df = sensor_geometry.copy()\n    df['string_id'] = df['sensor_id'] // 60\n    df['sensor_string_index'] = df['sensor_id'] % 60\n    repurposed_string_ids = [35, 26, 25, 45, 34, 44, 36]\n    df['sensor_type'] = 0\n    df['deepcore_region'] = 0\n    # Label DeepCore sensor types\n    df.loc[df.string_id >= 78, 'sensor_type'] = 2\n    deepcore_above_dust_top = df.query('sensor_type == 2 and sensor_string_index == 0')['z'].max()\n    deepcore_above_dust_bottom = df.query('sensor_type == 2 and sensor_string_index == 9')['z'].min()\n    deepcore_below_dust_top = df.query('sensor_type == 2 and sensor_string_index == 10')['z'].max()\n    deepcore_below_dust_bottom = df.query('sensor_type == 2 and sensor_string_index == 59')['z'].min()\n    df.loc[df.string_id.isin(repurposed_string_ids) & (df.z >= deepcore_above_dust_bottom) & (df.z <= deepcore_above_dust_top), 'sensor_type'] = 1 \n    df.loc[df.string_id.isin(repurposed_string_ids) & (df.z >= deepcore_below_dust_bottom) & (df.z <= deepcore_below_dust_top), 'sensor_type'] = 1 \n    # Label DeepCore regions\n    df.loc[(df.sensor_type != 0) & (df.z >= deepcore_above_dust_bottom) & (df.z <= deepcore_above_dust_top), 'deepcore_region'] = 1\n    df.loc[(df.sensor_type != 0) & (df.z >= deepcore_below_dust_bottom) & (df.z <= deepcore_below_dust_top), 'deepcore_region'] = 2\n    return df[['sensor_id', 'sensor_type', 'deepcore_region']]","metadata":{"execution":{"iopub.status.busy":"2023-02-02T04:42:20.102037Z","iopub.execute_input":"2023-02-02T04:42:20.102467Z","iopub.status.idle":"2023-02-02T04:42:20.114385Z","shell.execute_reply.started":"2023-02-02T04:42:20.102430Z","shell.execute_reply":"2023-02-02T04:42:20.113080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A test which illustrates the sensors that are selected by the above function.","metadata":{}},{"cell_type":"code","source":"PATH_TO_SENSOR_GEOMETRY = '../input/icecube-neutrinos-in-deep-ice/sensor_geometry.csv'\n\nsensor_geometry = pd.read_csv(PATH_TO_SENSOR_GEOMETRY)\ndeepcore_flags = get_deepcore_flags(sensor_geometry)\nsensor_geometry = sensor_geometry.merge(deepcore_flags, on = 'sensor_id', how = 'inner')\n\n\n\ndef plot_sensor_geometry(sensor_geometry, deepcore_query_string):\n    deepcore = sensor_geometry.query(deepcore_query_string)\n    sensor_geometry = sensor_geometry.query('sensor_id not in @deepcore.sensor_id')\n    ax = plt.figure(figsize=(10, 10)).add_subplot(projection='3d')\n    ax.set_xlabel('x')\n    ax.set_ylabel('y')\n    ax.set_zlabel('z')\n    ax.set_box_aspect([1,1,1])\n    ax.view_init(azim=-35, elev=25)\n    ax.scatter(sensor_geometry.x, sensor_geometry.y, sensor_geometry.z, s=10, color='black', alpha=0.2)\n    ax.scatter(deepcore.x, deepcore.y, deepcore.z, s=100, color='blue', alpha=0.2)\n    ax.legend()\n    plt.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2023-02-02T04:42:21.404292Z","iopub.execute_input":"2023-02-02T04:42:21.404682Z","iopub.status.idle":"2023-02-02T04:42:21.445906Z","shell.execute_reply.started":"2023-02-02T04:42:21.404651Z","shell.execute_reply":"2023-02-02T04:42:21.444746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Full DeepCore subdetector\nplot_sensor_geometry(sensor_geometry, 'sensor_type > 0')","metadata":{"execution":{"iopub.status.busy":"2023-02-02T04:42:22.680909Z","iopub.execute_input":"2023-02-02T04:42:22.681323Z","iopub.status.idle":"2023-02-02T04:42:23.162897Z","shell.execute_reply.started":"2023-02-02T04:42:22.681289Z","shell.execute_reply":"2023-02-02T04:42:23.159967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Only new DeepCore strings, mostly containing HQE PMTs at high densities.\nplot_sensor_geometry(sensor_geometry, 'sensor_type == 1')","metadata":{"execution":{"iopub.status.busy":"2023-02-02T04:42:24.175414Z","iopub.execute_input":"2023-02-02T04:42:24.176323Z","iopub.status.idle":"2023-02-02T04:42:24.563529Z","shell.execute_reply.started":"2023-02-02T04:42:24.176278Z","shell.execute_reply":"2023-02-02T04:42:24.562143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Main subdetector below dust layer\nplot_sensor_geometry(sensor_geometry, 'deepcore_region == 2')","metadata":{"execution":{"iopub.status.busy":"2023-02-02T04:42:24.635917Z","iopub.execute_input":"2023-02-02T04:42:24.636337Z","iopub.status.idle":"2023-02-02T04:42:25.146742Z","shell.execute_reply.started":"2023-02-02T04:42:24.636304Z","shell.execute_reply":"2023-02-02T04:42:25.145277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The rest of the notebook is a walkthrough of how the appropriate sensors were identified.","metadata":{}},{"cell_type":"markdown","source":"### First look at the sensor geometry","metadata":{"execution":{"iopub.status.busy":"2023-02-02T02:17:53.174380Z","iopub.execute_input":"2023-02-02T02:17:53.174728Z","iopub.status.idle":"2023-02-02T02:17:54.844920Z","shell.execute_reply.started":"2023-02-02T02:17:53.174700Z","shell.execute_reply":"2023-02-02T02:17:54.844265Z"}}},{"cell_type":"code","source":"PATH_TO_SENSOR_GEOMETRY = '../input/icecube-neutrinos-in-deep-ice/sensor_geometry.csv'\n\nsensor_geometry = pd.read_csv(PATH_TO_SENSOR_GEOMETRY)\n\nax = plt.figure(figsize=(10, 10)).add_subplot(projection='3d')\nax.set_xlabel('x')\nax.set_ylabel('y')\nax.set_zlabel('z')\nax.set_box_aspect([1,1,1])\nax.view_init(azim=-35, elev=25)\nax.scatter(sensor_geometry.x, sensor_geometry.y, sensor_geometry.z, s=10, color='black', alpha=0.2)\nax.legend()\nplt.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2023-02-02T04:42:26.460660Z","iopub.execute_input":"2023-02-02T04:42:26.461101Z","iopub.status.idle":"2023-02-02T04:42:26.960435Z","shell.execute_reply.started":"2023-02-02T04:42:26.461057Z","shell.execute_reply":"2023-02-02T04:42:26.959439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The DeepCore can be clearly seen towards the center of the image. \n\nThe higher density region of DOMs is broken into two regions by z. This is because the higher density strings are purposefully not instrumented in the area between regions due to the presence of a dust layer in the ice that reduces measurement quality. We can identify HQE DOMs by looking at the distance between consecutive detectors on a string.","metadata":{}},{"cell_type":"markdown","source":"### Labelling HQE DOMs","metadata":{}},{"cell_type":"code","source":"sensor_geometry['string_id'] = sensor_geometry['sensor_id'] // 60\nsensor_geometry['sensor_string_index'] = sensor_geometry['sensor_id'] % 60\nsensor_geometry['z_next_sensor_on_string'] = sensor_geometry.sort_values(by='z', ascending=False).groupby('string_id')['z'].shift(-1)\nsensor_geometry['delta_z'] = sensor_geometry['z'] - sensor_geometry['z_next_sensor_on_string']\nsensor_geometry.groupby('string_id').min()['delta_z'].plot()","metadata":{"execution":{"iopub.status.busy":"2023-02-02T04:42:27.121148Z","iopub.execute_input":"2023-02-02T04:42:27.122136Z","iopub.status.idle":"2023-02-02T04:42:27.359710Z","shell.execute_reply.started":"2023-02-02T04:42:27.122081Z","shell.execute_reply":"2023-02-02T04:42:27.358405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sensor_geometry.groupby('string_id').min()['delta_z'].tail(15)","metadata":{"execution":{"iopub.status.busy":"2023-02-02T04:42:27.373540Z","iopub.execute_input":"2023-02-02T04:42:27.374427Z","iopub.status.idle":"2023-02-02T04:42:27.388320Z","shell.execute_reply.started":"2023-02-02T04:42:27.374382Z","shell.execute_reply":"2023-02-02T04:42:27.386704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Strings 78-85 contain DOMS spaced ~7m apart, clearly indicating that these are the expected 8 higher density strings that were added in to form DeepCore.\n\nNow we find the neighboring strings that were repurposed for DeepCore, and select the sensors on those strings that are at the same depth as the HQE sensors.","metadata":{}},{"cell_type":"markdown","source":"### DeepCore Repurposed Strings","metadata":{}},{"cell_type":"markdown","source":"First we find the depth of the DeepCore sensors.","metadata":{"execution":{"iopub.status.busy":"2023-02-02T03:08:36.992003Z","iopub.execute_input":"2023-02-02T03:08:36.992378Z","iopub.status.idle":"2023-02-02T03:08:36.997531Z","shell.execute_reply.started":"2023-02-02T03:08:36.992352Z","shell.execute_reply":"2023-02-02T03:08:36.996637Z"}}},{"cell_type":"code","source":"deepcore_hqe = sensor_geometry.query('string_id >= 78')\n# First 10 sensors are above the dust layer, last 50 are below. This creates a discontinuity indicated by the large delta_z for points where sensor_string_index == 9.\ndeepcore_hqe.query('sensor_string_index == 9 or sensor_string_index == 10')","metadata":{"execution":{"iopub.status.busy":"2023-02-02T04:42:28.228147Z","iopub.execute_input":"2023-02-02T04:42:28.228592Z","iopub.status.idle":"2023-02-02T04:42:28.261733Z","shell.execute_reply.started":"2023-02-02T04:42:28.228555Z","shell.execute_reply":"2023-02-02T04:42:28.260685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"deepcore_above_dust_top = deepcore_hqe.query('sensor_string_index == 0')['z'].max()\ndeepcore_above_dust_bottom = deepcore_hqe.query('sensor_string_index == 9')['z'].min()\ndeepcore_below_dust_top = deepcore_hqe.query('sensor_string_index == 10')['z'].max()\ndeepcore_below_dust_bottom = deepcore_hqe.query('sensor_string_index == 59')['z'].min()","metadata":{"execution":{"iopub.status.busy":"2023-02-02T04:42:28.426466Z","iopub.execute_input":"2023-02-02T04:42:28.427601Z","iopub.status.idle":"2023-02-02T04:42:28.448593Z","shell.execute_reply.started":"2023-02-02T04:42:28.427549Z","shell.execute_reply":"2023-02-02T04:42:28.447102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next we find the strings neighboring the HQE strings.","metadata":{}},{"cell_type":"code","source":"non_hqe_strings = sensor_geometry.query('string_id < 78')\nplt.scatter(non_hqe_strings.query('sensor_string_index == 0').x,  non_hqe_strings.query('sensor_string_index == 0').y, c = 'b')\nplt.scatter(deepcore_hqe.query('sensor_string_index == 0').x,  deepcore_hqe.query('sensor_string_index == 0').y, c = 'orange')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-02T04:42:28.823655Z","iopub.execute_input":"2023-02-02T04:42:28.824217Z","iopub.status.idle":"2023-02-02T04:42:29.072151Z","shell.execute_reply.started":"2023-02-02T04:42:28.824172Z","shell.execute_reply":"2023-02-02T04:42:29.070679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"string_geometry = sensor_geometry.groupby('string_id').mean()[['x', 'y']]\nstring_geometry['deepcore_x'] = string_geometry.query('string_id >= 78')['x'].mean()\nstring_geometry['deepcore_y'] = string_geometry.query('string_id >= 78')['y'].mean()\n\n# Distance to center of DeepCore\nstring_geometry['distance_to_deepcore_squared'] = (string_geometry.x - string_geometry.deepcore_x) ** 2 + (string_geometry.y - string_geometry.deepcore_y) ** 2\nstring_geometry.query('string_id < 78').sort_values('distance_to_deepcore_squared', ascending=True).head(20)","metadata":{"execution":{"iopub.status.busy":"2023-02-02T04:42:29.074474Z","iopub.execute_input":"2023-02-02T04:42:29.074980Z","iopub.status.idle":"2023-02-02T04:42:29.124436Z","shell.execute_reply.started":"2023-02-02T04:42:29.074933Z","shell.execute_reply":"2023-02-02T04:42:29.122995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We know that 7 strings were repurposed for DeepCore, and there is a clear discontinuity that appears in squared distance for the 7th and 8th string. So, we've found the repurposed DeepCore strings: 35, 26, 25, 45, 34, 44, 36. To find the specific sensors at the depth of DeepCore, we would just need query on z values found above.","metadata":{}},{"cell_type":"code","source":"repurposed_string_ids = [35, 26, 25, 45, 34, 44, 36]\nnon_hqe_strings = sensor_geometry.query('string_id < 78 and string_id not in @repurposed_string_ids')\nrepurposed_strings = sensor_geometry.query('string_id in @repurposed_string_ids')\nplt.scatter(non_hqe_strings.query('sensor_string_index == 0').x,  non_hqe_strings.query('sensor_string_index == 0').y, c = 'orange')\nplt.scatter(deepcore_hqe.query('sensor_string_index == 0').x,  deepcore_hqe.query('sensor_string_index == 0').y, c = 'b')\nplt.scatter(repurposed_strings.query('sensor_string_index == 0').x,  repurposed_strings.query('sensor_string_index == 0').y, c = 'cyan')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-02T04:42:29.469760Z","iopub.execute_input":"2023-02-02T04:42:29.470731Z","iopub.status.idle":"2023-02-02T04:42:29.693263Z","shell.execute_reply.started":"2023-02-02T04:42:29.470683Z","shell.execute_reply":"2023-02-02T04:42:29.692124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Aligns with Fig 6 of linked arxiv paper.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}