{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Can we predict neutrino movement by connecting two dots?\n\n## Idea\n\nWe know that neutrinos are extremely difficult to detect, due to their low mass and lack of electric charge. So what does IceCube actually detect? \nIt detects charges particles such as electrons and muons as neutrinos interact with the particles of waters. This is called Cherenkov's effect and the radiation wave coming from this kind of interaction is actually what we are detecting in IceCube. \n\nOf course the actual path of leptons travelling through IceCube is very convoluted and thus we have this problem in front of us. \n\nBut the idea in this notebook - how good can two dots represents a neutrino's path?\n\nNow of course, this is kind of a naive approach but worth trying. \n\n## What will we do?\n\n1. Install Polars and use Polars to speed things up\n2. Calculate MAE (mean angular error) for each of the pairs of sensors in events\n3. Analyze the results. \n4. Try to predict neutrino's movement","metadata":{}},{"cell_type":"code","source":"#!pip install - q polars\n!pip install -q ../input/polars01516/typing_extensions-4.4.0-py3-none-any.whl\n!pip install -q ../input/polars01516/polars-0.15.16-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-03-02T18:18:20.025516Z","iopub.execute_input":"2023-03-02T18:18:20.026447Z","iopub.status.idle":"2023-03-02T18:19:39.304352Z","shell.execute_reply.started":"2023-03-02T18:18:20.026387Z","shell.execute_reply":"2023-03-02T18:19:39.302365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport polars as pl\nimport seaborn as sns\nimport math\n\nfrom pathlib import Path\nfrom matplotlib import pyplot as plt\nfrom tqdm import tqdm\nfrom sklearn.model_selection import KFold, GridSearchCV\nfrom catboost import CatBoostRegressor, Pool, metrics, cv","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:19:39.307889Z","iopub.execute_input":"2023-03-02T18:19:39.308302Z","iopub.status.idle":"2023-03-02T18:19:39.316948Z","shell.execute_reply.started":"2023-03-02T18:19:39.308257Z","shell.execute_reply":"2023-03-02T18:19:39.315705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Configuration\nTEST_MODE = False\nPATH_INPUT = Path(\"/kaggle/input/icecube-neutrinos-in-deep-ice\")","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:19:39.318266Z","iopub.execute_input":"2023-03-02T18:19:39.318588Z","iopub.status.idle":"2023-03-02T18:19:39.331811Z","shell.execute_reply.started":"2023-03-02T18:19:39.318557Z","shell.execute_reply":"2023-03-02T18:19:39.330716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Constructing training dataset\n\n* We will construct a training dataset by finding the best pairs in each event with minimum MAE (mean angular error)\n\n1. We take a random batch on which we will train the dataset\n2. We generate all possible pairs of points (sensor 1, sensor 2) where we can fit a line\n3. Join the datasets\n4. Calculate azimuth and zenith for a given pair of dots\n5. Calculate MAE\n6. We take smaller training sample - only 10k events to make it time efficient\n\nEverything is done in Polars through pipe function","metadata":{}},{"cell_type":"code","source":"np.random.seed(0)\ntrain_batch_id = np.random.choice(range(660))\nprint('Training batch', train_batch_id)\nbatch_path = \"train/batch_\" + str(train_batch_id)+ \".parquet\" \ntrain_batch = pl.scan_parquet(PATH_INPUT / batch_path).lazy()\ndf_train_meta = pl.scan_parquet(PATH_INPUT / \"train_meta.parquet\").lazy()\ndf_sensor_geometry = pl.scan_csv(PATH_INPUT / 'sensor_geometry.csv').with_columns(pl.col('sensor_id').cast(pl.Int16)).lazy()","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:19:39.333665Z","iopub.execute_input":"2023-03-02T18:19:39.334269Z","iopub.status.idle":"2023-03-02T18:19:39.357358Z","shell.execute_reply.started":"2023-03-02T18:19:39.334220Z","shell.execute_reply":"2023-03-02T18:19:39.356324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def generate_pairs(dataf):\n    return dataf.groupby(['event_id']).agg(pl.col('sensor_id').unique().alias('sensor_id_1')).explode(pl.col('sensor_id_1')\n        ).join(dataf.groupby(['event_id']).agg(pl.col('sensor_id').unique().alias('sensor_id_2')), on='event_id').explode('sensor_id_2'\n        ).unique().filter(pl.col('sensor_id_1') != pl.col('sensor_id_2'))","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:19:39.361084Z","iopub.execute_input":"2023-03-02T18:19:39.361542Z","iopub.status.idle":"2023-03-02T18:19:39.369051Z","shell.execute_reply.started":"2023-03-02T18:19:39.361490Z","shell.execute_reply":"2023-03-02T18:19:39.367799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def split_by_events(dataf, number_of_events = 0, event_ids = []):\n    assert number_of_events > 0 or len(event_ids) > 0\n    if len(event_ids) == 0:\n        np.random.seed(0)\n        event_ids = np.random.choice(np.array(dataf.select(pl.col('event_id')).unique().collect()).reshape(-1), size = number_of_events, replace=False)\n        return dataf.filter(\n            pl.col('event_id').is_in(pl.Series(event_ids))\n        ).with_columns([\n            (pl.col('time') - pl.col('time').min()).over('event_id')\n        ])\n    else:\n        return dataf.filter(\n            pl.col('event_id').is_in(pl.Series(event_ids))\n        ).with_columns([\n            (pl.col('time') - pl.col('time').min()).over('event_id')\n        ])            ","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:19:39.371957Z","iopub.execute_input":"2023-03-02T18:19:39.372443Z","iopub.status.idle":"2023-03-02T18:19:39.383724Z","shell.execute_reply.started":"2023-03-02T18:19:39.372391Z","shell.execute_reply":"2023-03-02T18:19:39.382630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def join_data(dataf, metaf, TEST_MODE=False):\n    if TEST_MODE == True:\n        return dataf.join(df_sensor_geometry,left_on = 'sensor_id_1', right_on ='sensor_id'\n             ).join(df_sensor_geometry,left_on = 'sensor_id_2', right_on ='sensor_id'\n             ).with_columns([\n        pl.col('x').alias('x1'),\n        pl.col('y').alias('y1'),\n        pl.col('z').alias('z1'),\n        pl.col('x_right').alias('x2'),\n        pl.col('y_right').alias('y2'),\n        pl.col('z_right').alias('z2')\n        ]).join(metaf, on='event_id'\n        ).with_columns([\n        pl.col('azimuth').alias('az_true'),\n        pl.col('zenith').alias('zen_true')\n        ])\n    \n    if TEST_MODE == False:\n        return dataf.join(df_sensor_geometry,left_on = 'sensor_id_1', right_on ='sensor_id'\n             ).join(df_sensor_geometry,left_on = 'sensor_id_2', right_on ='sensor_id'\n             ).with_columns([\n        pl.col('x').alias('x1'),\n        pl.col('y').alias('y1'),\n        pl.col('z').alias('z1'),\n        pl.col('x_right').alias('x2'),\n        pl.col('y_right').alias('y2'),\n        pl.col('z_right').alias('z2')\n        ]).join(metaf, on='event_id'\n        )","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:19:39.385475Z","iopub.execute_input":"2023-03-02T18:19:39.385832Z","iopub.status.idle":"2023-03-02T18:19:39.400503Z","shell.execute_reply.started":"2023-03-02T18:19:39.385798Z","shell.execute_reply":"2023-03-02T18:19:39.399202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def normalize(dataf):\n    return dataf.with_columns([\n    (pl.col('x1') - pl.col('x2')).alias('x'),\n    (pl.col('y1') - pl.col('y2')).alias('y'),\n    (pl.col('z1') - pl.col('z2')).alias('z'),\n    ]).with_columns([\n    pl.col('x') / (pl.col('x') ** 2 + pl.col('y') ** 2 + pl.col('z') ** 2) ** 0.5,\n    pl.col('y') / (pl.col('x') ** 2 + pl.col('y') ** 2 + pl.col('z') ** 2) ** 0.5,\n    pl.col('z') / (pl.col('x') ** 2 + pl.col('y') ** 2 + pl.col('z') ** 2) ** 0.5\n    ])\n\n\ndef add_azimuth_zenith(dataf):\n    return dataf.with_columns([\n    pl.col('z').arccos().alias('zenith')\n]).with_columns([\n    pl.when(pl.col(\"z\").round(2).abs() == 1).then(0).otherwise((pl.col('x') / (pl.col('zenith').sin())).arccos()).alias('azimuth')\n]).with_columns([\n        pl.col('azimuth').fill_nan(0).alias('az_pred'),\n        pl.col('zenith').fill_nan(0).alias('zen_pred')\n    ])","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:19:39.402349Z","iopub.execute_input":"2023-03-02T18:19:39.403213Z","iopub.status.idle":"2023-03-02T18:19:39.416009Z","shell.execute_reply.started":"2023-03-02T18:19:39.403171Z","shell.execute_reply":"2023-03-02T18:19:39.414689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def calculate_mae(dataf):\n    return dataf.with_columns([\n    pl.col('az_true').sin().alias('sa1'),\n    pl.col('az_true').cos().alias('ca1'),\n    pl.col('zen_true').sin().alias('sz1'),\n    pl.col('zen_true').cos().alias('cz1'),\n    \n    pl.col('az_pred').sin().alias('sa2'),\n    pl.col('az_pred').cos().alias('ca2'),\n    pl.col('zen_pred').sin().alias('sz2'),\n    pl.col('zen_pred').cos().alias('cz2')\n]).with_columns([\n        (pl.col('sz1')*pl.col('sz2')*(pl.col('ca1')*pl.col('ca2') + pl.col('sa1')*pl.col('sa2')) + (pl.col('cz1')*pl.col('cz2'))).arccos().abs().alias('mae')\n])","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:19:39.418136Z","iopub.execute_input":"2023-03-02T18:19:39.418499Z","iopub.status.idle":"2023-03-02T18:19:39.434882Z","shell.execute_reply.started":"2023-03-02T18:19:39.418461Z","shell.execute_reply":"2023-03-02T18:19:39.433654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def generate_features(dataf,number_of_events=0, event_ids=[]):\n    return dataf.pipe(split_by_events, number_of_events, event_ids).with_columns([\n    pl.col('time').min().over(['event_id','sensor_id']).alias('sensor_min_time'),\n    pl.col('time').mean().over(['event_id','sensor_id']).alias('sensor_mean_time'),\n    pl.col('time').max().over(['event_id','sensor_id']).alias('sensor_max_time'),\n    pl.col('charge').min().over(['event_id','sensor_id']).alias('sensor_min_charge'),\n    pl.col('charge').mean().over(['event_id','sensor_id']).alias('sensor_mean_charge'),\n    pl.col('charge').max().over(['event_id','sensor_id']).alias('sensor_max_charge'),\n    pl.col('time').count().over(['event_id','sensor_id']).alias('sensor_count'),\n    pl.col('auxiliary').sum().over(['event_id','sensor_id']).alias('sensor_aux_sum'),\n    pl.col('auxiliary').sum().over(['event_id']).alias('overall_aux_sum'),\n    pl.col('time').count().over(['event_id']).alias('overall_count'),\n    pl.col('time').max().over(['event_id']).alias('time_overall'),\n    pl.col('charge').max().over(['event_id']).alias('max_charge_overall'),\n    pl.col('charge').mean().over(['event_id']).alias('mean_charge_overall'),\n    pl.col('charge').min().over(['event_id']).alias('min_charge_overall')\n]).with_columns([\n    pl.when(pl.col('time_overall') != 0).then(pl.col('sensor_min_time') / pl.col('time_overall')).otherwise(1).alias('sensor_min_time_ratio'),\n    pl.when(pl.col('time_overall') != 0).then(pl.col('sensor_mean_time') / pl.col('time_overall')).otherwise(1).alias('sensor_mean_time_ratio'),\n    pl.when(pl.col('time_overall') != 0).then(pl.col('sensor_max_time') / pl.col('time_overall')).otherwise(1).alias('sensor_max_time_ratio'),\n    pl.when(pl.col('overall_count') != 0).then(pl.col('sensor_count') / pl.col('overall_count')).otherwise(1).alias('sensor_count_ratio'),\n    pl.when(pl.col('overall_count') != 0).then(pl.col('sensor_aux_sum') / pl.col('overall_count')).otherwise(1).alias('sensor_aux_count_ratio'),\n    pl.when(pl.col('mean_charge_overall') != 0).then(pl.col('sensor_min_charge') / pl.col('mean_charge_overall')).otherwise(1).alias('sensor_min_charge_ratio'),\n    pl.when(pl.col('mean_charge_overall') != 0).then(pl.col('sensor_mean_charge') / pl.col('mean_charge_overall')).otherwise(1).alias('sensor_mean_charge_ratio'),\n    pl.when(pl.col('mean_charge_overall') != 0).then(pl.col('sensor_max_charge') / pl.col('mean_charge_overall')).otherwise(1).alias('sensor_max_charge_ratio')\n]).groupby(['event_id', 'sensor_id']).agg([\n    pl.col('sensor_min_time').first(),\n    pl.col('sensor_mean_time').first(),\n    pl.col('sensor_max_time').first(),\n    pl.col('sensor_min_charge').first(),\n    pl.col('sensor_mean_charge').first(),\n    pl.col('sensor_max_charge').first(),\n    pl.col('sensor_count').first(),\n    pl.col('sensor_aux_sum').first(),\n    pl.col('overall_aux_sum').first(),\n    pl.col('overall_count').first(),\n    pl.col('time_overall').first(),\n    pl.col('max_charge_overall').first(),\n    pl.col('mean_charge_overall').first(),\n    pl.col('min_charge_overall').first(),\n    pl.col('sensor_min_time_ratio').first(),\n    pl.col('sensor_mean_time_ratio').first(),\n    pl.col('sensor_max_time_ratio').first(),\n    pl.col('sensor_aux_count_ratio').first(),\n    pl.col('sensor_count_ratio').first(),\n    (pl.col('sensor_aux_sum').first() / (pl.col('overall_aux_sum').first() + 1)).alias('sensor_aux_ratio'),\n    pl.col('sensor_min_charge_ratio').first(),\n    pl.col('sensor_mean_charge_ratio').first(),\n    pl.col('sensor_max_charge_ratio').first()\n])\n","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:19:39.436823Z","iopub.execute_input":"2023-03-02T18:19:39.437174Z","iopub.status.idle":"2023-03-02T18:19:39.462079Z","shell.execute_reply.started":"2023-03-02T18:19:39.437141Z","shell.execute_reply":"2023-03-02T18:19:39.460997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def join_features(dataf,initial_df,number_of_events=0, event_ids = []):\n    columns_to_return = [\n             'event_id',\n             'batch_id',\n             'x',\n             'y',\n             'z',\n             'x1',\n             'y1',\n             'z1',\n             'x2',\n             'y2',\n             'z2',\n             'az_pred',\n             'zen_pred',\n             'sensor_min_time',\n             'sensor_mean_time',\n             'sensor_max_time',\n             'sensor_min_charge',\n             'sensor_mean_charge',\n             'sensor_max_charge',\n             'sensor_count',\n             'sensor_aux_sum',\n             'overall_aux_sum',\n             'overall_count',\n             'time_overall',\n             'max_charge_overall',\n             'mean_charge_overall',\n             'min_charge_overall',\n             'sensor_min_time_ratio',\n             'sensor_mean_time_ratio',\n             'sensor_max_time_ratio',\n             'sensor_count_ratio',\n             'sensor_aux_ratio',\n             'sensor_aux_count_ratio',\n             'sensor_min_charge_ratio',\n             'sensor_mean_charge_ratio',\n             'sensor_max_charge_ratio',\n             'sensor_min_time_right',\n             'sensor_mean_time_right',\n             'sensor_max_time_right',\n             'sensor_min_charge_right',\n             'sensor_mean_charge_right',\n             'sensor_max_charge_right',\n             'sensor_count_right',\n             'sensor_aux_sum_right',\n             'sensor_min_time_ratio_right',\n             'sensor_mean_time_ratio_right',\n             'sensor_max_time_ratio_right',\n             'sensor_count_ratio_right',\n             'sensor_aux_count_ratio_right',\n             'sensor_aux_ratio_right',\n             'sensor_min_charge_ratio_right',\n             'sensor_mean_charge_ratio_right',\n             'sensor_max_charge_ratio_right',\n        ]\n    if len(event_ids) == 0:\n        columns_to_return = columns_to_return + ['mae']\n\n    return dataf.join(\n        initial_df.pipe(generate_features, number_of_events, event_ids), left_on=['event_id', 'sensor_id_1'], right_on=['event_id', 'sensor_id']).join(\n        initial_df.pipe(generate_features, number_of_events, event_ids), left_on=['event_id', 'sensor_id_2'], right_on=['event_id', 'sensor_id']\n    ).select(columns_to_return)","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:19:39.463899Z","iopub.execute_input":"2023-03-02T18:19:39.465154Z","iopub.status.idle":"2023-03-02T18:19:39.484200Z","shell.execute_reply.started":"2023-03-02T18:19:39.465113Z","shell.execute_reply":"2023-03-02T18:19:39.482529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def make_smaller_sample(dataf, size = 10):\n    return dataf.filter(\n    (pl.arange(0, pl.count()).shuffle(seed=0).over(\"event_id\") < 10) |\n    (pl.col('mae') == pl.col('mae').min().over(pl.col('event_id'))) |\n    (pl.col('mae') == pl.col('mae').max().over(pl.col('event_id')))\n).unique()","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:19:39.486170Z","iopub.execute_input":"2023-03-02T18:19:39.486803Z","iopub.status.idle":"2023-03-02T18:19:39.504106Z","shell.execute_reply.started":"2023-03-02T18:19:39.486758Z","shell.execute_reply":"2023-03-02T18:19:39.502703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"number_of_events=10000\ntrain_data = train_batch.pipe(split_by_events, number_of_events\n                       ).pipe(generate_pairs\n                       ).pipe(join_data, df_train_meta, True\n                       ).pipe(normalize\n                       ).pipe(add_azimuth_zenith\n                       ).pipe(calculate_mae\n                       ).pipe(make_smaller_sample\n                       ).pipe(join_features,train_batch, number_of_events)","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:19:39.505706Z","iopub.execute_input":"2023-03-02T18:19:39.506369Z","iopub.status.idle":"2023-03-02T18:19:43.566299Z","shell.execute_reply.started":"2023-03-02T18:19:39.506326Z","shell.execute_reply":"2023-03-02T18:19:43.565177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Hyperparameters tuning","metadata":{}},{"cell_type":"markdown","source":"* We have to convert the dataset back to numpy in order to train our CatBoost model\n* We did hyperparameters tuning already (number of iterations and learning rate)","metadata":{}},{"cell_type":"code","source":"train_sample_np = np.array(train_data.collect())\nfeatures_train = train_sample_np[:][2:len(train_sample_np)-1].T\ntarget_train = train_sample_np[:][len(train_sample_np)-1].T\n\ndel train_sample_np\ndel train_data","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:19:43.570120Z","iopub.execute_input":"2023-03-02T18:19:43.570471Z","iopub.status.idle":"2023-03-02T18:21:18.227459Z","shell.execute_reply.started":"2023-03-02T18:19:43.570439Z","shell.execute_reply":"2023-03-02T18:21:18.226232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_catboost = CatBoostRegressor(\n    iterations=3300,\n    loss_function ='RMSE',\n    learning_rate = 0.3,\n    random_seed = 1,\n    od_type = \"Iter\",\n    od_wait = 200,\n    depth = 6,\n    #task_type = \"GPU\",\n    #devices = '0:1',\n    save_snapshot = False,\n)\n\nmodel_catboost.fit(\n    features_train, target_train,\n    verbose=1000,\n);","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:21:18.228952Z","iopub.execute_input":"2023-03-02T18:21:18.229288Z","iopub.status.idle":"2023-03-02T18:24:00.501362Z","shell.execute_reply.started":"2023-03-02T18:21:18.229257Z","shell.execute_reply":"2023-03-02T18:24:00.499932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Fit test data","metadata":{}},{"cell_type":"markdown","source":"* Fitting the test data. The test batches are split into 50 parts in order to fit the memory limit","metadata":{}},{"cell_type":"code","source":"TEST_MODE == False","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:24:00.503504Z","iopub.execute_input":"2023-03-02T18:24:00.504025Z","iopub.status.idle":"2023-03-02T18:24:00.512569Z","shell.execute_reply.started":"2023-03-02T18:24:00.503975Z","shell.execute_reply":"2023-03-02T18:24:00.511269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if TEST_MODE == True:\n    df_test_meta = pl.scan_parquet(PATH_INPUT /  'train_meta.parquet')\n    test_batch_ids = [120, 121, 122]\nelse:\n    df_test_meta = pl.scan_parquet(PATH_INPUT /  'test_meta.parquet')\n    test_batch_ids = [test_event_id for test_event_id in np.array(df_test_meta.select('batch_id').unique().collect()).reshape(-1)]","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:24:00.514579Z","iopub.execute_input":"2023-03-02T18:24:00.515415Z","iopub.status.idle":"2023-03-02T18:24:00.530468Z","shell.execute_reply.started":"2023-03-02T18:24:00.515361Z","shell.execute_reply":"2023-03-02T18:24:00.529018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_test_batch(test_batch_id):\n    if TEST_MODE:\n        test_batch = pl.scan_parquet(PATH_INPUT / ('train/batch_' + str(test_batch_id) +'.parquet'))\n    else:\n        test_batch = pl.scan_parquet(PATH_INPUT / ('test/batch_' + str(test_batch_id) +'.parquet'))\n    return test_batch","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:24:00.532041Z","iopub.execute_input":"2023-03-02T18:24:00.532411Z","iopub.status.idle":"2023-03-02T18:24:00.540239Z","shell.execute_reply.started":"2023-03-02T18:24:00.532375Z","shell.execute_reply":"2023-03-02T18:24:00.538637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def predict_azimuth_zenith(dataf):\n    features_test = np.array(dataf)[:][2:].T\n    return dataf.with_columns([\n    pl.Series(model_catboost.predict(features_test)).alias('mae_predict')\n    ]).select([\n    pl.col(['event_id', 'az_pred', 'zen_pred', 'mae_predict']).sort_by(by=[pl.col('mae_predict')]).head(1).list().over(pl.col('event_id')).flatten()\n]).join(df_train_meta, on = 'event_id').with_columns([\n    pl.col(['event_id', 'az_pred', 'zen_pred', 'mae_predict']),\n    pl.col('azimuth').alias('az_true'),\n    pl.col('zenith').alias('zen_true')])","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:24:00.541967Z","iopub.execute_input":"2023-03-02T18:24:00.542745Z","iopub.status.idle":"2023-03-02T18:24:00.555286Z","shell.execute_reply.started":"2023-03-02T18:24:00.542699Z","shell.execute_reply":"2023-03-02T18:24:00.553476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def predict_azimuth_zenith_test(dataf):\n    features_test = np.array(dataf)[:][2:].T\n    return dataf.with_columns([\n    pl.Series(model_catboost.predict(features_test)).alias('mae_predict')\n    ]).select([\n    pl.col(['event_id', 'az_pred', 'zen_pred', 'mae_predict']).sort_by(by=[pl.col('mae_predict')]).head(1).list().over(pl.col('event_id')).flatten()])","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:24:00.556876Z","iopub.execute_input":"2023-03-02T18:24:00.557281Z","iopub.status.idle":"2023-03-02T18:24:00.573267Z","shell.execute_reply.started":"2023-03-02T18:24:00.557223Z","shell.execute_reply":"2023-03-02T18:24:00.571676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def make_smaller_sample_test(dataf):\n    return dataf.filter(\n    (pl.arange(0, pl.count()).shuffle(seed=0).over(\"event_id\") < 100))","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:24:00.575439Z","iopub.execute_input":"2023-03-02T18:24:00.575910Z","iopub.status.idle":"2023-03-02T18:24:00.585888Z","shell.execute_reply.started":"2023-03-02T18:24:00.575867Z","shell.execute_reply":"2023-03-02T18:24:00.584424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result_df = pl.DataFrame()\nnumber_of_events = 0\n\nif TEST_MODE == True:\n    for batch_sample_id in test_batch_ids:\n        all_event_ids = np.array((df_test_meta.filter(pl.col('batch_id') == batch_sample_id)).select(pl.col('event_id')).unique().collect()).reshape(-1)\n        all_event_ids = np.array_split(all_event_ids, 50)\n        for i, batch_event_ids in tqdm(enumerate(all_event_ids)):\n            test_batch = load_test_batch(batch_sample_id)\n            test_batch = test_batch.pipe(split_by_events, number_of_events, batch_event_ids\n                                        ).pipe(generate_pairs\n                                   ).pipe(join_data\n                                   ).pipe(normalize\n                                   ).pipe(add_azimuth_zenith\n                                   ).pipe(join_features,load_test_batch(batch_sample_id), number_of_events, batch_event_ids\n                                   ).make_smaller_sample_test(dataf).collect()\n            test_batch = test_batch.pipe(predict_azimuth_zenith\n                                   ).pipe(calculate_mae).select(['event_id', 'az_pred', 'zen_pred', 'mae'])\n\n            if len(result_df) == 0:\n                result_df = test_batch\n            else:\n                result_df = pl.concat([result_df, test_batch])\n            del test_batch\n            \nif TEST_MODE == False:\n        for batch_sample_id in test_batch_ids:\n            all_event_ids = np.array((df_test_meta.filter(pl.col('batch_id') == batch_sample_id)).select(pl.col('event_id')).unique().collect()).reshape(-1)\n            all_event_ids = np.array_split(all_event_ids, 20)\n            for i, batch_event_ids in tqdm(enumerate(all_event_ids)):\n                if len(batch_event_ids) == 0:\n                    break\n                test_batch = load_test_batch(batch_sample_id)\n                test_batch = test_batch.pipe(split_by_events, number_of_events, batch_event_ids\n                                            ).pipe(generate_pairs\n                                       ).pipe(join_data, df_test_meta\n                                       ).pipe(normalize\n                                       ).pipe(add_azimuth_zenith\n                                       ).pipe(join_features,load_test_batch(batch_sample_id), number_of_events, batch_event_ids\n                                       ).pipe(make_smaller_sample_test).collect()\n                test_batch = test_batch.pipe(predict_azimuth_zenith_test\n                                       ).select([\n                                        pl.col('event_id'), \n                                        pl.col('az_pred').alias('azimuth'), \n                                        pl.col('zen_pred').alias('zenith')])\n\n                if len(result_df) == 0:\n                    result_df = test_batch\n                else:\n                    result_df = pl.concat([result_df, test_batch])\n                del test_batch\n    ","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:25:25.306441Z","iopub.execute_input":"2023-03-02T18:25:25.307383Z","iopub.status.idle":"2023-03-02T18:25:25.489249Z","shell.execute_reply.started":"2023-03-02T18:25:25.307325Z","shell.execute_reply":"2023-03-02T18:25:25.488185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result_df.sort('event_id').write_csv('submission.csv')","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:25:27.820287Z","iopub.execute_input":"2023-03-02T18:25:27.820751Z","iopub.status.idle":"2023-03-02T18:25:27.827989Z","shell.execute_reply.started":"2023-03-02T18:25:27.820708Z","shell.execute_reply":"2023-03-02T18:25:27.826527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.read_csv('submission.csv')","metadata":{"execution":{"iopub.status.busy":"2023-03-02T18:25:28.538457Z","iopub.execute_input":"2023-03-02T18:25:28.538897Z","iopub.status.idle":"2023-03-02T18:25:28.555915Z","shell.execute_reply.started":"2023-03-02T18:25:28.538857Z","shell.execute_reply":"2023-03-02T18:25:28.554137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion\n\n* In this excersise we tried to do a naive 2-dots method. There are a lot of limitation with this method but mostly is due to computation complexity. The training sample explodes into N^2 in size which makes already quite a heavy dataset - impossible to manage with the given Kaggle restriction.\n* The idea was to experiment and see how well it can do. It obviously doesn't do well, however, it was a nice practice to try out Polars library and recieve some kind of baseline","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}