{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"In this notebook we will train an LGBM Ranker.\n\nIn his very informative post, [Recommendation Systems for Large Datasets](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364721) [@ravishah1](https://www.kaggle.com/ravishah1) explains how re-ranking models are the industry standard for dealing with datasets like we are presented with in this competition, that is ones with high cardinality categories!\n\nEarlier in this competition I shared a notebook [co-visitation matrix - simplified, imprvd logic 🔥](https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic) which introduces the co-visitation matrix that can be used for candidate generation and scoring. (to read more about co-visitation matrices and how they work, please see [💡 What is the co-visiation matrix, really?](https://www.kaggle.com/competitions/otto-recommender-system/discussion/365358))\n\nHere, we will only look at ranking. I don't expect this notebook to achieve a particularly good score, but it will provide all the low level plumbing needed for training ranking models. One will be able to build on it and improve the result (via for instance adding new candidates generated using co-visitation matrices!).\n\nFor data processing we will use [polars](https://www.pola.rs/). Polars is a very interesting library that I wanted to try for a very long time now. It is written in Rust and embraces running on multiple cores. And I must say it delivers! I liked the API quite a bit and its speed (though in that department `cudf` would still be my first choice!). I am however not touching my GPU quata on Kaggle just yet as I have a couple of things lined up that I would like to share with you that definitely will require the GPU! 🙂\n\nTo simplify the code, I am using a version of the dataset that I shared [here](https://www.kaggle.com/datasets/radek1/otto-train-and-test-data-for-local-validation). No need for dealing with `jsonl` files any longer as it's all `parquet` files now! (Specifically, I am using a version of this dataset that I preprared for local validation [in this notebook](https://www.kaggle.com/code/radek1/a-robust-local-validation-framework).)\n\n## You might also find useful:\n\n* [💡 Training an XGBoost Ranker on the GPU with Merlin Models 🔥🔥🔥](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368848)\n* [How to train a Word2Vec model 🚀🚀🚀](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368384)\n* [💡 Can you beat static rules with a ranker model without additional features?](https://www.kaggle.com/competitions/otto-recommender-system/discussion/366474)\n* [🐘 the elephant in the room -- high cardinality of targets and what to do about this](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364722)\n* [📖 What are some good resources to learn about how gradient-boosted tree ranking models work?](https://www.kaggle.com/competitions/otto-recommender-system/discussion/366477)\n* [💡How to ensemble predictions -- a key component to every strong solution 🏅](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368747)\n* [from zero to 60 in 2 seconds or less 🏎️🚓🚓🚓](https://www.kaggle.com/competitions/otto-recommender-system/discussion/367058)\n* [💡What is a good initial goal in the competition? How to improve beyond it? 📈](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368685)\n* [💡How to improve the results of your Approximate Nearest Neighbor search! (annoy)](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368385)\n* [📅 Dataset for local validation created using organizer's repository (parquet files)](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364534)\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"# Data Processing","metadata":{}},{"cell_type":"code","source":"!pip install polars\n!pip install datetime","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:37:39.458239Z","iopub.execute_input":"2023-01-12T11:37:39.458742Z","iopub.status.idle":"2023-01-12T11:38:01.968604Z","shell.execute_reply.started":"2023-01-12T11:37:39.458707Z","shell.execute_reply":"2023-01-12T11:38:01.967261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import polars as pl\nimport datetime\nimport pandas as pd","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:01.971491Z","iopub.execute_input":"2023-01-12T11:38:01.971875Z","iopub.status.idle":"2023-01-12T11:38:01.977653Z","shell.execute_reply.started":"2023-01-12T11:38:01.971837Z","shell.execute_reply":"2023-01-12T11:38:01.976448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"train = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/test.parquet')\ntrain_labels = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/test_labels.parquet')\nprint(train['ts'].max())\nprint(train['ts'].min())\n","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:01.979286Z","iopub.execute_input":"2023-01-12T11:38:01.979635Z","iopub.status.idle":"2023-01-12T11:38:02.506225Z","shell.execute_reply.started":"2023-01-12T11:38:01.979602Z","shell.execute_reply":"2023-01-12T11:38:02.504826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are calculating the scores that we used for creating co-vistation matrices! We know they carry signal, so let's provde this information to our `LGBM Ranker`!","metadata":{}},{"cell_type":"code","source":"def add_action_num_reverse_chrono(df):\n    return df.select([\n        pl.col('*'),\n        pl.col('session').cumcount().reverse().over('session').alias('action_num_reverse_chrono')\n    ])\n\ndef add_session_length(df):\n    return df.select([\n        pl.col('*'),\n        pl.col('session').count().over('session').alias('session_length')\n    ])\n\ndef add_log_recency_score(df):\n    linear_interpolation = 0.1 + ((1-0.1) / (df['session_length']-1)) * (df['session_length']-df['action_num_reverse_chrono']-1)\n    #li = 1 - 0.9*(df['chrono']/ ( df['session_length'] -1))\n    return df.with_columns(pl.Series(2**linear_interpolation - 1).alias('log_recency_score')).fill_nan(1)\n\ndef add_type_weighted_log_recency_score(df):\n    type_weights = {0:1, 1:6, 2:3}\n    type_weighted_log_recency_score = pl.Series(df['type'].apply(lambda x: type_weights[x]) * df['log_recency_score'])\n    return df.with_column(type_weighted_log_recency_score.alias('type_weighted_log_recency_score'))\n\ndef diff_ts(df):\n    df = df.select([\n        pl.col('*'),\n        pl.col('ts').diff().over('session').fill_null(0).alias('ts_diff')\n    ])\n    #df.unique(subset='session', keep='first')['ts_diff'] = 0\n    #hoge = df.select(pl.col(['session', 'ts_diff']).is_first(maintain_order=True))\n    #.col('ts_diff').apply(lambda x:x*0)\n    #print(hoge)\n    #print(df.unique(subset=['session'], keep='first')['ts_diff'])\n    #= df[pl.col('ts_diff').is_duplicated()].fill(0)\n    return df\n\ndef apply(df, pipeline):\n    for f in pipeline:\n        df = f(df)\n    return df\n\n\n# weight調整\n# 1時間を絶対ではなく間隔にする。 \n# 2季節にする。\n# 3商品の出現時間\n# 4商品ごとのfrequency(option: 期間, 重み)\n# 5ある期間内にclick, cart, orderされた回数\n# テストの時間を調べて絶対時間を利用するか検討する。\n\n# 7商品ごと or セッションごとにクラスタリング\n# 商品のカテゴリをまたいだ特徴(?) ゲームを買う人はレッドブルをよく飲み(ゲーム→飲み物)\n# {(aid): (frequencey), (aid2): (f2), (aid3): (f3), ...}\n#\n#           aid1, aid2, aid3, ...\n#session1   2,    3,    4, ...\n#session2   3,    4,    2, ...\n#session3   ..................\n#\n#              ↓\n#          category1, c2, c3, c4, c5\n#session1          1,  0,  1,  1,  0\n#session2          .................\n#session3\n# .\n# .\n# .\n#         category\n# aid1           1\n# aid2           3\n# .\n# .\n# .\n# (設計)\n# 意味付け\n# 具体的な数値変換\n# 特徴ベクトルの作り方\n\n#(開発)\n# コーディング\n\n# cfにから出てくるベクトル\n# co-visitation matrix","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:02.509342Z","iopub.execute_input":"2023-01-12T11:38:02.510298Z","iopub.status.idle":"2023-01-12T11:38:02.523189Z","shell.execute_reply.started":"2023-01-12T11:38:02.510228Z","shell.execute_reply":"2023-01-12T11:38:02.522234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pipeline = [add_action_num_reverse_chrono, add_session_length, add_log_recency_score, add_type_weighted_log_recency_score, diff_ts]","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:02.525796Z","iopub.execute_input":"2023-01-12T11:38:02.526564Z","iopub.status.idle":"2023-01-12T11:38:02.542312Z","shell.execute_reply.started":"2023-01-12T11:38:02.526512Z","shell.execute_reply":"2023-01-12T11:38:02.540935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = apply(train, pipeline)\nnew = (train['ts'].cast(pl.Int64)*1000).alias('ts')\ntrain = train.with_column(new)\ndatetime = train['ts'].cast(pl.Datetime).dt.with_time_unit(\"ms\").alias(\"datetime\")\n\ntrain = train.with_column(datetime)\nhour_label = train.select(pl\\\n                   .when((0 <= pl.col(\"datetime\").dt.hour()) & (pl.col(\"datetime\").dt.hour() < 6))\\\n                   .then(1)\\\n                   .when((6 <= pl.col(\"datetime\").dt.hour()) & (pl.col(\"datetime\").dt.hour() < 12))\\\n                   .then(2)\\\n                   .when((12 <= pl.col(\"datetime\").dt.hour()) & (pl.col(\"datetime\").dt.hour() < 18))\\\n                   .then(3)\\\n                   .when((18 <= pl.col(\"datetime\").dt.hour()) & (pl.col(\"datetime\").dt.hour() < 24))\\\n                   .then(4)\\\n                   .otherwise(-1)\n                   .alias('hour_label')\n                   )['hour_label']\ntrain = train.with_column(hour_label)\ntrain = train.to_dummies(columns=\"hour_label\")\nprint(train.head(50))","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:02.544226Z","iopub.execute_input":"2023-01-12T11:38:02.545027Z","iopub.status.idle":"2023-01-12T11:38:15.162247Z","shell.execute_reply.started":"2023-01-12T11:38:02.544988Z","shell.execute_reply":"2023-01-12T11:38:15.161235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All done!","metadata":{}},{"cell_type":"code","source":"train.head(30)\n\n# X(cell4)の作り方はわかった\n\n# ???\n# train_labelになにが入っているかわからない。\n# \n","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:15.163287Z","iopub.execute_input":"2023-01-12T11:38:15.163587Z","iopub.status.idle":"2023-01-12T11:38:15.190162Z","shell.execute_reply.started":"2023-01-12T11:38:15.163562Z","shell.execute_reply":"2023-01-12T11:38:15.188903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we need to process our labels a little bit and merge them onto our train set.","metadata":{}},{"cell_type":"code","source":"type2id = {\"clicks\": 0, \"carts\": 1, \"orders\": 2}","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:15.191443Z","iopub.execute_input":"2023-01-12T11:38:15.191758Z","iopub.status.idle":"2023-01-12T11:38:15.197572Z","shell.execute_reply.started":"2023-01-12T11:38:15.191729Z","shell.execute_reply":"2023-01-12T11:38:15.196210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels = train_labels.explode('ground_truth').with_columns([\n    pl.col('ground_truth').alias('aid'),\n    pl.col('type').apply(lambda x: type2id[x])\n])[['session', 'type', 'aid']]","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:15.199257Z","iopub.execute_input":"2023-01-12T11:38:15.199983Z","iopub.status.idle":"2023-01-12T11:38:16.672050Z","shell.execute_reply.started":"2023-01-12T11:38:15.199938Z","shell.execute_reply":"2023-01-12T11:38:16.670934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels = train_labels.with_columns([\n    pl.col('session').cast(pl.datatypes.Int32),\n    pl.col('type').cast(pl.datatypes.UInt8),\n    pl.col('aid').cast(pl.datatypes.Int32)\n])","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:16.673640Z","iopub.execute_input":"2023-01-12T11:38:16.674307Z","iopub.status.idle":"2023-01-12T11:38:16.720689Z","shell.execute_reply.started":"2023-01-12T11:38:16.674270Z","shell.execute_reply":"2023-01-12T11:38:16.719659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels = train_labels.with_column(pl.lit(1).alias('gt'))\ntrain_labels","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:16.725126Z","iopub.execute_input":"2023-01-12T11:38:16.725538Z","iopub.status.idle":"2023-01-12T11:38:16.745092Z","shell.execute_reply.started":"2023-01-12T11:38:16.725504Z","shell.execute_reply":"2023-01-12T11:38:16.744053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.join(train_labels, how='left', on=['session', 'type', 'aid']).with_column(pl.col('gt').fill_null(0))","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:16.746698Z","iopub.execute_input":"2023-01-12T11:38:16.747353Z","iopub.status.idle":"2023-01-12T11:38:17.839968Z","shell.execute_reply.started":"2023-01-12T11:38:16.747318Z","shell.execute_reply":"2023-01-12T11:38:17.838967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:17.841636Z","iopub.execute_input":"2023-01-12T11:38:17.842359Z","iopub.status.idle":"2023-01-12T11:38:17.855614Z","shell.execute_reply.started":"2023-01-12T11:38:17.842317Z","shell.execute_reply":"2023-01-12T11:38:17.854091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok, so we now have our preprocessed dataset, a column with ground truth, which means that the only thing we are missing for our Ranker is... information how to group individual rows into sessions!","metadata":{}},{"cell_type":"code","source":"def get_session_lenghts(df):\n    return df.groupby('session').agg([\n        pl.col('session').count().alias('session_length')\n    ])['session_length'].to_numpy()","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:17.861840Z","iopub.execute_input":"2023-01-12T11:38:17.862305Z","iopub.status.idle":"2023-01-12T11:38:17.870789Z","shell.execute_reply.started":"2023-01-12T11:38:17.862267Z","shell.execute_reply":"2023-01-12T11:38:17.869233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"session_lengths_train = get_session_lenghts(train)","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:17.872173Z","iopub.execute_input":"2023-01-12T11:38:17.872544Z","iopub.status.idle":"2023-01-12T11:38:18.857805Z","shell.execute_reply.started":"2023-01-12T11:38:17.872511Z","shell.execute_reply":"2023-01-12T11:38:18.856910Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model training","metadata":{}},{"cell_type":"code","source":"from lightgbm.sklearn import LGBMRanker","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:18.858897Z","iopub.execute_input":"2023-01-12T11:38:18.859419Z","iopub.status.idle":"2023-01-12T11:38:20.402785Z","shell.execute_reply.started":"2023-01-12T11:38:18.859382Z","shell.execute_reply":"2023-01-12T11:38:20.401506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ranker = LGBMRanker(\n    objective=\"lambdarank\",\n    metric=\"ndcg\",\n    boosting_type=\"dart\",\n    n_estimators=20,\n    importance_type='gain',\n)","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:20.404242Z","iopub.execute_input":"2023-01-12T11:38:20.404978Z","iopub.status.idle":"2023-01-12T11:38:20.411754Z","shell.execute_reply.started":"2023-01-12T11:38:20.404933Z","shell.execute_reply":"2023-01-12T11:38:20.410308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.columns","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:20.413492Z","iopub.execute_input":"2023-01-12T11:38:20.413817Z","iopub.status.idle":"2023-01-12T11:38:20.425307Z","shell.execute_reply.started":"2023-01-12T11:38:20.413789Z","shell.execute_reply":"2023-01-12T11:38:20.424106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_cols = ['aid', 'type', 'action_num_reverse_chrono', 'session_length', 'log_recency_score', 'type_weighted_log_recency_score', 'ts_diff', 'hour_label_1','hour_label_2','hour_label_3','hour_label_4']\ntarget = 'gt'","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:20.426702Z","iopub.execute_input":"2023-01-12T11:38:20.427917Z","iopub.status.idle":"2023-01-12T11:38:20.435925Z","shell.execute_reply.started":"2023-01-12T11:38:20.427871Z","shell.execute_reply":"2023-01-12T11:38:20.434497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ranker = ranker.fit(\n    train[feature_cols].to_pandas(),\n    train[target].to_pandas(),\n    group=session_lengths_train,\n)","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:38:20.437123Z","iopub.execute_input":"2023-01-12T11:38:20.437536Z","iopub.status.idle":"2023-01-12T11:38:38.839628Z","shell.execute_reply.started":"2023-01-12T11:38:20.437504Z","shell.execute_reply":"2023-01-12T11:38:38.838462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predict on test data","metadata":{}},{"cell_type":"markdown","source":"Let's load our test set, process it and predict on it.","metadata":{}},{"cell_type":"code","source":"test = pl.read_parquet('../input/otto-full-optimized-memory-footprint/test.parquet')\ntest = apply(test, pipeline)\n\nnew = (test['ts'].cast(pl.Int64)*1000).alias('ts')\ntest = test.with_column(new)\ndatetime = test['ts'].cast(pl.Datetime).dt.with_time_unit(\"ms\").alias(\"datetime\")\n\ntest = test.with_column(datetime)\nhour_label = test.select(pl\\\n                   .when((0 <= pl.col(\"datetime\").dt.hour()) & (pl.col(\"datetime\").dt.hour() < 6))\\\n                   .then(1)\\\n                   .when((6 <= pl.col(\"datetime\").dt.hour()) & (pl.col(\"datetime\").dt.hour() < 12))\\\n                   .then(2)\\\n                   .when((12 <= pl.col(\"datetime\").dt.hour()) & (pl.col(\"datetime\").dt.hour() < 18))\\\n                   .then(3)\\\n                   .when((18 <= pl.col(\"datetime\").dt.hour()) & (pl.col(\"datetime\").dt.hour() < 24))\\\n                   .then(4)\\\n                   .otherwise(-1)\n                   .alias('hour_label')\n                   )['hour_label']\ntest = test.with_column(hour_label)\ntest = test.to_dummies(columns=\"hour_label\")\nprint(test.head(50))","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:44:28.967600Z","iopub.execute_input":"2023-01-12T11:44:28.968065Z","iopub.status.idle":"2023-01-12T11:44:40.900115Z","shell.execute_reply.started":"2023-01-12T11:44:28.968027Z","shell.execute_reply":"2023-01-12T11:44:40.899183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores = ranker.predict(test[feature_cols].to_pandas())","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:44:40.901419Z","iopub.execute_input":"2023-01-12T11:44:40.901729Z","iopub.status.idle":"2023-01-12T11:44:44.283957Z","shell.execute_reply.started":"2023-01-12T11:44:40.901702Z","shell.execute_reply":"2023-01-12T11:44:44.282858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create submission","metadata":{}},{"cell_type":"code","source":"test = test.with_columns(pl.Series(name='score', values=scores))\ntest_predictions = test.sort(['session', 'score'], reverse=True).groupby('session').agg([\n    pl.col('aid').limit(20).list()\n])","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:44:49.687612Z","iopub.execute_input":"2023-01-12T11:44:49.688025Z","iopub.status.idle":"2023-01-12T11:44:50.539084Z","shell.execute_reply.started":"2023-01-12T11:44:49.687991Z","shell.execute_reply":"2023-01-12T11:44:50.537675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"session_types = []\nlabels = []\n\nfor session, preds in zip(test_predictions['session'].to_numpy(), test_predictions['aid'].to_numpy()):\n    l = ' '.join(str(p) for p in preds)\n    for session_type in ['clicks', 'carts', 'orders']:\n        labels.append(l)\n        session_types.append(f'{session}_{session_type}')","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:44:50.540949Z","iopub.execute_input":"2023-01-12T11:44:50.541320Z","iopub.status.idle":"2023-01-12T11:45:01.386205Z","shell.execute_reply.started":"2023-01-12T11:44:50.541290Z","shell.execute_reply":"2023-01-12T11:45:01.384989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pl.DataFrame({'session_type': session_types, 'labels': labels})\nsubmission.write_csv('submission.csv')","metadata":{"execution":{"iopub.status.busy":"2023-01-12T11:45:01.388243Z","iopub.execute_input":"2023-01-12T11:45:01.389049Z","iopub.status.idle":"2023-01-12T11:45:04.273269Z","shell.execute_reply.started":"2023-01-12T11:45:01.389008Z","shell.execute_reply":"2023-01-12T11:45:04.272369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}