{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"In this notebook we will train an LGBM Ranker.\n\nIn his very informative post, [Recommendation Systems for Large Datasets](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364721) [@ravishah1](https://www.kaggle.com/ravishah1) explains how re-ranking models are the industry standard for dealing with datasets like we are presented with in this competition, that is ones with high cardinality categories!\n\nEarlier in this competition I shared a notebook [co-visitation matrix - simplified, imprvd logic 🔥](https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic) which introduces the co-visitation matrix that can be used for candidate generation and scoring. (to read more about co-visitation matrices and how they work, please see [💡 What is the co-visiation matrix, really?](https://www.kaggle.com/competitions/otto-recommender-system/discussion/365358))\n\nHere, we will only look at ranking. I don't expect this notebook to achieve a particularly good score, but it will provide all the low level plumbing needed for training ranking models. One will be able to build on it and improve the result (via for instance adding new candidates generated using co-visitation matrices!).\n\nFor data processing we will use [polars](https://www.pola.rs/). Polars is a very interesting library that I wanted to try for a very long time now. It is written in Rust and embraces running on multiple cores. And I must say it delivers! I liked the API quite a bit and its speed (though in that department `cudf` would still be my first choice!). I am however not touching my GPU quata on Kaggle just yet as I have a couple of things lined up that I would like to share with you that definitely will require the GPU! 🙂\n\nTo simplify the code, I am using a version of the dataset that I shared [here](https://www.kaggle.com/datasets/radek1/otto-train-and-test-data-for-local-validation). No need for dealing with `jsonl` files any longer as it's all `parquet` files now! (Specifically, I am using a version of this dataset that I preprared for local validation [in this notebook](https://www.kaggle.com/code/radek1/a-robust-local-validation-framework).)\n\n## Other resources you might find useful:\n\n* [💡 [2 methods] How-to ensemble predictions 🏅🏅🏅](https://www.kaggle.com/code/radek1/2-methods-how-to-ensemble-predictions)\n* [co-visitation matrix - simplified, imprvd logic 🔥](https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic)\n* [💡 Word2Vec How-to [training and submission]🚀🚀🚀](https://www.kaggle.com/code/radek1/word2vec-how-to-training-and-submission)\n* [local validation tracks public LB perfecty -- here is the setup](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364991)\n* [💡 For my friends from Twitter and LinkedIn -- here is how to dive into this competition 🐳](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368560)\n* [Full dataset processed to CSV/parquet files with optimized memory footprint](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363843)\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"# Data Processing","metadata":{}},{"cell_type":"code","source":"!pip install polars","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:31:24.616341Z","iopub.execute_input":"2023-10-23T08:31:24.617103Z","iopub.status.idle":"2023-10-23T08:31:45.919595Z","shell.execute_reply.started":"2023-10-23T08:31:24.617041Z","shell.execute_reply":"2023-10-23T08:31:45.918166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import polars as pl","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:31:45.923765Z","iopub.execute_input":"2023-10-23T08:31:45.924299Z","iopub.status.idle":"2023-10-23T08:31:46.006847Z","shell.execute_reply.started":"2023-10-23T08:31:45.924240Z","shell.execute_reply":"2023-10-23T08:31:46.004395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/test.parquet')\ntrain_labels = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/test_labels.parquet')\ntesting = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/train.parquet')","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:31:46.009195Z","iopub.execute_input":"2023-10-23T08:31:46.009714Z","iopub.status.idle":"2023-10-23T08:31:58.036434Z","shell.execute_reply.started":"2023-10-23T08:31:46.009677Z","shell.execute_reply":"2023-10-23T08:31:58.035087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:31:58.037928Z","iopub.execute_input":"2023-10-23T08:31:58.038314Z","iopub.status.idle":"2023-10-23T08:31:58.050625Z","shell.execute_reply.started":"2023-10-23T08:31:58.038278Z","shell.execute_reply":"2023-10-23T08:31:58.049403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"testing.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:36:38.788293Z","iopub.execute_input":"2023-10-23T08:36:38.788753Z","iopub.status.idle":"2023-10-23T08:36:38.801656Z","shell.execute_reply.started":"2023-10-23T08:36:38.788712Z","shell.execute_reply":"2023-10-23T08:36:38.800041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:31:58.053842Z","iopub.execute_input":"2023-10-23T08:31:58.054260Z","iopub.status.idle":"2023-10-23T08:31:58.068348Z","shell.execute_reply.started":"2023-10-23T08:31:58.054221Z","shell.execute_reply":"2023-10-23T08:31:58.067086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are calculating the scores that we used for creating co-vistation matrices! We know they carry signal, so let's provde this information to our `LGBM Ranker`!","metadata":{}},{"cell_type":"code","source":"def add_action_num_reverse_chrono(df):\n    return df.select([\n        pl.col('*'),\n        pl.col('session').cumcount().reverse().over('session').alias('action_num_reverse_chrono')\n    ])\n\ndef add_session_length(df):\n    return df.select([\n        pl.col('*'),\n        pl.col('session').count().over('session').alias('session_length')\n    ])\n\ndef add_log_recency_score(df):\n    linear_interpolation = 0.1 + ((1-0.1) / (df['session_length']-1)) * (df['session_length']-df['action_num_reverse_chrono']-1)\n    return df.with_columns(pl.Series(2**linear_interpolation - 1).alias('log_recency_score')).fill_nan(1)\n\ndef add_type_weighted_log_recency_score(df):\n    type_weights = {0:1, 1:6, 2:3}\n    type_weighted_log_recency_score = pl.Series(df['log_recency_score'] / df['type'].apply(lambda x: type_weights[x]))\n    return df.with_columns(type_weighted_log_recency_score.alias('type_weighted_log_recency_score'))\n\ndef apply(df, pipeline):\n    for f in pipeline:\n        df = f(df)\n    return df","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:31:58.069850Z","iopub.execute_input":"2023-10-23T08:31:58.070662Z","iopub.status.idle":"2023-10-23T08:31:58.084199Z","shell.execute_reply.started":"2023-10-23T08:31:58.070589Z","shell.execute_reply":"2023-10-23T08:31:58.082910Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pipeline = [add_action_num_reverse_chrono, add_session_length, add_log_recency_score, add_type_weighted_log_recency_score]","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:31:58.086081Z","iopub.execute_input":"2023-10-23T08:31:58.086516Z","iopub.status.idle":"2023-10-23T08:31:58.096601Z","shell.execute_reply.started":"2023-10-23T08:31:58.086468Z","shell.execute_reply":"2023-10-23T08:31:58.095136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = apply(train, pipeline)","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:31:58.098226Z","iopub.execute_input":"2023-10-23T08:31:58.098727Z","iopub.status.idle":"2023-10-23T08:32:06.161868Z","shell.execute_reply.started":"2023-10-23T08:31:58.098678Z","shell.execute_reply":"2023-10-23T08:32:06.160654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All done!","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:06.163515Z","iopub.execute_input":"2023-10-23T08:32:06.164431Z","iopub.status.idle":"2023-10-23T08:32:06.179398Z","shell.execute_reply.started":"2023-10-23T08:32:06.164378Z","shell.execute_reply":"2023-10-23T08:32:06.177985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we need to process our labels a little bit and merge them onto our train set.","metadata":{}},{"cell_type":"code","source":"type2id = {\"clicks\": 0, \"carts\": 1, \"orders\": 2}","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:06.181011Z","iopub.execute_input":"2023-10-23T08:32:06.181369Z","iopub.status.idle":"2023-10-23T08:32:06.187231Z","shell.execute_reply.started":"2023-10-23T08:32:06.181336Z","shell.execute_reply":"2023-10-23T08:32:06.186072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels = train_labels.explode('ground_truth').with_columns([\n    pl.col('ground_truth').alias('aid'),\n    pl.col('type').apply(lambda x: type2id[x])\n])[['session', 'type', 'aid']]","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:06.189474Z","iopub.execute_input":"2023-10-23T08:32:06.190005Z","iopub.status.idle":"2023-10-23T08:32:07.951906Z","shell.execute_reply.started":"2023-10-23T08:32:06.189937Z","shell.execute_reply":"2023-10-23T08:32:07.950659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels = train_labels.with_columns([\n    pl.col('session').cast(pl.datatypes.Int32),\n    pl.col('type').cast(pl.datatypes.UInt8),\n    pl.col('aid').cast(pl.datatypes.Int32)\n])","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:07.953256Z","iopub.execute_input":"2023-10-23T08:32:07.954075Z","iopub.status.idle":"2023-10-23T08:32:07.999419Z","shell.execute_reply.started":"2023-10-23T08:32:07.954039Z","shell.execute_reply":"2023-10-23T08:32:07.998478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels = train_labels.with_columns(pl.lit(1).alias('gt'))","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:08.000634Z","iopub.execute_input":"2023-10-23T08:32:08.001196Z","iopub.status.idle":"2023-10-23T08:32:08.015016Z","shell.execute_reply.started":"2023-10-23T08:32:08.001159Z","shell.execute_reply":"2023-10-23T08:32:08.013618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.join(train_labels, how='left', on=['session', 'type', 'aid']).with_columns(pl.col('gt').fill_null(0))","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:08.019874Z","iopub.execute_input":"2023-10-23T08:32:08.020303Z","iopub.status.idle":"2023-10-23T08:32:08.949938Z","shell.execute_reply.started":"2023-10-23T08:32:08.020264Z","shell.execute_reply":"2023-10-23T08:32:08.948840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-23T09:31:56.577881Z","iopub.execute_input":"2023-10-23T09:31:56.578929Z","iopub.status.idle":"2023-10-23T09:31:56.591798Z","shell.execute_reply.started":"2023-10-23T09:31:56.578890Z","shell.execute_reply":"2023-10-23T09:31:56.590343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok, so we now have our preprocessed dataset, a column with ground truth, which means that the only thing we are missing for our Ranker is... information how to group individual rows into sessions!","metadata":{}},{"cell_type":"code","source":"def get_session_lenghts(df):\n    return df.groupby('session').agg([\n        pl.col('session').count().alias('session_length')\n    ])['session_length'].to_numpy()","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:08.971313Z","iopub.execute_input":"2023-10-23T08:32:08.972230Z","iopub.status.idle":"2023-10-23T08:32:08.980108Z","shell.execute_reply.started":"2023-10-23T08:32:08.972159Z","shell.execute_reply":"2023-10-23T08:32:08.978936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"session_lengths_train = get_session_lenghts(train)","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:08.981695Z","iopub.execute_input":"2023-10-23T08:32:08.982051Z","iopub.status.idle":"2023-10-23T08:32:09.329002Z","shell.execute_reply.started":"2023-10-23T08:32:08.982019Z","shell.execute_reply":"2023-10-23T08:32:09.327988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(session_lengths_train)","metadata":{"execution":{"iopub.status.busy":"2023-10-23T09:19:26.322762Z","iopub.execute_input":"2023-10-23T09:19:26.323541Z","iopub.status.idle":"2023-10-23T09:19:26.331748Z","shell.execute_reply.started":"2023-10-23T09:19:26.323501Z","shell.execute_reply":"2023-10-23T09:19:26.330293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model training","metadata":{}},{"cell_type":"code","source":"from lightgbm.sklearn import LGBMRanker","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:09.333518Z","iopub.execute_input":"2023-10-23T08:32:09.334953Z","iopub.status.idle":"2023-10-23T08:32:10.880294Z","shell.execute_reply.started":"2023-10-23T08:32:09.334899Z","shell.execute_reply":"2023-10-23T08:32:10.879293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ranker = LGBMRanker(\n    objective=\"lambdarank\",\n    metric=\"ndcg\",\n    boosting_type=\"dart\",\n    n_estimators=20,\n    importance_type='gain',\n)","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:10.881913Z","iopub.execute_input":"2023-10-23T08:32:10.882635Z","iopub.status.idle":"2023-10-23T08:32:10.888335Z","shell.execute_reply.started":"2023-10-23T08:32:10.882592Z","shell.execute_reply":"2023-10-23T08:32:10.887001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.columns","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:10.889662Z","iopub.execute_input":"2023-10-23T08:32:10.890047Z","iopub.status.idle":"2023-10-23T08:32:10.903435Z","shell.execute_reply.started":"2023-10-23T08:32:10.890012Z","shell.execute_reply":"2023-10-23T08:32:10.902142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_cols = ['aid', 'type', 'action_num_reverse_chrono', 'session_length', 'log_recency_score', 'type_weighted_log_recency_score']\ntarget = 'gt'","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:10.905693Z","iopub.execute_input":"2023-10-23T08:32:10.906684Z","iopub.status.idle":"2023-10-23T08:32:10.915552Z","shell.execute_reply.started":"2023-10-23T08:32:10.906631Z","shell.execute_reply":"2023-10-23T08:32:10.914652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ranker = ranker.fit(\n    train[feature_cols].to_pandas(),\n    train[target].to_pandas(),\n    group=session_lengths_train,\n)","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:10.917330Z","iopub.execute_input":"2023-10-23T08:32:10.918783Z","iopub.status.idle":"2023-10-23T08:32:29.789350Z","shell.execute_reply.started":"2023-10-23T08:32:10.918708Z","shell.execute_reply":"2023-10-23T08:32:29.788367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predict on test data","metadata":{}},{"cell_type":"markdown","source":"Let's load our test set, process it and predict on it.","metadata":{}},{"cell_type":"code","source":"test = pl.read_parquet('../input/otto-full-optimized-memory-footprint/test.parquet')\ntest = apply(test, pipeline)","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:29.790894Z","iopub.execute_input":"2023-10-23T08:32:29.795349Z","iopub.status.idle":"2023-10-23T08:32:38.008954Z","shell.execute_reply.started":"2023-10-23T08:32:29.795295Z","shell.execute_reply":"2023-10-23T08:32:38.007672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores = ranker.predict(test[feature_cols].to_pandas())\nprint(scores)","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:38.013426Z","iopub.execute_input":"2023-10-23T08:32:38.014330Z","iopub.status.idle":"2023-10-23T08:32:41.114135Z","shell.execute_reply.started":"2023-10-23T08:32:38.014286Z","shell.execute_reply":"2023-10-23T08:32:41.113099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create submission","metadata":{}},{"cell_type":"code","source":"test = test.with_columns(pl.Series(name='score', values=scores))\ntest_predictions = test.sort(['session', 'score'], descending =True).groupby('session').agg([\n    pl.col(\"aid\").limit(20)\n])","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:41.119699Z","iopub.execute_input":"2023-10-23T08:32:41.122312Z","iopub.status.idle":"2023-10-23T08:32:41.781293Z","shell.execute_reply.started":"2023-10-23T08:32:41.122254Z","shell.execute_reply":"2023-10-23T08:32:41.779936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-23T10:10:54.717740Z","iopub.execute_input":"2023-10-23T10:10:54.719077Z","iopub.status.idle":"2023-10-23T10:10:54.735291Z","shell.execute_reply.started":"2023-10-23T10:10:54.718999Z","shell.execute_reply":"2023-10-23T10:10:54.733304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_predictions.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-23T10:07:57.193873Z","iopub.execute_input":"2023-10-23T10:07:57.194337Z","iopub.status.idle":"2023-10-23T10:07:57.202999Z","shell.execute_reply.started":"2023-10-23T10:07:57.194295Z","shell.execute_reply":"2023-10-23T10:07:57.201864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"session_types = []\nlabels = []\n\nfor session, preds in zip(test_predictions['session'].to_numpy(), test_predictions['aid'].to_numpy()):\n    l = ' '.join(str(p) for p in preds)\n    for session_type in ['clicks', 'carts', 'orders']:\n        labels.append(l)\n        session_types.append(f'{session}_{session_type}')","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:41.783232Z","iopub.execute_input":"2023-10-23T08:32:41.783656Z","iopub.status.idle":"2023-10-23T08:32:58.531856Z","shell.execute_reply.started":"2023-10-23T08:32:41.783609Z","shell.execute_reply":"2023-10-23T08:32:58.530553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pl.DataFrame({'session_type': session_types, 'labels': labels})\nsubmission.write_csv('submission.csv')","metadata":{"execution":{"iopub.status.busy":"2023-10-23T08:32:58.533421Z","iopub.execute_input":"2023-10-23T08:32:58.533826Z","iopub.status.idle":"2023-10-23T08:33:02.045649Z","shell.execute_reply.started":"2023-10-23T08:32:58.533788Z","shell.execute_reply":"2023-10-23T08:33:02.044554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-10-23T10:04:59.586900Z","iopub.execute_input":"2023-10-23T10:04:59.587308Z","iopub.status.idle":"2023-10-23T10:04:59.598366Z","shell.execute_reply.started":"2023-10-23T10:04:59.587274Z","shell.execute_reply":"2023-10-23T10:04:59.596899Z"},"trusted":true},"execution_count":null,"outputs":[]}]}