{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This notebook is a translated version of [@radek's notebook](https://www.kaggle.com/code/radek1/polars-proof-of-concept-lgbm-ranker) from Polars to Pandas/cuDF, since many here are more familiar with Pandas and can better understand how to build off of @radek's work. If you enjoy this notebook, please leave an upvote on his notebook as well.","metadata":{}},{"cell_type":"markdown","source":"In this notebook we will train an LGBM Ranker.\n\nIn his very informative post, [Recommendation Systems for Large Datasets](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364721) [@ravishah1](https://www.kaggle.com/ravishah1) explains how re-ranking models are the industry standard for dealing with datasets like we are presented with in this competition, that is ones with high cardinality categories!\n\nEarlier in this competition I shared a notebook [co-visitation matrix - simplified, imprvd logic 🔥](https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic) which introduces the co-visitation matrix that can be used for candidate generation and scoring. (to read more about co-visitation matrices and how they work, please see [💡 What is the co-visiation matrix, really?](https://www.kaggle.com/competitions/otto-recommender-system/discussion/365358))\n\nHere, we will only look at ranking. I don't expect this notebook to achieve a particularly good score, but it will provide all the low level plumbing needed for training ranking models. One will be able to build on it and improve the result (via for instance adding new candidates generated using co-visitation matrices!).\n\nTo simplify the code, I am using a version of the dataset that I shared [here](https://www.kaggle.com/datasets/radek1/otto-train-and-test-data-for-local-validation). No need for dealing with `jsonl` files any longer as it's all `parquet` files now! (Specifically, I am using a version of this dataset that I preprared for local validation [in this notebook](https://www.kaggle.com/code/radek1/a-robust-local-validation-framework).)\n\n## Other resources you might find useful:\n\n* [💡 Training an XGBoost Ranker on the GPU with Merlin Models 🔥🔥🔥](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368848)\n* [💡 [2 methods] How-to ensemble predictions 🏅🏅🏅](https://www.kaggle.com/code/radek1/2-methods-how-to-ensemble-predictions)\n* [How to train a Word2Vec model 🚀🚀🚀](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368384)\n* [📖 What are some good resources to learn about how gradient-boosted tree ranking models work?](https://www.kaggle.com/competitions/otto-recommender-system/discussion/366477)\n* [from zero to 60 in 2 seconds or less 🏎️🚓🚓🚓](https://www.kaggle.com/competitions/otto-recommender-system/discussion/367058)\n* [💡What is a good initial goal in the competition? How to improve beyond it? 📈](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368685)\n* [💡How to improve the results of your Approximate Nearest Neighbor search! (annoy)](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368385)\n* [📅 Dataset for local validation created using organizer's repository (parquet files)](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364534)\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"# Data Processing","metadata":{}},{"cell_type":"code","source":"import cudf # import pandas if you're not using gpu","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:17:52.131182Z","iopub.execute_input":"2022-12-11T18:17:52.131556Z","iopub.status.idle":"2022-12-11T18:17:52.347491Z","shell.execute_reply.started":"2022-12-11T18:17:52.131526Z","shell.execute_reply":"2022-12-11T18:17:52.346192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = cudf.read_parquet('../input/otto-train-and-test-data-for-local-validation/test.parquet')\ntrain_labels = cudf.read_parquet('../input/otto-train-and-test-data-for-local-validation/test_labels.parquet')","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:17:52.515474Z","iopub.execute_input":"2022-12-11T18:17:52.516373Z","iopub.status.idle":"2022-12-11T18:17:53.169557Z","shell.execute_reply.started":"2022-12-11T18:17:52.516329Z","shell.execute_reply":"2022-12-11T18:17:53.168585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are calculating the scores that we used for creating co-vistation matrices! We know they carry signal, so let's provde this information to our `LGBM Ranker`!","metadata":{}},{"cell_type":"code","source":"def add_session_length(df):\n    # If not using cuDF, remove .to_pandas()\n    df['session_length'] = df.to_pandas().groupby('session')['ts'].transform('count')\n    return df\n\ndef add_action_num_reverse_chrono(df):\n    df['action_num_reverse_chrono'] = df.session_length - df.groupby('session').cumcount() - 1\n    return df\n\ndef add_log_recency_score(df):\n    linear_interpolation = 0.1 + ((1-0.1) / (df['session_length']-1)) * (df['session_length']-df['action_num_reverse_chrono']-1)\n    df['log_recency_score'] = (2 ** linear_interpolation - 1).fillna(1.0)\n    return df\n\ndef add_type_weighted_log_recency_score(df):\n    type_weights = {0:1, 1:6, 2:3}\n    df['type_weighted_log_recency_score'] = df['log_recency_score'] / df['type'].map(type_weights)\n    return df\n    \ndef apply(df, pipeline):\n    for f in pipeline:\n        df = f(df)\n    return df","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:18:00.205242Z","iopub.execute_input":"2022-12-11T18:18:00.205600Z","iopub.status.idle":"2022-12-11T18:18:00.214133Z","shell.execute_reply.started":"2022-12-11T18:18:00.205570Z","shell.execute_reply":"2022-12-11T18:18:00.213031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pipeline = [add_session_length, add_action_num_reverse_chrono, add_log_recency_score, add_type_weighted_log_recency_score]","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:18:00.362078Z","iopub.execute_input":"2022-12-11T18:18:00.364025Z","iopub.status.idle":"2022-12-11T18:18:00.369946Z","shell.execute_reply.started":"2022-12-11T18:18:00.363995Z","shell.execute_reply":"2022-12-11T18:18:00.368957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = apply(train, pipeline)","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:18:00.549366Z","iopub.execute_input":"2022-12-11T18:18:00.549687Z","iopub.status.idle":"2022-12-11T18:18:01.216303Z","shell.execute_reply.started":"2022-12-11T18:18:00.549620Z","shell.execute_reply":"2022-12-11T18:18:01.215362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All done!","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:18:01.409036Z","iopub.execute_input":"2022-12-11T18:18:01.409355Z","iopub.status.idle":"2022-12-11T18:18:01.437461Z","shell.execute_reply.started":"2022-12-11T18:18:01.409327Z","shell.execute_reply":"2022-12-11T18:18:01.436603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we need to process our labels a little bit and merge them onto our train set.","metadata":{}},{"cell_type":"code","source":"type2id = {\"clicks\": 0, \"carts\": 1, \"orders\": 2}","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:18:02.128249Z","iopub.execute_input":"2022-12-11T18:18:02.128613Z","iopub.status.idle":"2022-12-11T18:18:02.133501Z","shell.execute_reply.started":"2022-12-11T18:18:02.128581Z","shell.execute_reply":"2022-12-11T18:18:02.132554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels = train_labels.explode('ground_truth')\ntrain_labels = train_labels.rename(columns={'ground_truth': 'aid'})\ntrain_labels['type'] = train_labels.type.map(type2id)","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:18:02.531272Z","iopub.execute_input":"2022-12-11T18:18:02.531587Z","iopub.status.idle":"2022-12-11T18:18:02.588823Z","shell.execute_reply.started":"2022-12-11T18:18:02.531559Z","shell.execute_reply":"2022-12-11T18:18:02.587947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels['aid'] = train_labels.aid.astype('int32')\ntrain_labels['type'] = train_labels.type.astype('uint8')\ntrain_labels['session'] = train_labels.session.astype('int32')","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:18:04.562245Z","iopub.execute_input":"2022-12-11T18:18:04.563389Z","iopub.status.idle":"2022-12-11T18:18:04.572948Z","shell.execute_reply.started":"2022-12-11T18:18:04.563342Z","shell.execute_reply":"2022-12-11T18:18:04.571964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels['gt'] = 1","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:18:04.904461Z","iopub.execute_input":"2022-12-11T18:18:04.905179Z","iopub.status.idle":"2022-12-11T18:18:04.912137Z","shell.execute_reply.started":"2022-12-11T18:18:04.905141Z","shell.execute_reply":"2022-12-11T18:18:04.911112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.merge(train_labels, on=['session', 'type', 'aid'], how='left')\ntrain['gt'] = train.gt.fillna(0)\ntrain = train.sort_values('session').reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:18:06.516883Z","iopub.execute_input":"2022-12-11T18:18:06.517704Z","iopub.status.idle":"2022-12-11T18:18:06.571025Z","shell.execute_reply.started":"2022-12-11T18:18:06.517659Z","shell.execute_reply":"2022-12-11T18:18:06.569090Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:18:08.069056Z","iopub.execute_input":"2022-12-11T18:18:08.069339Z","iopub.status.idle":"2022-12-11T18:18:08.098031Z","shell.execute_reply.started":"2022-12-11T18:18:08.069314Z","shell.execute_reply":"2022-12-11T18:18:08.096978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok, so we now have our preprocessed dataset, a column with ground truth, which means that the only thing we are missing for our Ranker is... information how to group individual rows into sessions!","metadata":{}},{"cell_type":"code","source":"def get_session_lengths(df): # Fixed function name typo in original notebook\n    # In radek's version, each execution of this function returns a different list\n    # I don't think that is the intended behavior, and is fixed here\n    return df.groupby('session')['session_length'].count().sort_index().to_pandas().values","metadata":{"execution":{"iopub.status.busy":"2022-12-11T17:56:53.170579Z","iopub.execute_input":"2022-12-11T17:56:53.170946Z","iopub.status.idle":"2022-12-11T17:56:53.177661Z","shell.execute_reply.started":"2022-12-11T17:56:53.170912Z","shell.execute_reply":"2022-12-11T17:56:53.176699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"session_lengths_train = get_session_lengths(train)","metadata":{"execution":{"iopub.status.busy":"2022-12-11T17:56:53.178950Z","iopub.execute_input":"2022-12-11T17:56:53.179552Z","iopub.status.idle":"2022-12-11T17:56:53.224468Z","shell.execute_reply.started":"2022-12-11T17:56:53.179516Z","shell.execute_reply":"2022-12-11T17:56:53.223667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model training","metadata":{}},{"cell_type":"code","source":"from lightgbm.sklearn import LGBMRanker","metadata":{"execution":{"iopub.status.busy":"2022-12-11T17:56:53.225892Z","iopub.execute_input":"2022-12-11T17:56:53.226243Z","iopub.status.idle":"2022-12-11T17:56:54.100926Z","shell.execute_reply.started":"2022-12-11T17:56:53.226210Z","shell.execute_reply":"2022-12-11T17:56:54.099979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ranker = LGBMRanker(\n    objective=\"lambdarank\",\n    metric=\"ndcg\",\n    boosting_type=\"dart\",\n    n_estimators=20,\n    importance_type='gain',\n)","metadata":{"execution":{"iopub.status.busy":"2022-12-11T17:56:54.103299Z","iopub.execute_input":"2022-12-11T17:56:54.103578Z","iopub.status.idle":"2022-12-11T17:56:54.107995Z","shell.execute_reply.started":"2022-12-11T17:56:54.103553Z","shell.execute_reply":"2022-12-11T17:56:54.107038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.columns","metadata":{"execution":{"iopub.status.busy":"2022-12-11T17:56:54.109592Z","iopub.execute_input":"2022-12-11T17:56:54.110624Z","iopub.status.idle":"2022-12-11T17:56:54.122121Z","shell.execute_reply.started":"2022-12-11T17:56:54.110548Z","shell.execute_reply":"2022-12-11T17:56:54.121080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_cols = ['aid', 'type', 'action_num_reverse_chrono', 'session_length', 'log_recency_score', 'type_weighted_log_recency_score']\ntarget = 'gt'","metadata":{"execution":{"iopub.status.busy":"2022-12-11T17:56:54.123480Z","iopub.execute_input":"2022-12-11T17:56:54.124034Z","iopub.status.idle":"2022-12-11T17:56:54.130956Z","shell.execute_reply.started":"2022-12-11T17:56:54.124000Z","shell.execute_reply":"2022-12-11T17:56:54.130134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ranker = ranker.fit(\n    train[feature_cols].to_pandas(),\n    train[target].to_pandas(),\n    group=session_lengths_train,\n)","metadata":{"execution":{"iopub.status.busy":"2022-12-11T17:56:54.132472Z","iopub.execute_input":"2022-12-11T17:56:54.133270Z","iopub.status.idle":"2022-12-11T17:57:21.132595Z","shell.execute_reply.started":"2022-12-11T17:56:54.133235Z","shell.execute_reply":"2022-12-11T17:57:21.131787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predict on test data","metadata":{}},{"cell_type":"markdown","source":"Let's load our test set, process it and predict on it.","metadata":{}},{"cell_type":"code","source":"test = cudf.read_parquet('../input/otto-full-optimized-memory-footprint/test.parquet')\ntest = apply(test, pipeline)","metadata":{"execution":{"iopub.status.busy":"2022-12-11T17:57:47.707087Z","iopub.execute_input":"2022-12-11T17:57:47.707490Z","iopub.status.idle":"2022-12-11T17:57:49.379273Z","shell.execute_reply.started":"2022-12-11T17:57:47.707457Z","shell.execute_reply":"2022-12-11T17:57:49.378274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores = ranker.predict(test[feature_cols].to_pandas())","metadata":{"execution":{"iopub.status.busy":"2022-12-11T17:57:49.381003Z","iopub.execute_input":"2022-12-11T17:57:49.381449Z","iopub.status.idle":"2022-12-11T17:57:53.457625Z","shell.execute_reply.started":"2022-12-11T17:57:49.381412Z","shell.execute_reply":"2022-12-11T17:57:53.456801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create submission","metadata":{}},{"cell_type":"code","source":"test['score'] = scores\ntest_predictions = test.sort_values(by=['session', 'score'], \n                                    ascending=False)[['session', 'aid']] \\\n                                    .reset_index(drop=True)\n\ntest_predictions = test_predictions.to_pandas().groupby('session').head(20) \\\n                                   .groupby('session').agg(list) \\\n                                   .reset_index(drop=False) ","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:31:51.781898Z","iopub.execute_input":"2022-12-11T18:31:51.782320Z","iopub.status.idle":"2022-12-11T18:31:52.022188Z","shell.execute_reply.started":"2022-12-11T18:31:51.782282Z","shell.execute_reply":"2022-12-11T18:31:52.021220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"session_types = []\nlabels = []\n\nfor session, preds in zip(test_predictions['session'].to_numpy(), test_predictions['aid'].to_numpy()):\n    l = ' '.join(str(p) for p in preds)\n    for session_type in ['clicks', 'carts', 'orders']:\n        labels.append(l)\n        session_types.append(f'{session}_{session_type}')","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:32:11.367740Z","iopub.execute_input":"2022-12-11T18:32:11.368864Z","iopub.status.idle":"2022-12-11T18:32:17.801381Z","shell.execute_reply.started":"2022-12-11T18:32:11.368817Z","shell.execute_reply":"2022-12-11T18:32:17.800332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = cudf.DataFrame({'session_type': session_types, 'labels': labels})\nsubmission.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-12-11T18:32:44.682493Z","iopub.execute_input":"2022-12-11T18:32:44.682957Z","iopub.status.idle":"2022-12-11T18:32:45.854678Z","shell.execute_reply.started":"2022-12-11T18:32:44.682920Z","shell.execute_reply":"2022-12-11T18:32:45.853595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.head()","metadata":{},"execution_count":null,"outputs":[]}]}