![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F582742%2F3ed3ae4f60217e6622235469056e439a%2Fmodel.png?generation=1676287838345268&alt=media\")
thanks to Otto and Kaggle to organize the competition. The competition and dataset were well organized without any leakage or shake up. I competed in previous RecSys competition, but this was my first Kaggle competition. It was great to work with @cdeotte @titericz and @theoviel . You can read Chris's solution and Theo's solution. I learned a lot during this competition, again.
\nEach of us developed their own model and we ensembled our final models by adding the ranks. I will focus on my model. Similar to many solutions, I used a tree-based reranker (XGBoost). I will focus my write up about:
\n| Model | \nLocal CV | \nPublic LB | \n
|---|---|---|
| Chris Baseline | \n0.567 | \n0.575 | \n
| XGBoost Reranker: trained on 4th week truncated) | \n0.576 | \n0.583 | \n
| XGBoost Reranker: trained on 20% 4th week truncated | \n0.585 | \n0.591 | \n
| XGBoost Reranker: trained on 20% 4th week truncated + activity history | \n0.587 | \n0.594 | \n
| XGBoost Reranker: trained on 100% 4th week truncated + activity history | \n0.592 | \n0.598 | \n
| XGBoost Reranker: trained on 100% 4th week truncated + activity history + Word2Vec | \n0.593 | \n0.599 | \n
| (I will explain the models in the post) | \n\n | \n |
My final model is a similar pipeline as proposed by Chris and described by many top solution. I will briefly summarize my solution:
\nClicks: Union of Top80 Candidates per session for each of ~6 different CoVisitation Matrix + items which were in the session history
\nCarts+Orders: Union of Top80 Candidates per session all CoVisitation Matrix + session history and keep only candidates, which were generated from at least 2 different sources
Note: Increasing the number of candidates for Carts+Clicks improved my score from 0.599 to 0.601
\nI had around ~160 features and around avg of 100-150 candidates per session.
\nSome additional notes:
\nThe main question for me was: how to split the data? Which data should be used for feature engineering and which data should be used to train the reranker?
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F582742%2F2d46eb5384967948c6a7068306f31a85%2Fdatasplit.png?generation=1676288119588241&alt=media\")
I used Radek's local CV strategy and run the host's dataset myself to split the training dataset into 3 full weeks and 4th week of truncated sessions. I used week 1-3, truncated 4th week (without labels) and test dataset to engineer features (e.g. co-visitation matrix). I trained the XGBoost reranker on the 4th week truncated sessions, where the session data are used to generate a X dataframe and the labels are the targets.
\nI noticed that I lose too much information by truncating all sessions of the 4th week. One strategy is to generate features by running the pipeline for train dataset and submission dataset with different inputs. Train dataset uses the truncated versions and submission dataset uses the full train dataset. In previous RecSys competition, I had bad experience because the feature distribution can shift. I wanted that my train and submission dataset uses features generated from the same dataset (e.g. same co-visitation matrix).
\nI decided to split the 4th week into 5 folds by session ID. For one fold, I will truncate the sessions and the remaining 4 folds are based on the original train dataset. Therefore, I have more data for generating the features and enough data for training my XGBoost model.
\nI ran the pipeline of XGBoost Reranker: trained on 20% 4th week truncated for each fold - having 100% of the 4th week as truncated sessions, which can be used for training the Reranker. Each fold is generated with full week 1-3, 80% of full 4th week, 20% of truncated 4th week and test sessions.
\nIt significantly improved my LB score. Unfortunately, running experiments became really slow. Running the pipeline for each fold was complex - managing the files, executing the pipeline and computation time.
\nI started with transformer-based architectures to predict the next item using Transformer4Rec. It provided great results without a complicated pipeline of feature engineering, candidate generation and reranker. Some observations:
\nUnfortunately, at the time, I truncated the full 4th week, removing a lot of data. I did not split sessions into subsessions.
\nAfter I added the Word2Vec embeddings to my reranker, I wanted to use a GRU model to learn an aggregation function of embeddings -> final score. I trained the GRU model with pre-trained word2vec embeddings and random initialized embeddings. I created sub datasets by excluding the last interactions of a session as the target and keeping the remaining interactions as input data. I repeated the process multiple times. I used negative sampling strategies to reduce training time.
", "messages": [ { "id": "2142205", "postDate": "02/13/2023 11:35:41", "content": "thanks to Otto and Kaggle to organize the competition. The competition and dataset were well organized without any leakage or shake up. I competed in previous RecSys competition, but this was my first Kaggle competition. It was great to work with @cdeotte @titericz and @theoviel . You can read Chris's solution and Theo's solution. I learned a lot during this competition, again.
\nEach of us developed their own model and we ensembled our final models by adding the ranks. I will focus on my model. Similar to many solutions, I used a tree-based reranker (XGBoost). I will focus my write up about:
\n| Model | \nLocal CV | \nPublic LB | \n
|---|---|---|
| Chris Baseline | \n0.567 | \n0.575 | \n
| XGBoost Reranker: trained on 4th week truncated) | \n0.576 | \n0.583 | \n
| XGBoost Reranker: trained on 20% 4th week truncated | \n0.585 | \n0.591 | \n
| XGBoost Reranker: trained on 20% 4th week truncated + activity history | \n0.587 | \n0.594 | \n
| XGBoost Reranker: trained on 100% 4th week truncated + activity history | \n0.592 | \n0.598 | \n
| XGBoost Reranker: trained on 100% 4th week truncated + activity history + Word2Vec | \n0.593 | \n0.599 | \n
| (I will explain the models in the post) | \n\n | \n |
My final model is a similar pipeline as proposed by Chris and described by many top solution. I will briefly summarize my solution:
\nClicks: Union of Top80 Candidates per session for each of ~6 different CoVisitation Matrix + items which were in the session history
\nCarts+Orders: Union of Top80 Candidates per session all CoVisitation Matrix + session history and keep only candidates, which were generated from at least 2 different sources
Note: Increasing the number of candidates for Carts+Clicks improved my score from 0.599 to 0.601
\nI had around ~160 features and around avg of 100-150 candidates per session.
\nSome additional notes:
\nThe main question for me was: how to split the data? Which data should be used for feature engineering and which data should be used to train the reranker?
\n
I used Radek's local CV strategy and run the host's dataset myself to split the training dataset into 3 full weeks and 4th week of truncated sessions. I used week 1-3, truncated 4th week (without labels) and test dataset to engineer features (e.g. co-visitation matrix). I trained the XGBoost reranker on the 4th week truncated sessions, where the session data are used to generate a X dataframe and the labels are the targets.
\nI noticed that I lose too much information by truncating all sessions of the 4th week. One strategy is to generate features by running the pipeline for train dataset and submission dataset with different inputs. Train dataset uses the truncated versions and submission dataset uses the full train dataset. In previous RecSys competition, I had bad experience because the feature distribution can shift. I wanted that my train and submission dataset uses features generated from the same dataset (e.g. same co-visitation matrix).
\nI decided to split the 4th week into 5 folds by session ID. For one fold, I will truncate the sessions and the remaining 4 folds are based on the original train dataset. Therefore, I have more data for generating the features and enough data for training my XGBoost model.
\nI ran the pipeline of XGBoost Reranker: trained on 20% 4th week truncated for each fold - having 100% of the 4th week as truncated sessions, which can be used for training the Reranker. Each fold is generated with full week 1-3, 80% of full 4th week, 20% of truncated 4th week and test sessions.
\nIt significantly improved my LB score. Unfortunately, running experiments became really slow. Running the pipeline for each fold was complex - managing the files, executing the pipeline and computation time.
\nI started with transformer-based architectures to predict the next item using Transformer4Rec. It provided great results without a complicated pipeline of feature engineering, candidate generation and reranker. Some observations:
\nUnfortunately, at the time, I truncated the full 4th week, removing a lot of data. I did not split sessions into subsessions.
\nAfter I added the Word2Vec embeddings to my reranker, I wanted to use a GRU model to learn an aggregation function of embeddings -> final score. I trained the GRU model with pre-trained word2vec embeddings and random initialized embeddings. I created sub datasets by excluding the last interactions of a session as the target and keeping the remaining interactions as input data. I repeated the process multiple times. I used negative sampling strategies to reduce training time.
", "rawMarkdown": "thanks to Otto and Kaggle to organize the competition. The competition and dataset were well organized without any leakage or shake up. I competed in previous RecSys competition, but this was my first Kaggle competition. It was great to work with @cdeotte @titericz and @theoviel . You can read [Chris's solution](https://www.kaggle.com/competitions/otto-recommender-system/discussion/383013) and [Theo's solution](https://www.kaggle.com/competitions/otto-recommender-system/discussion/382975). I learned a lot during this competition, again.\n\n# Summary\n\nEach of us developed their own model and we ensembled our final models by adding the ranks. I will focus on my model. Similar to many solutions, I used a tree-based reranker (XGBoost). I will focus my write up about:\n- Short Description of Reranker\n- Data Split\n- Transformer4Rec\n- GRU\n\n| Model | Local CV | Public LB |\n| --- | --- | --- |\n| Chris Baseline | 0.567 | 0.575 |\n| XGBoost Reranker: trained on 4th week truncated) | 0.576 | 0.583 |\n| XGBoost Reranker: trained on 20% 4th week truncated | 0.585 | 0.591 |\n| XGBoost Reranker: trained on 20% 4th week truncated + activity history | 0.587 | 0.594 |\n| XGBoost Reranker: trained on 100% 4th week truncated + activity history | 0.592 | 0.598 |\n| XGBoost Reranker: trained on 100% 4th week truncated + activity history + Word2Vec | 0.593 | 0.599 |\n(I will explain the models in the post)\n\n# Reranker - LB 0.601\n\n\n\nMy final model is a similar pipeline as [proposed by Chris](https://www.kaggle.com/competitions/otto-recommender-system/discussion/370210) and described by many top solution. I will briefly summarize my solution:\n\n1. Feature Engineering:\n- Generating CoVisitation Matrix (7 different ones)\n- After merging teams, I added Chris’ CoVisitation Matrix (only 10)\n- Training a Transformer4Rec model to predict scores (next item prediction)\n- Training a GRU model to predict scores (next item prediction)\n- Train a [Word2Vec model](https://www.kaggle.com/code/duuuscha/train-submit-word2vec-optimized-hparams) generating embeddings\n- Session Features - length of session, day, time, etc.\n- Candidate Features - how many clicks, addtocarts, orders on same day, previous 7 days, previous 13 days, etc.\n\n2. Generating Candidates:\nI used a different candidate generation process pre target. I noticed increasing the number of candidates improved local CV and LB score, but the clicks dataset is too large and I haven’t enough time to refactor my pipeline to support more candidates for clicks.\n\nClicks: Union of Top80 Candidates per session for each of ~6 different CoVisitation Matrix + items which were in the session history\nCarts+Orders: Union of Top80 Candidates per session all CoVisitation Matrix + session history and keep only candidates, which were generated from at least 2 different sources\n\nNote: Increasing the number of candidates for Carts+Clicks improved my score from 0.599 to 0.601\n\n3. Add Features/Scores for session x candidate pairs:\nAs [Theo wrote](https://www.kaggle.com/competitions/otto-recommender-system/discussion/382975) - how to generate unique scores?\nI used different combinations between - max, mean, weighted mean by rank or linear weights, sum, count to generate the final score. For example, I calculate the cosine similarity between all viewed items in a session and the candidate and then groupby session, candidate with aggregating the cosine similarity with max, mean, weighted mean.\n\nI had around ~160 features and around avg of 100-150 candidates per session.\n\n4. Training a XGBoost model\n\nSome additional notes:\n- I added hierarchical predictions. I used all sessions that have a click target and NO carts nor order targets to train a XGBoost model and add the click probability as an input to my carts and orders model. Similarly I added carts probability to the orders model. I used the hierarchical prediction only for orders.\n- Although I had already added Transformer4Rec scores, adding Word2Vec embeddings improved my LB score\n- I tried NN reranker but it didnt improve my local CV. I should have tried to ensemble with XGBoost reranker.\n- I tried different feature selection methods. However, removing features decreased my local CV score.\n- The pipeline was implemented with [RAPIDs cuDF](https://rapids.ai/) using a single GPU with 32GB memory. I ran some code in parallel by using multiple workers with each 1x GPU. \n\n# Data Split\nThe main question for me was: how to split the data? Which data should be used for feature engineering and which data should be used to train the reranker?\n\n\n\n### XGBoost Reranker: trained on 4th week truncated - LB 0.584\nI used Radek's local CV strategy and run the host's dataset myself to split the training dataset into 3 full weeks and 4th week of truncated sessions. I used week 1-3, truncated 4th week (without labels) and test dataset to engineer features (e.g. co-visitation matrix). I trained the XGBoost reranker on the 4th week truncated sessions, where the session data are used to generate a X dataframe and the labels are the targets.\n\n### XGBoost Reranker: trained on 20% 4th week truncated - LB 0.591\nI noticed that I lose too much information by truncating all sessions of the 4th week. One strategy is to generate features by running the pipeline for train dataset and submission dataset with different inputs. Train dataset uses the truncated versions and submission dataset uses the full train dataset. In previous RecSys competition, I had bad experience because the feature distribution can shift. I wanted that my train and submission dataset uses features generated from the same dataset (e.g. same co-visitation matrix). \n\nI decided to split the 4th week into 5 folds by session ID. For one fold, I will truncate the sessions and the remaining 4 folds are based on the original train dataset. Therefore, I have more data for generating the features and enough data for training my XGBoost model.\n\n### XGBoost Reranker: trained on 100% 4th week truncated + activity history - LB 0.598\nI ran the pipeline of *XGBoost Reranker: trained on 20% 4th week truncated* for each fold - having 100% of the 4th week as truncated sessions, which can be used for training the Reranker. Each fold is generated with full week 1-3, 80% of full 4th week, 20% of truncated 4th week and test sessions. \n\nIt significantly improved my LB score. Unfortunately, running experiments became really slow. Running the pipeline for each fold was complex - managing the files, executing the pipeline and computation time.\n\n# Transformer4Rec \n\nI started with transformer-based architectures to predict the next item using [Transformer4Rec](https://github.com/NVIDIA-Merlin/Transformers4Rec/). It provided great results without a complicated pipeline of feature engineering, candidate generation and reranker. Some observations:\n- Increasing embedding width improved local CV (maximum possible was 256)\n- Using Dot-Product (weight-tying) as the output layer of the full item catalog was inefficient. I implemented a negative sampling strategy, which significantly reduced training time. I will create a PR for Transformer4Rec\n- I optimized ~25 hyperparameters\n\nUnfortunately, at the time, I truncated the full 4th week, removing a lot of data. I did not split sessions into subsessions.\n\n# GRU\n\nAfter I added the Word2Vec embeddings to my reranker, I wanted to use a GRU model to learn an aggregation function of embeddings -> final score. I trained the GRU model with pre-trained word2vec embeddings and random initialized embeddings. I created sub datasets by excluding the last interactions of a session as the target and keeping the remaining interactions as input data. I repeated the process multiple times. I used negative sampling strategies to reduce training time.", "votes": null }, { "id": "2145268", "postDate": "02/14/2023 21:53:46", "content": "I uploaded my code to github: https://github.com/bschifferer/Kaggle-Otto-Comp
", "rawMarkdown": "I uploaded my code to github: https://github.com/bschifferer/Kaggle-Otto-Comp", "votes": null }, { "id": "2243399", "postDate": "05/02/2023 21:55:48", "content": "Hi Benny, thanks so much for sharing the amazing code! I was learning your code and had a question:
\nIn this file 01e_FE_Transformer/02_T4R_Train-v2.ipynb, from custom_t4r import *. Whereas I didn't see where the custom_t4r is, could you help with this? Appreciate your reply!
I uploaded my code to github: https://github.com/bschifferer/Kaggle-Otto-Comp
", "votes": null, "replies": [] }, { "id": 2243399, "author_name": "sherrypomi", "author_url": "", "post_date": "05/02/2023 21:55:48", "content": "Hi Benny, thanks so much for sharing the amazing code! I was learning your code and had a question:
\nIn this file 01e_FE_Transformer/02_T4R_Train-v2.ipynb, from custom_t4r import *. Whereas I didn't see where the custom_t4r is, could you help with this? Appreciate your reply!