First of all, we would like to thank OTTO for organizing this competition. It was the first RecSys experience for most of us and we learned a lot. I'll try to talk about the things we've all tried at each stage of our solution, and the ones that didn't work at the end.
\nIn addition, @tetsuro731 has opened a separate thread for the stacking method he applied
\nIn addition, the links to our public repos are below:
\nAnil: GitHub Repo
\nGunes: GitHub Repo
\nTetsuro: GitHub Repo
Generated features for following data subsets:
\nAll of those aggregations generated for each type, last week, last 1, 2, 3, 4, 5, 6, 7 days (all of the aid aggregations with count and nunique are replaced with rank percentile because two datasets had different sizes)
\nThese aggregations generated per last 2 week and type.
\nThese aggregations generated per type.
\nThese aggregations generated using all of the types and per type.
\nOur blending approach was pretty straightforward.
\nFirst of all, we would like to thank OTTO for organizing this competition. It was the first RecSys experience for most of us and we learned a lot. I'll try to talk about the things we've all tried at each stage of our solution, and the ones that didn't work at the end.
\nIn addition, @tetsuro731 has opened a separate thread for the stacking method he applied
\nIn addition, the links to our public repos are below:
\nAnil: GitHub Repo
\nGunes: GitHub Repo
\nTetsuro: GitHub Repo
Generated features for following data subsets:
\nAll of those aggregations generated for each type, last week, last 1, 2, 3, 4, 5, 6, 7 days (all of the aid aggregations with count and nunique are replaced with rank percentile because two datasets had different sizes)
\nThese aggregations generated per last 2 week and type.
\nThese aggregations generated per type.
\nThese aggregations generated using all of the types and per type.
\nOur blending approach was pretty straightforward.
\nCongrats on your impressive score!
", "rawMarkdown": "Congrats on your impressive score!", "votes": null }, { "id": "2124810", "postDate": "02/01/2023 08:17:38", "content": "Hi @nlztrk and team, thanks for sharing the code as well. There is so much to learn here. Congratulations on a strong silver! 🎉
", "rawMarkdown": "Hi @nlztrk and team, thanks for sharing the code as well. There is so much to learn here. Congratulations on a strong silver! 🎉", "votes": null }, { "id": "2126111", "postDate": "02/02/2023 05:20:38", "content": "Congratulations! There is so much in this writeup as well, Thank you for sharing!
\nThe Devastator.
", "rawMarkdown": "Congratulations! There is so much in this writeup as well, Thank you for sharing!\n\n\nThe Devastator.", "votes": null }, { "id": "2127451", "postDate": "02/02/2023 22:28:22", "content": "So Recbole didn't work… it was on my backlog during the entire competition and I never found the time to try that lib. Thanks for this info and congrats to the team!
", "rawMarkdown": "So Recbole didn't work... it was on my backlog during the entire competition and I never found the time to try that lib. Thanks for this info and congrats to the team!", "votes": null } ], "comments": [ { "id": 2124620, "author_name": "tahamhaider", "author_url": "", "post_date": "02/01/2023 05:18:07", "content": "Congrats on your impressive score!
", "votes": null, "replies": [] }, { "id": 2124810, "author_name": "parthpankajtiwary", "author_url": "", "post_date": "02/01/2023 08:17:38", "content": "Hi @nlztrk and team, thanks for sharing the code as well. There is so much to learn here. Congratulations on a strong silver! 🎉
", "votes": null, "replies": [] }, { "id": 2126111, "author_name": "thedevastator", "author_url": "", "post_date": "02/02/2023 05:20:38", "content": "Congratulations! There is so much in this writeup as well, Thank you for sharing!
\nThe Devastator.
", "votes": null, "replies": [] }, { "id": 2127451, "author_name": "hinepo", "author_url": "", "post_date": "02/02/2023 22:28:22", "content": "So Recbole didn't work… it was on my backlog during the entire competition and I never found the time to try that lib. Thanks for this info and congrats to the team!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2124586": "First of all, we would like to thank OTTO for organizing this competition. It was the first RecSys experience for most of us and we learned a lot. I'll try to talk about the things we've all tried at each stage of our solution, and the ones that didn't work at the end.\n\nIn addition, @tetsuro731 has opened a separate thread for [the stacking method he applied](https://www.kaggle.com/competitions/otto-recommender-system/discussion/382955)\n\n**In addition, the links to our public repos are below:**\nAnil: [GitHub Repo](https://github.com/nlztrk/OTTO-Multi-Objective-Recommender-System)\nGunes: [GitHub Repo](https://github.com/gunesevitan/otto-multi-objective-recommender-system) \nTetsuro: [GitHub Repo](https://github.com/tetsuro731/OTTO-kaggle-tetsuro731) \n***\n## Generating Covisitation Matrices\n#### Anil & Ayberk\n- Generated **Top-100** AIDs for three well-known covisitation schemes given in: [link](https://www.kaggle.com/code/tuongkhang/otto-pipeline2-lb-0-576)\n#### Gunes\n- 7 covisitation matrices like the ones in public but slightly different (click/cart/order weighted, only click/cart, click/order, cart/order and time weighted)\n- Created a counter from **7** covisitation matrices and **50** nearest neighbor of last aid in session using fasttext model + annoy\n#### Tetsuro\n- Generated co-visitation matrix with several parameters/weights based on the public notebook.\n***\n## Splitting the Data as Train/Val\n#### Anil & Ayberk\n- Used the [local validation scheme](https://www.kaggle.com/datasets/radek1/otto-train-and-test-data-for-local-validation) given by [Radek](https://www.kaggle.com/radek1). The local train data was also splitted into two by sessions in order to avoid possible leakage during model training and score calculation. The implementation can be seen in the corresponding notebook.\n#### Gunes\n- Radek’s split is used as the validation set. Every approach is applied on two different datasets; training + validation and entire dataset. First one is used for the validation and second one is used for the submission.\n#### Tetsuro\n- Didn’t use 5 weeks and used 4 weeks for training because the data for the prediction have only 4 weeks.\n***\n## Generating Co-Occurrence Matrices\n#### Anil\n- Generated all pair occurrences for all AIDs among all sessions for all action pairs (click-cart, cart-order, etc.). This is used for feature extraction later.\n***\n## Candidate Generation\n#### Anil & Ayberk\n- Used the [public candidate generation script](https://www.kaggle.com/code/tuongkhang/otto-pipeline2-lb-0-576) and generated **100** candidates for all action types. The rank of the candidates generation with several parameters/weights are also used for the features for the ranking model.\n#### Gunes\n- Generated co-visitation matrices slightly different than publicly shared ones\n - 12 hour difference 15x click weighted\n - 12 hour difference 15x cart weighted\n - 12 hour difference 15x order weighted\n - 12 hour difference only click and cart\n - 12 hour difference only click and order\n - 14 day difference only cart and order\n - 24 hour difference time weighted\n- Trained and tuned a FastText (0.547 lb score) skipgram which worked better than gensim word2vec and it was faster because of c++ bindings\n- Created a counter from 7 covisitation matrices and 50 nearest neighbors of last aid in the session using FastText model and annoy (100 trees)\n- 80-100 candidates are selected using the approach above and session unique aids are concatenated to candidates for each session\n#### Tetsuro\n- Created **Top-100** candidates.\n- Added click/cart/order Top-50 Popular items for each candidate. Finally, 150 candidates are generated which are used for the next ranking phase. The rank of the candidates generation with several parameters/weights are also used for the features for the ranking model.\n***\n## Feature Extraction\n\nGenerated features for following data subsets:\n- Items\n- Sessions\n- Item-Session Combinations\n- Covisitation and Co-Occurrence Statistics\n\n### Item Features\n####Anil\n- Statistics generated from hour, weekday and weekend status\n- Count features (bool for >0 and >1, rank among all)\n- Unique count features (unique count and rank among all)\n- Distribution of action types in percentiles\n- Inclusion rate by all sessions\n- Occurrence rate in the last week of data\n- Average number of times seen in the same sessions at different times\n- All of the above with filtered separately for all action types\n#### Gunes\n- type mean/std\n- day of week mean\n- hour mean\n- is session start/end mean\n- candidate score mean/std/min/max\n- timestamp ratio/difference\n\nAll of those aggregations generated for each type, last week, last 1, 2, 3, 4, 5, 6, 7 days (all of the aid aggregations with count and nunique are replaced with rank percentile because two datasets had different sizes)\n#### Ayberk\n- Unique count features\n- Count features\n- Time features\n- Rank of a item in sessions\n- How many times item clicked after bought\n- Session based features like how many time item clicked per session\n- Weekly change rates\n- click-to-cart , cart-to-buy ratios\n\nThese aggregations generated per last 2 week and type.\n\n#### Tetsuro\n- Popular item feature\n- The number/rank/unique number of click/cart/order during 1/2/4 weeks\n- Counts/Unique counts of click/cart/order for each session\n- The ratio of these features.\n\n### Session Features\n#### Anil\n- Statistics generated from hour, weekday and weekend status\n- Count features (bool for >0 and >1, rank among all)\n- Unique count features (unique count and rank among all)\n- Distribution of action types in percentiles\n- Length of the session\n- Features generated by extracting mini-sessions according to the time differences between actions\n- Statistics generated from multiple purchases made in a single basket\n- Rates of taking products to the next action within the same session (click->cart, cart->order)\n- All of the above with filtered separately for all action types\n#### Gunes\n- Aid count\n- Unique count\n- Count of last aid\n- Type of last aid\n- Last aid itself\n- Timestamp ratio/difference\n#### Ayberk\n- Session unique aid count, length, time features\n- Session click-to-cart , cart-to-buy ratios\n\nThese aggregations generated per type.\n#### Tetsuro\n- Mean number of click/cart/order counts/unique counts\n- Session duration (min and max diff of session timestamp)\n\n### Item-Session Combination Features\n####Anil\n- Statistics generated from hour, weekday and weekend status\n- Count features (bool for >0 and >1, rank among all)\n- Unique count features (unique count and rank among all)\n- Distribution of action types in percentiles\n- Reversed order of the item in the session\n- Time difference between the latest occurrence of the item and the start - end of the session\n#### Gunes\n- Candidate count in session\n- Candidate click, cart and order count in session\n#### Ayberk\n- Count features\n- How much time passed since last action.\n- Inverse rank place in session\n\nThese aggregations generated using all of the types and per type.\n#### Tetsuro\n- Diff/ratio between item and session which are related to similarity.\n\n### Covisitation and Co-Occurrence Statistics\n#### Anil\n- Statistics generated from covisitation and co-occurrence scores between candidate items and items in the session's history\n#### Ayberk\n- Co-visitation scores used.\n***\n## Training\n#### Anil\n- **Model:** XGBoost\n- **Fold Scheme:** 5-Fold (Grouped by \"session\")\n- **Negative Sampling Fraction:** 15%\n- Dropped sessions with no positive labels\n- Used the first half of splitted local training set\n#### Gunes\n- **Model:** XGBoost and LightGBM\n- **Fold Scheme:** 5-Fold (Grouped by \"session\")\n- **Negative Sampling Fraction:** 30%\n- Dropped sessions with no positive labels\n#### Ayberk\n- **Model:** LightGBM\n- **Fold Scheme:** 5-Fold (Grouped by \"session\")\n- **Negative Sampling Fraction:** 20%\n- Early stopping used.\n- Dropped sessions with no positive labels\n- Used the first half of splitted local training set\n#### Tetsuro\n- **Model:** LightGBM\n - Metrics: nDCG@20\n- **Fold Scheme:** 5-Fold (Grouped by \"session\")\n - Checked recall for each fold\n- **Negative Sampling:** sampling negative to 2.5% positive.\n- Dropped sessions with no positive labels\n***\n## Inference\n#### Anil\n- Used mean blending\n- Executed on the second half of splitted local training set when running local validation\n#### Gunes\n- Used mean blending\n#### Ayberk\n- Used mean blending\n- Executed on the second half of splitted local training set when running local validation\n- Filling orders predictions in the end with carts first non duplicate predictions (last 2)\n#### Tetsuro\n- Used mean blending\n***\n## Submission Blending\nOur blending approach was pretty straightforward.\n- Scale each prediction with robust scaler\n- Outer join different predictions on session and aid\n- Fill missing values with 0\n- Do a weighted sum based on OOF scores\n***\n## Didn't Work & Improve\n#### Anil\n- Weekday-Specific aggregations\n- Word2Vec features\n- Different models (CatBoost, LGBM)\n- Comprehensive pair scores (because of OOM errors)\n- Max-median blending\n- Early-stopping\n- Higher negative fractions\n- Different objective metrics\n- Different fold counts\n#### Gunes\n- **Collaborative filtering:** Either the scores aren't good or the embeddings become too large and inference is too slow\n- **Matrix factorization:** Same as collaborative filtering\n- **Doc2vec:** Model doesn't learn anything\n- **Sequential models from recbole library (GRU4Rec, BERT4Rec and etc.):** Training is too slow and they are not competitive\n- **General models from recbole library (BPR, CF and MF models):** Training is fast but inference is too slow because models don't scale\n- **Models from surprise library:** Same as general models from recbole library\n- **TF-IDF + pairwise similarity:** Very slow inference time since I was using argsort to get top 20 most similar aids\n#### Ayberk\n- Daily aid features\n- User2User Similarities\n- Tried MLP architecture with various settings (pairwise type prediction, session-based multi class)\nbut couldnt beat heuristic approach by Chris and as i used only kaggle resources it was hard for me to keep improving so i dropped this approach at early stages.\n- Candidate generation using item embeddings from MLP architecture\n- Higher negative fractions\n- Candidate ensemble with best @20 recall and best@N recall as it would rank candidates better but didnt work\n#### Tetsuro\n- Weekday-Specific aggregations\n- improve nDCG but local recall hadn’t improved\n- Wanted to try W2C based method but I didn’t have time to do.", "2124620": "Congrats on your impressive score!", "2124810": "Hi @nlztrk and team, thanks for sharing the code as well. There is so much to learn here. Congratulations on a strong silver! 🎉", "2126111": "Congratulations! There is so much in this writeup as well, Thank you for sharing!\n\n\nThe Devastator.", "2127451": "So Recbole didn't work... it was on my backlog during the entire competition and I never found the time to try that lib. Thanks for this info and congrats to the team!" }, "source": "meta" }