It's too long since I last see the competition about recommendations on Kaggle. So great thanks to the Kaggle team and H&M for hosting such an interesting competition. The high correlation between CV and LB, big training data for deep data mining, no leaking of test information, friendly staff and participants, all of these make the successful competition.
\nThe key idea of my solution will be illustrated here for fast reading purposes.
\ntrain = transactions.loc[transactions[\"t_dat\"] < \"2020-09-16\"]\nvalid = transactions.loc[transactions[\"t_dat\"] >= \"2020-09-16\"]\nIt is better to use k-fold validation in this post to avoid overfitting the last week if you have enough computing power.
\nBut from my experience in this competition, LB score and CV score are highly correlated using last week only as a validation set.
\n| Recall method | \nlink | \n
|---|---|
| SAR | \nnotebook | \n
| Trending | \nnotebook | \n
| Most popular items based on age | \n/ | \n
LightGBM trained with 20 weeks of data using around 300 features
\nI have no time to train all 5 models with k fold split on the train set, only 3 of them are completed before the deadline. So my final submission is based on 1 model trained on all data samples and the other 3 models trained based on k fold split mentioned before (4 models in total).
\n| Model | \npublic score | \nprivate score | \n
|---|---|---|
| one model | \n0.0338 | \n0.0341 | \n
| ensemble | \n0.0343 | \n0.0346 | \n
Using cudf and Forest Inference Library from Rapids for fast feature generation and fast tree inference
\n| Hardware | \nvalue | \n
|---|---|
| CPU | \n32core | \n
| Memory | \n128G | \n
| GPU | \nV100 32G | \n
train_condition = (trans[\"t_dat\"] >= \"2020-08-16\") & (trans[\"t_dat\"] < \"2020-09-16\")\nitems_in_valid = trans.loc[trans[\"t_dat\"] >= \"2020-09-16\", \"article_id\"].unique()\nitems_last_month = trans.loc[train_condition, \"article_id\"].unique()\nprint(len(set(items_in_valid) & set(items_last_month)) / len(set(items_in_valid)))\n# 0.92\ntrain_condition = (trans[\"t_dat\"] >= \"2020-08-16\") & (trans[\"t_dat\"] < \"2020-09-16\")\nusers_in_valid = trans.loc[trans[\"t_dat\"] >= \"2020-09-16\", \"customer_id\"].unique()\nusers_last_month = trans.loc[train_condition, \"customer_id\"].unique()\nprint(len(set(users_in_valid) & set(users_last_month)) / len(set(users_in_valid)))\n# 0.47\nConsidering these results, we can focus on recommending items bought in the last 30 days, especially for those customers who have transaction records.
\nOnly mapk of each recall method on the local validation set is recorded. Most of these ideas are coming from public notebooks or discussions and I will give the link if so.
\nSimply sorting items by transaction records. This is also the basic rule for those customers has no age data
\n| Recall method | \nmapk | \n
|---|---|
| Top 12 items | \n0.00814889 | \n
We are doing personalized recommendations, so it's better to use some customers' features based on the rule for recall. Unfortunately, only 2 features seem valuable to me at first glance, age, and postal code. It's impossible to get the location feature from the postal code since it is encoded. Finally, the only choice is age.
\nbins = [0, 18, 22, 28, 35, 45, 55, 65, 200]\nlast_week_transactions[\"age_bin\"] = pd.cut(user[\"age\"], bins=bins)\nlast_week_transactions_popular = last_week_transactions.groupby(\"age_bin\").size()\nBins are simply tuned with last week's data. The mapk score is slightly higher than only using the most popular items.
\n| Recall method | \nmapk | \n
|---|---|
| Top 12 items | \n0.0081488 | \n
| Top 12 items by age | \n0.0091866 | \n
The first method is just adding items that customers have bought in the last 4 weeks. Similar to this notebook
\n| Recall method | \nmapk | \n
|---|---|
| Top 12 items | \n0.0081488 | \n
| Top 12 items by age | \n0.0091866 | \n
| bought last 4 week +Top 12 items by age | \n0.0245722 | \n
There is a significant increase in mapk. But since we know only half of the customers have transaction history in the last 30 days, we can do better. According to this notebook, we have a wonderful method to compute the score of each item to all customers without the limitation of 30 days.
\n| Recall method | \nmapk | \n
|---|---|
| Top 12 items | \n0.0081488 | \n
| Top 12 items by age | \n0.0091866 | \n
| bought last 4 week +Top 12 items by age | \n0.0245722 | \n
| Trending + Top 12 items by age | \n0.0255676 | \n
SAR is a kind of item-based collaborative filtering. As mentioned in this post, we can get 0.27+ with SAR. Here is the example from the Microsoft official repo. My SAR is trained with the last 30 days. Adding more data will make the performance worse in my case.
\n| experiment | \nmapk | \n
|---|---|
| Top 12 items | \n0.0081488 | \n
| Top 12 items by age | \n0.0091866 | \n
| bought last 4 week +Top 12 items by age | \n0.0245722 | \n
| Trending + Top 12 items by age | \n0.0255676 | \n
| SAR + Trending + Top 12 items by age | \n0.0278576 | \n
Let's say we are going to recommend K items for each customer. My Final strategy for the recall is
\nLightGBM classifier is used as a ranking model. There is no significant difference between classifier and lambda ranker. But from my previous experience, the classifier will give a slightly higher performance in general. So I haven't tried lambda ranker in this competition, but it's a good way for ensembling.
\nNothing special here, just group by user/item, (user, item) and do some aggregation.
\n| Feature | \nDescription | \n
|---|---|
| user feature | \naverage prices of items this customer has bought in the last 7/30/90 days | \n
| item feature | \naverage age of customers who have bought this item in the last 7/30/90 days | \n
| time | \nhow many days after the customer's last transaction record | \n
| interaction | \nwhether the customer has bought the current item before | \n
| others | \n… | \n
We can treat the transaction history of a customer as a sentence, then use some NLP technique to pretrain the embedding of customer/item. My solution using word2vec to train the embedding and simply averaging the item embedding of a customer transaction history as one of the customer features (be care of leaking problems, do not use items in the current week)
\ntrain with the last 20 weeks and generate 100 candidates for each customer (a bug was fixed in the submission phase several hours before the deadline, so the gap should be smaller)
\n| validation | \nscore | \n
|---|---|
| local cv | \n0.0363 | \n
| public score | \n0.0310 | \n
| private score | \n0.0312 | \n
Feature engineering with cudf is 30X faster than using pandas in my case. It allows me to generate more candidates and get the feature in the submission phase. Load LightGBM model with Forest Inference Library and inference with GPU is 40X faster than using 32 core CPUs.
\nGenerate 200 candidates for each customer, keeping all positive samples and random choosing half of the negative samples in the last 20 weeks
\nGenerate 400 candidates for each customer, making prediction scores with 4 LightGBM models. Averaging them as the final score and getting top12 items
", "messages": [ { "id": "1783261", "postDate": "05/10/2022 08:35:00", "content": "It's too long since I last see the competition about recommendations on Kaggle. So great thanks to the Kaggle team and H&M for hosting such an interesting competition. The high correlation between CV and LB, big training data for deep data mining, no leaking of test information, friendly staff and participants, all of these make the successful competition.
\nThe key idea of my solution will be illustrated here for fast reading purposes.
\ntrain = transactions.loc[transactions[\"t_dat\"] < \"2020-09-16\"]\nvalid = transactions.loc[transactions[\"t_dat\"] >= \"2020-09-16\"]\nIt is better to use k-fold validation in this post to avoid overfitting the last week if you have enough computing power.
\nBut from my experience in this competition, LB score and CV score are highly correlated using last week only as a validation set.
\n| Recall method | \nlink | \n
|---|---|
| SAR | \nnotebook | \n
| Trending | \nnotebook | \n
| Most popular items based on age | \n/ | \n
LightGBM trained with 20 weeks of data using around 300 features
\nI have no time to train all 5 models with k fold split on the train set, only 3 of them are completed before the deadline. So my final submission is based on 1 model trained on all data samples and the other 3 models trained based on k fold split mentioned before (4 models in total).
\n| Model | \npublic score | \nprivate score | \n
|---|---|---|
| one model | \n0.0338 | \n0.0341 | \n
| ensemble | \n0.0343 | \n0.0346 | \n
Using cudf and Forest Inference Library from Rapids for fast feature generation and fast tree inference
\n| Hardware | \nvalue | \n
|---|---|
| CPU | \n32core | \n
| Memory | \n128G | \n
| GPU | \nV100 32G | \n
train_condition = (trans[\"t_dat\"] >= \"2020-08-16\") & (trans[\"t_dat\"] < \"2020-09-16\")\nitems_in_valid = trans.loc[trans[\"t_dat\"] >= \"2020-09-16\", \"article_id\"].unique()\nitems_last_month = trans.loc[train_condition, \"article_id\"].unique()\nprint(len(set(items_in_valid) & set(items_last_month)) / len(set(items_in_valid)))\n# 0.92\ntrain_condition = (trans[\"t_dat\"] >= \"2020-08-16\") & (trans[\"t_dat\"] < \"2020-09-16\")\nusers_in_valid = trans.loc[trans[\"t_dat\"] >= \"2020-09-16\", \"customer_id\"].unique()\nusers_last_month = trans.loc[train_condition, \"customer_id\"].unique()\nprint(len(set(users_in_valid) & set(users_last_month)) / len(set(users_in_valid)))\n# 0.47\nConsidering these results, we can focus on recommending items bought in the last 30 days, especially for those customers who have transaction records.
\nOnly mapk of each recall method on the local validation set is recorded. Most of these ideas are coming from public notebooks or discussions and I will give the link if so.
\nSimply sorting items by transaction records. This is also the basic rule for those customers has no age data
\n| Recall method | \nmapk | \n
|---|---|
| Top 12 items | \n0.00814889 | \n
We are doing personalized recommendations, so it's better to use some customers' features based on the rule for recall. Unfortunately, only 2 features seem valuable to me at first glance, age, and postal code. It's impossible to get the location feature from the postal code since it is encoded. Finally, the only choice is age.
\nbins = [0, 18, 22, 28, 35, 45, 55, 65, 200]\nlast_week_transactions[\"age_bin\"] = pd.cut(user[\"age\"], bins=bins)\nlast_week_transactions_popular = last_week_transactions.groupby(\"age_bin\").size()\nBins are simply tuned with last week's data. The mapk score is slightly higher than only using the most popular items.
\n| Recall method | \nmapk | \n
|---|---|
| Top 12 items | \n0.0081488 | \n
| Top 12 items by age | \n0.0091866 | \n
The first method is just adding items that customers have bought in the last 4 weeks. Similar to this notebook
\n| Recall method | \nmapk | \n
|---|---|
| Top 12 items | \n0.0081488 | \n
| Top 12 items by age | \n0.0091866 | \n
| bought last 4 week +Top 12 items by age | \n0.0245722 | \n
There is a significant increase in mapk. But since we know only half of the customers have transaction history in the last 30 days, we can do better. According to this notebook, we have a wonderful method to compute the score of each item to all customers without the limitation of 30 days.
\n| Recall method | \nmapk | \n
|---|---|
| Top 12 items | \n0.0081488 | \n
| Top 12 items by age | \n0.0091866 | \n
| bought last 4 week +Top 12 items by age | \n0.0245722 | \n
| Trending + Top 12 items by age | \n0.0255676 | \n
SAR is a kind of item-based collaborative filtering. As mentioned in this post, we can get 0.27+ with SAR. Here is the example from the Microsoft official repo. My SAR is trained with the last 30 days. Adding more data will make the performance worse in my case.
\n| experiment | \nmapk | \n
|---|---|
| Top 12 items | \n0.0081488 | \n
| Top 12 items by age | \n0.0091866 | \n
| bought last 4 week +Top 12 items by age | \n0.0245722 | \n
| Trending + Top 12 items by age | \n0.0255676 | \n
| SAR + Trending + Top 12 items by age | \n0.0278576 | \n
Let's say we are going to recommend K items for each customer. My Final strategy for the recall is
\nLightGBM classifier is used as a ranking model. There is no significant difference between classifier and lambda ranker. But from my previous experience, the classifier will give a slightly higher performance in general. So I haven't tried lambda ranker in this competition, but it's a good way for ensembling.
\nNothing special here, just group by user/item, (user, item) and do some aggregation.
\n| Feature | \nDescription | \n
|---|---|
| user feature | \naverage prices of items this customer has bought in the last 7/30/90 days | \n
| item feature | \naverage age of customers who have bought this item in the last 7/30/90 days | \n
| time | \nhow many days after the customer's last transaction record | \n
| interaction | \nwhether the customer has bought the current item before | \n
| others | \n… | \n
We can treat the transaction history of a customer as a sentence, then use some NLP technique to pretrain the embedding of customer/item. My solution using word2vec to train the embedding and simply averaging the item embedding of a customer transaction history as one of the customer features (be care of leaking problems, do not use items in the current week)
\ntrain with the last 20 weeks and generate 100 candidates for each customer (a bug was fixed in the submission phase several hours before the deadline, so the gap should be smaller)
\n| validation | \nscore | \n
|---|---|
| local cv | \n0.0363 | \n
| public score | \n0.0310 | \n
| private score | \n0.0312 | \n
Feature engineering with cudf is 30X faster than using pandas in my case. It allows me to generate more candidates and get the feature in the submission phase. Load LightGBM model with Forest Inference Library and inference with GPU is 40X faster than using 32 core CPUs.
\nGenerate 200 candidates for each customer, keeping all positive samples and random choosing half of the negative samples in the last 20 weeks
\nGenerate 400 candidates for each customer, making prediction scores with 4 LightGBM models. Averaging them as the final score and getting top12 items
", "rawMarkdown": "It's too long since I last see the competition about recommendations on Kaggle. So great thanks to the Kaggle team and H&M for hosting such an interesting competition. The high correlation between CV and LB, big training data for deep data mining, no leaking of test information, friendly staff and participants, all of these make the successful competition.\n\n# Summary\n\nThe key idea of my solution will be illustrated here for fast reading purposes.\n\n##### Validation strategy\n\n```python\ntrain = transactions.loc[transactions[\"t_dat\"] < \"2020-09-16\"]\nvalid = transactions.loc[transactions[\"t_dat\"] >= \"2020-09-16\"]\n```\n\nIt is better to use k-fold validation in [this post](https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/308919) to avoid overfitting the last week if you have enough computing power.\n\nBut from my experience in this competition, LB score and CV score are highly correlated using last week only as a validation set.\n\n\n##### Recall\n\n| Recall method | link |\n| ------------------------------- | ----------------------------------------------------------------------------------------------------------- |\n| SAR | [notebook](https://github.com/microsoft/recommenders/blob/main/examples/00_quick_start/sar_movielens.ipynb) |\n| Trending | [notebook](https://www.kaggle.com/code/byfone/h-m-trending-products-weekly) |\n| Most popular items based on age | / |\n\n\n##### Ranking\n\nLightGBM trained with 20 weeks of data using around 300 features\n\n\n##### Ensemble\n\nI have no time to train all 5 models with k fold split on the train set, only 3 of them are completed before the deadline. So my final submission is based on 1 model trained on all data samples and the other 3 models trained based on k fold split mentioned before (4 models in total).\n\n| Model | public score | private score |\n| --------- | ------------ | ------------- |\n| one model | 0.0338 | 0.0341 |\n| ensemble | 0.0343 | 0.0346 |\n\n\n##### Optimization\n\nUsing cudf and Forest Inference Library from Rapids for fast feature generation and fast tree inference\n\n\n##### Hardware\n\n| Hardware | value |\n| -------- | -------- |\n| CPU | 32core |\n| Memory | 128G |\n| GPU | V100 32G |\n\n\n# Data Analysis\n\n1. We can find 90%+ items in the validation set also in the last 30 days\n\n```python\ntrain_condition = (trans[\"t_dat\"] >= \"2020-08-16\") & (trans[\"t_dat\"] < \"2020-09-16\")\nitems_in_valid = trans.loc[trans[\"t_dat\"] >= \"2020-09-16\", \"article_id\"].unique()\nitems_last_month = trans.loc[train_condition, \"article_id\"].unique()\nprint(len(set(items_in_valid) & set(items_last_month)) / len(set(items_in_valid)))\n# 0.92\n```\n\n2. Approximately 50% of customers in the validation set have transaction records in the last 30 days\n\n```python\ntrain_condition = (trans[\"t_dat\"] >= \"2020-08-16\") & (trans[\"t_dat\"] < \"2020-09-16\")\nusers_in_valid = trans.loc[trans[\"t_dat\"] >= \"2020-09-16\", \"customer_id\"].unique()\nusers_last_month = trans.loc[train_condition, \"customer_id\"].unique()\nprint(len(set(users_in_valid) & set(users_last_month)) / len(set(users_in_valid)))\n# 0.47\n```\n\nConsidering these results, we can focus on recommending items bought in the last 30 days, especially for those customers who have transaction records.\n\n# Recall\n\nOnly mapk of each recall method on the local validation set is recorded. Most of these ideas are coming from public notebooks or discussions and I will give the link if so.\n\n##### Most popular items last 7 days\n\nSimply sorting items by transaction records. This is also the basic rule for those customers has no age data\n\n| Recall method | mapk |\n| ------------- | ---------- |\n| Top 12 items | 0.00814889 |\n\n##### Most popular itmes last 7 days by age\n\nWe are doing personalized recommendations, so it's better to use some customers' features based on the rule for recall. Unfortunately, only 2 features seem valuable to me at first glance, age, and postal code. It's impossible to get the location feature from the postal code since it is encoded. Finally, the only choice is age.\n\n```python\nbins = [0, 18, 22, 28, 35, 45, 55, 65, 200]\nlast_week_transactions[\"age_bin\"] = pd.cut(user[\"age\"], bins=bins)\nlast_week_transactions_popular = last_week_transactions.groupby(\"age_bin\").size()\n```\n\nBins are simply tuned with last week's data. The mapk score is slightly higher than only using the most popular items.\n\n| Recall method | mapk |\n| ------------------- | --------- |\n| Top 12 items | 0.0081488 |\n| Top 12 items by age | 0.0091866 |\n\n##### Consider items that customers have bought before\n\nThe first method is just adding items that customers have bought in the last 4 weeks. Similar to [this notebook](https://www.kaggle.com/code/hengzheng/time-is-our-best-friend-v2)\n\n| Recall method | mapk |\n| --------------------------------------- | --------- |\n| Top 12 items | 0.0081488 |\n| Top 12 items by age | 0.0091866 |\n| bought last 4 week +Top 12 items by age | 0.0245722 |\n\n##### Use Trending\n\nThere is a significant increase in mapk. But since we know only half of the customers have transaction history in the last 30 days, we can do better. According to [this notebook](https://www.kaggle.com/code/byfone/h-m-trending-products-weekly), we have a wonderful method to compute the score of each item to all customers without the limitation of 30 days.\n\n\n| Recall method | mapk |\n| --------------------------------------- | --------- |\n| Top 12 items | 0.0081488 |\n| Top 12 items by age | 0.0091866 |\n| bought last 4 week +Top 12 items by age | 0.0245722 |\n| Trending + Top 12 items by age | 0.0255676 |\n\n##### Recommend with SAR\n\nSAR is a kind of item-based collaborative filtering. As mentioned in [this post](https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/316205), we can get 0.27+ with SAR. Here is the [example](https://github.com/microsoft/recommenders/blob/main/examples/00_quick_start/sar_movielens.ipynb) from the Microsoft official repo. My SAR is trained with the last 30 days. Adding more data will make the performance worse in my case.\n\n| experiment | mapk |\n| --------------------------------------- | --------- |\n| Top 12 items | 0.0081488 |\n| Top 12 items by age | 0.0091866 |\n| bought last 4 week +Top 12 items by age | 0.0245722 |\n| Trending + Top 12 items by age | 0.0255676 |\n| SAR + Trending + Top 12 items by age | 0.0278576 |\n\n##### Overall recall strategy\n\nLet's say we are going to recommend K items for each customer. My Final strategy for the recall is\n1. Recommend with SAR if the customer has a transaction in the last 30 days\n2. If the number of candidates is less than K, pick items from the trending method (remove items that have already been in the first step)\n3. If the number of candidates is still less than K, fill with most popular items by age (using most popular items overall if the age data is missing)\n4. Each recall method has a score and we will use them as features in the ranking phase.\n\n# Ranking\n\nLightGBM classifier is used as a ranking model. There is no significant difference between classifier and lambda ranker. But from my previous experience, the classifier will give a slightly higher performance in general. So I haven't tried lambda ranker in this competition, but it's a good way for ensembling.\n\n##### Aggregation features\n\nNothing special here, just group by user/item, (user, item) and do some aggregation.\n| Feature | Description |\n| ------------ | --------------------------------------------------------------------------- |\n| user feature | average prices of items this customer has bought in the last 7/30/90 days |\n| item feature | average age of customers who have bought this item in the last 7/30/90 days |\n| time | how many days after the customer's last transaction record |\n| interaction | whether the customer has bought the current item before |\n| others | ... |\n\n##### Embedding feature\n\nWe can treat the transaction history of a customer as a sentence, then use some NLP technique to pretrain the embedding of customer/item. My solution using word2vec to train the embedding and simply averaging the item embedding of a customer transaction history as one of the customer features (be care of leaking problems, do not use items in the current week)\n\n##### CV vs LB\n\ntrain with the last 20 weeks and generate 100 candidates for each customer (a bug was fixed in the submission phase several hours before the deadline, so the gap should be smaller)\n\n| validation | score |\n| ------------- | ------ |\n| local cv | 0.0363 |\n| public score | 0.0310 |\n| private score | 0.0312 |\n\n# Optimization\n\nFeature engineering with cudf is 30X faster than using pandas in my case. It allows me to generate more candidates and get the feature in the submission phase. Load LightGBM model with Forest Inference Library and inference with GPU is 40X faster than using 32 core CPUs.\n\n\n# Final Train & Inference\n\n- Train\n\nGenerate 200 candidates for each customer, keeping all positive samples and random choosing half of the negative samples in the last 20 weeks\n\n- Inference\n\nGenerate 400 candidates for each customer, making prediction scores with 4 LightGBM models. Averaging them as the final score and getting top12 items", "votes": null }, { "id": "1783546", "postDate": "05/10/2022 13:19:02", "content": "Great job!
", "rawMarkdown": "Great job!", "votes": null }, { "id": "1783648", "postDate": "05/10/2022 14:52:53", "content": "Well done! I'm glad SAR worked for you as well… we knew it was a good way to generate candidates but unfortunately we did something wrong in our pipeline and we couldn't improve our score.
", "rawMarkdown": "Well done! I'm glad SAR worked for you as well... we knew it was a good way to generate candidates but unfortunately we did something wrong in our pipeline and we couldn't improve our score.", "votes": null }, { "id": "1783653", "postDate": "05/10/2022 14:58:18", "content": "Thanks for posting and congratz for gold!
\nThere's something I don't understood from \"Overall recall strategy\"
\n\nLet's say we are going to recommend K items for each customer. My Final strategy for the recall is
\n\n
\n- Recommend with SAR if the customer has a transaction in the last 30 days
\n- If the number of candidates is less than K, pick items from the trending method (remove items that have already been in the first step)
\n
Can you cite when \"If the number of candidates is less than K\" could happen?
", "rawMarkdown": "Thanks for posting and congratz for gold!\nThere's something I don't understood from \"Overall recall strategy\"\n\n> Let's say we are going to recommend K items for each customer. My Final strategy for the recall is\n1. Recommend with SAR if the customer has a transaction in the last 30 days\n2. If the number of candidates is less than K, pick items from the trending method (remove items that have already been in the first step)\n\nCan you cite when \"If the number of candidates is less than K\" could happen?", "votes": null }, { "id": "1783679", "postDate": "05/10/2022 15:21:56", "content": "Congratz!!!
", "rawMarkdown": "Congratz!!!", "votes": null }, { "id": "1783686", "postDate": "05/10/2022 15:32:47", "content": "Great job !!!
", "rawMarkdown": "Great job !!!", "votes": null }, { "id": "1783788", "postDate": "05/10/2022 17:11:15", "content": "Thanks for this great explanation, great job!
", "rawMarkdown": "Thanks for this great explanation, great job!", "votes": null }, { "id": "1784205", "postDate": "05/11/2022 02:54:51", "content": "There is a threshold for each recall method tuned with my local validation set. If the prediction score of SAR/trending method is less than a value (tuned locally), these items will be removed from my candidates. So actually SAR won't recall K items for each customer especially for those with few transactions (score will be lower).
", "rawMarkdown": "There is a threshold for each recall method tuned with my local validation set. If the prediction score of SAR/trending method is less than a value (tuned locally), these items will be removed from my candidates. So actually SAR won't recall K items for each customer especially for those with few transactions (score will be lower).", "votes": null }, { "id": "1784214", "postDate": "05/11/2022 03:03:11", "content": "Thank you for pointing out the performance of SAR in the early stage, this method not only generates good candidates but also outputs the score which can be used as a golden feature for my ranking model. Hope you can get a better result in the next RecSys competition!
", "rawMarkdown": "Thank you for pointing out the performance of SAR in the early stage, this method not only generates good candidates but also outputs the score which can be used as a golden feature for my ranking model. Hope you can get a better result in the next RecSys competition!", "votes": null }, { "id": "1785882", "postDate": "05/12/2022 13:19:31", "content": "Great Work! Thanks for sharing.
\n\n\nGenerate 200 candidates for each customer, keeping all positive samples and random choosing half of the negative samples in the last 20 weeks
\n
Does it mean that if get 50 positive candidates and 150 negative candidates , label 50 all positive as 1 and 75 negative as 0?
\nMy idea are very similarity with you especially on Data Analysis (90%,50%).But I have no ability to implement because I'm a newbie here.Could you share your code on github ? If convenient, it will help a lot of people like me. I will appreciate it so much.BTY,I found you are in shanghai,and I'm studying there now.
\n\nDoes it mean that if get 50 positive candidates and 150 negative candidates , label 50 all positive as 1 and 75 negative as 0?
\n
Yes, do this properly for each customer because the number of candidates for each customer is different. There may be a better method but I have no time to focus on this.
\n\n\nCould you share your code on github ?
\n
I have no plan to release my messy code so far. I may try to clean my code and add some comments if I have time to do that.
", "rawMarkdown": "> Does it mean that if get 50 positive candidates and 150 negative candidates , label 50 all positive as 1 and 75 negative as 0?\n\nYes, do this properly for each customer because the number of candidates for each customer is different. There may be a better method but I have no time to focus on this.\n\n> Could you share your code on github ?\n\nI have no plan to release my messy code so far. I may try to clean my code and add some comments if I have time to do that.", "votes": null }, { "id": "1790493", "postDate": "05/15/2022 02:50:06", "content": "Thansk for sharing, I have one problem:
\nin the SAR recall model, how do you control the [<Event Weight>] in a interaction?
A good job! Could you share the sorted importance of features used in lightgbm? Besides, How long is the embedding dimension?
", "rawMarkdown": "A good job! Could you share the sorted importance of features used in lightgbm? Besides, How long is the embedding dimension?", "votes": null }, { "id": "1790798", "postDate": "05/15/2022 10:34:45", "content": "\n\nEach event type can be assigned a different weight, for example, we might assign a “buy” event a weight of 10, while a “view” event might only have a weight of 1.
\n
As shown in MS official example, the event weight shows the importance of this interaction. Thus, I give the same value to every behavior since all of them are \"buy\".
", "rawMarkdown": "> Each event type can be assigned a different weight, for example, we might assign a “buy” event a weight of 10, while a “view” event might only have a weight of 1.\n\nAs shown in MS official example, the event weight shows the importance of this interaction. Thus, I give the same value to every behavior since all of them are \"buy\".", "votes": null }, { "id": "1790803", "postDate": "05/15/2022 10:49:57", "content": "\n\nCould you share the sorted importance of features used in lightgbm?
\n
As is expected, the most important features in top 10 are related to repurchase such as how many days since the customer last bought this item/product and how many times this customer bought this item/product.
\n\n\nHow long is the embedding dimension?
\n
I have tried sizes 32 and 64 and the performance of size 64 is slightly better than size 32 on my local CV. But I finally use size 32 considering the limitation of my hardware.
", "rawMarkdown": "> Could you share the sorted importance of features used in lightgbm?\n\nAs is expected, the most important features in top 10 are related to repurchase such as how many days since the customer last bought this item/product and how many times this customer bought this item/product. \n\n> How long is the embedding dimension?\n\nI have tried sizes 32 and 64 and the performance of size 64 is slightly better than size 32 on my local CV. But I finally use size 32 considering the limitation of my hardware.", "votes": null }, { "id": "1790902", "postDate": "05/15/2022 13:04:08", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": null }, { "id": "1791013", "postDate": "05/15/2022 15:10:35", "content": "Thanks for your reply, learned a lot!
", "rawMarkdown": "Thanks for your reply, learned a lot!", "votes": null } ], "comments": [ { "id": 1783546, "author_name": "xiamaozi11", "author_url": "", "post_date": "05/10/2022 13:19:02", "content": "Great job!
", "votes": null, "replies": [] }, { "id": 1783648, "author_name": "igormunizims", "author_url": "", "post_date": "05/10/2022 14:52:53", "content": "Well done! I'm glad SAR worked for you as well… we knew it was a good way to generate candidates but unfortunately we did something wrong in our pipeline and we couldn't improve our score.
", "votes": null, "replies": [ { "id": 1784214, "author_name": "yzheng21", "author_url": "", "post_date": "05/11/2022 03:03:11", "content": "Thank you for pointing out the performance of SAR in the early stage, this method not only generates good candidates but also outputs the score which can be used as a golden feature for my ranking model. Hope you can get a better result in the next RecSys competition!
", "votes": null, "replies": [] } ] }, { "id": 1783653, "author_name": "igorkf", "author_url": "", "post_date": "05/10/2022 14:58:18", "content": "Thanks for posting and congratz for gold!
\nThere's something I don't understood from \"Overall recall strategy\"
\n\nLet's say we are going to recommend K items for each customer. My Final strategy for the recall is
\n\n
\n- Recommend with SAR if the customer has a transaction in the last 30 days
\n- If the number of candidates is less than K, pick items from the trending method (remove items that have already been in the first step)
\n
Can you cite when \"If the number of candidates is less than K\" could happen?
", "votes": null, "replies": [ { "id": 1784205, "author_name": "yzheng21", "author_url": "", "post_date": "05/11/2022 02:54:51", "content": "There is a threshold for each recall method tuned with my local validation set. If the prediction score of SAR/trending method is less than a value (tuned locally), these items will be removed from my candidates. So actually SAR won't recall K items for each customer especially for those with few transactions (score will be lower).
", "votes": null, "replies": [] } ] }, { "id": 1783679, "author_name": "renxingkai", "author_url": "", "post_date": "05/10/2022 15:21:56", "content": "Congratz!!!
", "votes": null, "replies": [] }, { "id": 1783686, "author_name": "mohamedaliguirat", "author_url": "", "post_date": "05/10/2022 15:32:47", "content": "Great job !!!
", "votes": null, "replies": [] }, { "id": 1783788, "author_name": "ryanrose1", "author_url": "", "post_date": "05/10/2022 17:11:15", "content": "Thanks for this great explanation, great job!
", "votes": null, "replies": [] }, { "id": 1785882, "author_name": "jiahongxie", "author_url": "", "post_date": "05/12/2022 13:19:31", "content": "Great Work! Thanks for sharing.
\n\n\nGenerate 200 candidates for each customer, keeping all positive samples and random choosing half of the negative samples in the last 20 weeks
\n
Does it mean that if get 50 positive candidates and 150 negative candidates , label 50 all positive as 1 and 75 negative as 0?
\nMy idea are very similarity with you especially on Data Analysis (90%,50%).But I have no ability to implement because I'm a newbie here.Could you share your code on github ? If convenient, it will help a lot of people like me. I will appreciate it so much.BTY,I found you are in shanghai,and I'm studying there now.
\n\nDoes it mean that if get 50 positive candidates and 150 negative candidates , label 50 all positive as 1 and 75 negative as 0?
\n
Yes, do this properly for each customer because the number of candidates for each customer is different. There may be a better method but I have no time to focus on this.
\n\n\nCould you share your code on github ?
\n
I have no plan to release my messy code so far. I may try to clean my code and add some comments if I have time to do that.
", "votes": null, "replies": [] } ] }, { "id": 1790493, "author_name": "cherrizhu", "author_url": "", "post_date": "05/15/2022 02:50:06", "content": "Thansk for sharing, I have one problem:
\nin the SAR recall model, how do you control the [<Event Weight>] in a interaction?
\n\nEach event type can be assigned a different weight, for example, we might assign a “buy” event a weight of 10, while a “view” event might only have a weight of 1.
\n
As shown in MS official example, the event weight shows the importance of this interaction. Thus, I give the same value to every behavior since all of them are \"buy\".
", "votes": null, "replies": [] }, { "id": 1791013, "author_name": "cherrizhu", "author_url": "", "post_date": "05/15/2022 15:10:35", "content": "Thanks for your reply, learned a lot!
", "votes": null, "replies": [] } ] }, { "id": 1790659, "author_name": "dengniewei", "author_url": "", "post_date": "05/15/2022 07:12:43", "content": "A good job! Could you share the sorted importance of features used in lightgbm? Besides, How long is the embedding dimension?
", "votes": null, "replies": [ { "id": 1790803, "author_name": "yzheng21", "author_url": "", "post_date": "05/15/2022 10:49:57", "content": "\n\nCould you share the sorted importance of features used in lightgbm?
\n
As is expected, the most important features in top 10 are related to repurchase such as how many days since the customer last bought this item/product and how many times this customer bought this item/product.
\n\n\nHow long is the embedding dimension?
\n
I have tried sizes 32 and 64 and the performance of size 64 is slightly better than size 32 on my local CV. But I finally use size 32 considering the limitation of my hardware.
", "votes": null, "replies": [] }, { "id": 1790902, "author_name": "dengniewei", "author_url": "", "post_date": "05/15/2022 13:04:08", "content": "Thanks for sharing.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1783261": "It's too long since I last see the competition about recommendations on Kaggle. So great thanks to the Kaggle team and H&M for hosting such an interesting competition. The high correlation between CV and LB, big training data for deep data mining, no leaking of test information, friendly staff and participants, all of these make the successful competition.\n\n# Summary\n\nThe key idea of my solution will be illustrated here for fast reading purposes.\n\n##### Validation strategy\n\n```python\ntrain = transactions.loc[transactions[\"t_dat\"] < \"2020-09-16\"]\nvalid = transactions.loc[transactions[\"t_dat\"] >= \"2020-09-16\"]\n```\n\nIt is better to use k-fold validation in [this post](https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/308919) to avoid overfitting the last week if you have enough computing power.\n\nBut from my experience in this competition, LB score and CV score are highly correlated using last week only as a validation set.\n\n\n##### Recall\n\n| Recall method | link |\n| ------------------------------- | ----------------------------------------------------------------------------------------------------------- |\n| SAR | [notebook](https://github.com/microsoft/recommenders/blob/main/examples/00_quick_start/sar_movielens.ipynb) |\n| Trending | [notebook](https://www.kaggle.com/code/byfone/h-m-trending-products-weekly) |\n| Most popular items based on age | / |\n\n\n##### Ranking\n\nLightGBM trained with 20 weeks of data using around 300 features\n\n\n##### Ensemble\n\nI have no time to train all 5 models with k fold split on the train set, only 3 of them are completed before the deadline. So my final submission is based on 1 model trained on all data samples and the other 3 models trained based on k fold split mentioned before (4 models in total).\n\n| Model | public score | private score |\n| --------- | ------------ | ------------- |\n| one model | 0.0338 | 0.0341 |\n| ensemble | 0.0343 | 0.0346 |\n\n\n##### Optimization\n\nUsing cudf and Forest Inference Library from Rapids for fast feature generation and fast tree inference\n\n\n##### Hardware\n\n| Hardware | value |\n| -------- | -------- |\n| CPU | 32core |\n| Memory | 128G |\n| GPU | V100 32G |\n\n\n# Data Analysis\n\n1. We can find 90%+ items in the validation set also in the last 30 days\n\n```python\ntrain_condition = (trans[\"t_dat\"] >= \"2020-08-16\") & (trans[\"t_dat\"] < \"2020-09-16\")\nitems_in_valid = trans.loc[trans[\"t_dat\"] >= \"2020-09-16\", \"article_id\"].unique()\nitems_last_month = trans.loc[train_condition, \"article_id\"].unique()\nprint(len(set(items_in_valid) & set(items_last_month)) / len(set(items_in_valid)))\n# 0.92\n```\n\n2. Approximately 50% of customers in the validation set have transaction records in the last 30 days\n\n```python\ntrain_condition = (trans[\"t_dat\"] >= \"2020-08-16\") & (trans[\"t_dat\"] < \"2020-09-16\")\nusers_in_valid = trans.loc[trans[\"t_dat\"] >= \"2020-09-16\", \"customer_id\"].unique()\nusers_last_month = trans.loc[train_condition, \"customer_id\"].unique()\nprint(len(set(users_in_valid) & set(users_last_month)) / len(set(users_in_valid)))\n# 0.47\n```\n\nConsidering these results, we can focus on recommending items bought in the last 30 days, especially for those customers who have transaction records.\n\n# Recall\n\nOnly mapk of each recall method on the local validation set is recorded. Most of these ideas are coming from public notebooks or discussions and I will give the link if so.\n\n##### Most popular items last 7 days\n\nSimply sorting items by transaction records. This is also the basic rule for those customers has no age data\n\n| Recall method | mapk |\n| ------------- | ---------- |\n| Top 12 items | 0.00814889 |\n\n##### Most popular itmes last 7 days by age\n\nWe are doing personalized recommendations, so it's better to use some customers' features based on the rule for recall. Unfortunately, only 2 features seem valuable to me at first glance, age, and postal code. It's impossible to get the location feature from the postal code since it is encoded. Finally, the only choice is age.\n\n```python\nbins = [0, 18, 22, 28, 35, 45, 55, 65, 200]\nlast_week_transactions[\"age_bin\"] = pd.cut(user[\"age\"], bins=bins)\nlast_week_transactions_popular = last_week_transactions.groupby(\"age_bin\").size()\n```\n\nBins are simply tuned with last week's data. The mapk score is slightly higher than only using the most popular items.\n\n| Recall method | mapk |\n| ------------------- | --------- |\n| Top 12 items | 0.0081488 |\n| Top 12 items by age | 0.0091866 |\n\n##### Consider items that customers have bought before\n\nThe first method is just adding items that customers have bought in the last 4 weeks. Similar to [this notebook](https://www.kaggle.com/code/hengzheng/time-is-our-best-friend-v2)\n\n| Recall method | mapk |\n| --------------------------------------- | --------- |\n| Top 12 items | 0.0081488 |\n| Top 12 items by age | 0.0091866 |\n| bought last 4 week +Top 12 items by age | 0.0245722 |\n\n##### Use Trending\n\nThere is a significant increase in mapk. But since we know only half of the customers have transaction history in the last 30 days, we can do better. According to [this notebook](https://www.kaggle.com/code/byfone/h-m-trending-products-weekly), we have a wonderful method to compute the score of each item to all customers without the limitation of 30 days.\n\n\n| Recall method | mapk |\n| --------------------------------------- | --------- |\n| Top 12 items | 0.0081488 |\n| Top 12 items by age | 0.0091866 |\n| bought last 4 week +Top 12 items by age | 0.0245722 |\n| Trending + Top 12 items by age | 0.0255676 |\n\n##### Recommend with SAR\n\nSAR is a kind of item-based collaborative filtering. As mentioned in [this post](https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/316205), we can get 0.27+ with SAR. Here is the [example](https://github.com/microsoft/recommenders/blob/main/examples/00_quick_start/sar_movielens.ipynb) from the Microsoft official repo. My SAR is trained with the last 30 days. Adding more data will make the performance worse in my case.\n\n| experiment | mapk |\n| --------------------------------------- | --------- |\n| Top 12 items | 0.0081488 |\n| Top 12 items by age | 0.0091866 |\n| bought last 4 week +Top 12 items by age | 0.0245722 |\n| Trending + Top 12 items by age | 0.0255676 |\n| SAR + Trending + Top 12 items by age | 0.0278576 |\n\n##### Overall recall strategy\n\nLet's say we are going to recommend K items for each customer. My Final strategy for the recall is\n1. Recommend with SAR if the customer has a transaction in the last 30 days\n2. If the number of candidates is less than K, pick items from the trending method (remove items that have already been in the first step)\n3. If the number of candidates is still less than K, fill with most popular items by age (using most popular items overall if the age data is missing)\n4. Each recall method has a score and we will use them as features in the ranking phase.\n\n# Ranking\n\nLightGBM classifier is used as a ranking model. There is no significant difference between classifier and lambda ranker. But from my previous experience, the classifier will give a slightly higher performance in general. So I haven't tried lambda ranker in this competition, but it's a good way for ensembling.\n\n##### Aggregation features\n\nNothing special here, just group by user/item, (user, item) and do some aggregation.\n| Feature | Description |\n| ------------ | --------------------------------------------------------------------------- |\n| user feature | average prices of items this customer has bought in the last 7/30/90 days |\n| item feature | average age of customers who have bought this item in the last 7/30/90 days |\n| time | how many days after the customer's last transaction record |\n| interaction | whether the customer has bought the current item before |\n| others | ... |\n\n##### Embedding feature\n\nWe can treat the transaction history of a customer as a sentence, then use some NLP technique to pretrain the embedding of customer/item. My solution using word2vec to train the embedding and simply averaging the item embedding of a customer transaction history as one of the customer features (be care of leaking problems, do not use items in the current week)\n\n##### CV vs LB\n\ntrain with the last 20 weeks and generate 100 candidates for each customer (a bug was fixed in the submission phase several hours before the deadline, so the gap should be smaller)\n\n| validation | score |\n| ------------- | ------ |\n| local cv | 0.0363 |\n| public score | 0.0310 |\n| private score | 0.0312 |\n\n# Optimization\n\nFeature engineering with cudf is 30X faster than using pandas in my case. It allows me to generate more candidates and get the feature in the submission phase. Load LightGBM model with Forest Inference Library and inference with GPU is 40X faster than using 32 core CPUs.\n\n\n# Final Train & Inference\n\n- Train\n\nGenerate 200 candidates for each customer, keeping all positive samples and random choosing half of the negative samples in the last 20 weeks\n\n- Inference\n\nGenerate 400 candidates for each customer, making prediction scores with 4 LightGBM models. Averaging them as the final score and getting top12 items", "1783546": "Great job!", "1783648": "Well done! I'm glad SAR worked for you as well... we knew it was a good way to generate candidates but unfortunately we did something wrong in our pipeline and we couldn't improve our score.", "1783653": "Thanks for posting and congratz for gold!\nThere's something I don't understood from \"Overall recall strategy\"\n\n> Let's say we are going to recommend K items for each customer. My Final strategy for the recall is\n1. Recommend with SAR if the customer has a transaction in the last 30 days\n2. If the number of candidates is less than K, pick items from the trending method (remove items that have already been in the first step)\n\nCan you cite when \"If the number of candidates is less than K\" could happen?", "1783679": "Congratz!!!", "1783686": "Great job !!!", "1783788": "Thanks for this great explanation, great job!", "1784205": "There is a threshold for each recall method tuned with my local validation set. If the prediction score of SAR/trending method is less than a value (tuned locally), these items will be removed from my candidates. So actually SAR won't recall K items for each customer especially for those with few transactions (score will be lower).", "1784214": "Thank you for pointing out the performance of SAR in the early stage, this method not only generates good candidates but also outputs the score which can be used as a golden feature for my ranking model. Hope you can get a better result in the next RecSys competition!", "1785882": "Great Work! Thanks for sharing.\n> Generate 200 candidates for each customer, keeping all positive samples and random choosing half of the negative samples in the last 20 weeks\n\n\nDoes it mean that if get 50 positive candidates and 150 negative candidates , label 50 all positive as 1 and 75 negative as 0?\nMy idea are very similarity with you especially on Data Analysis (90%,50%).But I have no ability to implement because I'm a newbie here.Could you share your code on github ? If convenient, it will help a lot of people like me. I will appreciate it so much.BTY,I found you are in shanghai,and I'm studying there now.", "1786577": "> Does it mean that if get 50 positive candidates and 150 negative candidates , label 50 all positive as 1 and 75 negative as 0?\n\nYes, do this properly for each customer because the number of candidates for each customer is different. There may be a better method but I have no time to focus on this.\n\n> Could you share your code on github ?\n\nI have no plan to release my messy code so far. I may try to clean my code and add some comments if I have time to do that.", "1790493": "Thansk for sharing, I have one problem: \nin the SAR recall model, how do you control the `[