This thread has (happily) evolved from the topic title to a more general sharing by @lihaorocky and @paweljankiewicz (both very high-ranking on the LB) on their methods in this competition - see the sub-threads below.
\nUpdate per the original post:\nFigured it out - there was silent leakage going on, when I was merging the labels in.\nSee this post for details.
\nI'm using the default training loss (lambdarank), and have tried both the default evaluation metric (NDCG) as well as MAP, but the score deteriorates as training progresses, even on the training set.
\nHow is it possible that the loss is not correlated with the evaluation metric?
\nWhat does one do in such a case?
This thread has (happily) evolved from the topic title to a more general sharing by @lihaorocky and @paweljankiewicz (both very high-ranking on the LB) on their methods in this competition - see the sub-threads below.
\nUpdate per the original post:\nFigured it out - there was silent leakage going on, when I was merging the labels in.\nSee this post for details.
\nI'm using the default training loss (lambdarank), and have tried both the default evaluation metric (NDCG) as well as MAP, but the score deteriorates as training progresses, even on the training set.
\nHow is it possible that the loss is not correlated with the evaluation metric?
\nWhat does one do in such a case?
I would suggest you doing the follow steps:
\n1: Say you have a column in your feature dataframe called \"target_week\" to represent the following 7-days window. Then sort the dataframe
\ndf_feat_train.sort_values(['target_week', 'customer_id'], ignore_index=True, inplace=True)
\n2: After splitting the training and validation dataset, try:
\ntrain_baskets = df_feat_train.loc[train_idx, :].groupby(['target_week', 'customer_id'])['article_id'].count().values
\nto get the train baskets. And do the same for the validation dataset.
\n3: Lower the learning rate, in my case, the training loss is early-stopped in sort of 150 training rounds with learning_rate 0.03 when using only one week for training and one week for validation.
\nHope it helps.
@lihaorocky - thank you for replying, and for sharing.
\nI'm only using one target week as of now, so I don't need to group by target_week, but I'm doing exactly that.
\nMy code is:
features_df = features_df.sort_values(\"customer_id\").reset_index(drop=True)\ngroups = list(features_df.groupby(\"customer_id\")[\"article_id\"].count())\nI can't get the metric to improve even for 2 training rounds.
\nMaybe I just need to move away from prototyping and adding more features and data…
", "rawMarkdown": "@lihaorocky - thank you for replying, and for sharing.\n\nI'm only using one target week as of now, so I don't need to group by target_week, but I'm doing exactly that.\nMy code is:\n\n```\nfeatures_df = features_df.sort_values(\"customer_id\").reset_index(drop=True)\ngroups = list(features_df.groupby(\"customer_id\")[\"article_id\"].count())\n```\n\nI can't get the metric to improve even for 2 training rounds.\n\nMaybe I just need to move away from prototyping and adding more features and data..." }, { "id": 1749561, "postDate": "2022-04-08T17:45:52.487Z", "content": "Hi, @lihaorocky . Thanks for sharing
\nAs I understood target_week is a binary relevance (1 or 0) to denote if customer bought or not an article, right?
\nIf this is the case, there's any reason to sort values by target_week? This way seems you are putting 0's first in the whole dataset.
Not really. I use target_week column to represent the target week period. For example, on an observation date 20200915, your target week will be 20200916-20200922, I give it a label 0. And for an observation date 20200908, the target week will be 20200909-20200915, I give it a label 1 etc. This way you will get lots of target_weeks 0 1 2 3 4 5….I use the data before observation date to create candidate samples and sample features, while using the data during target week to create sample labels(bought or not). I sort the samples based on target_week and customer_id because I have a lot of weeks' samples to be input to LightGBMRanker. So it need to be sorted to create baskets.
", "rawMarkdown": "Not really. I use target_week column to represent the target week period. For example, on an observation date 20200915, your target week will be 20200916-20200922, I give it a label 0. And for an observation date 20200908, the target week will be 20200909-20200915, I give it a label 1 etc. This way you will get lots of target_weeks 0 1 2 3 4 5....I use the data before observation date to create candidate samples and sample features, while using the data during target week to create sample labels(bought or not). I sort the samples based on target_week and customer_id because I have a lot of weeks' samples to be input to LightGBMRanker. So it need to be sorted to create baskets.", "votes": 10 }, { "id": 1750082, "postDate": "2022-04-09T10:03:30.087Z", "content": "Hi， @lihaorocky . Thanks for sharing
\nI‘d like to ask ， \"target_weeks\" this column just for building pos/neg samples ?
And in the sorting model, I only use some existing features on the item side and user side. I'm a little confused about the sorting feature and don't know how to construct it, can you give me some advice?
", "rawMarkdown": "Hi， @lihaorocky . Thanks for sharing\nI‘d like to ask ， \"target_weeks\" this column just for building pos/neg samples ?\n\nAnd in the sorting model, I only use some existing features on the item side and user side. I'm a little confused about the sorting feature and don't know how to construct it, can you give me some advice?" }, { "id": 1750110, "postDate": "2022-04-09T10:33:36.200Z", "content": "For your first question, yes, and it also eases filtering training and validation samples based on it.
\nWell, there are a lot of ways to construct features. Besides the features of customer and article alone, you can measure the similarity between customers and articles. I will not talk too much about this but I would say it will give huge performance improvement.
Thanks for the answer @lihaorocky. Seems that LGBMRanker is limited to a relevance of 31, so your target_weeks lies in the range [0, max(31, n_weeks)]?
Limited by the Ram size, my current score is using only 5 weeks for training. I'm trying adding more weeks.
", "rawMarkdown": "Limited by the Ram size, my current score is using only 5 weeks for training. I'm trying adding more weeks.", "votes": 6 }, { "id": 1750196, "postDate": "2022-04-09T12:11:13.747Z", "content": "I got it. Thanks for the infos @lihaorocky . You helped me a lot.
", "rawMarkdown": "I got it. Thanks for the infos @lihaorocky . You helped me a lot." }, { "id": 1750922, "postDate": "2022-04-10T08:17:31.963Z", "content": "Thanks for your sharing, it really helps me a lot! But I have a question, do you use the popularity features of articles for LGBMRanker training? Because I find the importance of this feature is much higher than other features, such as the similarity between customers and articles, and it reduces the score of my final prediction, I wonder if you meet this situation, thank you! @lihaorocky
", "rawMarkdown": "Thanks for your sharing, it really helps me a lot! But I have a question, do you use the popularity features of articles for LGBMRanker training? Because I find the importance of this feature is much higher than other features, such as the similarity between customers and articles, and it reduces the score of my final prediction, I wonder if you meet this situation, thank you! @lihaorocky ", "votes": 2 }, { "id": 1750950, "postDate": "2022-04-10T08:44:56.767Z", "content": "Yes, I did use. I think the way you generated this kind of features (popularity of articles, similarity of articles and customers) are very important. And it's also depending on the strategy you created candidate pairs. In my case, the similarity features are always the most important ones.
", "rawMarkdown": "Yes, I did use. I think the way you generated this kind of features (popularity of articles, similarity of articles and customers) are very important. And it's also depending on the strategy you created candidate pairs. In my case, the similarity features are always the most important ones.", "votes": 4 }, { "id": 1751089, "postDate": "2022-04-10T11:04:11.340Z", "content": "Thanks for your reply, it's really helpful!
", "rawMarkdown": "Thanks for your reply, it's really helpful!" }, { "id": 1754264, "postDate": "2022-04-13T14:02:16.813Z", "content": "Hi @lihaorocky
\nI'm bit confused on how to split train/validation sets after generating the dataset.
\nTo create train baskets you do:
train_baskets = df_feat_train.loc[train_idx, :].groupby(['target_week', 'customer_id'])['article_id'].count().values\nIn this case, train_idx should be created in a group split way?
\nI mean, all rows from a customer must be in train OR in val or you're doing just a holdout (e.g. 80/20) split?
I split the train/validation sets based on time. Specifically I split them based on the value of target_week(0 as validation set and >1 as train set).
", "rawMarkdown": "I split the train/validation sets based on time. Specifically I split them based on the value of target_week(0 as validation set and >1 as train set).", "votes": 3 }, { "id": 1754302, "postDate": "2022-04-13T14:27:21.540Z", "content": "Thank you very much @lihaorocky . This really makes sense…I was overcomplicating.
", "rawMarkdown": "Thank you very much @lihaorocky . This really makes sense...I was overcomplicating." }, { "id": 1754358, "postDate": "2022-04-13T15:32:25.403Z", "content": "Hi @lihaorocky ,thanks for your reply.
\nPlease allow me to ask two more questions
During the recall phase, did you use multiple recalls?
If each user recalls nearly 50 articles, then more than 1 million users will have more than 50 million recall data, and then I am sorting these recall data. But I am a little confused now, how to solve such a large amount of data?
1: Yes, I use multiple recalls
\n2: As others posted in some discussions, if you are using LGBMRanker you can drop the users' recalls if they haven't any purchase in next 7 days or your recalls don't contains any real purchase that will really happen, which will hugely reduce the number of candidates to be input to your ranking model.
As to the inference stage, you can't use the two above mentioned tricks to reduce candidates to be predicted. And yes, there will be \"more than 50 million recall data\", but you can generate features and predict for these recalls batch by batch(for example in each batch you generate features and predict for 1 million recalls), which actually doesn't cost too much time. In my case, predicting for 160 million recalls cost overall 2 hours including generating features and model prediction.
", "rawMarkdown": "1: Yes, I use multiple recalls\n2: As others posted in some discussions, if you are using LGBMRanker you can drop the users' recalls if they haven't any purchase in next 7 days or your recalls don't contains any real purchase that will really happen, which will hugely reduce the number of candidates to be input to your ranking model.\n \nAs to the inference stage, you can't use the two above mentioned tricks to reduce candidates to be predicted. And yes, there will be \"more than 50 million recall data\", but you can generate features and predict for these recalls batch by batch(for example in each batch you generate features and predict for 1 million recalls), which actually doesn't cost too much time. In my case, predicting for 160 million recalls cost overall 2 hours including generating features and model prediction.", "votes": 6 }, { "id": 1754547, "postDate": "2022-04-13T18:52:16.883Z", "content": "@lihaorocky could be as well describing my process. I have 1000 batches for the submission and the prediction is the slowest part - it takes about 5 hours. You did well on the candidate generation - I think I have close to 1 billion candidates.
", "rawMarkdown": "@lihaorocky could be as well describing my process. I have 1000 batches for the submission and the prediction is the slowest part - it takes about 5 hours. You did well on the candidate generation - I think I have close to 1 billion candidates.", "votes": 5 }, { "id": 1754567, "postDate": "2022-04-13T19:02:55.870Z", "content": "If you don't mind:
\nI'm using Rust for all the feature generation. Overall it takes somewhere around 2.5 hours on 32 threads. I think I could reduce it to 1 hour if I parallelized more calculations.
", "rawMarkdown": "I'm using Rust for all the feature generation. Overall it takes somewhere around 2.5 hours on 32 threads. I think I could reduce it to 1 hour if I parallelized more calculations.", "votes": 2 }, { "id": 1756464, "postDate": "2022-04-15T14:38:38.147Z", "content": "@panadam @lihaorocky
\nI have a question about your interesting discussion.
\nWhat does 'recalls' exactly mean?
In recomendations system, normally it consists two stages: recall and ranking.
\nRecalls here means finding some candidate articles one customers will have interations with. In this stage, you want to contains as many as articles one customers will bought(in this competition) using as few as candidates. You can develop multiple strategies to creating candidate articles which are mentioned in a lot of posts.
\nIn second stage, you want to rank those candidates based on the possibility being purchased buy customers.
@lihaorocky
\nThanks for reply!
\nThen, I have interpreted 'multiple recalls' means that you used multiple strategies to make candidates in the SAME exam. How do you compare possibilities from different strategies? I guess, do you use standarization?
Actually, I give the choice mostly to the ranking model. He (or she) is much smarter than me.😂
", "rawMarkdown": "Actually, I give the choice mostly to the ranking model. He (or she) is much smarter than me.😂", "votes": 3 }, { "id": 1756539, "postDate": "2022-04-15T15:37:31.610Z", "content": "@lihaorocky
\nAI overtakes humans :D
\nDo you suggest to give ['possibility', 'strategy', …(other features)] to ranking model?
Yes, for example when you develop the strategy based on article's hotness. The article hottness can be measured easily using some method like purchased count in previous week. I use some thing like this. But from the performance comparison using or not using those measurement feature in ranking model, the improvement of adding those features is quite small. That's why I say, mostly I leave it to the ranking model.
", "rawMarkdown": "Yes, for example when you develop the strategy based on article's hotness. The article hottness can be measured easily using some method like purchased count in previous week. I use some thing like this. But from the performance comparison using or not using those measurement feature in ranking model, the improvement of adding those features is quite small. That's why I say, mostly I leave it to the ranking model. ", "votes": 4 }, { "id": 1756568, "postDate": "2022-04-15T16:11:19.640Z", "content": "Hi @lihaorocky thanks for your shares.
\nThere's a way to know if the strategy can generate good candidates?
Yes, for example on an observation date 20200915, from one of your candidate generating strategy you create 1 million pairs(customer_id, article_id, target_week), and from the next 7-days transations window 20200916~20200922, you can label those pairs with 1(purchase happend) and 0(purchase not happen). Say totally you have 50 thousands actually happend pairs. Then the \"matching ratio\" will be 50000 / 1000000 = 0.05. Clearly if keeping the number of total candidate pairs fixed, the bigger the numebr actully happend purchase pairs the better. So you can use this above mentioned ratio to measure how good you strategy is.
", "rawMarkdown": "Yes, for example on an observation date 20200915, from one of your candidate generating strategy you create 1 million pairs(customer_id, article_id, target_week), and from the next 7-days transations window 20200916~20200922, you can label those pairs with 1(purchase happend) and 0(purchase not happen). Say totally you have 50 thousands actually happend pairs. Then the \"matching ratio\" will be 50000 / 1000000 = 0.05. Clearly if keeping the number of total candidate pairs fixed, the bigger the numebr actully happend purchase pairs the better. So you can use this above mentioned ratio to measure how good you strategy is.", "votes": 5 }, { "id": 1756645, "postDate": "2022-04-15T18:04:45.777Z", "content": "Hi @lihaorocky
\nThank you very much for your explanation. This totally makes sense
@lihaorocky
\nAfter all, my understanding of ranking model improved very well thanks to your clear explanation.
\nLet's do our best together!
@lihaorocky Hi, roughly what percentage of actual purchases can you generate in the recall stage, with 160 million candidates?
", "rawMarkdown": "@lihaorocky Hi, roughly what percentage of actual purchases can you generate in the recall stage, with 160 million candidates? ", "votes": 1 }, { "id": 1759099, "postDate": "2022-04-18T10:41:58.393Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1759101, "postDate": "2022-04-18T10:42:50.867Z", "content": "Well, the 160 million candidates is in the prediction stage, which mean the actually purchases is not known until 20200929. If you mean what the percentage ratio of candidates in training stage is, different recall strategies acts very differently, which range from 0.3~0.001.
", "rawMarkdown": "Well, the 160 million candidates is in the prediction stage, which mean the actually purchases is not known until 20200929. If you mean what the percentage ratio of candidates in training stage is, different recall strategies acts very differently, which range from 0.3~0.001.", "votes": 4 }, { "id": 1759122, "postDate": "2022-04-18T11:04:24.343Z", "content": "@lihaorocky I think he meant the recall of real sales from the candidates altogether. So if there was 300k transactions in the last week how many you can generate using your candidates. I must check this too.
", "rawMarkdown": "@lihaorocky I think he meant the recall of real sales from the candidates altogether. So if there was 300k transactions in the last week how many you can generate using your candidates. I must check this too.", "votes": 2 }, { "id": 1759145, "postDate": "2022-04-18T11:23:58.870Z", "content": "@paweljankiewicz Yes that's what I mean. @lihaorocky 30% is striking, I haven't been woking on this that much but saw a post saying that he could hardly exceed 7.5%. https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/314458
", "rawMarkdown": "@paweljankiewicz Yes that's what I mean. @lihaorocky 30% is striking, I haven't been woking on this that much but saw a post saying that he could hardly exceed 7.5%. https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/314458", "votes": 1 }, { "id": 1759148, "postDate": "2022-04-18T11:27:12.257Z", "content": "I just realized you mean recall. In my case, it's about 25%.
", "rawMarkdown": "I just realized you mean recall. In my case, it's about 25%.", "votes": 2 } ] }, { "id": 1755632, "postDate": "2022-04-14T19:29:27.600Z", "content": "Hi. I'm thinking how much your CV boosted after ranking the set of candidates with LGBMRanker or other model.
\nCurrently I'm getting just +0.0015 boost, so I think something is wrong.
Could you guys @paweljankiewicz @lihaorocky @jacob34 share this if you don't mind?
", "rawMarkdown": "Hi. I'm thinking how much your CV boosted after ranking the set of candidates with LGBMRanker or other model. \nCurrently I'm getting just +0.0015 boost, so I think something is wrong.\n\nCould you guys @paweljankiewicz @lihaorocky @jacob34 share this if you don't mind?", "votes": 1, "replies": [ { "id": 1755675, "postDate": "2022-04-14T21:04:12.523Z", "content": "There is no CV before the ranking. Ranking model is a must. I have more than 1000 candidate items for each customer. How else would I be able to use them without the ranker? Single strategies to generate candidates are pretty useless on their own. You need to have multiple candidate generation strategies like I explained in the other thread.
", "rawMarkdown": "There is no CV before the ranking. Ranking model is a must. I have more than 1000 candidate items for each customer. How else would I be able to use them without the ranker? Single strategies to generate candidates are pretty useless on their own. You need to have multiple candidate generation strategies like I explained in the other thread.", "votes": 2 }, { "id": 1755683, "postDate": "2022-04-14T21:23:21.233Z", "content": "Thanks for the answer @paweljankiewicz.
\nI understood that we need to have many candidates generation strategies, but there's some points not clear to me yet.
How do I know if a candidate strategy is good? I suppose that good strategy would generate more 1's (item was bought by the customer in the target week) than a strategy with bad candidates. For instance, good strategy A generated 1000 candidates (85% is label=0, 15% is label=1), meanwhile strategy B generated 1000 candidates (95% is label=0, 5% is label=1)
How should we assign the label? Here's a pseudo code:
for each row in candidates:\n if this candidate item was bought by this customer in the next week:\n label = 1\n else:\n label = 0\nIs this right?
\nSorry for the (possible) obvious questions…this is the first time we're using a ranking model.
", "rawMarkdown": "Thanks for the answer @paweljankiewicz. \nI understood that we need to have many candidates generation strategies, but there's some points not clear to me yet.\n1. How do I know if a candidate strategy is good? I suppose that good strategy would generate more 1's (item was bought by the customer in the target week) than a strategy with bad candidates. For instance, good strategy A generated 1000 candidates (85% is label=0, 15% is label=1), meanwhile strategy B generated 1000 candidates (95% is label=0, 5% is label=1)\n\n2. How should we assign the label? Here's a pseudo code:\n```\nfor each row in candidates:\n if this candidate item was bought by this customer in the next week:\n label = 1\n else:\n label = 0\n```\nIs this right?\n\nSorry for the (possible) obvious questions...this is the first time we're using a ranking model.\n", "votes": 1 }, { "id": 1756598, "postDate": "2022-04-15T16:41:47.153Z", "content": "No worries. Ranking model are quite helpful in many situations.
\nThe pseudo code to generate the training data would be something like this for 2 strategies
\nall_data = []\nfor query_id, customer in enumerate(customers):\n candidates_a = generate_candidates_a(customer)\n candidates_b = generate_candidates_b(customer)\n all_candidates = union(candidates_a, candidates_b) # you generate a list of all unique candidates\n customer_df = pd.DataFrame({\n query_id: query_id,\n customer_id: [customer_id]*len(all_candidates),\n article_id: all_candidates,\n bought_next_week: all_candidates.isin(customer_sales_next_week.get(customer, [])), # this is your label\n article_in_strategy_a: all_candidates.isin(candidates_a),\n article_in_strategy_b: all_candidates.isin(candidates_b),\n })\n all_data.append(customer_df)\nA dataframe like this for each customer is a query for a ranking model. It is like a separate part of your data that is evaluated by the ranking model. Unlike some other methods like regression and classification you want to properly rank items withing customer query.
\nSo going back to your first question - the ranking model sees the articles, whether they were bought or not and from which strategies they came so it can assess how good the candidate is based on this information. So you want to have precise strategies without adding too many negative observations.
", "rawMarkdown": "No worries. Ranking model are quite helpful in many situations.\n\nThe pseudo code to generate the training data would be something like this for 2 strategies\n\n```python\nall_data = []\nfor query_id, customer in enumerate(customers):\n candidates_a = generate_candidates_a(customer)\n candidates_b = generate_candidates_b(customer)\n all_candidates = union(candidates_a, candidates_b) # you generate a list of all unique candidates\n customer_df = pd.DataFrame({\n query_id: query_id,\n customer_id: [customer_id]*len(all_candidates),\n article_id: all_candidates,\n bought_next_week: all_candidates.isin(customer_sales_next_week.get(customer, [])), # this is your label\n article_in_strategy_a: all_candidates.isin(candidates_a),\n article_in_strategy_b: all_candidates.isin(candidates_b),\n })\n all_data.append(customer_df)\n```\n\nA dataframe like this for each customer is a query for a ranking model. It is like a separate part of your data that is evaluated by the ranking model. Unlike some other methods like regression and classification you want to properly rank items withing customer query. \n\nSo going back to your first question - the ranking model sees the articles, whether they were bought or not and from which strategies they came so it can assess how good the candidate is based on this information. So you want to have precise strategies without adding too many negative observations.", "votes": 9 }, { "id": 1756641, "postDate": "2022-04-15T17:54:01.930Z", "content": "Hi @paweljankiewicz
\nThank you very much for your help. I would say that is not a pseudo code, but a 100%-real code 😄
\nNow it's more clear how to combine multiple candidates. You rock
In the first line: for query_id, customer in enumerate(customers),
\ncustomers is a list of customers from current week or customers from the next week?
Current week. Labels are from next week.
", "rawMarkdown": "Current week. Labels are from next week.", "votes": 3 }, { "id": 1756750, "postDate": "2022-04-15T19:54:36.323Z", "content": "Thanks for your help @paweljankiewicz
", "rawMarkdown": "Thanks for your help @paweljankiewicz " }, { "id": 1756875, "postDate": "2022-04-16T02:35:23.467Z", "content": "@paweljankiewicz Thanks for your sharing! But I have a little confusion about your pseudo code. In my previous idea, I union the articles from all strategies, and add other features (like article popularity, and similarity between articles and customers) to these articles, but in your code, you just point out what strategies these articles came from. Do you think it is necessary to add other features?
", "rawMarkdown": "@paweljankiewicz Thanks for your sharing! But I have a little confusion about your pseudo code. In my previous idea, I union the articles from all strategies, and add other features (like article popularity, and similarity between articles and customers) to these articles, but in your code, you just point out what strategies these articles came from. Do you think it is necessary to add other features?", "votes": 2 }, { "id": 1757146, "postDate": "2022-04-16T10:06:20.433Z", "content": "@rock139 Of course you need to add other features as well as I explained in the other thread. This was only a question about how to combine the strategies into training data.
", "rawMarkdown": "@rock139 Of course you need to add other features as well as I explained in the other thread. This was only a question about how to combine the strategies into training data.", "votes": 2 }, { "id": 1757159, "postDate": "2022-04-16T10:29:50.023Z", "content": "Thanks for your help!
", "rawMarkdown": "Thanks for your help!" } ] }, { "id": 1759500, "postDate": "2022-04-18T17:14:40.880Z", "content": "Update:
\nI've found that after deteriorating for a few steps, the evaluation metric does start improving (although it never gets as good as what it was after the first step).
\nAt that point, it does appear to correlate with the competition metric, and across cv/LB.
\nSo I'm working with that.
I still don't know why it's acting this way originally, but I'm assuming it's some artifact of slight difference between loss function and evaluation metrics that is showing because of the small size of my training set and weak features.
", "rawMarkdown": "Update:\n\nI've found that after deteriorating for a few steps, the evaluation metric does start improving (although it never gets as good as what it was after the first step).\nAt that point, it does appear to correlate with the competition metric, and across cv/LB.\nSo I'm working with that.\n\nI still don't know why it's acting this way originally, but I'm assuming it's some artifact of slight difference between loss function and evaluation metrics that is showing because of the small size of my training set and weak features.\n", "votes": 2 }, { "id": 1777718, "postDate": "2022-05-04T18:30:10.893Z", "content": "Update:
\nFigured it out - there was silent leakage going on, when I was merging the labels in.
\nSee this post for details.
@lihaorocky well, thanks for your reply! I guess I have only one question. I didn't recall any candidates, I just use week 1 as label and data in week 5, 4, 3, 2 as candidates for week 1, which means I label data in week 1 as positive and data in week 5, 4, 3, 2 as negative after remove samples which customer_id not appear in week 1. So, do you think this is why my candidates is bad?
\nThank you again!
@lihaorocky Hi, cound you explain more detailed about how do you create the training and testing set? From my understanding, for example, there are 5, 4, 3, 2, 1, 0 weeks, and you just need take the week 1 data as label, to label data before week 1 (such as week 5, 4, 3, 2), this can be used as train data. And take the week 0 data to label data before week 0, this can be used as valid data. Is that right?
\nIf that's right, then I come with same problem, my NDCG deteriorates as training progresses…
", "rawMarkdown": "@lihaorocky Hi, cound you explain more detailed about how do you create the training and testing set? From my understanding, for example, there are 5, 4, 3, 2, 1, 0 weeks, and you just need take the week 1 data as label, to label data before week 1 (such as week 5, 4, 3, 2), this can be used as train data. And take the week 0 data to label data before week 0, this can be used as valid data. Is that right?\n\nIf that's right, then I come with same problem, my NDCG deteriorates as training progresses...", "replies": [ { "id": 1760772, "postDate": "2022-04-19T14:04:51.783Z", "content": "For your question, yes, I think your description was correct.
\nAnd as to the problem you mentioned, I didn't meet this problem. But if you did all other things right, I guess it happend maybe because of the quality of the candidates you generated, the amount of training dataset is not big enough, or the features you created are too few or not good enough. Just be patient and good luck.
I would suggest you doing the follow steps:
\n1: Say you have a column in your feature dataframe called \"target_week\" to represent the following 7-days window. Then sort the dataframe
\ndf_feat_train.sort_values(['target_week', 'customer_id'], ignore_index=True, inplace=True)
\n2: After splitting the training and validation dataset, try:
\ntrain_baskets = df_feat_train.loc[train_idx, :].groupby(['target_week', 'customer_id'])['article_id'].count().values
\nto get the train baskets. And do the same for the validation dataset.
\n3: Lower the learning rate, in my case, the training loss is early-stopped in sort of 150 training rounds with learning_rate 0.03 when using only one week for training and one week for validation.
\nHope it helps.
@lihaorocky - thank you for replying, and for sharing.
\nI'm only using one target week as of now, so I don't need to group by target_week, but I'm doing exactly that.
\nMy code is:
features_df = features_df.sort_values(\"customer_id\").reset_index(drop=True)\ngroups = list(features_df.groupby(\"customer_id\")[\"article_id\"].count())\nI can't get the metric to improve even for 2 training rounds.
\nMaybe I just need to move away from prototyping and adding more features and data…
", "votes": 0, "replies": [] }, { "id": 1749561, "author_name": "Igor Kuivjogi Fernandes", "author_url": "", "post_date": "2022-04-08T17:45:52.487000", "content": "Hi, @lihaorocky . Thanks for sharing
\nAs I understood target_week is a binary relevance (1 or 0) to denote if customer bought or not an article, right?
\nIf this is the case, there's any reason to sort values by target_week? This way seems you are putting 0's first in the whole dataset.
Not really. I use target_week column to represent the target week period. For example, on an observation date 20200915, your target week will be 20200916-20200922, I give it a label 0. And for an observation date 20200908, the target week will be 20200909-20200915, I give it a label 1 etc. This way you will get lots of target_weeks 0 1 2 3 4 5….I use the data before observation date to create candidate samples and sample features, while using the data during target week to create sample labels(bought or not). I sort the samples based on target_week and customer_id because I have a lot of weeks' samples to be input to LightGBMRanker. So it need to be sorted to create baskets.
", "votes": 10, "replies": [] }, { "id": 1750082, "author_name": "Pan Adam", "author_url": "", "post_date": "2022-04-09T10:03:30.087000", "content": "Hi， @lihaorocky . Thanks for sharing
\nI‘d like to ask ， \"target_weeks\" this column just for building pos/neg samples ?
And in the sorting model, I only use some existing features on the item side and user side. I'm a little confused about the sorting feature and don't know how to construct it, can you give me some advice?
", "votes": 0, "replies": [] }, { "id": 1750110, "author_name": "HAO", "author_url": "", "post_date": "2022-04-09T10:33:36.200000", "content": "For your first question, yes, and it also eases filtering training and validation samples based on it.
\nWell, there are a lot of ways to construct features. Besides the features of customer and article alone, you can measure the similarity between customers and articles. I will not talk too much about this but I would say it will give huge performance improvement.
Thanks for the answer @lihaorocky. Seems that LGBMRanker is limited to a relevance of 31, so your target_weeks lies in the range [0, max(31, n_weeks)]?
Limited by the Ram size, my current score is using only 5 weeks for training. I'm trying adding more weeks.
", "votes": 6, "replies": [] }, { "id": 1750196, "author_name": "Igor Kuivjogi Fernandes", "author_url": "", "post_date": "2022-04-09T12:11:13.747000", "content": "I got it. Thanks for the infos @lihaorocky . You helped me a lot.
", "votes": 0, "replies": [] }, { "id": 1750922, "author_name": "Rock", "author_url": "", "post_date": "2022-04-10T08:17:31.963000", "content": "Thanks for your sharing, it really helps me a lot! But I have a question, do you use the popularity features of articles for LGBMRanker training? Because I find the importance of this feature is much higher than other features, such as the similarity between customers and articles, and it reduces the score of my final prediction, I wonder if you meet this situation, thank you! @lihaorocky
", "votes": 2, "replies": [] }, { "id": 1750950, "author_name": "HAO", "author_url": "", "post_date": "2022-04-10T08:44:56.767000", "content": "Yes, I did use. I think the way you generated this kind of features (popularity of articles, similarity of articles and customers) are very important. And it's also depending on the strategy you created candidate pairs. In my case, the similarity features are always the most important ones.
", "votes": 4, "replies": [] }, { "id": 1751089, "author_name": "Rock", "author_url": "", "post_date": "2022-04-10T11:04:11.340000", "content": "Thanks for your reply, it's really helpful!
", "votes": 0, "replies": [] }, { "id": 1754264, "author_name": "Igor Kuivjogi Fernandes", "author_url": "", "post_date": "2022-04-13T14:02:16.813000", "content": "Hi @lihaorocky
\nI'm bit confused on how to split train/validation sets after generating the dataset.
\nTo create train baskets you do:
train_baskets = df_feat_train.loc[train_idx, :].groupby(['target_week', 'customer_id'])['article_id'].count().values\nIn this case, train_idx should be created in a group split way?
\nI mean, all rows from a customer must be in train OR in val or you're doing just a holdout (e.g. 80/20) split?
I split the train/validation sets based on time. Specifically I split them based on the value of target_week(0 as validation set and >1 as train set).
", "votes": 3, "replies": [] }, { "id": 1754302, "author_name": "Igor Kuivjogi Fernandes", "author_url": "", "post_date": "2022-04-13T14:27:21.540000", "content": "Thank you very much @lihaorocky . This really makes sense…I was overcomplicating.
", "votes": 0, "replies": [] }, { "id": 1754358, "author_name": "Pan Adam", "author_url": "", "post_date": "2022-04-13T15:32:25.403000", "content": "Hi @lihaorocky ,thanks for your reply.
\nPlease allow me to ask two more questions
During the recall phase, did you use multiple recalls?
If each user recalls nearly 50 articles, then more than 1 million users will have more than 50 million recall data, and then I am sorting these recall data. But I am a little confused now, how to solve such a large amount of data?
1: Yes, I use multiple recalls
\n2: As others posted in some discussions, if you are using LGBMRanker you can drop the users' recalls if they haven't any purchase in next 7 days or your recalls don't contains any real purchase that will really happen, which will hugely reduce the number of candidates to be input to your ranking model.
As to the inference stage, you can't use the two above mentioned tricks to reduce candidates to be predicted. And yes, there will be \"more than 50 million recall data\", but you can generate features and predict for these recalls batch by batch(for example in each batch you generate features and predict for 1 million recalls), which actually doesn't cost too much time. In my case, predicting for 160 million recalls cost overall 2 hours including generating features and model prediction.
", "votes": 6, "replies": [] }, { "id": 1754547, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-04-13T18:52:16.883000", "content": "@lihaorocky could be as well describing my process. I have 1000 batches for the submission and the prediction is the slowest part - it takes about 5 hours. You did well on the candidate generation - I think I have close to 1 billion candidates.
", "votes": 5, "replies": [] }, { "id": 1754567, "author_name": "Igor Kuivjogi Fernandes", "author_url": "", "post_date": "2022-04-13T19:02:55.870000", "content": "If you don't mind:
\nI'm using Rust for all the feature generation. Overall it takes somewhere around 2.5 hours on 32 threads. I think I could reduce it to 1 hour if I parallelized more calculations.
", "votes": 2, "replies": [] }, { "id": 1756464, "author_name": "Y-Haneji", "author_url": "", "post_date": "2022-04-15T14:38:38.147000", "content": "@panadam @lihaorocky
\nI have a question about your interesting discussion.
\nWhat does 'recalls' exactly mean?
In recomendations system, normally it consists two stages: recall and ranking.
\nRecalls here means finding some candidate articles one customers will have interations with. In this stage, you want to contains as many as articles one customers will bought(in this competition) using as few as candidates. You can develop multiple strategies to creating candidate articles which are mentioned in a lot of posts.
\nIn second stage, you want to rank those candidates based on the possibility being purchased buy customers.
@lihaorocky
\nThanks for reply!
\nThen, I have interpreted 'multiple recalls' means that you used multiple strategies to make candidates in the SAME exam. How do you compare possibilities from different strategies? I guess, do you use standarization?
Actually, I give the choice mostly to the ranking model. He (or she) is much smarter than me.😂
", "votes": 3, "replies": [] }, { "id": 1756539, "author_name": "Y-Haneji", "author_url": "", "post_date": "2022-04-15T15:37:31.610000", "content": "@lihaorocky
\nAI overtakes humans :D
\nDo you suggest to give ['possibility', 'strategy', …(other features)] to ranking model?
Yes, for example when you develop the strategy based on article's hotness. The article hottness can be measured easily using some method like purchased count in previous week. I use some thing like this. But from the performance comparison using or not using those measurement feature in ranking model, the improvement of adding those features is quite small. That's why I say, mostly I leave it to the ranking model.
", "votes": 4, "replies": [] }, { "id": 1756568, "author_name": "Igor Kuivjogi Fernandes", "author_url": "", "post_date": "2022-04-15T16:11:19.640000", "content": "Hi @lihaorocky thanks for your shares.
\nThere's a way to know if the strategy can generate good candidates?
Yes, for example on an observation date 20200915, from one of your candidate generating strategy you create 1 million pairs(customer_id, article_id, target_week), and from the next 7-days transations window 20200916~20200922, you can label those pairs with 1(purchase happend) and 0(purchase not happen). Say totally you have 50 thousands actually happend pairs. Then the \"matching ratio\" will be 50000 / 1000000 = 0.05. Clearly if keeping the number of total candidate pairs fixed, the bigger the numebr actully happend purchase pairs the better. So you can use this above mentioned ratio to measure how good you strategy is.
", "votes": 5, "replies": [] }, { "id": 1756645, "author_name": "Igor Kuivjogi Fernandes", "author_url": "", "post_date": "2022-04-15T18:04:45.777000", "content": "Hi @lihaorocky
\nThank you very much for your explanation. This totally makes sense
@lihaorocky
\nAfter all, my understanding of ranking model improved very well thanks to your clear explanation.
\nLet's do our best together!
@lihaorocky Hi, roughly what percentage of actual purchases can you generate in the recall stage, with 160 million candidates?
", "votes": 1, "replies": [] }, { "id": 1759099, "author_name": "", "author_url": "", "post_date": "2022-04-18T10:41:58.393000", "content": "", "votes": 0, "replies": [] }, { "id": 1759101, "author_name": "HAO", "author_url": "", "post_date": "2022-04-18T10:42:50.867000", "content": "Well, the 160 million candidates is in the prediction stage, which mean the actually purchases is not known until 20200929. If you mean what the percentage ratio of candidates in training stage is, different recall strategies acts very differently, which range from 0.3~0.001.
", "votes": 4, "replies": [] }, { "id": 1759122, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-04-18T11:04:24.343000", "content": "@lihaorocky I think he meant the recall of real sales from the candidates altogether. So if there was 300k transactions in the last week how many you can generate using your candidates. I must check this too.
", "votes": 2, "replies": [] }, { "id": 1759145, "author_name": "Homoalways", "author_url": "", "post_date": "2022-04-18T11:23:58.870000", "content": "@paweljankiewicz Yes that's what I mean. @lihaorocky 30% is striking, I haven't been woking on this that much but saw a post saying that he could hardly exceed 7.5%. https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/314458
", "votes": 1, "replies": [] }, { "id": 1759148, "author_name": "HAO", "author_url": "", "post_date": "2022-04-18T11:27:12.257000", "content": "I just realized you mean recall. In my case, it's about 25%.
", "votes": 2, "replies": [] } ] }, { "id": 1755632, "author_name": "Igor Kuivjogi Fernandes", "author_url": "", "post_date": "2022-04-14T19:29:27.600000", "content": "Hi. I'm thinking how much your CV boosted after ranking the set of candidates with LGBMRanker or other model.
\nCurrently I'm getting just +0.0015 boost, so I think something is wrong.
Could you guys @paweljankiewicz @lihaorocky @jacob34 share this if you don't mind?
", "votes": 1, "replies": [ { "id": 1755675, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-04-14T21:04:12.523000", "content": "There is no CV before the ranking. Ranking model is a must. I have more than 1000 candidate items for each customer. How else would I be able to use them without the ranker? Single strategies to generate candidates are pretty useless on their own. You need to have multiple candidate generation strategies like I explained in the other thread.
", "votes": 2, "replies": [] }, { "id": 1755683, "author_name": "Igor Kuivjogi Fernandes", "author_url": "", "post_date": "2022-04-14T21:23:21.233000", "content": "Thanks for the answer @paweljankiewicz.
\nI understood that we need to have many candidates generation strategies, but there's some points not clear to me yet.
How do I know if a candidate strategy is good? I suppose that good strategy would generate more 1's (item was bought by the customer in the target week) than a strategy with bad candidates. For instance, good strategy A generated 1000 candidates (85% is label=0, 15% is label=1), meanwhile strategy B generated 1000 candidates (95% is label=0, 5% is label=1)
How should we assign the label? Here's a pseudo code:
for each row in candidates:\n if this candidate item was bought by this customer in the next week:\n label = 1\n else:\n label = 0\nIs this right?
\nSorry for the (possible) obvious questions…this is the first time we're using a ranking model.
", "votes": 1, "replies": [] }, { "id": 1756598, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-04-15T16:41:47.153000", "content": "No worries. Ranking model are quite helpful in many situations.
\nThe pseudo code to generate the training data would be something like this for 2 strategies
\nall_data = []\nfor query_id, customer in enumerate(customers):\n candidates_a = generate_candidates_a(customer)\n candidates_b = generate_candidates_b(customer)\n all_candidates = union(candidates_a, candidates_b) # you generate a list of all unique candidates\n customer_df = pd.DataFrame({\n query_id: query_id,\n customer_id: [customer_id]*len(all_candidates),\n article_id: all_candidates,\n bought_next_week: all_candidates.isin(customer_sales_next_week.get(customer, [])), # this is your label\n article_in_strategy_a: all_candidates.isin(candidates_a),\n article_in_strategy_b: all_candidates.isin(candidates_b),\n })\n all_data.append(customer_df)\nA dataframe like this for each customer is a query for a ranking model. It is like a separate part of your data that is evaluated by the ranking model. Unlike some other methods like regression and classification you want to properly rank items withing customer query.
\nSo going back to your first question - the ranking model sees the articles, whether they were bought or not and from which strategies they came so it can assess how good the candidate is based on this information. So you want to have precise strategies without adding too many negative observations.
", "votes": 9, "replies": [] }, { "id": 1756641, "author_name": "Igor Kuivjogi Fernandes", "author_url": "", "post_date": "2022-04-15T17:54:01.930000", "content": "Hi @paweljankiewicz
\nThank you very much for your help. I would say that is not a pseudo code, but a 100%-real code 😄
\nNow it's more clear how to combine multiple candidates. You rock
In the first line: for query_id, customer in enumerate(customers),
\ncustomers is a list of customers from current week or customers from the next week?
Current week. Labels are from next week.
", "votes": 3, "replies": [] }, { "id": 1756750, "author_name": "Igor Kuivjogi Fernandes", "author_url": "", "post_date": "2022-04-15T19:54:36.323000", "content": "Thanks for your help @paweljankiewicz
", "votes": 0, "replies": [] }, { "id": 1756875, "author_name": "Rock", "author_url": "", "post_date": "2022-04-16T02:35:23.467000", "content": "@paweljankiewicz Thanks for your sharing! But I have a little confusion about your pseudo code. In my previous idea, I union the articles from all strategies, and add other features (like article popularity, and similarity between articles and customers) to these articles, but in your code, you just point out what strategies these articles came from. Do you think it is necessary to add other features?
", "votes": 2, "replies": [] }, { "id": 1757146, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-04-16T10:06:20.433000", "content": "@rock139 Of course you need to add other features as well as I explained in the other thread. This was only a question about how to combine the strategies into training data.
", "votes": 2, "replies": [] }, { "id": 1757159, "author_name": "Rock", "author_url": "", "post_date": "2022-04-16T10:29:50.023000", "content": "Thanks for your help!
", "votes": 0, "replies": [] } ] }, { "id": 1759500, "author_name": "Clear n' Simple", "author_url": "", "post_date": "2022-04-18T17:14:40.880000", "content": "Update:
\nI've found that after deteriorating for a few steps, the evaluation metric does start improving (although it never gets as good as what it was after the first step).
\nAt that point, it does appear to correlate with the competition metric, and across cv/LB.
\nSo I'm working with that.
I still don't know why it's acting this way originally, but I'm assuming it's some artifact of slight difference between loss function and evaluation metrics that is showing because of the small size of my training set and weak features.
", "votes": 2, "replies": [] }, { "id": 1777718, "author_name": "Clear n' Simple", "author_url": "", "post_date": "2022-05-04T18:30:10.893000", "content": "Update:
\nFigured it out - there was silent leakage going on, when I was merging the labels in.
\nSee this post for details.
@lihaorocky well, thanks for your reply! I guess I have only one question. I didn't recall any candidates, I just use week 1 as label and data in week 5, 4, 3, 2 as candidates for week 1, which means I label data in week 1 as positive and data in week 5, 4, 3, 2 as negative after remove samples which customer_id not appear in week 1. So, do you think this is why my candidates is bad?
\nThank you again!
@lihaorocky Hi, cound you explain more detailed about how do you create the training and testing set? From my understanding, for example, there are 5, 4, 3, 2, 1, 0 weeks, and you just need take the week 1 data as label, to label data before week 1 (such as week 5, 4, 3, 2), this can be used as train data. And take the week 0 data to label data before week 0, this can be used as valid data. Is that right?
\nIf that's right, then I come with same problem, my NDCG deteriorates as training progresses…
", "votes": 0, "replies": [ { "id": 1760772, "author_name": "HAO", "author_url": "", "post_date": "2022-04-19T14:04:51.783000", "content": "For your question, yes, I think your description was correct.
\nAnd as to the problem you mentioned, I didn't meet this problem. But if you did all other things right, I guess it happend maybe because of the quality of the candidates you generated, the amount of training dataset is not big enough, or the features you created are too few or not good enough. Just be patient and good luck.