Just to give you some background - my team won RecSys 2019 so I'm pretty familiar with problems like these. Now that we have this out of the way…
\nTo simplify the discussion I propose to call the set of articles that a customer bought on a given date a basket. And then this problem can be really summarized as predict next basket.
\nGeneral approach towards problems like this could be as follows. At least mine will be.
\nThese candidates are your negative and positive examples. I'm saying this because someone on the forum asked where are the negative observations. And the answer is you generate them.
\nThe strategies to generate those candidates are probably the most important aspect of this competition. The strategies can be based on:
\nComing up with different strategies is a fun challenge :). I cannot imagine that a winning solution will be based on a closed form model representing one strategy. Once you have all the different strategies ready for each customer then it is time to create a huge table with columns like these:
\nOnce you have a table like this it all becomes very familiar. There were many competitions involving click through predictions. They have almost an identical structure the only difference is the source of observations.
\nRanking is a different problem than classification but in general classification models can be used as well. My favourite ranking model is LightGBM (we used it in RecSys 2019) https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRanker.html.
\nThe customer id can be treated as a query in ranking model because you are interested in optimizing the scores at a customer level.
\nSample the data as much as possible - most of the competition you should spend validating your features on the sample. Increase the sample you are using when you are no longer seeing the correlation between local validation and the leaderboard.
Test different ideas quickly. Fast local validation is key here because you must do as much experiments as possible.
Good luck :)
\nUpdate 1. Added code for sampling here https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples
", "messages": [ { "id": 1688473, "postDate": "2022-02-13T16:40:50.790Z", "content": "Just to give you some background - my team won RecSys 2019 so I'm pretty familiar with problems like these. Now that we have this out of the way…
\nTo simplify the discussion I propose to call the set of articles that a customer bought on a given date a basket. And then this problem can be really summarized as predict next basket.
\nGeneral approach towards problems like this could be as follows. At least mine will be.
\nThese candidates are your negative and positive examples. I'm saying this because someone on the forum asked where are the negative observations. And the answer is you generate them.
\nThe strategies to generate those candidates are probably the most important aspect of this competition. The strategies can be based on:
\nComing up with different strategies is a fun challenge :). I cannot imagine that a winning solution will be based on a closed form model representing one strategy. Once you have all the different strategies ready for each customer then it is time to create a huge table with columns like these:
\nOnce you have a table like this it all becomes very familiar. There were many competitions involving click through predictions. They have almost an identical structure the only difference is the source of observations.
\nRanking is a different problem than classification but in general classification models can be used as well. My favourite ranking model is LightGBM (we used it in RecSys 2019) https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRanker.html.
\nThe customer id can be treated as a query in ranking model because you are interested in optimizing the scores at a customer level.
\nSample the data as much as possible - most of the competition you should spend validating your features on the sample. Increase the sample you are using when you are no longer seeing the correlation between local validation and the leaderboard.
Test different ideas quickly. Fast local validation is key here because you must do as much experiments as possible.
Good luck :)
\nUpdate 1. Added code for sampling here https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples
", "rawMarkdown": "Just to give you some background - my team won RecSys 2019 so I'm pretty familiar with problems like these. Now that we have this out of the way...\n\nTo simplify the discussion I propose to call the set of articles that a customer bought on a given date a **basket**. And then this problem can be really summarized as predict next basket.\n\nGeneral approach towards problems like this could be as follows. At least mine will be.\n\n## 1. For each customer generate candidates that they can buy. \n\nThese candidates are your negative and positive examples. I'm saying this because someone on the forum asked where are the negative observations. And the answer is you generate them. \n\nThe strategies to generate those candidates are probably the most important aspect of this competition. The strategies can be based on:\n- item to item similarities between customer previous baskets - so image similarity is not out of the question as somebody suggested\n- user based collaborative filtering\n- last baskets (with the hope that the user will buy them again)\n- model based predictions with customers without the transaction history\n- etc\n\nComing up with different strategies is a fun challenge :). I cannot imagine that a winning solution will be based on a closed form model representing one strategy. Once you have all the different strategies ready for each customer then it is time to create a huge table with columns like these:\n\n- observation_date - the date for which you are making predictions - submission should be based on all the data before 2020-09-22 (my birthday btw).\n- customer_id\n- article_id\n- bought in the next week (this is your label)\n- customer static attributes\n- customer dynamic attributes\n- article static attributes\n- article dynamic attributes (maybe item popularity and such)\n- whether the article exist in the strategies and which ones\n- strategy score (whatever it is)\n- many other features that relate the item to the user\n\nOnce you have a table like this it all becomes very familiar. There were many competitions involving click through predictions. They have almost an identical structure the only difference is the source of observations.\n\n## 2. Build a ranking model that ranks the items within Customer \n\nRanking is a different problem than classification but in general classification models can be used as well. My favourite ranking model is LightGBM (we used it in RecSys 2019) https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRanker.html.\n\nThe customer id can be treated as a query in ranking model because you are interested in optimizing the scores at a customer level.\n\n## General tips\n\n1. Sample the data as much as possible - most of the competition you should spend validating your features on the sample. Increase the sample you are using when you are no longer seeing the correlation between local validation and the leaderboard.\n\n2. Test different ideas quickly. Fast local validation is key here because you must do as much experiments as possible.\n\nGood luck :)\n\nUpdate 1. Added code for sampling here https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples", "votes": 430 }, { "id": 1688506, "postDate": "2022-02-13T17:03:05.520Z", "content": "For anyone that might not be familiar RecSys is one of the most respected competitions in the Recommender space.
\n@paweljankiewicz I was trying to find your winning solution but I could only find the press release content. Is it available anywhere?
\nTIA! :)
", "rawMarkdown": "For anyone that might not be familiar [RecSys](https://recsys.acm.org) is one of the most respected competitions in the Recommender space. \n\n@paweljankiewicz I was trying to find your winning solution but I could only find the press release content. Is it available anywhere? \n\nTIA! :)", "votes": 9, "replies": [ { "id": 1688518, "postDate": "2022-02-13T17:12:18.343Z", "content": "Yeah here it is - https://github.com/logicai-io/recsys2019.
", "rawMarkdown": "Yeah here it is - https://github.com/logicai-io/recsys2019.", "votes": 16 }, { "id": 1695896, "postDate": "2022-02-18T12:47:09.383Z", "content": "Hi, @paweljankiewicz , i try to download the data in Recsys 2019
\n--2022-02-18 20:38:52-- https://storage.googleapis.com/logicai-recsys2019/trivagoRecSysChallengeData2019_v2.zip\n正在解析主机 storage.googleapis.com (storage.googleapis.com)... 142.250.204.112, 142.250.204.48, 142.250.66.144, ...\n正在连接 storage.googleapis.com (storage.googleapis.com)|142.250.204.112|:443... 已连接。\n已发出 HTTP 请求，正在等待回应... 404 Not Found\n2022-02-18 20:38:54 错误 404：Not Found。I don't have the original dataset. Also the code is written in such a way that it needs close to 500gb of ram to train the models.
", "rawMarkdown": "I don't have the original dataset. Also the code is written in such a way that it needs close to 500gb of ram to train the models.", "votes": 5 }, { "id": 1696214, "postDate": "2022-02-18T16:57:10.007Z", "content": "For anyone else surprised by:
\n\n\nAlso the code is written in such a way that it needs close to 500gb of ram to train the models.
\n
When I had the opportunity to do an interview for another RecSys winning team, I learned this is quite a common number for such problems. 😅
\nThe embedding sizes based on number of users and recommendations just eats up A LOT of memory by the nature of their size
", "rawMarkdown": "For anyone else surprised by:\n\n> Also the code is written in such a way that it needs close to 500gb of ram to train the models.\n\nWhen I had the opportunity to do an interview for another [RecSys winning team](https://www.youtube.com/watch?v=W3aWEXqIkWk), I learned this is quite a common number for such problems. 😅\n\nThe embedding sizes based on number of users and recommendations just eats up A LOT of memory by the nature of their size", "votes": 7 }, { "id": 1696389, "postDate": "2022-02-18T19:52:49.823Z", "content": "BTW I found the data here on Kaggle (but don't seem to have the link to it now), it didn't seem to be available anywhere else.
\nBut I got the most mileage out of reading the code in the repo, didn't get around to doing much with the generated data.
", "rawMarkdown": "BTW I found the data here on Kaggle (but don't seem to have the link to it now), it didn't seem to be available anywhere else.\n\nBut I got the most mileage out of reading the code in the repo, didn't get around to doing much with the generated data.", "votes": 1 }, { "id": 1696410, "postDate": "2022-02-18T20:07:22.843Z", "content": "@radek1 I believe this is the link to the dataset?
", "rawMarkdown": "@radek1 I believe [this](https://www.kaggle.com/pranavmahajan725/trivagorecsyschallengedata2019) is the link to the dataset?", "votes": 2 }, { "id": 1696492, "postDate": "2022-02-18T22:30:40.160Z", "content": "yes, you found it! :) I think it was that one
\nwith the data I downloaded the preprocessing steps ran without issue
", "rawMarkdown": "yes, you found it! :) I think it was that one\n\nwith the data I downloaded the preprocessing steps ran without issue" }, { "id": 1696542, "postDate": "2022-02-18T23:16:10.543Z", "content": "@radek1 @init27 the most important concept in RecSys 2019 was the notion of accumulators classes which have 2 methods:
\nWhen you process events by date you first extract the information from the accumulator and then update it with new information. This way there is no possibility to leak any info. The accumulators themselves can track information about the item, customer, item+customer, other groups of information. Really anything you need.
\nThe code could be summarized as follows:
\nfor transaction in transactions:\n for accumulator in accumulators:\n get features from accumulator - really important that it is before update\n update the accumulator\nThere is an opportunity to parallelize the extraction by reversing the loop so you can process and extract features from accumulators in parallel.
", "rawMarkdown": "@radek1 @init27 the most important concept in RecSys 2019 was the notion of accumulators classes which have 2 methods:\n\n- update - updates the state of accumulator\n- get - extracts the state of accumulator\n\nWhen you process events by date you first extract the information from the accumulator and then update it with new information. This way there is no possibility to leak any info. The accumulators themselves can track information about the item, customer, item+customer, other groups of information. Really anything you need.\n\nThe code could be summarized as follows:\n\n```python\nfor transaction in transactions:\n for accumulator in accumulators:\n get features from accumulator - really important that it is before update\n update the accumulator\n```\n\nThere is an opportunity to parallelize the extraction by reversing the loop so you can process and extract features from accumulators in parallel.", "votes": 4 }, { "id": 1696572, "postDate": "2022-02-19T00:18:28.483Z", "content": "Ah, I see! @paweljankiewicz thank you very much for sharing this, this is wonderful! 😊
\nIf I am reading this right, the accumulators should be anything -- items of the given type purchased by a customer up to this point, count of transactions, etc.
\nI have completely not thought of processing data like this, this is very cool, eager to give it a try!
\nSo far, I have been attempting to generate features (and candidate transactions) using dataframes, groupby, shift, and merging. But that takes a ridiculous amount of RAM and the code is very hard to reason about and prone to having bugs!
\nWould love to go back to straight-up Python for more of that :)
\nI am not sure I understand the parallelization bit, where we reverse the loop? There is a first pass through the dataset that we cannot parallelize, to compute the accumulators. And we can only parallelize adding features to records in the second pass? (split the df into n_chunks, apply features from accumulator to each chunk, combine?) I am thinking I might be missing something here :) I am also thinking we probably want the accumulators to be simple so that they can only be applied as they go through the list of data in chronological order -- I guess I am really lost on the parallelization bit :)
\nOnce we get all these features, do we do feature selection at all? Or do we just see how correlated columns are and drop the ones that don't seem to add a lot of signal?
\nI saw in the repo you shared that it seems models were run on different train sets? (maybe I misread this). Would you grab some numbers of columns and train different models on different subsets? I am wondering if I didn't misread this as I think there is a way to make lgbm randomly sample a subset of columns to grow each tree, but maybe such an approach is nice for blending?
\n@paweljankiewicz your posts are legendary and this thread is an absolute treasure trove of information for people just getting started with recsys, thank you so much for all your help!!! 🙏
", "rawMarkdown": "Ah, I see! @paweljankiewicz thank you very much for sharing this, this is wonderful! 😊\n\nIf I am reading this right, the accumulators should be anything -- items of the given type purchased by a customer up to this point, count of transactions, etc.\n\nI have completely not thought of processing data like this, this is very cool, eager to give it a try!\n\nSo far, I have been attempting to generate features (and candidate transactions) using dataframes, groupby, shift, and merging. But that takes a ridiculous amount of RAM and the code is very hard to reason about and prone to having bugs!\n\nWould love to go back to straight-up Python for more of that :) \n\nI am not sure I understand the parallelization bit, where we reverse the loop? There is a first pass through the dataset that we cannot parallelize, to compute the accumulators. And we can only parallelize adding features to records in the second pass? (split the df into n_chunks, apply features from accumulator to each chunk, combine?) I am thinking I might be missing something here :) I am also thinking we probably want the accumulators to be simple so that they can only be applied as they go through the list of data in chronological order -- I guess I am really lost on the parallelization bit :) \n\nOnce we get all these features, do we do feature selection at all? Or do we just see how correlated columns are and drop the ones that don't seem to add a lot of signal?\n\nI saw in the repo you shared that it seems models were run on different train sets? (maybe I misread this). Would you grab some numbers of columns and train different models on different subsets? I am wondering if I didn't misread this as I think there is a way to make lgbm randomly sample a subset of columns to grow each tree, but maybe such an approach is nice for blending?\n\n@paweljankiewicz your posts are legendary and this thread is an absolute treasure trove of information for people just getting started with recsys, thank you so much for all your help!!! 🙏", "votes": 1 }, { "id": 1696593, "postDate": "2022-02-19T00:33:44.940Z", "content": "I think I might have understood it now! :)
\nWe don't reverse the loop in the sense that we run from end to finish, we just reverse it by moving the inside of the loop outside, like this:
\nfor accumulator in accumulators:\n for transaction in transactions:\n run acc and apply features\nWe can walk the column/dataframe in parallel by multiple accumulators and then combine the results!
\nI would love to try this with something like dask and delayed, I suspect this might be a nice way to work around the limitations of dask's API, really curious how well this could work :)
Great work! Respect to all of you!
", "rawMarkdown": "Great work! Respect to all of you!" }, { "id": 1717998, "postDate": "2022-03-10T12:21:07.620Z", "content": "So is it true that if I don't have a machine with lots of memory, I cannot become one of the winners😂
", "rawMarkdown": "So is it true that if I don't have a machine with lots of memory, I cannot become one of the winners😂" } ] }, { "id": 1776281, "postDate": "2022-05-03T21:35:19.013Z", "content": "Hi, thank you so much for this post, it is very useful. I had one question about the generation of the positive examples you use to train the ranking model. Do you use the same method for finding these as with the final test set which means there are missing positive examples or are you including all positive examples as they are known in the training data?
", "rawMarkdown": "Hi, thank you so much for this post, it is very useful. I had one question about the generation of the positive examples you use to train the ranking model. Do you use the same method for finding these as with the final test set which means there are missing positive examples or are you including all positive examples as they are known in the training data?", "votes": 3, "replies": [ { "id": 1776316, "postDate": "2022-05-03T22:24:40.767Z", "content": "That's a good question. I tried adding all positives but the score didn't improve. I'm using only positives which are associated with some strategy.
", "rawMarkdown": "That's a good question. I tried adding all positives but the score didn't improve. I'm using only positives which are associated with some strategy.", "votes": 5 } ] }, { "id": 1778497, "postDate": "2022-05-05T11:20:42.293Z", "content": "Thank you for your posting. This post is very helpful.
\nI would like to ask you positive-negative sample rate during training process.
\nYou said that about 1000 candidates are collected for each customers in your strategy.
\nBut, when training model, this is unbalanced-data. I think down-sampling is necessary.
\nIf you don't mind, please tell me your positive-negative rate.
You are right. At the time of writing I thought that increasing the number of candidates can be good. Then I got to a point where adding more candidates doesn't help. I have way too many candidates right now. During training I take only 5-10% of negative examples. @lihaorocky commented that he is using only 200 candidates which makes sense. I think most of the true positives that we are able to predict are just popular items. With 5% downsampling I have about 4% positives.
", "rawMarkdown": "You are right. At the time of writing I thought that increasing the number of candidates can be good. Then I got to a point where adding more candidates doesn't help. I have way too many candidates right now. During training I take only 5-10% of negative examples. @lihaorocky commented that he is using only 200 candidates which makes sense. I think most of the true positives that we are able to predict are just popular items. With 5% downsampling I have about 4% positives.", "votes": 3 }, { "id": 1778653, "postDate": "2022-05-05T14:58:26.153Z", "content": "Hi @paweljankiewicz ;) I'm optimizing number of candidates with Bayesian Optimization. For every candidate generation method I have a k parameter determining how many articles can be returned. Then I optimize Precision@50 for a baseline model (no reranking model) where the k's are hyperparameters. After a while you can clearly see which generators should be switched off or limited.
Interesting approach. I think that this competition shows very well that most cf recommendation algorithms are almost useless in this setting. It is more like a symbolic recommendation system where you don't rely on particular items but rather pools of items created from recent transaction history. So recommendation can be based on:
\n0.5top_50_articles_department + 0.5most_similar_articles_to_the_last_basket_that_are_within_1000_most_popular_articles
\nThere are some obvious flaws in the competition like guessing the availability of items but it is a valuable experience nonetheless.
", "rawMarkdown": "Interesting approach. I think that this competition shows very well that most cf recommendation algorithms are almost useless in this setting. It is more like a symbolic recommendation system where you don't rely on particular items but rather pools of items created from recent transaction history. So recommendation can be based on:\n\n0.5*top_50_articles_department + 0.5*most_similar_articles_to_the_last_basket_that_are_within_1000_most_popular_articles\n\nThere are some obvious flaws in the competition like guessing the availability of items but it is a valuable experience nonetheless.\n\n", "votes": 3 }, { "id": 1779443, "postDate": "2022-05-06T12:57:13.977Z", "content": "@paweljankiewicz - thanks for sharing that!
\nWhat does top_50_articles_department mean?
It is just an example of some strategy. In this case it would be something like: take popular items from some specific item category that the user has in the historical transactions.
", "rawMarkdown": "It is just an example of some strategy. In this case it would be something like: take popular items from some specific item category that the user has in the historical transactions. ", "votes": 1 } ] }, { "id": 1767890, "postDate": "2022-04-25T18:37:44.290Z", "content": "Hi @paweljankiewicz, very very helpful post. Thanks
\nJust one question, what do you mean with “model based predictions with customers without the transaction history” ?
", "rawMarkdown": "Hi @paweljankiewicz, very very helpful post. Thanks \n\nJust one question, what do you mean with “model based predictions with customers without the transaction history” ?\n", "votes": 1 }, { "id": 1761945, "postDate": "2022-04-20T10:22:19.840Z", "content": "Hi Pawl,
\nthanks for your informative post. I would like to ask you what do you mean by dynamic/static attributes for customer and articles? if illustrated with an example will be more helpful. Thanks!
", "rawMarkdown": "Hi Pawl, \n\nthanks for your informative post. I would like to ask you what do you mean by dynamic/static attributes for customer and articles? if illustrated with an example will be more helpful. Thanks!", "votes": 1 }, { "id": 1742077, "postDate": "2022-04-01T12:44:09.713Z", "content": "@paweljankiewicz
\nThanks for great idea!
\nI have a question.
\nWhat does observation_date exactly mean? Please give me some example.
Observation date is the date you treat as \"now\" so you can use all the data before this date. So for validation you need to take a date like 2020-09-15. For submission the observation date is of course the last available date 2020-09-22.
", "rawMarkdown": "Observation date is the date you treat as \"now\" so you can use all the data before this date. So for validation you need to take a date like 2020-09-15. For submission the observation date is of course the last available date 2020-09-22.", "votes": 2 }, { "id": 1750050, "postDate": "2022-04-09T09:14:06.753Z", "content": "@paweljankiewicz
\nThanks! I have completely understood.
Thank you for sharing your idea. This is really a great approach. I am getting \"run out of memory\" problem. I am using my local PC which has 16GB RAM. Can you suggest how to deal with this situation. Thank You!
", "rawMarkdown": "Thank you for sharing your idea. This is really a great approach. I am getting \"run out of memory\" problem. I am using my local PC which has 16GB RAM. Can you suggest how to deal with this situation. Thank You! ", "votes": 1, "replies": [ { "id": 1700865, "postDate": "2022-02-22T10:18:45.013Z", "content": "16GB RAM is probably not enough to comfortably work with this dataset - at least the approach I suggested needs a lot more memory. You can experiment on a smaller dataset https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples. Even using 5% sample should give you similar performance to 100% on this data.
", "rawMarkdown": "16GB RAM is probably not enough to comfortably work with this dataset - at least the approach I suggested needs a lot more memory. You can experiment on a smaller dataset https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples. Even using 5% sample should give you similar performance to 100% on this data. ", "votes": 5 }, { "id": 1701108, "postDate": "2022-02-22T14:24:37.200Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" }, { "id": 1704464, "postDate": "2022-02-25T14:47:55.280Z", "content": "Hi @paweljankiewicz :)
\nThank you for sharing
\nYou might find this code useful to make a stratified Sample using many columns.
\nhttps://github.com/flaboss/python_stratified_sampling/blob/master/stratifiedSample.py
\nAll the best
", "rawMarkdown": "Hi @paweljankiewicz :)\n\nThank you for sharing\n\nYou might find this code useful to make a stratified Sample using many columns.\n\nhttps://github.com/flaboss/python_stratified_sampling/blob/master/stratifiedSample.py\n\nAll the best ", "votes": 1 } ] }, { "id": 1698165, "postDate": "2022-02-20T07:42:19.007Z", "content": "I am new to the recommender system, do you have any references for me? Thanks.
", "rawMarkdown": "I am new to the recommender system, do you have any references for me? Thanks.", "votes": 1, "replies": [ { "id": 1699049, "postDate": "2022-02-20T21:36:41.370Z", "content": "I suggest to start with the most famous datasets for recommendations - something like movielens - https://grouplens.org/datasets/movielens/. Follow some tutorial using these datasets. This competition is on the harder spectrum because it is not a very typical dataset. I remember watching a part of this course https://www.coursera.org/specializations/recommender-systems and it was decent.
", "rawMarkdown": "I suggest to start with the most famous datasets for recommendations - something like movielens - https://grouplens.org/datasets/movielens/. Follow some tutorial using these datasets. This competition is on the harder spectrum because it is not a very typical dataset. I remember watching a part of this course https://www.coursera.org/specializations/recommender-systems and it was decent.", "votes": 12 } ] }, { "id": 1696455, "postDate": "2022-02-18T21:03:01.053Z", "content": "how could you create negative samples?
", "rawMarkdown": "how could you create negative samples? ", "votes": 1, "replies": [ { "id": 1696519, "postDate": "2022-02-18T22:52:07.610Z", "content": "As I have written there are many ways to create negative samples. For example items in the previous basket can be a source of negative samples. Popular items can be too. There are countless of examples.
", "rawMarkdown": "As I have written there are many ways to create negative samples. For example items in the previous basket can be a source of negative samples. Popular items can be too. There are countless of examples.", "votes": 2 }, { "id": 1715321, "postDate": "2022-03-07T21:50:41.483Z", "content": "Thanks for the inspiring post @paweljankiewicz It is very helpful especially on the negative sampling.
\nI have another question about negative sampling. Shall we sample from items which are not similar to positive samples as negative samples? I am asking because it may confuse the model if we sample popular or previously purchased items as negative samples. They could be potentially positive samples in the test set. On the other hand, they could represent an interest shift of the customers. Do you have suggestions from your experience? Thank you ahead.
He said 'use items in the previous basket', it might mean 'user other customer's basket as the negative samples for the current customer', make sense? I think it's ok.
\nBut what if these two customers bought the same items, does it confuse the model?
", "rawMarkdown": "He said 'use items in the previous basket', it might mean 'user other customer's basket as the negative samples for the current customer', make sense? I think it's ok.\n\nBut what if these two customers bought the same items, does it confuse the model? " } ] }, { "id": 1707473, "postDate": "2022-02-28T13:49:40.003Z", "content": "Pawel, thanks so much for putting this together, this is awesome! 🙏
\nI have a question for you about search strategies. Is there any reason why, instead of separately implementing a search strategy and inserting the result as a column in the training set, it couldn't also work to just directly add the variables that the strategy depends on into the training set?
\nSo for example if strategy S is dependent on 10 variables x1, x2… x10, instead of adding S as a column to the training set, adding x1 - x10. Although I suppose this would only be reasonable for simpler strategies and wouldn't work for strategies like image detection.
", "rawMarkdown": "Pawel, thanks so much for putting this together, this is awesome! 🙏\n\nI have a question for you about search strategies. Is there any reason why, instead of separately implementing a search strategy and inserting the result as a column in the training set, it couldn't also work to just directly add the variables that the strategy depends on into the training set? \n\nSo for example if strategy S is dependent on 10 variables x1, x2... x10, instead of adding S as a column to the training set, adding x1 - x10. Although I suppose this would only be reasonable for simpler strategies and wouldn't work for strategies like image detection.", "votes": 2, "replies": [ { "id": 1707500, "postDate": "2022-02-28T14:14:09.507Z", "content": "This is a good question. I think there are several problems
\n1) For each customer you must consider 100k items if you really want to include all variables with which the strategy was created.
\n2) What would you do in a case that you want to relate a set of items from the history with those candidates. The strategy can really on more complex feature set than x1,…,x10.
A search strategy is a way to reduce the number of candidates that you consider for each customer.
", "rawMarkdown": "This is a good question. I think there are several problems\n\n1) For each customer you must consider 100k items if you really want to include all variables with which the strategy was created.\n2) What would you do in a case that you want to relate a set of items from the history with those candidates. The strategy can really on more complex feature set than x1,...,x10.\n\nA search strategy is a way to reduce the number of candidates that you consider for each customer.", "votes": 1 }, { "id": 1707630, "postDate": "2022-02-28T16:23:25.903Z", "content": "That makes sense to me. Thanks!
", "rawMarkdown": "That makes sense to me. Thanks!" } ] }, { "id": 1698993, "postDate": "2022-02-20T20:11:23.767Z", "content": "As I keep working on this and learning about it, I keep coming to this thread and rereading it. This is such a wealth of information!
\nThank you again @paweljankiewicz!
\nWhat blew my mind today was using ‘customer_id’ as a query! I thought we might want to query on [‘customer_id’, ‘observation_date’] to essentially query on baskets?
\nAlso, the list of transactions can be a good staying set of positive examples? Meaning, from the post it might seem like we need to do something extra (‘the label indicates whether the article will be purchased next week’) but this is essentially what the list of transactions does? It is an observation date and if we add an indicator column such as ‘purchased’ we will have everything that we need? I am just worried I might be missing something subtle here.
", "rawMarkdown": "As I keep working on this and learning about it, I keep coming to this thread and rereading it. This is such a wealth of information!\n\nThank you again @paweljankiewicz!\n\nWhat blew my mind today was using ‘customer_id’ as a query! I thought we might want to query on [‘customer_id’, ‘observation_date’] to essentially query on baskets?\n\nAlso, the list of transactions can be a good staying set of positive examples? Meaning, from the post it might seem like we need to do something extra (‘the label indicates whether the article will be purchased next week’) but this is essentially what the list of transactions does? It is an observation date and if we add an indicator column such as ‘purchased’ we will have everything that we need? I am just worried I might be missing something subtle here.", "votes": 2, "replies": [ { "id": 1699032, "postDate": "2022-02-20T21:09:57.727Z", "content": "No you are right for more than 1 observation date the query should be customer_id + observation_dt. Because I have everything sorted by date I didn't notice that I'm implicitly including observation_dt in the query.
", "rawMarkdown": "No you are right for more than 1 observation date the query should be customer_id + observation_dt. Because I have everything sorted by date I didn't notice that I'm implicitly including observation_dt in the query.", "votes": 2 }, { "id": 1699040, "postDate": "2022-02-20T21:18:58.783Z", "content": "@radek1 I think you are missing something else. When you generate candidates, it is not 100% sure that you will find any transactions. Let's say your only strategy is to include the last basket items. If you have a customer which didn't buy anything twice then you can discard whole customer + observation_dt. The way the ranking models work they need positive examples in the query. This is a source of an important optimization in my opinion.
\nThe process should be like this: you generate a candidates set something like Dict[Strategy, List[ArticleId]]. A unique set of ArticleId is your number of observations that you have to consider for the customer.
As I have said for training purposes you can discard all the queries without positive examples. But for validation and submission you must include all of your candidates.
", "rawMarkdown": "@radek1 I think you are missing something else. When you generate candidates, it is not 100% sure that you will find any transactions. Let's say your only strategy is to include the last basket items. If you have a customer which didn't buy anything twice then you can discard whole customer + observation_dt. The way the ranking models work they need positive examples in the query. This is a source of an important optimization in my opinion.\n\nThe process should be like this: you generate a candidates set something like `Dict[Strategy, List[ArticleId]]`. A unique set of ArticleId is your number of observations that you have to consider for the customer. \n\nAs I have said for training purposes you can discard all the queries without positive examples. But for validation and submission you must include all of your candidates.", "votes": 3 }, { "id": 1699125, "postDate": "2022-02-21T00:08:45.827Z", "content": "Thank you so much for these additional details, @paweljankiewicz! 😊 All this is of great help
", "rawMarkdown": "Thank you so much for these additional details, @paweljankiewicz! 😊 All this is of great help" } ] }, { "id": 1758919, "postDate": "2022-04-18T06:50:27.390Z", "content": "nice work!!
", "rawMarkdown": "nice work!!" }, { "id": 1757936, "postDate": "2022-04-17T06:43:41.253Z", "content": "@paweljankiewicz
\nInspired by you, I have been working on rank model these days.
\nI have a question about mixing several strategy to make candidates. How can we put \"strategy score\" of candidate from different strategy together? Or make a column for each strategy?
\nTake an example of mixing \"recent transaction strategy\"(strategy1) and \"LSTM strategy\"(strategy2). I can simply make candidates of [customer, article, strategy1: bool, strategy2: bool] columns.
\nBUT, I want to use features from strategies, like \"transaction count\" from strategy1 or \"LSTM's probability\" from strategy2. I think this is \"strategy score\".
I think it is worth having boolean flags for each strategy and of course you can use additional features from the strategy like counts and probabilities. The only thing you have to watch for is data leakage, so make sure you use only the data before the observation date to train your models.
", "rawMarkdown": "I think it is worth having boolean flags for each strategy and of course you can use additional features from the strategy like counts and probabilities. The only thing you have to watch for is data leakage, so make sure you use only the data before the observation date to train your models.", "votes": 2 }, { "id": 1758408, "postDate": "2022-04-17T16:43:06.213Z", "content": "@paweljankiewicz
\nThanks for reply!
\nI understand that to avoid leakage is important.
\nI have adopted naive strategies, but will develop some elaborate strategies from now on and mix them.
@paweljankiewicz - I'm using LGBMRanker, and printing out the ndcg at each step, and I'm finding that as the model trains, it's getting worse, even on the training set!
\nSame when using map.
That's really strange. You probably have some sort of a bug. Are you sure you are providing a proper group parameter? I can vouch for LGBMRanker.
", "rawMarkdown": "That's really strange. You probably have some sort of a bug. Are you sure you are providing a proper group parameter? I can vouch for LGBMRanker. ", "votes": 1 }, { "id": 1747673, "postDate": "2022-04-06T21:36:46.240Z", "content": "We had the same problem and so far I couldn't find a solution for it. Not sure but I think it might be related to the way we are building our dataset which Idk if it's right. We are considering 1 week only as the label and all data before this week as train. Maybe there is a better way to carry time information than that.
", "rawMarkdown": "We had the same problem and so far I couldn't find a solution for it. Not sure but I think it might be related to the way we are building our dataset which Idk if it's right. We are considering 1 week only as the label and all data before this week as train. Maybe there is a better way to carry time information than that.\n\n\n" }, { "id": 1748364, "postDate": "2022-04-07T14:11:57.113Z", "content": "@paweljankiewicz - thanks for letting me know that it shouldn't be happening, I'll keep digging until I find out what's wrong.
\nI went back and checked, the group parameter looks good.
\nMaybe it's because I haven't added customer features yet, only article and customer-article ones?
", "rawMarkdown": "@paweljankiewicz - thanks for letting me know that it shouldn't be happening, I'll keep digging until I find out what's wrong.\n\nI went back and checked, the group parameter looks good.\n\nMaybe it's because I haven't added customer features yet, only article and customer-article ones?" }, { "id": 1748384, "postDate": "2022-04-07T14:37:02.650Z", "content": "@igormunizims, I'm also only considering one week as label for starters.
\nI don't see why that should be a problem, though.
\nWhen you have very little data, you overfit quickly, but it should still improve on the training set.
@jacob34 there are some optimizations you can do when you are training the ranking model. For example you can consider only the queries with at least 1 positive example. For the ranking model the queries with all negative observations are useless. If you didn't remove them maybe you can check if this is the reason why the ranking model is behaving in a strange way. It may be that after removing all such observations the number of observations is really low and it does indeed is overfiiting a lot.
\nUnfortunately when you are predicting for submission you must use all the observations which is painful (in my case I need to predict 100gb of compressed dataframes and it is the slowest part of the process).
", "rawMarkdown": "@jacob34 there are some optimizations you can do when you are training the ranking model. For example you can consider only the queries with at least 1 positive example. For the ranking model the queries with all negative observations are useless. If you didn't remove them maybe you can check if this is the reason why the ranking model is behaving in a strange way. It may be that after removing all such observations the number of observations is really low and it does indeed is overfiiting a lot.\n\nUnfortunately when you are predicting for submission you must use all the observations which is painful (in my case I need to predict 100gb of compressed dataframes and it is the slowest part of the process).", "votes": 5 }, { "id": 1748487, "postDate": "2022-04-07T16:29:31.913Z", "content": "Yes, I did follow your advice on that.
\nFor a single fold, I have 390k rows, for 12k customers, with 15k positives.
\nI realize it may be overfitting, but evaluation metric should still go down on the training set, no?
Wow - 100gb is massive.
", "rawMarkdown": "Yes, I did follow your advice on that.\n\nFor a single fold, I have 390k rows, for 12k customers, with 15k positives.\nI realize it may be overfitting, but evaluation metric should still go down on the training set, no?\n\nWow - 100gb is massive.", "votes": 1 }, { "id": 1748580, "postDate": "2022-04-07T18:15:37.703Z", "content": "Yes. It should go down. Are the results when you are checking the metric manually the same?
", "rawMarkdown": "Yes. It should go down. Are the results when you are checking the metric manually the same?", "votes": 1 }, { "id": 1759512, "postDate": "2022-04-18T17:30:53.483Z", "content": "Yes, checking manually gives me the same.
\nI find that after a few steps, it starts improving again (although never getting as good at it originally was), and at that point, the different evaluation metrics correlate, and cv correlates with LB, so I'm working with that.
\nStill not sure why it's happening, but I suspect it's because of weak features and small training set size.
", "rawMarkdown": "Yes, checking manually gives me the same.\n\nI find that after a few steps, it starts improving again (although never getting as good at it originally was), and at that point, the different evaluation metrics correlate, and cv correlates with LB, so I'm working with that.\n\nStill not sure why it's happening, but I suspect it's because of weak features and small training set size." }, { "id": 1777731, "postDate": "2022-05-04T18:35:47.290Z", "content": "I finally figured it out - it was a nasty, silent leakage of the label, in the form of positive exampled getting put earlier than negative ones.
\nThe details are here.
", "rawMarkdown": "I finally figured it out - it was a nasty, silent leakage of the label, in the form of positive exampled getting put earlier than negative ones.\n\nThe details are [here](https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/323081)." } ] }, { "id": 1720857, "postDate": "2022-03-13T07:52:42.817Z", "content": "Thank you very much! But I have a question. Sampling 5% data could keep the original data distribution?
", "rawMarkdown": "Thank you very much! But I have a question. Sampling 5% data could keep the original data distribution?" }, { "id": 1718932, "postDate": "2022-03-11T09:58:46.230Z", "content": "Great tips, Thanks a lot ! How do you think of transformer based model in this competition?
\njust like riiid 2021。
\nDoes the Magnitudes of users 、items 、orders and dates satisfy ？
@paweljankiewicz , thanks for the great post and learned alot…
\nwanted to know how do we emphasize on ranking/relevance of articles for the customer
\nbecause there are lot customers who are returning or rebuying the same stuff. So,, how do we differentiate.
I think ranking of relevance of articles to customers may take care of that.
", "rawMarkdown": "I think ranking of relevance of articles to customers may take care of that." } ] }, { "id": 1702159, "postDate": "2022-02-23T11:29:44.417Z", "content": "My apologies in advance but is it fair to tie a recommendation to purchase? Is it a given fact that if the Customer made a purchase, the attribution should go to the recommendation. Is it more of a prediction than a recommendation?
", "rawMarkdown": "My apologies in advance but is it fair to tie a recommendation to purchase? Is it a given fact that if the Customer made a purchase, the attribution should go to the recommendation. Is it more of a prediction than a recommendation?" }, { "id": 1699304, "postDate": "2022-02-21T05:13:50.970Z", "content": "this was very helpful
", "rawMarkdown": "this was very helpful\n" }, { "id": 1691296, "postDate": "2022-02-15T10:38:22.437Z", "content": "Thank you for these great tips, this is my first competition in Kaggle (I have experience in smaller scale competitions in another platform). I wanted to know what do you mean by sampling data as much as possible ? Do you mean we should resample data every time we experiment a new thing ? Are there any notebooks or tutorials that talk about this in more detail ?
\nThank you again :)
", "rawMarkdown": "Thank you for these great tips, this is my first competition in Kaggle (I have experience in smaller scale competitions in another platform). I wanted to know what do you mean by sampling data as much as possible ? Do you mean we should resample data every time we experiment a new thing ? Are there any notebooks or tutorials that talk about this in more detail ? \n\nThank you again :)", "replies": [ { "id": 1691508, "postDate": "2022-02-15T12:52:03.397Z", "content": "One sample is enough for quick tests. Added the code to sample the data https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples
", "rawMarkdown": "One sample is enough for quick tests. Added the code to sample the data https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples", "votes": 1 }, { "id": 1692185, "postDate": "2022-02-15T22:10:10.660Z", "content": "Thank you so much, but the notebook is empty, I think it did not run :/
", "rawMarkdown": "Thank you so much, but the notebook is empty, I think it did not run :/" }, { "id": 1693237, "postDate": "2022-02-16T14:38:29.617Z", "content": "It should be good now.
", "rawMarkdown": "It should be good now." } ] }, { "id": 1728123, "postDate": "2022-03-18T15:51:18.240Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 1728274, "postDate": "2022-03-18T18:21:30.870Z", "content": "Ad 1.
\nIn my setting I propose to convert the problem to a table where you have a list of item candidates and you mark the sales as either 0 or 1 depending whether the item was sold in the next week.
\nImagine you want to create negative samples as a list of 1000 most popular items in the last week. Some of those items will be bought by the customers.
\nSo the table you need to create is:
\nI'm not saying this is the best approach here but it is more or less what I'm doing. This is a technique where you create a set of explicit negative items. There are also techniques for implicit recommendations but all of them assume some sort of a strategy to generate negative samples.
\nAd 2.
\nPlease see sampling process here https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples.
", "rawMarkdown": "Ad 1.\n\nIn my setting I propose to convert the problem to a table where you have a list of item candidates and you mark the sales as either 0 or 1 depending whether the item was sold in the next week.\n\nImagine you want to create negative samples as a list of 1000 most popular items in the last week. Some of those items will be bought by the customers.\n\nSo the table you need to create is:\n- customer_id\n- label = whether it was bought or not\n- article_id from the list of 1000 most popular items\n\nI'm not saying this is the best approach here but it is more or less what I'm doing. This is a technique where you create a set of explicit negative items. There are also techniques for implicit recommendations but all of them assume some sort of a strategy to generate negative samples.\n\nAd 2.\n\nPlease see sampling process here https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples.", "votes": 10 }, { "id": 1748764, "postDate": "2022-04-08T00:35:29.250Z", "content": "I hava a question.
\nDid you mean I need to create 1000 most popular items for each customer and label = whether it was bought or not by each customer? Should the length of train table be len(nunique(customer))×1000?
", "rawMarkdown": "I hava a question. \n\n Did you mean I need to create 1000 most popular items for each customer and label = whether it was bought or not by each customer? Should the length of train table be len(nunique(customer))×1000?", "votes": 1 }, { "id": 1749400, "postDate": "2022-04-08T14:38:42.747Z", "content": "Pretty much yes, except for training you can discard customers without transactions. Only for making the submission predictions you need to consider nunique(customer)×1000 candidates.
", "rawMarkdown": "Pretty much yes, except for training you can discard customers without transactions. Only for making the submission predictions you need to consider nunique(customer)×1000 candidates.", "votes": 4 } ] }, { "id": 1690140, "postDate": "2022-02-14T18:09:42.267Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1690153, "postDate": "2022-02-14T18:26:49.887Z", "content": "\"articles\" are products in french
", "rawMarkdown": "\"articles\" are products in french", "votes": 4 } ] }, { "id": 1782309, "postDate": "2022-05-09T13:13:07.437Z", "content": "Thank you for sharing, learn a lot!
", "rawMarkdown": "Thank you for sharing, learn a lot!" }, { "id": 1763307, "postDate": "2022-04-21T12:25:22.520Z", "content": "It is very nice! thanks for you advice!
", "rawMarkdown": "It is very nice! thanks for you advice!" }, { "id": 1762021, "postDate": "2022-04-20T11:57:43.520Z", "content": "Thank you for sharing! nice work!
", "rawMarkdown": "Thank you for sharing! nice work!" }, { "id": 1743427, "postDate": "2022-04-03T01:26:36.553Z", "content": "thanks for your sampling code
", "rawMarkdown": "thanks for your sampling code" }, { "id": 1723072, "postDate": "2022-03-15T05:27:28.673Z", "content": "Thank you for sharing your great idea.
", "rawMarkdown": "Thank you for sharing your great idea." } ], "comments": [ { "id": 1688506, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2022-02-13T17:03:05.520000", "content": "For anyone that might not be familiar RecSys is one of the most respected competitions in the Recommender space.
\n@paweljankiewicz I was trying to find your winning solution but I could only find the press release content. Is it available anywhere?
\nTIA! :)
", "votes": 9, "replies": [ { "id": 1688518, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-02-13T17:12:18.343000", "content": "Yeah here it is - https://github.com/logicai-io/recsys2019.
", "votes": 16, "replies": [] }, { "id": 1695896, "author_name": "chuan", "author_url": "", "post_date": "2022-02-18T12:47:09.383000", "content": "Hi, @paweljankiewicz , i try to download the data in Recsys 2019
\n--2022-02-18 20:38:52-- https://storage.googleapis.com/logicai-recsys2019/trivagoRecSysChallengeData2019_v2.zip\n正在解析主机 storage.googleapis.com (storage.googleapis.com)... 142.250.204.112, 142.250.204.48, 142.250.66.144, ...\n正在连接 storage.googleapis.com (storage.googleapis.com)|142.250.204.112|:443... 已连接。\n已发出 HTTP 请求，正在等待回应... 404 Not Found\n2022-02-18 20:38:54 错误 404：Not Found。I don't have the original dataset. Also the code is written in such a way that it needs close to 500gb of ram to train the models.
", "votes": 5, "replies": [] }, { "id": 1696214, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2022-02-18T16:57:10.007000", "content": "For anyone else surprised by:
\n\n\nAlso the code is written in such a way that it needs close to 500gb of ram to train the models.
\n
When I had the opportunity to do an interview for another RecSys winning team, I learned this is quite a common number for such problems. 😅
\nThe embedding sizes based on number of users and recommendations just eats up A LOT of memory by the nature of their size
", "votes": 7, "replies": [] }, { "id": 1696389, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-02-18T19:52:49.823000", "content": "BTW I found the data here on Kaggle (but don't seem to have the link to it now), it didn't seem to be available anywhere else.
\nBut I got the most mileage out of reading the code in the repo, didn't get around to doing much with the generated data.
", "votes": 1, "replies": [] }, { "id": 1696410, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2022-02-18T20:07:22.843000", "content": "@radek1 I believe this is the link to the dataset?
", "votes": 2, "replies": [] }, { "id": 1696492, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-02-18T22:30:40.160000", "content": "yes, you found it! :) I think it was that one
\nwith the data I downloaded the preprocessing steps ran without issue
", "votes": 0, "replies": [] }, { "id": 1696542, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-02-18T23:16:10.543000", "content": "@radek1 @init27 the most important concept in RecSys 2019 was the notion of accumulators classes which have 2 methods:
\nWhen you process events by date you first extract the information from the accumulator and then update it with new information. This way there is no possibility to leak any info. The accumulators themselves can track information about the item, customer, item+customer, other groups of information. Really anything you need.
\nThe code could be summarized as follows:
\nfor transaction in transactions:\n for accumulator in accumulators:\n get features from accumulator - really important that it is before update\n update the accumulator\nThere is an opportunity to parallelize the extraction by reversing the loop so you can process and extract features from accumulators in parallel.
", "votes": 4, "replies": [] }, { "id": 1696572, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-02-19T00:18:28.483000", "content": "Ah, I see! @paweljankiewicz thank you very much for sharing this, this is wonderful! 😊
\nIf I am reading this right, the accumulators should be anything -- items of the given type purchased by a customer up to this point, count of transactions, etc.
\nI have completely not thought of processing data like this, this is very cool, eager to give it a try!
\nSo far, I have been attempting to generate features (and candidate transactions) using dataframes, groupby, shift, and merging. But that takes a ridiculous amount of RAM and the code is very hard to reason about and prone to having bugs!
\nWould love to go back to straight-up Python for more of that :)
\nI am not sure I understand the parallelization bit, where we reverse the loop? There is a first pass through the dataset that we cannot parallelize, to compute the accumulators. And we can only parallelize adding features to records in the second pass? (split the df into n_chunks, apply features from accumulator to each chunk, combine?) I am thinking I might be missing something here :) I am also thinking we probably want the accumulators to be simple so that they can only be applied as they go through the list of data in chronological order -- I guess I am really lost on the parallelization bit :)
\nOnce we get all these features, do we do feature selection at all? Or do we just see how correlated columns are and drop the ones that don't seem to add a lot of signal?
\nI saw in the repo you shared that it seems models were run on different train sets? (maybe I misread this). Would you grab some numbers of columns and train different models on different subsets? I am wondering if I didn't misread this as I think there is a way to make lgbm randomly sample a subset of columns to grow each tree, but maybe such an approach is nice for blending?
\n@paweljankiewicz your posts are legendary and this thread is an absolute treasure trove of information for people just getting started with recsys, thank you so much for all your help!!! 🙏
", "votes": 1, "replies": [] }, { "id": 1696593, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-02-19T00:33:44.940000", "content": "I think I might have understood it now! :)
\nWe don't reverse the loop in the sense that we run from end to finish, we just reverse it by moving the inside of the loop outside, like this:
\nfor accumulator in accumulators:\n for transaction in transactions:\n run acc and apply features\nWe can walk the column/dataframe in parallel by multiple accumulators and then combine the results!
\nI would love to try this with something like dask and delayed, I suspect this might be a nice way to work around the limitations of dask's API, really curious how well this could work :)
Great work! Respect to all of you!
", "votes": 0, "replies": [] }, { "id": 1717998, "author_name": "Homoalways", "author_url": "", "post_date": "2022-03-10T12:21:07.620000", "content": "So is it true that if I don't have a machine with lots of memory, I cannot become one of the winners😂
", "votes": 0, "replies": [] } ] }, { "id": 1776281, "author_name": "Daan Le", "author_url": "", "post_date": "2022-05-03T21:35:19.013000", "content": "Hi, thank you so much for this post, it is very useful. I had one question about the generation of the positive examples you use to train the ranking model. Do you use the same method for finding these as with the final test set which means there are missing positive examples or are you including all positive examples as they are known in the training data?
", "votes": 3, "replies": [ { "id": 1776316, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-05-03T22:24:40.767000", "content": "That's a good question. I tried adding all positives but the score didn't improve. I'm using only positives which are associated with some strategy.
", "votes": 5, "replies": [] } ] }, { "id": 1778497, "author_name": "Ryota", "author_url": "", "post_date": "2022-05-05T11:20:42.293000", "content": "Thank you for your posting. This post is very helpful.
\nI would like to ask you positive-negative sample rate during training process.
\nYou said that about 1000 candidates are collected for each customers in your strategy.
\nBut, when training model, this is unbalanced-data. I think down-sampling is necessary.
\nIf you don't mind, please tell me your positive-negative rate.
You are right. At the time of writing I thought that increasing the number of candidates can be good. Then I got to a point where adding more candidates doesn't help. I have way too many candidates right now. During training I take only 5-10% of negative examples. @lihaorocky commented that he is using only 200 candidates which makes sense. I think most of the true positives that we are able to predict are just popular items. With 5% downsampling I have about 4% positives.
", "votes": 3, "replies": [] }, { "id": 1778653, "author_name": "Piotr Gabrys", "author_url": "", "post_date": "2022-05-05T14:58:26.153000", "content": "Hi @paweljankiewicz ;) I'm optimizing number of candidates with Bayesian Optimization. For every candidate generation method I have a k parameter determining how many articles can be returned. Then I optimize Precision@50 for a baseline model (no reranking model) where the k's are hyperparameters. After a while you can clearly see which generators should be switched off or limited.
Interesting approach. I think that this competition shows very well that most cf recommendation algorithms are almost useless in this setting. It is more like a symbolic recommendation system where you don't rely on particular items but rather pools of items created from recent transaction history. So recommendation can be based on:
\n0.5top_50_articles_department + 0.5most_similar_articles_to_the_last_basket_that_are_within_1000_most_popular_articles
\nThere are some obvious flaws in the competition like guessing the availability of items but it is a valuable experience nonetheless.
", "votes": 3, "replies": [] }, { "id": 1779443, "author_name": "Clear n' Simple", "author_url": "", "post_date": "2022-05-06T12:57:13.977000", "content": "@paweljankiewicz - thanks for sharing that!
\nWhat does top_50_articles_department mean?
It is just an example of some strategy. In this case it would be something like: take popular items from some specific item category that the user has in the historical transactions.
", "votes": 1, "replies": [] } ] }, { "id": 1767890, "author_name": "Fnoa", "author_url": "", "post_date": "2022-04-25T18:37:44.290000", "content": "Hi @paweljankiewicz, very very helpful post. Thanks
\nJust one question, what do you mean with “model based predictions with customers without the transaction history” ?
", "votes": 1, "replies": [] }, { "id": 1761945, "author_name": "MohammedAlsayed", "author_url": "", "post_date": "2022-04-20T10:22:19.840000", "content": "Hi Pawl,
\nthanks for your informative post. I would like to ask you what do you mean by dynamic/static attributes for customer and articles? if illustrated with an example will be more helpful. Thanks!
", "votes": 1, "replies": [] }, { "id": 1742077, "author_name": "Y-Haneji", "author_url": "", "post_date": "2022-04-01T12:44:09.713000", "content": "@paweljankiewicz
\nThanks for great idea!
\nI have a question.
\nWhat does observation_date exactly mean? Please give me some example.
Observation date is the date you treat as \"now\" so you can use all the data before this date. So for validation you need to take a date like 2020-09-15. For submission the observation date is of course the last available date 2020-09-22.
", "votes": 2, "replies": [] }, { "id": 1750050, "author_name": "Y-Haneji", "author_url": "", "post_date": "2022-04-09T09:14:06.753000", "content": "@paweljankiewicz
\nThanks! I have completely understood.
Thank you for sharing your idea. This is really a great approach. I am getting \"run out of memory\" problem. I am using my local PC which has 16GB RAM. Can you suggest how to deal with this situation. Thank You!
", "votes": 1, "replies": [ { "id": 1700865, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-02-22T10:18:45.013000", "content": "16GB RAM is probably not enough to comfortably work with this dataset - at least the approach I suggested needs a lot more memory. You can experiment on a smaller dataset https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples. Even using 5% sample should give you similar performance to 100% on this data.
", "votes": 5, "replies": [] }, { "id": 1701108, "author_name": "Pankaj Kumar", "author_url": "", "post_date": "2022-02-22T14:24:37.200000", "content": "Thank you!
", "votes": 0, "replies": [] }, { "id": 1704464, "author_name": "Faisal Alsrheed", "author_url": "", "post_date": "2022-02-25T14:47:55.280000", "content": "Hi @paweljankiewicz :)
\nThank you for sharing
\nYou might find this code useful to make a stratified Sample using many columns.
\nhttps://github.com/flaboss/python_stratified_sampling/blob/master/stratifiedSample.py
\nAll the best
", "votes": 1, "replies": [] } ] }, { "id": 1698165, "author_name": "Mahamadi NIKIEMA", "author_url": "", "post_date": "2022-02-20T07:42:19.007000", "content": "I am new to the recommender system, do you have any references for me? Thanks.
", "votes": 1, "replies": [ { "id": 1699049, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-02-20T21:36:41.370000", "content": "I suggest to start with the most famous datasets for recommendations - something like movielens - https://grouplens.org/datasets/movielens/. Follow some tutorial using these datasets. This competition is on the harder spectrum because it is not a very typical dataset. I remember watching a part of this course https://www.coursera.org/specializations/recommender-systems and it was decent.
", "votes": 12, "replies": [] } ] }, { "id": 1696455, "author_name": "Jianing ", "author_url": "", "post_date": "2022-02-18T21:03:01.053000", "content": "how could you create negative samples?
", "votes": 1, "replies": [ { "id": 1696519, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-02-18T22:52:07.610000", "content": "As I have written there are many ways to create negative samples. For example items in the previous basket can be a source of negative samples. Popular items can be too. There are countless of examples.
", "votes": 2, "replies": [] }, { "id": 1715321, "author_name": "HongweiLuan", "author_url": "", "post_date": "2022-03-07T21:50:41.483000", "content": "Thanks for the inspiring post @paweljankiewicz It is very helpful especially on the negative sampling.
\nI have another question about negative sampling. Shall we sample from items which are not similar to positive samples as negative samples? I am asking because it may confuse the model if we sample popular or previously purchased items as negative samples. They could be potentially positive samples in the test set. On the other hand, they could represent an interest shift of the customers. Do you have suggestions from your experience? Thank you ahead.
He said 'use items in the previous basket', it might mean 'user other customer's basket as the negative samples for the current customer', make sense? I think it's ok.
\nBut what if these two customers bought the same items, does it confuse the model?
", "votes": 0, "replies": [] } ] }, { "id": 1707473, "author_name": "Ethan Kershner", "author_url": "", "post_date": "2022-02-28T13:49:40.003000", "content": "Pawel, thanks so much for putting this together, this is awesome! 🙏
\nI have a question for you about search strategies. Is there any reason why, instead of separately implementing a search strategy and inserting the result as a column in the training set, it couldn't also work to just directly add the variables that the strategy depends on into the training set?
\nSo for example if strategy S is dependent on 10 variables x1, x2… x10, instead of adding S as a column to the training set, adding x1 - x10. Although I suppose this would only be reasonable for simpler strategies and wouldn't work for strategies like image detection.
", "votes": 2, "replies": [ { "id": 1707500, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-02-28T14:14:09.507000", "content": "This is a good question. I think there are several problems
\n1) For each customer you must consider 100k items if you really want to include all variables with which the strategy was created.
\n2) What would you do in a case that you want to relate a set of items from the history with those candidates. The strategy can really on more complex feature set than x1,…,x10.
A search strategy is a way to reduce the number of candidates that you consider for each customer.
", "votes": 1, "replies": [] }, { "id": 1707630, "author_name": "Ethan Kershner", "author_url": "", "post_date": "2022-02-28T16:23:25.903000", "content": "That makes sense to me. Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 1698993, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-02-20T20:11:23.767000", "content": "As I keep working on this and learning about it, I keep coming to this thread and rereading it. This is such a wealth of information!
\nThank you again @paweljankiewicz!
\nWhat blew my mind today was using ‘customer_id’ as a query! I thought we might want to query on [‘customer_id’, ‘observation_date’] to essentially query on baskets?
\nAlso, the list of transactions can be a good staying set of positive examples? Meaning, from the post it might seem like we need to do something extra (‘the label indicates whether the article will be purchased next week’) but this is essentially what the list of transactions does? It is an observation date and if we add an indicator column such as ‘purchased’ we will have everything that we need? I am just worried I might be missing something subtle here.
", "votes": 2, "replies": [ { "id": 1699032, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-02-20T21:09:57.727000", "content": "No you are right for more than 1 observation date the query should be customer_id + observation_dt. Because I have everything sorted by date I didn't notice that I'm implicitly including observation_dt in the query.
", "votes": 2, "replies": [] }, { "id": 1699040, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-02-20T21:18:58.783000", "content": "@radek1 I think you are missing something else. When you generate candidates, it is not 100% sure that you will find any transactions. Let's say your only strategy is to include the last basket items. If you have a customer which didn't buy anything twice then you can discard whole customer + observation_dt. The way the ranking models work they need positive examples in the query. This is a source of an important optimization in my opinion.
\nThe process should be like this: you generate a candidates set something like Dict[Strategy, List[ArticleId]]. A unique set of ArticleId is your number of observations that you have to consider for the customer.
As I have said for training purposes you can discard all the queries without positive examples. But for validation and submission you must include all of your candidates.
", "votes": 3, "replies": [] }, { "id": 1699125, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-02-21T00:08:45.827000", "content": "Thank you so much for these additional details, @paweljankiewicz! 😊 All this is of great help
", "votes": 0, "replies": [] } ] }, { "id": 1758919, "author_name": "WENDY LIU", "author_url": "", "post_date": "2022-04-18T06:50:27.390000", "content": "nice work!!
", "votes": 0, "replies": [] }, { "id": 1757936, "author_name": "Y-Haneji", "author_url": "", "post_date": "2022-04-17T06:43:41.253000", "content": "@paweljankiewicz
\nInspired by you, I have been working on rank model these days.
\nI have a question about mixing several strategy to make candidates. How can we put \"strategy score\" of candidate from different strategy together? Or make a column for each strategy?
\nTake an example of mixing \"recent transaction strategy\"(strategy1) and \"LSTM strategy\"(strategy2). I can simply make candidates of [customer, article, strategy1: bool, strategy2: bool] columns.
\nBUT, I want to use features from strategies, like \"transaction count\" from strategy1 or \"LSTM's probability\" from strategy2. I think this is \"strategy score\".
I think it is worth having boolean flags for each strategy and of course you can use additional features from the strategy like counts and probabilities. The only thing you have to watch for is data leakage, so make sure you use only the data before the observation date to train your models.
", "votes": 2, "replies": [] }, { "id": 1758408, "author_name": "Y-Haneji", "author_url": "", "post_date": "2022-04-17T16:43:06.213000", "content": "@paweljankiewicz
\nThanks for reply!
\nI understand that to avoid leakage is important.
\nI have adopted naive strategies, but will develop some elaborate strategies from now on and mix them.
@paweljankiewicz - I'm using LGBMRanker, and printing out the ndcg at each step, and I'm finding that as the model trains, it's getting worse, even on the training set!
\nSame when using map.
That's really strange. You probably have some sort of a bug. Are you sure you are providing a proper group parameter? I can vouch for LGBMRanker.
", "votes": 1, "replies": [] }, { "id": 1747673, "author_name": "IgorMuniz", "author_url": "", "post_date": "2022-04-06T21:36:46.240000", "content": "We had the same problem and so far I couldn't find a solution for it. Not sure but I think it might be related to the way we are building our dataset which Idk if it's right. We are considering 1 week only as the label and all data before this week as train. Maybe there is a better way to carry time information than that.
", "votes": 0, "replies": [] }, { "id": 1748364, "author_name": "Clear n' Simple", "author_url": "", "post_date": "2022-04-07T14:11:57.113000", "content": "@paweljankiewicz - thanks for letting me know that it shouldn't be happening, I'll keep digging until I find out what's wrong.
\nI went back and checked, the group parameter looks good.
\nMaybe it's because I haven't added customer features yet, only article and customer-article ones?
", "votes": 0, "replies": [] }, { "id": 1748384, "author_name": "Clear n' Simple", "author_url": "", "post_date": "2022-04-07T14:37:02.650000", "content": "@igormunizims, I'm also only considering one week as label for starters.
\nI don't see why that should be a problem, though.
\nWhen you have very little data, you overfit quickly, but it should still improve on the training set.
@jacob34 there are some optimizations you can do when you are training the ranking model. For example you can consider only the queries with at least 1 positive example. For the ranking model the queries with all negative observations are useless. If you didn't remove them maybe you can check if this is the reason why the ranking model is behaving in a strange way. It may be that after removing all such observations the number of observations is really low and it does indeed is overfiiting a lot.
\nUnfortunately when you are predicting for submission you must use all the observations which is painful (in my case I need to predict 100gb of compressed dataframes and it is the slowest part of the process).
", "votes": 5, "replies": [] }, { "id": 1748487, "author_name": "Clear n' Simple", "author_url": "", "post_date": "2022-04-07T16:29:31.913000", "content": "Yes, I did follow your advice on that.
\nFor a single fold, I have 390k rows, for 12k customers, with 15k positives.
\nI realize it may be overfitting, but evaluation metric should still go down on the training set, no?
Wow - 100gb is massive.
", "votes": 1, "replies": [] }, { "id": 1748580, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-04-07T18:15:37.703000", "content": "Yes. It should go down. Are the results when you are checking the metric manually the same?
", "votes": 1, "replies": [] }, { "id": 1759512, "author_name": "Clear n' Simple", "author_url": "", "post_date": "2022-04-18T17:30:53.483000", "content": "Yes, checking manually gives me the same.
\nI find that after a few steps, it starts improving again (although never getting as good at it originally was), and at that point, the different evaluation metrics correlate, and cv correlates with LB, so I'm working with that.
\nStill not sure why it's happening, but I suspect it's because of weak features and small training set size.
", "votes": 0, "replies": [] }, { "id": 1777731, "author_name": "Clear n' Simple", "author_url": "", "post_date": "2022-05-04T18:35:47.290000", "content": "I finally figured it out - it was a nasty, silent leakage of the label, in the form of positive exampled getting put earlier than negative ones.
\nThe details are here.
", "votes": 0, "replies": [] } ] }, { "id": 1720857, "author_name": "G!", "author_url": "", "post_date": "2022-03-13T07:52:42.817000", "content": "Thank you very much! But I have a question. Sampling 5% data could keep the original data distribution?
", "votes": 0, "replies": [] }, { "id": 1718932, "author_name": "ctscsu", "author_url": "", "post_date": "2022-03-11T09:58:46.230000", "content": "Great tips, Thanks a lot ! How do you think of transformer based model in this competition?
\njust like riiid 2021。
\nDoes the Magnitudes of users 、items 、orders and dates satisfy ？
@paweljankiewicz , thanks for the great post and learned alot…
\nwanted to know how do we emphasize on ranking/relevance of articles for the customer
\nbecause there are lot customers who are returning or rebuying the same stuff. So,, how do we differentiate.
I think ranking of relevance of articles to customers may take care of that.
", "votes": 0, "replies": [] } ] }, { "id": 1702159, "author_name": "AtulVerma", "author_url": "", "post_date": "2022-02-23T11:29:44.417000", "content": "My apologies in advance but is it fair to tie a recommendation to purchase? Is it a given fact that if the Customer made a purchase, the attribution should go to the recommendation. Is it more of a prediction than a recommendation?
", "votes": 0, "replies": [] }, { "id": 1699304, "author_name": "kwalpratham", "author_url": "", "post_date": "2022-02-21T05:13:50.970000", "content": "this was very helpful
", "votes": 0, "replies": [] }, { "id": 1691296, "author_name": "Mohamed Annis SOUAMES", "author_url": "", "post_date": "2022-02-15T10:38:22.437000", "content": "Thank you for these great tips, this is my first competition in Kaggle (I have experience in smaller scale competitions in another platform). I wanted to know what do you mean by sampling data as much as possible ? Do you mean we should resample data every time we experiment a new thing ? Are there any notebooks or tutorials that talk about this in more detail ?
\nThank you again :)
", "votes": 0, "replies": [ { "id": 1691508, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-02-15T12:52:03.397000", "content": "One sample is enough for quick tests. Added the code to sample the data https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples
", "votes": 1, "replies": [] }, { "id": 1692185, "author_name": "Mohamed Annis SOUAMES", "author_url": "", "post_date": "2022-02-15T22:10:10.660000", "content": "Thank you so much, but the notebook is empty, I think it did not run :/
", "votes": 0, "replies": [] }, { "id": 1693237, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-02-16T14:38:29.617000", "content": "It should be good now.
", "votes": 0, "replies": [] } ] }, { "id": 1728123, "author_name": "", "author_url": "", "post_date": "2022-03-18T15:51:18.240000", "content": "", "votes": 1, "replies": [ { "id": 1728274, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-03-18T18:21:30.870000", "content": "Ad 1.
\nIn my setting I propose to convert the problem to a table where you have a list of item candidates and you mark the sales as either 0 or 1 depending whether the item was sold in the next week.
\nImagine you want to create negative samples as a list of 1000 most popular items in the last week. Some of those items will be bought by the customers.
\nSo the table you need to create is:
\nI'm not saying this is the best approach here but it is more or less what I'm doing. This is a technique where you create a set of explicit negative items. There are also techniques for implicit recommendations but all of them assume some sort of a strategy to generate negative samples.
\nAd 2.
\nPlease see sampling process here https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples.
", "votes": 10, "replies": [] }, { "id": 1748764, "author_name": "oriori4244", "author_url": "", "post_date": "2022-04-08T00:35:29.250000", "content": "I hava a question.
\nDid you mean I need to create 1000 most popular items for each customer and label = whether it was bought or not by each customer? Should the length of train table be len(nunique(customer))×1000?
", "votes": 1, "replies": [] }, { "id": 1749400, "author_name": "Paweł Jankiewicz", "author_url": "", "post_date": "2022-04-08T14:38:42.747000", "content": "Pretty much yes, except for training you can discard customers without transactions. Only for making the submission predictions you need to consider nunique(customer)×1000 candidates.
", "votes": 4, "replies": [] } ] }, { "id": 1690140, "author_name": "", "author_url": "", "post_date": "2022-02-14T18:09:42.267000", "content": "", "votes": 0, "replies": [ { "id": 1690153, "author_name": "Ala Eddine Ayadi", "author_url": "", "post_date": "2022-02-14T18:26:49.887000", "content": "\"articles\" are products in french
", "votes": 4, "replies": [] } ] }, { "id": 1782309, "author_name": "Lukan", "author_url": "", "post_date": "2022-05-09T13:13:07.437000", "content": "Thank you for sharing, learn a lot!
", "votes": 0, "replies": [] }, { "id": 1763307, "author_name": "Zhicheng Wang", "author_url": "", "post_date": "2022-04-21T12:25:22.520000", "content": "It is very nice! thanks for you advice!
", "votes": 0, "replies": [] }, { "id": 1762021, "author_name": "amandus-kim", "author_url": "", "post_date": "2022-04-20T11:57:43.520000", "content": "Thank you for sharing! nice work!
", "votes": 0, "replies": [] }, { "id": 1743427, "author_name": "stallone", "author_url": "", "post_date": "2022-04-03T01:26:36.553000", "content": "thanks for your sampling code
", "votes": 0, "replies": [] }, { "id": 1723072, "author_name": "Tact Suetsugu", "author_url": "", "post_date": "2022-03-15T05:27:28.673000", "content": "Thank you for sharing your great idea.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1688473": "Just to give you some background - my team won RecSys 2019 so I'm pretty familiar with problems like these. Now that we have this out of the way...\n\nTo simplify the discussion I propose to call the set of articles that a customer bought on a given date a **basket**. And then this problem can be really summarized as predict next basket.\n\nGeneral approach towards problems like this could be as follows. At least mine will be.\n\n## 1. For each customer generate candidates that they can buy. \n\nThese candidates are your negative and positive examples. I'm saying this because someone on the forum asked where are the negative observations. And the answer is you generate them. \n\nThe strategies to generate those candidates are probably the most important aspect of this competition. The strategies can be based on:\n- item to item similarities between customer previous baskets - so image similarity is not out of the question as somebody suggested\n- user based collaborative filtering\n- last baskets (with the hope that the user will buy them again)\n- model based predictions with customers without the transaction history\n- etc\n\nComing up with different strategies is a fun challenge :). I cannot imagine that a winning solution will be based on a closed form model representing one strategy. Once you have all the different strategies ready for each customer then it is time to create a huge table with columns like these:\n\n- observation_date - the date for which you are making predictions - submission should be based on all the data before 2020-09-22 (my birthday btw).\n- customer_id\n- article_id\n- bought in the next week (this is your label)\n- customer static attributes\n- customer dynamic attributes\n- article static attributes\n- article dynamic attributes (maybe item popularity and such)\n- whether the article exist in the strategies and which ones\n- strategy score (whatever it is)\n- many other features that relate the item to the user\n\nOnce you have a table like this it all becomes very familiar. There were many competitions involving click through predictions. They have almost an identical structure the only difference is the source of observations.\n\n## 2. Build a ranking model that ranks the items within Customer \n\nRanking is a different problem than classification but in general classification models can be used as well. My favourite ranking model is LightGBM (we used it in RecSys 2019) https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRanker.html.\n\nThe customer id can be treated as a query in ranking model because you are interested in optimizing the scores at a customer level.\n\n## General tips\n\n1. Sample the data as much as possible - most of the competition you should spend validating your features on the sample. Increase the sample you are using when you are no longer seeing the correlation between local validation and the leaderboard.\n\n2. Test different ideas quickly. Fast local validation is key here because you must do as much experiments as possible.\n\nGood luck :)\n\nUpdate 1. Added code for sampling here https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples", "1688506": "For anyone that might not be familiar [RecSys](https://recsys.acm.org) is one of the most respected competitions in the Recommender space. \n\n@paweljankiewicz I was trying to find your winning solution but I could only find the press release content. Is it available anywhere? \n\nTIA! :)", "1776281": "Hi, thank you so much for this post, it is very useful. I had one question about the generation of the positive examples you use to train the ranking model. Do you use the same method for finding these as with the final test set which means there are missing positive examples or are you including all positive examples as they are known in the training data?", "1778497": "Thank you for your posting. This post is very helpful.\nI would like to ask you positive-negative sample rate during training process.\nYou said that about 1000 candidates are collected for each customers in your strategy.\nBut, when training model, this is unbalanced-data. I think down-sampling is necessary.\nIf you don't mind, please tell me your positive-negative rate.", "1767890": "Hi @paweljankiewicz, very very helpful post. Thanks \n\nJust one question, what do you mean with “model based predictions with customers without the transaction history” ?\n", "1761945": "Hi Pawl, \n\nthanks for your informative post. I would like to ask you what do you mean by dynamic/static attributes for customer and articles? if illustrated with an example will be more helpful. Thanks!", "1742077": "@paweljankiewicz \nThanks for great idea!\nI have a question.\nWhat does observation_date exactly mean? Please give me some example.", "1700846": "Thank you for sharing your idea. This is really a great approach. I am getting \"run out of memory\" problem. I am using my local PC which has 16GB RAM. Can you suggest how to deal with this situation. Thank You! ", "1698165": "I am new to the recommender system, do you have any references for me? Thanks.", "1696455": "how could you create negative samples? ", "1707473": "Pawel, thanks so much for putting this together, this is awesome! 🙏\n\nI have a question for you about search strategies. Is there any reason why, instead of separately implementing a search strategy and inserting the result as a column in the training set, it couldn't also work to just directly add the variables that the strategy depends on into the training set? \n\nSo for example if strategy S is dependent on 10 variables x1, x2... x10, instead of adding S as a column to the training set, adding x1 - x10. Although I suppose this would only be reasonable for simpler strategies and wouldn't work for strategies like image detection.", "1698993": "As I keep working on this and learning about it, I keep coming to this thread and rereading it. This is such a wealth of information!\n\nThank you again @paweljankiewicz!\n\nWhat blew my mind today was using ‘customer_id’ as a query! I thought we might want to query on [‘customer_id’, ‘observation_date’] to essentially query on baskets?\n\nAlso, the list of transactions can be a good staying set of positive examples? Meaning, from the post it might seem like we need to do something extra (‘the label indicates whether the article will be purchased next week’) but this is essentially what the list of transactions does? It is an observation date and if we add an indicator column such as ‘purchased’ we will have everything that we need? I am just worried I might be missing something subtle here.", "1758919": "nice work!!", "1757936": "@paweljankiewicz \nInspired by you, I have been working on rank model these days.\nI have a question about mixing several strategy to make candidates. How can we put \"strategy score\" of candidate from different strategy together? Or make a column for each strategy?\nTake an example of mixing \"recent transaction strategy\"(strategy1) and \"LSTM strategy\"(strategy2). I can simply make candidates of [customer, article, strategy1: bool, strategy2: bool] columns. \nBUT, I want to use features from strategies, like \"transaction count\" from strategy1 or \"LSTM's probability\" from strategy2. I think this is \"strategy score\".\n", "1747328": "@paweljankiewicz - I'm using LGBMRanker, and printing out the ndcg at each step, and I'm finding that as the model trains, it's getting worse, even on the training set!\n\nSame when using `map`.", "1720857": "Thank you very much! But I have a question. Sampling 5% data could keep the original data distribution?", "1718932": "Great tips, Thanks a lot ! How do you think of transformer based model in this competition?\njust like riiid 2021。\nDoes the Magnitudes of users 、items 、orders and dates satisfy ？", "1716637": "@paweljankiewicz , thanks for the great post and learned alot...\nwanted to know how do we emphasize on ranking/relevance of articles for the customer \nbecause there are lot customers who are returning or rebuying the same stuff. So,, how do we differentiate.", "1702159": "My apologies in advance but is it fair to tie a recommendation to purchase? Is it a given fact that if the Customer made a purchase, the attribution should go to the recommendation. Is it more of a prediction than a recommendation?", "1699304": "this was very helpful\n", "1691296": "Thank you for these great tips, this is my first competition in Kaggle (I have experience in smaller scale competitions in another platform). I wanted to know what do you mean by sampling data as much as possible ? Do you mean we should resample data every time we experiment a new thing ? Are there any notebooks or tutorials that talk about this in more detail ? \n\nThank you again :)", "1728123": "", "1690140": "", "1782309": "Thank you for sharing, learn a lot!", "1763307": "It is very nice! thanks for you advice!", "1762021": "Thank you for sharing! nice work!", "1743427": "thanks for your sampling code", "1723072": "Thank you for sharing your great idea." } }