Congrats to the 1st place winners and all the participants. Thanks to the competition organizers for this interesting competition. Here is my team mate @paweljankiewicz and my solution.
\nI didn't do a good job of recalling the product candidate. At the beginning, I tried recalling methods such as itemCF, but later found that the effect was not even as good as directly selecting popular products. Therefore, my solution is to directly select aboout 600 most popular products for each user (there will be some very simple strategies to refer to the user's history). Then train a lgb model and select 130 candidates for each user according to the model score, and then add all the products purchased by the user in the history as the final candidate
\nAfter that, train another lgb model to get the final result. Compare to the lgb model in recall stage, the features will be more complex, features can be mainly divided into the following groups:
\nThe score of the single model on the Leaderboard is 0.0355, after adding @paweljankiewicz candidates can be promoted to 0.0362, and the ensemble score is 0.0368.
\nI followed my advice from the thread mostly. I'm happy that my intution was correct. Overall until the merge with wht1996 I focused on generating a diverse set of recall strategies. The most interesting part of the solution wouldn't be the logic but that I have written the feature engineering part in Rust and create a mini framework for parallel feature extraction. I thought that I could get away with much more complex rules that way and it was still pretty fast.
\nI used all sorts of counters to measure the popularity of items in different groups:
\nAlso I used:
\nFeatures:
\nAt the end I ended up with 1000 candidates per customer which was way too many. I had trouble to improve beyond 0.0340 but with wht1996 help I managed to improve the single model to 0.0348.
\nI managed to write a lambdarankmap objective for LGBMRanker (mostly by copying the code from XGBoost) and it was slightly better than lambdarank (which uses NDCG objective). For the whole competition I trained the models using 2-3 months of transactions and last week to validate.
\nDeep learning models - I tried several recbole models but they weren't so good.
\nA large part of the competition was about predicting the availability of the products for different postal_codes. I think this was disguised as a recommendation system competition to some extent. For future competitions I would suggest:
\nThe cooperation was smooth. We exchanged our solutions and candidates (with scores).
", "messages": [ { "id": "1783562", "postDate": "05/10/2022 13:28:58", "content": "Congrats to the 1st place winners and all the participants. Thanks to the competition organizers for this interesting competition. Here is my team mate @paweljankiewicz and my solution.
\nI didn't do a good job of recalling the product candidate. At the beginning, I tried recalling methods such as itemCF, but later found that the effect was not even as good as directly selecting popular products. Therefore, my solution is to directly select aboout 600 most popular products for each user (there will be some very simple strategies to refer to the user's history). Then train a lgb model and select 130 candidates for each user according to the model score, and then add all the products purchased by the user in the history as the final candidate
\nAfter that, train another lgb model to get the final result. Compare to the lgb model in recall stage, the features will be more complex, features can be mainly divided into the following groups:
\nThe score of the single model on the Leaderboard is 0.0355, after adding @paweljankiewicz candidates can be promoted to 0.0362, and the ensemble score is 0.0368.
\nI followed my advice from the thread mostly. I'm happy that my intution was correct. Overall until the merge with wht1996 I focused on generating a diverse set of recall strategies. The most interesting part of the solution wouldn't be the logic but that I have written the feature engineering part in Rust and create a mini framework for parallel feature extraction. I thought that I could get away with much more complex rules that way and it was still pretty fast.
\nI used all sorts of counters to measure the popularity of items in different groups:
\nAlso I used:
\nFeatures:
\nAt the end I ended up with 1000 candidates per customer which was way too many. I had trouble to improve beyond 0.0340 but with wht1996 help I managed to improve the single model to 0.0348.
\nI managed to write a lambdarankmap objective for LGBMRanker (mostly by copying the code from XGBoost) and it was slightly better than lambdarank (which uses NDCG objective). For the whole competition I trained the models using 2-3 months of transactions and last week to validate.
\nDeep learning models - I tried several recbole models but they weren't so good.
\nA large part of the competition was about predicting the availability of the products for different postal_codes. I think this was disguised as a recommendation system competition to some extent. For future competitions I would suggest:
\nThe cooperation was smooth. We exchanged our solutions and candidates (with scores).
", "rawMarkdown": "Congrats to the 1st place winners and all the participants. Thanks to the competition organizers for this interesting competition. Here is my team mate @paweljankiewicz and my solution.\n\nwht1996 solution\n-----------\nI didn't do a good job of recalling the product candidate. At the beginning, I tried recalling methods such as itemCF, but later found that the effect was not even as good as directly selecting popular products. Therefore, my solution is to directly select aboout 600 most popular products for each user (there will be some very simple strategies to refer to the user's history). Then train a lgb model and select 130 candidates for each user according to the model score, and then add all the products purchased by the user in the history as the final candidate\n\nAfter that, train another lgb model to get the final result. Compare to the lgb model in recall stage, the features will be more complex, features can be mainly divided into the following groups:\n- User basic features: including num, price, sales_channel_id.\n- Product basic features: statistics based on each attribute of the product, including times, price, age, sales_channel_id, FN, Active, club_member_status, fashion_news_frequency, last purchase time, average purchase interval.\n- User product combination features: statistics based on each attribute of the product, including num, time, sales_channel_id, last purchase time, and average purchase interval.\n- Age product combination features: products popularity under each age group.\n- User product repurchase features: whether user will repurchase product and whether product will be repurchased.\n- Higher-order combinatorial features: For example, predict when the user will next purchase the product.\n- itemCF feature: calculate the similarity of each item through itemCF, and then calculate the score that whether user will buy the product.\n\nThe score of the single model on the Leaderboard is 0.0355, after adding @paweljankiewicz candidates can be promoted to 0.0362, and the ensemble score is 0.0368.\n\n@paweljankiewicz solution\n-----------\n\nI followed my advice from the thread mostly. I'm happy that my intution was correct. Overall until the merge with wht1996 I focused on generating a diverse set of recall strategies. The most interesting part of the solution wouldn't be the logic but that I have written the feature engineering part in Rust and create a mini framework for parallel feature extraction. I thought that I could get away with much more complex rules that way and it was still pretty fast.\n\n### Candidate strategies\n\nI used all sorts of counters to measure the popularity of items in different groups:\n- customer attributes: different combinations (what boosted my score significantly was including postal_code)\n- article attributes: different combinations and then intersect with the customer history - so if a customer bought an item with certain properties I looked for popular items with the same property\nThe counters were used for different recency like: 1,3,7,30,90 days.\n\nAlso I used:\n- image similarities using MobileNet embedding\n- cooccurences (inside basket and outside = basket to basket)\n- random graph walk over item-user graph\n\nFeatures:\n- customer / article attributes\n- all sorts of similarity based measures for example I calculated average similarity of items vs the items that the customer bought\n- streaks - this was inspired by some previous competitions posts - for many article attributes I calculated streaks measuring the number of times the customer bought a product, category, section etc in row.\n\nAt the end I ended up with 1000 candidates per customer which was way too many. I had trouble to improve beyond 0.0340 but with wht1996 help I managed to improve the single model to 0.0348.\n\n### Modeling \n\nI managed to write a lambdarankmap objective for LGBMRanker (mostly by copying the code from XGBoost) and it was slightly better than lambdarank (which uses NDCG objective). For the whole competition I trained the models using 2-3 months of transactions and last week to validate.\n\n### What didn't work\n\nDeep learning models - I tried several recbole models but they weren't so good.\n\n## How to improve competitions like this\n\nA large part of the competition was about predicting the availability of the products for different postal_codes. I think this was disguised as a recommendation system competition to some extent. For future competitions I would suggest:\n- providing the state of articles - which are available and where\n- to minimize the calculation needed to submit I would suggest to provide a list of customers with transactions in the subimission week. Predicting customers which didn't have transactions was very computation heavy task. The reason is that the models are the same but creating features and predicting takes time.\n\nHow we cooperated\n-----------\n\nThe cooperation was smooth. We exchanged our solutions and candidates (with scores).", "votes": null }, { "id": "1783572", "postDate": "05/10/2022 13:41:02", "content": "Congratulations to you both and many thanks to @paweljankiewicz, we all have learned a lot from you. When you say you used 2-3 months of transactions to train, is it mean you have 8-12 target weeks in your training set? And for candidate generation do you consider all transactions before the target week?
", "rawMarkdown": "Congratulations to you both and many thanks to @paweljankiewicz, we all have learned a lot from you. When you say you used 2-3 months of transactions to train, is it mean you have 8-12 target weeks in your training set? And for candidate generation do you consider all transactions before the target week?", "votes": null }, { "id": "1783579", "postDate": "05/10/2022 13:48:56", "content": "Yes exactly. So it is not maybe precise to say that I used last 2-3 months of transactions.
\nI used all transactions until the observation date. And only for ranking model I used last 2-3 months.
Congrats @paweljankiewicz and @wht1996 for the second place! Thanks for sharing your great solution.
", "rawMarkdown": "Congrats @paweljankiewicz and @wht1996 for the second place! Thanks for sharing your great solution.", "votes": null }, { "id": "1783651", "postDate": "05/10/2022 14:53:44", "content": "Congrats! Cool work
", "rawMarkdown": "Congrats! Cool work", "votes": null }, { "id": "1783688", "postDate": "05/10/2022 15:37:29", "content": "Thanks for sharing and congrats!
\nDefinitely agree with your improvement suggestions. Having to predict for all users was quite a slowdown.
Thanks for sharing!
\n@paweljankiewicz
\nI think this is the first place I saw someone mentioning postal code.
\nDid you do something special that made it work for you?
Also, you mentioned mid-competition that you were using labels from about 5 months worth of data, and using only 4 months score worsened - did that change at some point?
", "rawMarkdown": "Thanks for sharing!\n\n@paweljankiewicz \nI think this is the first place I saw someone mentioning postal code.\nDid you do something special that made it work for you?\n\nAlso, you mentioned mid-competition that you were using labels from about 5 months worth of data, and using only 4 months score worsened - did that change at some point?", "votes": null }, { "id": "1784093", "postDate": "05/10/2022 23:44:08", "content": "Great work! Can you share some details on the feature engineering part written in Rust and the mini framework for parallel feature extraction?
", "rawMarkdown": "Great work! Can you share some details on the feature engineering part written in Rust and the mini framework for parallel feature extraction?", "votes": null }, { "id": "1784118", "postDate": "05/11/2022 00:16:45", "content": "@wht1996 thank you for this write up and congratulations!
", "rawMarkdown": "wht1996 thank you for this write up and congratulations!", "votes": null }, { "id": "1784175", "postDate": "05/11/2022 01:59:42", "content": "Congrats! your improvement in LB of last 3 days was impressive and worried us.
", "rawMarkdown": "Congrats! your improvement in LB of last 3 days was impressive and worried us.", "votes": null }, { "id": "1784186", "postDate": "05/11/2022 02:36:11", "content": "Congrats for the second place finish! Very thorough feature engeering work. Learn a lot. Thanks!
", "rawMarkdown": "Congrats for the second place finish! Very thorough feature engeering work. Learn a lot. Thanks!", "votes": null }, { "id": "1784220", "postDate": "05/11/2022 03:13:47", "content": "\n\nto minimize the calculation needed to submit I would suggest to provide a list of customers with transactions in the subimission week.
\n
I raised that issue here, but nobody understood what I said, and computation time increased by a factor of 20, our time and money got lost in vain 😄👍
\nhttps://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/305952#1690119
I think if we know customers who have transactions at next week, we can create lots of features using future information to improve accuracy that will make this competition just become a competition not a real recommedation problem in industry.
", "rawMarkdown": "I think if we know customers who have transactions at next week, we can create lots of features using future information to improve accuracy that will make this competition just become a competition not a real recommedation problem in industry.", "votes": null }, { "id": "1784330", "postDate": "05/11/2022 05:50:00", "content": "Congrats on your 2nd place finish, especially for our new grandmaster @wht1996 .
\nTwo questions to @wht1996
\n\n\nHigher-order combinatorial features: For example, predict when the user will next purchase the product.
\n
Two questions to @paweljankiewicz
\n\n\ncustomer attributes: different combinations (what boosted my score significantly was including postal_code)
\n
\n\nimage similarities using MobileNet embedding
\n
Popularity in the last 1/3/7 days combined with postal_code + age or other item/customer attributes creates very precise candidates. Some combinations have 10-11% precision. About image embeddings I also didn't see any significant boost but it was one the strategies to generate candidates so I thought I should mention it.
", "rawMarkdown": "Popularity in the last 1/3/7 days combined with postal_code + age or other item/customer attributes creates very precise candidates. Some combinations have 10-11% precision. About image embeddings I also didn't see any significant boost but it was one the strategies to generate candidates so I thought I should mention it.", "votes": null }, { "id": "1784452", "postDate": "05/11/2022 07:32:00", "content": "That's because only at the end we combined our solutions fully. I'm quite happy about the synergy we got in our solutions.
", "rawMarkdown": "That's because only at the end we combined our solutions fully. I'm quite happy about the synergy we got in our solutions.", "votes": null }, { "id": "1784459", "postDate": "05/11/2022 07:36:34", "content": "To be honest I didn't experiment too much with different periods. @wht1996 used fewer weeks with great results but when I wanted to use the same my score worsened.
\nThese are a couple of scores/candidates using postal code. I also had a version with global popularities but these were very helpful too.
\npopular_postal_1d_score\npopular_postal_3d_score\npopular_postal_7d_score\npopular_postal_age_7d_score\npopular_postal_age_90d_score\npopular_postal_30d_score\npopular_postal_cust_str_attr_7d_score\npopular_postal_cust_all_attr_7d_score\npopular_postal_age_30d_score\npopular_postal_cust_str_attr_30d_score\npopular_postal_cust_all_attr_30d_score\npopular_global_dep_7d_score\npopular_global_dep_30d_score\nThe feature extraction is the same and model is the same so a large organization like H&M shouldn't have any problems applying this to larger scale. I would say that the data in a real system is much bigger. You have information about:
\nSo the problem is already greatly simplified. Maybe with fewer customers to predict we would see more useful notebooks. Also a competition like this is also not a good representative of industry applications. Recommendations are about exploration of strategies something that you can only do when you have control over live system. For example we could create the best recommendation strategy but the data is constrained to the strategies used by H&M to a large degree so there is not way to test it accurately using historical data.
", "rawMarkdown": "The feature extraction is the same and model is the same so a large organization like H&M shouldn't have any problems applying this to larger scale. I would say that the data in a real system is much bigger. You have information about:\n- interactions with items: watching the images/videos, hovering over items, reading comments, rating\n- search information and interaction with search: sorting by popularity, price\n- seeing recommendations through different channels\n- etc\n\nSo the problem is already greatly simplified. Maybe with fewer customers to predict we would see more useful notebooks. Also a competition like this is also not a good representative of industry applications. Recommendations are about exploration of strategies something that you can only do when you have control over live system. For example we could create the best recommendation strategy but the data is constrained to the strategies used by H&M to a large degree so there is not way to test it accurately using historical data.", "votes": null }, { "id": "1784526", "postDate": "05/11/2022 08:20:14", "content": "\n\nwe can create lots of features using future information to improve accuracy
\n
I just can't imagine how we can utilize those \"leak\". So I think our prediction won't change even if we already know which customer will make purchase actually. By any chance it would be leakage, we really need to predict all customers?
", "rawMarkdown": "> we can create lots of features using future information to improve accuracy\n\nI just can't imagine how we can utilize those \"leak\". So I think our prediction won't change even if we already know which customer will make purchase actually. By any chance it would be leakage, we really need to predict all customers?", "votes": null }, { "id": "1784563", "postDate": "05/11/2022 09:02:40", "content": "It is not leakage it is optimization of testing time.
", "rawMarkdown": "It is not leakage it is optimization of testing time.", "votes": null }, { "id": "1784579", "postDate": "05/11/2022 09:24:38", "content": "Thanks for sharing great work. I'm deeply impressed about your work.
\n@wht1996 Can you explain in more detail about how you select 600 most popular products for each user?
@paweljankiewicz
\nCongratulations for your gold!!
\nFirst, I want to say that your famous post helped me a lot with thinking about two stage prediction, generating candidates followed by ranking.
I have a question.
\nCould you give me some details about “ I calculated average similarity of items vs the items that the customer bought”? Do you mean some embedding vectors like Bert for description?
How do you compute streaks? I stored the number of times a customer buy each type of product in dictionary and look up the dictionary with article features. Cudf dataframe cannot be used and it is pretty slow&takes quite a lot of memory to do this with pandas.
", "rawMarkdown": "How do you compute streaks? I stored the number of times a customer buy each type of product in dictionary and look up the dictionary with article features. Cudf dataframe cannot be used and it is pretty slow&takes quite a lot of memory to do this with pandas.", "votes": null }, { "id": "1784598", "postDate": "05/11/2022 09:50:47", "content": "just a hypothesis, we get strong features by training all the previous transactions' user-item bipartite graph,if we use next week's purchased user to train a subgraph,maybe have additional improvement.hope to hear from organizers what they think.
", "rawMarkdown": "just a hypothesis, we get strong features by training all the previous transactions' user-item bipartite graph,if we use next week's purchased user to train a subgraph,maybe have additional improvement.hope to hear from organizers what they think.", "votes": null }, { "id": "1784628", "postDate": "05/11/2022 10:34:36", "content": "Glad my post was helpful to you.
\nThe actual comparison was actually simpler. I calculated average jaccard index of attributes. So when a customer had 3 items in the history:
\nitem_1 attributes A,B,C
\nitem_2 attributes B,C,D
\nitem_3 attributes A,B,D
Then you consider a new item_4 with attributes A,B so the average jaccard is calculated like this
\nitem_1 attributes A,B,C = jaccard 0.66
\nitem_2 attributes B,C,D = jaccard 0.33
\nitem_3 attributes A,B,D = jaccard 0.66
so the average similarity is 0.55
\nI had a different version to calculate \"fuzzy\" attribute similarity based on their cooccurence in baskets so if the attribute=X cooccured with attribute=Y in 50% of the baskets its similarity is 50% not 0 or 1 like in the original.
", "rawMarkdown": "Glad my post was helpful to you.\n\nThe actual comparison was actually simpler. I calculated average jaccard index of attributes. So when a customer had 3 items in the history:\n\nitem_1 attributes A,B,C\nitem_2 attributes B,C,D\nitem_3 attributes A,B,D\n\nThen you consider a new item_4 with attributes A,B so the average jaccard is calculated like this\n\nitem_1 attributes A,B,C = jaccard 0.66\nitem_2 attributes B,C,D = jaccard 0.33\nitem_3 attributes A,B,D = jaccard 0.66\n\nso the average similarity is 0.55\n\nI had a different version to calculate \"fuzzy\" attribute similarity based on their cooccurence in baskets so if the attribute=X cooccured with attribute=Y in 50% of the baskets its similarity is 50% not 0 or 1 like in the original.", "votes": null }, { "id": "1784640", "postDate": "05/11/2022 10:44:50", "content": "This is the power of a fast language like Rust. It is a fairly simple calculation.
\nThe pseudo code was something like this:
\nfor each customer:\n for each basket (in order):\n track set of attributes that are present in the basket\n if the attribute is present add 1 count to the attribute\n if the attribute is missing reset the counter to 1\nSo at the end you have a dictionary for each customer:
\n{\n \"product_code\": {a: 1, b: 2},\n \"department\": {x: 5, c: 4},\n ...\n}\nWhere the numbers are current streak values which you need to compare to the candidate item.
", "rawMarkdown": "This is the power of a fast language like Rust. It is a fairly simple calculation.\n\nThe pseudo code was something like this:\n\n```\nfor each customer:\n for each basket (in order):\n track set of attributes that are present in the basket\n if the attribute is present add 1 count to the attribute\n if the attribute is missing reset the counter to 1\n```\n\nSo at the end you have a dictionary for each customer:\n\n```\n{\n \"product_code\": {a: 1, b: 2},\n \"department\": {x: 5, c: 4},\n ...\n}\n```\n\nWhere the numbers are current streak values which you need to compare to the candidate item.", "votes": null }, { "id": "1784714", "postDate": "05/11/2022 11:59:56", "content": "1、For example, in the sorting stage, multiple windows of different sizes will be selected. In the recall stage, I only select a window of one size, and some complex features are missing, such as itemCF related features. The recall phase uses the binary objective and the sorting phase uses the lambdarank objective
\n2、It is calculated by numerical value. For example, I know the last time the user bought this product, and I also know the average interval of the purchase time of this product. The sum of the two is the estimated time of the next purchase.
In addition to the recent top-selling prodicts, I will also select the top-selling products under the section id that the user has historically purchased
", "rawMarkdown": "In addition to the recent top-selling prodicts, I will also select the top-selling products under the section id that the user has historically purchased", "votes": null }, { "id": "1784758", "postDate": "05/11/2022 12:51:29", "content": "Thank you for the reply from you both, I have learned a lot from your solution!
", "rawMarkdown": "Thank you for the reply from you both, I have learned a lot from your solution!", "votes": null }, { "id": "1784794", "postDate": "05/11/2022 13:37:08", "content": "@onodera @paweljankiewicz - Here's an idea:
\nThere may not be leakage, but if there was a subset of customers that were especially active during the evaluation period (i.e. a certain age, certain location, early/late-season buyers etc.) then models could be optimized towards that demographic, and might not generalize well to the general population.
", "rawMarkdown": "onodera @paweljankiewicz - Here's an idea:\n\nThere may not be leakage, but if there was a subset of customers that were especially active during the evaluation period (i.e. a certain age, certain location, early/late-season buyers etc.) then models could be optimized towards that demographic, and might not generalize well to the general population.", "votes": null }, { "id": "1784848", "postDate": "05/11/2022 14:34:54", "content": "I looked into using it, but thought that it for sure wouldn't help.
\nIf you look at the last 7 days of transactions, (and ignore where postal_code=='2c29ae653a9282cce4151bd87643c907644e09541abc28ae87dea0d1f6603b1c'), there are on average only 3 transactions per postal code, and over 90% of postal_code/article_id only occurred once.
\n![\"postal-purchase-counts.jpg\"](\"https://i.postimg.cc/9082gQfB/postal-purchase-counts.jpg\")
\nSo for the most part you're getting a binary feature, and probably not that indicative of popularity.
Are you grouping postal codes in any way?
", "rawMarkdown": "I looked into using it, but thought that it for sure wouldn't help.\n\nIf you look at the last 7 days of transactions, (and ignore where `postal_code=='2c29ae653a9282cce4151bd87643c907644e09541abc28ae87dea0d1f6603b1c'`), there are on average only 3 transactions per postal code, and over 90% of postal_code/article_id only occurred once. \n[](https://postimg.cc/SYMHJqCX) \nSo for the most part you're getting a binary feature, and probably not that indicative of popularity.\n\nAre you grouping postal codes in any way?", "votes": null }, { "id": "1785491", "postDate": "05/12/2022 06:18:58", "content": "Congrats! That is an amazing result!!!! 🥳
\n@paweljankiewicz when you say that you used the last week for validation, does it mean that you had a single end-to-end pipeline? Something that took in the data, generated features, trained the model, and outputted predictions both for the validation week and the test set?
\nThank you for this wonderful write-up and the legendary post with suggestions on how to approach this earlier on :)
", "rawMarkdown": "Congrats! That is an amazing result!!!! 🥳\n\n@paweljankiewicz when you say that you used the last week for validation, does it mean that you had a single end-to-end pipeline? Something that took in the data, generated features, trained the model, and outputted predictions both for the validation week and the test set?\n\nThank you for this wonderful write-up and the legendary post with suggestions on how to approach this earlier on :)", "votes": null }, { "id": "1785529", "postDate": "05/12/2022 07:06:10", "content": "I wouldn't call it a pipeline but indeed I had a script that did all of the things you mentioned.
", "rawMarkdown": "I wouldn't call it a pipeline but indeed I had a script that did all of the things you mentioned.", "votes": null }, { "id": "1785590", "postDate": "05/12/2022 08:35:33", "content": "@paweljankiewicz
\nI understand how did you treat similarity.
Could I ask you another question?
\nYou said that you used \"image similarities using MobileNet embedding\". Do you mean that you used AVERAGE similarities to the items that customers bought?
\nI tried image KNN to EACH item in the transaction but it didn't improved the score…
Yes exactly it is average similarity of items that someone bought (or only last basket) excluding the same item. I used it for generating candidates but it wasn't a very good strategy because it didn't take into account the popularity of items.
", "rawMarkdown": "Yes exactly it is average similarity of items that someone bought (or only last basket) excluding the same item. I used it for generating candidates but it wasn't a very good strategy because it didn't take into account the popularity of items.", "votes": null }, { "id": "1785642", "postDate": "05/12/2022 09:49:17", "content": "Actually interesting thing is that I used the tool I created in Rust to find similar images - it is open source https://github.com/recoai/visual-search.
", "rawMarkdown": "Actually interesting thing is that I used the tool I created in Rust to find similar images - it is open source https://github.com/recoai/visual-search.", "votes": null }, { "id": "1785660", "postDate": "05/12/2022 10:02:47", "content": "@paweljankiewicz
\nI see. Thank you for your answering!
\nNew items' recommendation seems more difficult than transactions' one in this competition… (and maybe also in another recommendation system.)
I stared your repo. I will try it in another competition. Thanks!
", "rawMarkdown": "I stared your repo. I will try it in another competition. Thanks!", "votes": null }, { "id": "1785711", "postDate": "05/12/2022 10:47:02", "content": "Thank you for your reply! If I am understanding this right, your submissions was trained as follows:
\nI am just wondering if that is a valid approach, if retaining this last week for validation is okay on models that you use for submission? Or does it make sense to first retain the last week for validation and find good hyperparams and then train on the entire train set (without a validation set?)
\nI am hoping that training on some number of weeks, using the last week for validation and then using the model for submission (prediction on the test set) is okay, but would like to doublecheck :)
\nTraining another model on the entire set based on hyperparams found earlier introduces a crazy amount of complexity into my pipeline…
", "rawMarkdown": "Thank you for your reply! If I am understanding this right, your submissions was trained as follows:\n\n- you trained on some number of weeks\n- retained the last week for validation\n- AND used that model for submission? (predictions on the test set?)\n\nI am just wondering if that is a valid approach, if retaining this last week for validation is okay on models that you use for submission? Or does it make sense to first retain the last week for validation and find good hyperparams and then train on the entire train set (without a validation set?)\n\nI am hoping that training on some number of weeks, using the last week for validation and then using the model for submission (prediction on the test set) is okay, but would like to doublecheck :)\n\nTraining another model on the entire set based on hyperparams found earlier introduces a crazy amount of complexity into my pipeline...", "votes": null }, { "id": "1785713", "postDate": "05/12/2022 10:48:04", "content": "Apologies for the additional bother on this and huge congrats again, @paweljankiewicz! 🙂
", "rawMarkdown": "Apologies for the additional bother on this and huge congrats again, @paweljankiewicz! 🙂", "votes": null }, { "id": "1785716", "postDate": "05/12/2022 10:51:13", "content": "For submission I retrained the model including the last week.
", "rawMarkdown": "For submission I retrained the model including the last week.", "votes": null }, { "id": "1785949", "postDate": "05/12/2022 14:09:45", "content": "Congrats for your winning! I didn't expect postal code would be a key!! Thanks for sharing
", "rawMarkdown": "Congrats for your winning! I didn't expect postal code would be a key!! Thanks for sharing", "votes": null }, { "id": "1786434", "postDate": "05/12/2022 22:29:27", "content": "@paweljankiewicz and @wht1996, once again, huge congrats on your super stellar performance 🙂
\nOne question if I may ask you, please. How do you handle duplicate purchases? Do you ever predict duplicate purchases?
\nThe way I went about this is I created baskets (all purchases per week per customer) and I got rid of duplicates, but not sure if that is a good way to handle this?
\nAlso, how did you combine your solutions? Seems that you shared some candidates, etc, but how did you go about combining your results?
\nThank you very much for all your help! 🙏
", "rawMarkdown": "paweljankiewicz and @wht1996, once again, huge congrats on your super stellar performance 🙂\n\nOne question if I may ask you, please. How do you handle duplicate purchases? Do you ever predict duplicate purchases?\n\nThe way I went about this is I created baskets (all purchases per week per customer) and I got rid of duplicates, but not sure if that is a good way to handle this?\n\nAlso, how did you combine your solutions? Seems that you shared some candidates, etc, but how did you go about combining your results?\n\nThank you very much for all your help! 🙏", "votes": null }, { "id": "1786444", "postDate": "05/12/2022 22:40:52", "content": "A duplicate purchase doesn't change anything in my opinion. We considered unique pairs of user/item.
\nWe combined the results on multiple levels. For example @wht1996 found a way to use my candidates without retraining the models. It was a huge time saving in testing the solutions. All in all the cooperation was like this:
\nCongratulations both… well done!! Thanks for sharing this… really interesting!
", "rawMarkdown": "Congratulations both... well done!! Thanks for sharing this... really interesting!", "votes": null }, { "id": "1786474", "postDate": "05/12/2022 23:53:15", "content": "Hi @jontix. Long time no see :)
", "rawMarkdown": "Hi @jontix. Long time no see :)", "votes": null }, { "id": "1786475", "postDate": "05/12/2022 23:55:09", "content": "Thank you very much for your answer!!! 🙂
", "rawMarkdown": "Thank you very much for your answer!!! 🙂", "votes": null }, { "id": "1786484", "postDate": "05/13/2022 00:25:38", "content": "Way too long @paweljankiewicz! Hope you and yours are well :)
", "rawMarkdown": "Way too long @paweljankiewicz! Hope you and yours are well :)", "votes": null }, { "id": "1787076", "postDate": "05/13/2022 14:57:19", "content": "a very new beginner, can we get the winner's code to study?
", "rawMarkdown": "a very new beginner, can we get the winner's code to study?", "votes": null }, { "id": "1790056", "postDate": "05/14/2022 13:14:24", "content": "Wow congrats!
", "rawMarkdown": "Wow congrats!", "votes": null }, { "id": "1795970", "postDate": "05/20/2022 10:13:57", "content": "Oh dang, I did NOt think about availability per post code (I assumed it was all online, and uniformly available). That's a good catch!
", "rawMarkdown": "Oh dang, I did NOt think about availability per post code (I assumed it was all online, and uniformly available). That's a good catch!", "votes": null } ], "comments": [ { "id": 1783572, "author_name": "igormunizims", "author_url": "", "post_date": "05/10/2022 13:41:02", "content": "Congratulations to you both and many thanks to @paweljankiewicz, we all have learned a lot from you. When you say you used 2-3 months of transactions to train, is it mean you have 8-12 target weeks in your training set? And for candidate generation do you consider all transactions before the target week?
", "votes": null, "replies": [ { "id": 1783579, "author_name": "paweljankiewicz", "author_url": "", "post_date": "05/10/2022 13:48:56", "content": "Yes exactly. So it is not maybe precise to say that I used last 2-3 months of transactions.
\nI used all transactions until the observation date. And only for ranking model I used last 2-3 months.
Congrats @paweljankiewicz and @wht1996 for the second place! Thanks for sharing your great solution.
", "votes": null, "replies": [] }, { "id": 1783651, "author_name": "sirius81", "author_url": "", "post_date": "05/10/2022 14:53:44", "content": "Congrats! Cool work
", "votes": null, "replies": [] }, { "id": 1783688, "author_name": "ferdinandlimburg", "author_url": "", "post_date": "05/10/2022 15:37:29", "content": "Thanks for sharing and congrats!
\nDefinitely agree with your improvement suggestions. Having to predict for all users was quite a slowdown.
Thanks for sharing!
\n@paweljankiewicz
\nI think this is the first place I saw someone mentioning postal code.
\nDid you do something special that made it work for you?
Also, you mentioned mid-competition that you were using labels from about 5 months worth of data, and using only 4 months score worsened - did that change at some point?
", "votes": null, "replies": [ { "id": 1784459, "author_name": "paweljankiewicz", "author_url": "", "post_date": "05/11/2022 07:36:34", "content": "To be honest I didn't experiment too much with different periods. @wht1996 used fewer weeks with great results but when I wanted to use the same my score worsened.
\nThese are a couple of scores/candidates using postal code. I also had a version with global popularities but these were very helpful too.
\npopular_postal_1d_score\npopular_postal_3d_score\npopular_postal_7d_score\npopular_postal_age_7d_score\npopular_postal_age_90d_score\npopular_postal_30d_score\npopular_postal_cust_str_attr_7d_score\npopular_postal_cust_all_attr_7d_score\npopular_postal_age_30d_score\npopular_postal_cust_str_attr_30d_score\npopular_postal_cust_all_attr_30d_score\npopular_global_dep_7d_score\npopular_global_dep_30d_score\nI looked into using it, but thought that it for sure wouldn't help.
\nIf you look at the last 7 days of transactions, (and ignore where postal_code=='2c29ae653a9282cce4151bd87643c907644e09541abc28ae87dea0d1f6603b1c'), there are on average only 3 transactions per postal code, and over 90% of postal_code/article_id only occurred once.
\n
\nSo for the most part you're getting a binary feature, and probably not that indicative of popularity.
Are you grouping postal codes in any way?
", "votes": null, "replies": [] } ] }, { "id": 1784093, "author_name": "", "author_url": "", "post_date": "05/10/2022 23:44:08", "content": "Great work! Can you share some details on the feature engineering part written in Rust and the mini framework for parallel feature extraction?
", "votes": null, "replies": [] }, { "id": 1784118, "author_name": "lachlangillian", "author_url": "", "post_date": "05/11/2022 00:16:45", "content": "@wht1996 thank you for this write up and congratulations!
", "votes": null, "replies": [] }, { "id": 1784175, "author_name": "senkin13", "author_url": "", "post_date": "05/11/2022 01:59:42", "content": "Congrats! your improvement in LB of last 3 days was impressive and worried us.
", "votes": null, "replies": [ { "id": 1784452, "author_name": "paweljankiewicz", "author_url": "", "post_date": "05/11/2022 07:32:00", "content": "That's because only at the end we combined our solutions fully. I'm quite happy about the synergy we got in our solutions.
", "votes": null, "replies": [] } ] }, { "id": 1784186, "author_name": "lihaorocky", "author_url": "", "post_date": "05/11/2022 02:36:11", "content": "Congrats for the second place finish! Very thorough feature engeering work. Learn a lot. Thanks!
", "votes": null, "replies": [] }, { "id": 1784220, "author_name": "onodera", "author_url": "", "post_date": "05/11/2022 03:13:47", "content": "\n\nto minimize the calculation needed to submit I would suggest to provide a list of customers with transactions in the subimission week.
\n
I raised that issue here, but nobody understood what I said, and computation time increased by a factor of 20, our time and money got lost in vain 😄👍
\nhttps://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/305952#1690119
I think if we know customers who have transactions at next week, we can create lots of features using future information to improve accuracy that will make this competition just become a competition not a real recommedation problem in industry.
", "votes": null, "replies": [] }, { "id": 1784484, "author_name": "paweljankiewicz", "author_url": "", "post_date": "05/11/2022 07:52:32", "content": "The feature extraction is the same and model is the same so a large organization like H&M shouldn't have any problems applying this to larger scale. I would say that the data in a real system is much bigger. You have information about:
\nSo the problem is already greatly simplified. Maybe with fewer customers to predict we would see more useful notebooks. Also a competition like this is also not a good representative of industry applications. Recommendations are about exploration of strategies something that you can only do when you have control over live system. For example we could create the best recommendation strategy but the data is constrained to the strategies used by H&M to a large degree so there is not way to test it accurately using historical data.
", "votes": null, "replies": [] }, { "id": 1784526, "author_name": "onodera", "author_url": "", "post_date": "05/11/2022 08:20:14", "content": "\n\nwe can create lots of features using future information to improve accuracy
\n
I just can't imagine how we can utilize those \"leak\". So I think our prediction won't change even if we already know which customer will make purchase actually. By any chance it would be leakage, we really need to predict all customers?
", "votes": null, "replies": [] }, { "id": 1784563, "author_name": "paweljankiewicz", "author_url": "", "post_date": "05/11/2022 09:02:40", "content": "It is not leakage it is optimization of testing time.
", "votes": null, "replies": [] }, { "id": 1784598, "author_name": "senkin13", "author_url": "", "post_date": "05/11/2022 09:50:47", "content": "just a hypothesis, we get strong features by training all the previous transactions' user-item bipartite graph,if we use next week's purchased user to train a subgraph,maybe have additional improvement.hope to hear from organizers what they think.
", "votes": null, "replies": [] }, { "id": 1784794, "author_name": "jacob34", "author_url": "", "post_date": "05/11/2022 13:37:08", "content": "@onodera @paweljankiewicz - Here's an idea:
\nThere may not be leakage, but if there was a subset of customers that were especially active during the evaluation period (i.e. a certain age, certain location, early/late-season buyers etc.) then models could be optimized towards that demographic, and might not generalize well to the general population.
", "votes": null, "replies": [] } ] }, { "id": 1784330, "author_name": "yzheng21", "author_url": "", "post_date": "05/11/2022 05:50:00", "content": "Congrats on your 2nd place finish, especially for our new grandmaster @wht1996 .
\nTwo questions to @wht1996
\n\n\nHigher-order combinatorial features: For example, predict when the user will next purchase the product.
\n
Two questions to @paweljankiewicz
\n\n\ncustomer attributes: different combinations (what boosted my score significantly was including postal_code)
\n
\n\nimage similarities using MobileNet embedding
\n
Popularity in the last 1/3/7 days combined with postal_code + age or other item/customer attributes creates very precise candidates. Some combinations have 10-11% precision. About image embeddings I also didn't see any significant boost but it was one the strategies to generate candidates so I thought I should mention it.
", "votes": null, "replies": [] }, { "id": 1784714, "author_name": "wht1996", "author_url": "", "post_date": "05/11/2022 11:59:56", "content": "1、For example, in the sorting stage, multiple windows of different sizes will be selected. In the recall stage, I only select a window of one size, and some complex features are missing, such as itemCF related features. The recall phase uses the binary objective and the sorting phase uses the lambdarank objective
\n2、It is calculated by numerical value. For example, I know the last time the user bought this product, and I also know the average interval of the purchase time of this product. The sum of the two is the estimated time of the next purchase.
Thank you for the reply from you both, I have learned a lot from your solution!
", "votes": null, "replies": [] } ] }, { "id": 1784579, "author_name": "hyungeunjo", "author_url": "", "post_date": "05/11/2022 09:24:38", "content": "Thanks for sharing great work. I'm deeply impressed about your work.
\n@wht1996 Can you explain in more detail about how you select 600 most popular products for each user?
In addition to the recent top-selling prodicts, I will also select the top-selling products under the section id that the user has historically purchased
", "votes": null, "replies": [] } ] }, { "id": 1784587, "author_name": "hanejiyuto", "author_url": "", "post_date": "05/11/2022 09:33:43", "content": "@paweljankiewicz
\nCongratulations for your gold!!
\nFirst, I want to say that your famous post helped me a lot with thinking about two stage prediction, generating candidates followed by ranking.
I have a question.
\nCould you give me some details about “ I calculated average similarity of items vs the items that the customer bought”? Do you mean some embedding vectors like Bert for description?
Glad my post was helpful to you.
\nThe actual comparison was actually simpler. I calculated average jaccard index of attributes. So when a customer had 3 items in the history:
\nitem_1 attributes A,B,C
\nitem_2 attributes B,C,D
\nitem_3 attributes A,B,D
Then you consider a new item_4 with attributes A,B so the average jaccard is calculated like this
\nitem_1 attributes A,B,C = jaccard 0.66
\nitem_2 attributes B,C,D = jaccard 0.33
\nitem_3 attributes A,B,D = jaccard 0.66
so the average similarity is 0.55
\nI had a different version to calculate \"fuzzy\" attribute similarity based on their cooccurence in baskets so if the attribute=X cooccured with attribute=Y in 50% of the baskets its similarity is 50% not 0 or 1 like in the original.
", "votes": null, "replies": [] }, { "id": 1785590, "author_name": "hanejiyuto", "author_url": "", "post_date": "05/12/2022 08:35:33", "content": "@paweljankiewicz
\nI understand how did you treat similarity.
Could I ask you another question?
\nYou said that you used \"image similarities using MobileNet embedding\". Do you mean that you used AVERAGE similarities to the items that customers bought?
\nI tried image KNN to EACH item in the transaction but it didn't improved the score…
Yes exactly it is average similarity of items that someone bought (or only last basket) excluding the same item. I used it for generating candidates but it wasn't a very good strategy because it didn't take into account the popularity of items.
", "votes": null, "replies": [] }, { "id": 1785642, "author_name": "paweljankiewicz", "author_url": "", "post_date": "05/12/2022 09:49:17", "content": "Actually interesting thing is that I used the tool I created in Rust to find similar images - it is open source https://github.com/recoai/visual-search.
", "votes": null, "replies": [] }, { "id": 1785660, "author_name": "hanejiyuto", "author_url": "", "post_date": "05/12/2022 10:02:47", "content": "@paweljankiewicz
\nI see. Thank you for your answering!
\nNew items' recommendation seems more difficult than transactions' one in this competition… (and maybe also in another recommendation system.)
I stared your repo. I will try it in another competition. Thanks!
", "votes": null, "replies": [] } ] }, { "id": 1784597, "author_name": "homoalways", "author_url": "", "post_date": "05/11/2022 09:50:21", "content": "How do you compute streaks? I stored the number of times a customer buy each type of product in dictionary and look up the dictionary with article features. Cudf dataframe cannot be used and it is pretty slow&takes quite a lot of memory to do this with pandas.
", "votes": null, "replies": [ { "id": 1784640, "author_name": "paweljankiewicz", "author_url": "", "post_date": "05/11/2022 10:44:50", "content": "This is the power of a fast language like Rust. It is a fairly simple calculation.
\nThe pseudo code was something like this:
\nfor each customer:\n for each basket (in order):\n track set of attributes that are present in the basket\n if the attribute is present add 1 count to the attribute\n if the attribute is missing reset the counter to 1\nSo at the end you have a dictionary for each customer:
\n{\n \"product_code\": {a: 1, b: 2},\n \"department\": {x: 5, c: 4},\n ...\n}\nWhere the numbers are current streak values which you need to compare to the candidate item.
", "votes": null, "replies": [] } ] }, { "id": 1785491, "author_name": "radek1", "author_url": "", "post_date": "05/12/2022 06:18:58", "content": "Congrats! That is an amazing result!!!! 🥳
\n@paweljankiewicz when you say that you used the last week for validation, does it mean that you had a single end-to-end pipeline? Something that took in the data, generated features, trained the model, and outputted predictions both for the validation week and the test set?
\nThank you for this wonderful write-up and the legendary post with suggestions on how to approach this earlier on :)
", "votes": null, "replies": [ { "id": 1785529, "author_name": "paweljankiewicz", "author_url": "", "post_date": "05/12/2022 07:06:10", "content": "I wouldn't call it a pipeline but indeed I had a script that did all of the things you mentioned.
", "votes": null, "replies": [] }, { "id": 1785711, "author_name": "radek1", "author_url": "", "post_date": "05/12/2022 10:47:02", "content": "Thank you for your reply! If I am understanding this right, your submissions was trained as follows:
\nI am just wondering if that is a valid approach, if retaining this last week for validation is okay on models that you use for submission? Or does it make sense to first retain the last week for validation and find good hyperparams and then train on the entire train set (without a validation set?)
\nI am hoping that training on some number of weeks, using the last week for validation and then using the model for submission (prediction on the test set) is okay, but would like to doublecheck :)
\nTraining another model on the entire set based on hyperparams found earlier introduces a crazy amount of complexity into my pipeline…
", "votes": null, "replies": [] }, { "id": 1785713, "author_name": "radek1", "author_url": "", "post_date": "05/12/2022 10:48:04", "content": "Apologies for the additional bother on this and huge congrats again, @paweljankiewicz! 🙂
", "votes": null, "replies": [] }, { "id": 1785716, "author_name": "paweljankiewicz", "author_url": "", "post_date": "05/12/2022 10:51:13", "content": "For submission I retrained the model including the last week.
", "votes": null, "replies": [] } ] }, { "id": 1785949, "author_name": "bell2psy", "author_url": "", "post_date": "05/12/2022 14:09:45", "content": "Congrats for your winning! I didn't expect postal code would be a key!! Thanks for sharing
", "votes": null, "replies": [] }, { "id": 1786434, "author_name": "radek1", "author_url": "", "post_date": "05/12/2022 22:29:27", "content": "@paweljankiewicz and @wht1996, once again, huge congrats on your super stellar performance 🙂
\nOne question if I may ask you, please. How do you handle duplicate purchases? Do you ever predict duplicate purchases?
\nThe way I went about this is I created baskets (all purchases per week per customer) and I got rid of duplicates, but not sure if that is a good way to handle this?
\nAlso, how did you combine your solutions? Seems that you shared some candidates, etc, but how did you go about combining your results?
\nThank you very much for all your help! 🙏
", "votes": null, "replies": [ { "id": 1786444, "author_name": "paweljankiewicz", "author_url": "", "post_date": "05/12/2022 22:40:52", "content": "A duplicate purchase doesn't change anything in my opinion. We considered unique pairs of user/item.
\nWe combined the results on multiple levels. For example @wht1996 found a way to use my candidates without retraining the models. It was a huge time saving in testing the solutions. All in all the cooperation was like this:
\nThank you very much for your answer!!! 🙂
", "votes": null, "replies": [] } ] }, { "id": 1786453, "author_name": "jontix", "author_url": "", "post_date": "05/12/2022 22:53:54", "content": "Congratulations both… well done!! Thanks for sharing this… really interesting!
", "votes": null, "replies": [ { "id": 1786474, "author_name": "paweljankiewicz", "author_url": "", "post_date": "05/12/2022 23:53:15", "content": "Hi @jontix. Long time no see :)
", "votes": null, "replies": [] }, { "id": 1786484, "author_name": "jontix", "author_url": "", "post_date": "05/13/2022 00:25:38", "content": "Way too long @paweljankiewicz! Hope you and yours are well :)
", "votes": null, "replies": [] } ] }, { "id": 1787076, "author_name": "aliciaworld", "author_url": "", "post_date": "05/13/2022 14:57:19", "content": "a very new beginner, can we get the winner's code to study?
", "votes": null, "replies": [] }, { "id": 1790056, "author_name": "ycding2001", "author_url": "", "post_date": "05/14/2022 13:14:24", "content": "Wow congrats!
", "votes": null, "replies": [] }, { "id": 1795970, "author_name": "danofer", "author_url": "", "post_date": "05/20/2022 10:13:57", "content": "Oh dang, I did NOt think about availability per post code (I assumed it was all online, and uniformly available). That's a good catch!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1783562": "Congrats to the 1st place winners and all the participants. Thanks to the competition organizers for this interesting competition. Here is my team mate @paweljankiewicz and my solution.\n\nwht1996 solution\n-----------\nI didn't do a good job of recalling the product candidate. At the beginning, I tried recalling methods such as itemCF, but later found that the effect was not even as good as directly selecting popular products. Therefore, my solution is to directly select aboout 600 most popular products for each user (there will be some very simple strategies to refer to the user's history). Then train a lgb model and select 130 candidates for each user according to the model score, and then add all the products purchased by the user in the history as the final candidate\n\nAfter that, train another lgb model to get the final result. Compare to the lgb model in recall stage, the features will be more complex, features can be mainly divided into the following groups:\n- User basic features: including num, price, sales_channel_id.\n- Product basic features: statistics based on each attribute of the product, including times, price, age, sales_channel_id, FN, Active, club_member_status, fashion_news_frequency, last purchase time, average purchase interval.\n- User product combination features: statistics based on each attribute of the product, including num, time, sales_channel_id, last purchase time, and average purchase interval.\n- Age product combination features: products popularity under each age group.\n- User product repurchase features: whether user will repurchase product and whether product will be repurchased.\n- Higher-order combinatorial features: For example, predict when the user will next purchase the product.\n- itemCF feature: calculate the similarity of each item through itemCF, and then calculate the score that whether user will buy the product.\n\nThe score of the single model on the Leaderboard is 0.0355, after adding @paweljankiewicz candidates can be promoted to 0.0362, and the ensemble score is 0.0368.\n\n@paweljankiewicz solution\n-----------\n\nI followed my advice from the thread mostly. I'm happy that my intution was correct. Overall until the merge with wht1996 I focused on generating a diverse set of recall strategies. The most interesting part of the solution wouldn't be the logic but that I have written the feature engineering part in Rust and create a mini framework for parallel feature extraction. I thought that I could get away with much more complex rules that way and it was still pretty fast.\n\n### Candidate strategies\n\nI used all sorts of counters to measure the popularity of items in different groups:\n- customer attributes: different combinations (what boosted my score significantly was including postal_code)\n- article attributes: different combinations and then intersect with the customer history - so if a customer bought an item with certain properties I looked for popular items with the same property\nThe counters were used for different recency like: 1,3,7,30,90 days.\n\nAlso I used:\n- image similarities using MobileNet embedding\n- cooccurences (inside basket and outside = basket to basket)\n- random graph walk over item-user graph\n\nFeatures:\n- customer / article attributes\n- all sorts of similarity based measures for example I calculated average similarity of items vs the items that the customer bought\n- streaks - this was inspired by some previous competitions posts - for many article attributes I calculated streaks measuring the number of times the customer bought a product, category, section etc in row.\n\nAt the end I ended up with 1000 candidates per customer which was way too many. I had trouble to improve beyond 0.0340 but with wht1996 help I managed to improve the single model to 0.0348.\n\n### Modeling \n\nI managed to write a lambdarankmap objective for LGBMRanker (mostly by copying the code from XGBoost) and it was slightly better than lambdarank (which uses NDCG objective). For the whole competition I trained the models using 2-3 months of transactions and last week to validate.\n\n### What didn't work\n\nDeep learning models - I tried several recbole models but they weren't so good.\n\n## How to improve competitions like this\n\nA large part of the competition was about predicting the availability of the products for different postal_codes. I think this was disguised as a recommendation system competition to some extent. For future competitions I would suggest:\n- providing the state of articles - which are available and where\n- to minimize the calculation needed to submit I would suggest to provide a list of customers with transactions in the subimission week. Predicting customers which didn't have transactions was very computation heavy task. The reason is that the models are the same but creating features and predicting takes time.\n\nHow we cooperated\n-----------\n\nThe cooperation was smooth. We exchanged our solutions and candidates (with scores).", "1783572": "Congratulations to you both and many thanks to @paweljankiewicz, we all have learned a lot from you. When you say you used 2-3 months of transactions to train, is it mean you have 8-12 target weeks in your training set? And for candidate generation do you consider all transactions before the target week?", "1783579": "Yes exactly. So it is not maybe precise to say that I used last 2-3 months of transactions.\nI used all transactions until the observation date. And only for ranking model I used last 2-3 months.", "1783631": "Congrats @paweljankiewicz and @wht1996 for the second place! Thanks for sharing your great solution.", "1783651": "Congrats! Cool work", "1783688": "Thanks for sharing and congrats!\nDefinitely agree with your improvement suggestions. Having to predict for all users was quite a slowdown.", "1783899": "Thanks for sharing!\n\n@paweljankiewicz \nI think this is the first place I saw someone mentioning postal code.\nDid you do something special that made it work for you?\n\nAlso, you mentioned mid-competition that you were using labels from about 5 months worth of data, and using only 4 months score worsened - did that change at some point?", "1784093": "Great work! Can you share some details on the feature engineering part written in Rust and the mini framework for parallel feature extraction?", "1784118": "wht1996 thank you for this write up and congratulations!", "1784175": "Congrats! your improvement in LB of last 3 days was impressive and worried us.", "1784186": "Congrats for the second place finish! Very thorough feature engeering work. Learn a lot. Thanks!", "1784220": "> to minimize the calculation needed to submit I would suggest to provide a list of customers with transactions in the subimission week.\n\nI raised that issue here, but nobody understood what I said, and computation time increased by a factor of 20, our time and money got lost in vain 😄👍\nhttps://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/305952#1690119", "1784266": "I think if we know customers who have transactions at next week, we can create lots of features using future information to improve accuracy that will make this competition just become a competition not a real recommedation problem in industry.", "1784330": "Congrats on your 2nd place finish, especially for our new grandmaster @wht1996 .\n\nTwo questions to @wht1996\n\n- As you mentioned that the feature of your second lightgbm model is more complex, what features are you using in the first lightgbm model to recall 130 candidates from 600? In addition, please tell me something about how you designed those two models if you are willing to. Because their objectives are different, one is to get top K candidates and the other is to ranking\n\n> Higher-order combinatorial features: For example, predict when the user will next purchase the product.\n\n- I am slightly confused about this feature since you said \"predict\". Does this feature derive from your first lightgbm model?\n\nTwo questions to @paweljankiewicz \n\n> customer attributes: different combinations (what boosted my score significantly was including postal_code)\n\n- I have tried to do some aggregation with the postal code feature for generating candidates but the performance is worse than using popular items only, could you please tell me some detail about how to use this feature? I am very interested in this since you said it helps improve your score a lot.\n\n> image similarities using MobileNet embedding\n\n- There is no significant improvement by using embedding derived from image data in my case. How much boost do you get from using image data?", "1784449": "Popularity in the last 1/3/7 days combined with postal_code + age or other item/customer attributes creates very precise candidates. Some combinations have 10-11% precision. About image embeddings I also didn't see any significant boost but it was one the strategies to generate candidates so I thought I should mention it.", "1784452": "That's because only at the end we combined our solutions fully. I'm quite happy about the synergy we got in our solutions.", "1784459": "To be honest I didn't experiment too much with different periods. @wht1996 used fewer weeks with great results but when I wanted to use the same my score worsened.\n\nThese are a couple of scores/candidates using postal code. I also had a version with global popularities but these were very helpful too.\n\n```\npopular_postal_1d_score\npopular_postal_3d_score\npopular_postal_7d_score\npopular_postal_age_7d_score\npopular_postal_age_90d_score\npopular_postal_30d_score\npopular_postal_cust_str_attr_7d_score\npopular_postal_cust_all_attr_7d_score\npopular_postal_age_30d_score\npopular_postal_cust_str_attr_30d_score\npopular_postal_cust_all_attr_30d_score\npopular_global_dep_7d_score\npopular_global_dep_30d_score\n```", "1784484": "The feature extraction is the same and model is the same so a large organization like H&M shouldn't have any problems applying this to larger scale. I would say that the data in a real system is much bigger. You have information about:\n- interactions with items: watching the images/videos, hovering over items, reading comments, rating\n- search information and interaction with search: sorting by popularity, price\n- seeing recommendations through different channels\n- etc\n\nSo the problem is already greatly simplified. Maybe with fewer customers to predict we would see more useful notebooks. Also a competition like this is also not a good representative of industry applications. Recommendations are about exploration of strategies something that you can only do when you have control over live system. For example we could create the best recommendation strategy but the data is constrained to the strategies used by H&M to a large degree so there is not way to test it accurately using historical data.", "1784526": "> we can create lots of features using future information to improve accuracy\n\nI just can't imagine how we can utilize those \"leak\". So I think our prediction won't change even if we already know which customer will make purchase actually. By any chance it would be leakage, we really need to predict all customers?", "1784563": "It is not leakage it is optimization of testing time.", "1784579": "Thanks for sharing great work. I'm deeply impressed about your work.\n@wht1996 Can you explain in more detail about how you select 600 most popular products for each user?", "1784587": "paweljankiewicz \nCongratulations for your gold!!\nFirst, I want to say that your famous post helped me a lot with thinking about two stage prediction, generating candidates followed by ranking.\n\nI have a question.\nCould you give me some details about “ I calculated average similarity of items vs the items that the customer bought”? Do you mean some embedding vectors like Bert for description?", "1784597": "How do you compute streaks? I stored the number of times a customer buy each type of product in dictionary and look up the dictionary with article features. Cudf dataframe cannot be used and it is pretty slow&takes quite a lot of memory to do this with pandas.", "1784598": "just a hypothesis, we get strong features by training all the previous transactions' user-item bipartite graph,if we use next week's purchased user to train a subgraph,maybe have additional improvement.hope to hear from organizers what they think.", "1784628": "Glad my post was helpful to you.\n\nThe actual comparison was actually simpler. I calculated average jaccard index of attributes. So when a customer had 3 items in the history:\n\nitem_1 attributes A,B,C\nitem_2 attributes B,C,D\nitem_3 attributes A,B,D\n\nThen you consider a new item_4 with attributes A,B so the average jaccard is calculated like this\n\nitem_1 attributes A,B,C = jaccard 0.66\nitem_2 attributes B,C,D = jaccard 0.33\nitem_3 attributes A,B,D = jaccard 0.66\n\nso the average similarity is 0.55\n\nI had a different version to calculate \"fuzzy\" attribute similarity based on their cooccurence in baskets so if the attribute=X cooccured with attribute=Y in 50% of the baskets its similarity is 50% not 0 or 1 like in the original.", "1784640": "This is the power of a fast language like Rust. It is a fairly simple calculation.\n\nThe pseudo code was something like this:\n\n```\nfor each customer:\n for each basket (in order):\n track set of attributes that are present in the basket\n if the attribute is present add 1 count to the attribute\n if the attribute is missing reset the counter to 1\n```\n\nSo at the end you have a dictionary for each customer:\n\n```\n{\n \"product_code\": {a: 1, b: 2},\n \"department\": {x: 5, c: 4},\n ...\n}\n```\n\nWhere the numbers are current streak values which you need to compare to the candidate item.", "1784714": "1、For example, in the sorting stage, multiple windows of different sizes will be selected. In the recall stage, I only select a window of one size, and some complex features are missing, such as itemCF related features. The recall phase uses the binary objective and the sorting phase uses the lambdarank objective\n2、It is calculated by numerical value. For example, I know the last time the user bought this product, and I also know the average interval of the purchase time of this product. The sum of the two is the estimated time of the next purchase.", "1784718": "In addition to the recent top-selling prodicts, I will also select the top-selling products under the section id that the user has historically purchased", "1784758": "Thank you for the reply from you both, I have learned a lot from your solution!", "1784794": "onodera @paweljankiewicz - Here's an idea:\n\nThere may not be leakage, but if there was a subset of customers that were especially active during the evaluation period (i.e. a certain age, certain location, early/late-season buyers etc.) then models could be optimized towards that demographic, and might not generalize well to the general population.", "1784848": "I looked into using it, but thought that it for sure wouldn't help.\n\nIf you look at the last 7 days of transactions, (and ignore where `postal_code=='2c29ae653a9282cce4151bd87643c907644e09541abc28ae87dea0d1f6603b1c'`), there are on average only 3 transactions per postal code, and over 90% of postal_code/article_id only occurred once. \n[](https://postimg.cc/SYMHJqCX) \nSo for the most part you're getting a binary feature, and probably not that indicative of popularity.\n\nAre you grouping postal codes in any way?", "1785491": "Congrats! That is an amazing result!!!! 🥳\n\n@paweljankiewicz when you say that you used the last week for validation, does it mean that you had a single end-to-end pipeline? Something that took in the data, generated features, trained the model, and outputted predictions both for the validation week and the test set?\n\nThank you for this wonderful write-up and the legendary post with suggestions on how to approach this earlier on :)", "1785529": "I wouldn't call it a pipeline but indeed I had a script that did all of the things you mentioned.", "1785590": "paweljankiewicz \nI understand how did you treat similarity.\n\nCould I ask you another question?\nYou said that you used \"image similarities using MobileNet embedding\". Do you mean that you used AVERAGE similarities to the items that customers bought?\nI tried image KNN to EACH item in the transaction but it didn't improved the score...", "1785640": "Yes exactly it is average similarity of items that someone bought (or only last basket) excluding the same item. I used it for generating candidates but it wasn't a very good strategy because it didn't take into account the popularity of items.", "1785642": "Actually interesting thing is that I used the tool I created in Rust to find similar images - it is open source https://github.com/recoai/visual-search.", "1785660": "paweljankiewicz\nI see. Thank you for your answering!\nNew items' recommendation seems more difficult than transactions' one in this competition... (and maybe also in another recommendation system.)", "1785661": "I stared your repo. I will try it in another competition. Thanks!", "1785711": "Thank you for your reply! If I am understanding this right, your submissions was trained as follows:\n\n- you trained on some number of weeks\n- retained the last week for validation\n- AND used that model for submission? (predictions on the test set?)\n\nI am just wondering if that is a valid approach, if retaining this last week for validation is okay on models that you use for submission? Or does it make sense to first retain the last week for validation and find good hyperparams and then train on the entire train set (without a validation set?)\n\nI am hoping that training on some number of weeks, using the last week for validation and then using the model for submission (prediction on the test set) is okay, but would like to doublecheck :)\n\nTraining another model on the entire set based on hyperparams found earlier introduces a crazy amount of complexity into my pipeline...", "1785713": "Apologies for the additional bother on this and huge congrats again, @paweljankiewicz! 🙂", "1785716": "For submission I retrained the model including the last week.", "1785949": "Congrats for your winning! I didn't expect postal code would be a key!! Thanks for sharing", "1786434": "paweljankiewicz and @wht1996, once again, huge congrats on your super stellar performance 🙂\n\nOne question if I may ask you, please. How do you handle duplicate purchases? Do you ever predict duplicate purchases?\n\nThe way I went about this is I created baskets (all purchases per week per customer) and I got rid of duplicates, but not sure if that is a good way to handle this?\n\nAlso, how did you combine your solutions? Seems that you shared some candidates, etc, but how did you go about combining your results?\n\nThank you very much for all your help! 🙏", "1786444": "A duplicate purchase doesn't change anything in my opinion. We considered unique pairs of user/item.\n\nWe combined the results on multiple levels. For example @wht1996 found a way to use my candidates without retraining the models. It was a huge time saving in testing the solutions. All in all the cooperation was like this:\n- we both took our scored candidates - @wht1996 created a prescoring model which I used as a source of candidate and also score\n- at the end we also combined our predictions together for example we have created a larger set of scored candidates for submission that we ensembled together", "1786453": "Congratulations both... well done!! Thanks for sharing this... really interesting!", "1786474": "Hi @jontix. Long time no see :)", "1786475": "Thank you very much for your answer!!! 🙂", "1786484": "Way too long @paweljankiewicz! Hope you and yours are well :)", "1787076": "a very new beginner, can we get the winner's code to study?", "1790056": "Wow congrats!", "1795970": "Oh dang, I did NOt think about availability per post code (I assumed it was all online, and uniformly available). That's a good catch!" }, "source": "meta" }