First, thanks to Kaggle and H&M for providing a very interesting and well-designed challenge. I can't think of another recent competition where the winning solutions employed such a variety of different techniques. Also thanks to the Kaggle community for sharing insights, especially @paweljankiewicz for providing the basic structure that I think just about everyone used. And congratulations to the winners!
\nI don't have much experience with Recommendation systems, so this competition gave me the opportunity to learn about a lot of different techniques. There's no point in detailing my solution since the winning solutions are superior, but here are a few things I discovered that may be of use in future projects:
\nI experimented with several Python CF packages. In the end I used Implicit (https://github.com/benfred/implicit), which gives good results and is faster than some of the others. cmfrec (https://github.com/david-cortes/cmfrec) is also good. One of the nice things about the Implict package is that in has a built-in topn estimator, which most of the other packages lack. (One note about the Implicit package: the current Kaggle environment has an old version that is missing some key functionality.)
For ranking, I used LightGBM and experimented with regression, logistic regression, and lambdarank (actually rank_xendcg). Surprisingly, I got the best results from simple regression. In theory, lambdarank should work much better but I couldn't get it to perform well. I'm wondering if there's something in the hyperparamer settings that I missed?
My candidate retrieval algorithms generated an average of about 130 candidates per customer, but for training the ranking model, I sampled the negatives. In my case, the best ratio of positives-to-negatives was 1-7.
I think most of us had the experience of a major improvement in our CV score, but no improvement on the LB. One thing I noticed is that the gap between my CV and LB gradually expanded, but also became more consistent, as my scores improved. I think others have noted this as well. I wonder what would cause this effect?
My final submission was:
\nCV: 0.03572
\nLB: 0.03277
\nPB: 0.03336
First, thanks to Kaggle and H&M for providing a very interesting and well-designed challenge. I can't think of another recent competition where the winning solutions employed such a variety of different techniques. Also thanks to the Kaggle community for sharing insights, especially @paweljankiewicz for providing the basic structure that I think just about everyone used. And congratulations to the winners!
\nI don't have much experience with Recommendation systems, so this competition gave me the opportunity to learn about a lot of different techniques. There's no point in detailing my solution since the winning solutions are superior, but here are a few things I discovered that may be of use in future projects:
\nI experimented with several Python CF packages. In the end I used Implicit (https://github.com/benfred/implicit), which gives good results and is faster than some of the others. cmfrec (https://github.com/david-cortes/cmfrec) is also good. One of the nice things about the Implict package is that in has a built-in topn estimator, which most of the other packages lack. (One note about the Implicit package: the current Kaggle environment has an old version that is missing some key functionality.)
For ranking, I used LightGBM and experimented with regression, logistic regression, and lambdarank (actually rank_xendcg). Surprisingly, I got the best results from simple regression. In theory, lambdarank should work much better but I couldn't get it to perform well. I'm wondering if there's something in the hyperparamer settings that I missed?
My candidate retrieval algorithms generated an average of about 130 candidates per customer, but for training the ranking model, I sampled the negatives. In my case, the best ratio of positives-to-negatives was 1-7.
I think most of us had the experience of a major improvement in our CV score, but no improvement on the LB. One thing I noticed is that the gap between my CV and LB gradually expanded, but also became more consistent, as my scores improved. I think others have noted this as well. I wonder what would cause this effect?
My final submission was:
\nCV: 0.03572
\nLB: 0.03277
\nPB: 0.03336
I didn't suspect the repercussions of my post :). I thought that it would be nice to clear things up for people who don't know how to approach this competition. Happy that it worked for many people.
", "rawMarkdown": "I didn't suspect the repercussions of my post :). I thought that it would be nice to clear things up for people who don't know how to approach this competition. Happy that it worked for many people.", "votes": null } ], "comments": [ { "id": 1783523, "author_name": "paweljankiewicz", "author_url": "", "post_date": "05/10/2022 12:57:46", "content": "I didn't suspect the repercussions of my post :). I thought that it would be nice to clear things up for people who don't know how to approach this competition. Happy that it worked for many people.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1783221": "First, thanks to Kaggle and H&M for providing a very interesting and well-designed challenge. I can't think of another recent competition where the winning solutions employed such a variety of different techniques. Also thanks to the Kaggle community for sharing insights, especially @paweljankiewicz for providing the basic structure that I think just about everyone used. And congratulations to the winners!\n\nI don't have much experience with Recommendation systems, so this competition gave me the opportunity to learn about a lot of different techniques. There's no point in detailing my solution since the winning solutions are superior, but here are a few things I discovered that may be of use in future projects:\n\n1. I experimented with several Python CF packages. In the end I used Implicit (https://github.com/benfred/implicit), which gives good results and is faster than some of the others. cmfrec (https://github.com/david-cortes/cmfrec) is also good. One of the nice things about the Implict package is that in has a built-in topn estimator, which most of the other packages lack. (One note about the Implicit package: the current Kaggle environment has an old version that is missing some key functionality.) \n\n2. For ranking, I used LightGBM and experimented with regression, logistic regression, and lambdarank (actually rank_xendcg). Surprisingly, I got the best results from simple regression. In theory, lambdarank should work much better but I couldn't get it to perform well. I'm wondering if there's something in the hyperparamer settings that I missed?\n\n3. My candidate retrieval algorithms generated an average of about 130 candidates per customer, but for training the ranking model, I sampled the negatives. In my case, the best ratio of positives-to-negatives was 1-7.\n\n4. I think most of us had the experience of a major improvement in our CV score, but no improvement on the LB. One thing I noticed is that the gap between my CV and LB gradually expanded, but also became more consistent, as my scores improved. I think others have noted this as well. I wonder what would cause this effect?\n\nMy final submission was:\n\nCV: 0.03572\nLB: 0.03277\nPB: 0.03336", "1783523": "I didn't suspect the repercussions of my post :). I thought that it would be nice to clear things up for people who don't know how to approach this competition. Happy that it worked for many people." }, "source": "meta" }