{ "id": 320617, "title": "Unexpected low LB score on RecBole", "url": "/competitions/h-and-m-personalized-fashion-recommendations/discussion/320617", "author_name": "", "post_date": "2022-04-22T14:49:49.728130Z", "votes": null, "comment_count": 5, "views": 0, "content": "

Hello everyone.

\n

I made a notebook which uses RecBole. Surprisingly, I got map@12 = 0.148 on eval set by running trainer.evaluate(test_data). But the LB score was only 0.0124. Do you have an idea where does this huge gap come from?

\n

As you may know, RecBole is an open-source recommendation library. It’s a kind of wrapper of PyTorch and you can build about 80 models easily.

\n

Let me share my strategy.

\n\n

The map@12 is 0.148. My model learned 450,255 users' interactions. And there’re 1,371,980 users in the submission file. If I’m correct, the LB score will be about 0.0485( = 0.148 * 450255 / 1371980). I must have made a mistake somewhere.

\n

I’m wondering that there’re some problems in my “making recommendations” section. I found this awesome notebook may improve my score. But still there will be a huge gap…

\n

And my another guess is that due to my strategy, the evaluation process was done in “super easy mode”, but submission process is “extremely hard mode”.

\n

Thanks for reading and taking time!

", "messages": [ { "id": "1764510", "postDate": "04/22/2022 14:49:49", "content": "

Hello everyone.

\n

I made a notebook which uses RecBole. Surprisingly, I got map@12 = 0.148 on eval set by running trainer.evaluate(test_data). But the LB score was only 0.0124. Do you have an idea where does this huge gap come from?

\n

As you may know, RecBole is an open-source recommendation library. It’s a kind of wrapper of PyTorch and you can build about 80 models easily.

\n

Let me share my strategy.

\n\n

The map@12 is 0.148. My model learned 450,255 users' interactions. And there’re 1,371,980 users in the submission file. If I’m correct, the LB score will be about 0.0485( = 0.148 * 450255 / 1371980). I must have made a mistake somewhere.

\n

I’m wondering that there’re some problems in my “making recommendations” section. I found this awesome notebook may improve my score. But still there will be a huge gap…

\n

And my another guess is that due to my strategy, the evaluation process was done in “super easy mode”, but submission process is “extremely hard mode”.

\n

Thanks for reading and taking time!

", "rawMarkdown": "Hello everyone.\n\nI made a [notebook](https://www.kaggle.com/code/peterpetrov826/fork-of-using-recbole/notebook) which uses RecBole. Surprisingly, I got map@12 = 0.148 on eval set by running `trainer.evaluate(test_data)`. But the LB score was only 0.0124. Do you have an idea where does this huge gap come from?\n\nAs you may know, [RecBole](https://recbole.io/) is an open-source recommendation library. It’s a kind of wrapper of PyTorch and you can build about 80 models easily.\n\nLet me share my strategy.\n- It’s difficult to recommend for users who rarely shop. So I extracted users who have bought more than 2 times and use them to train my model.\n- For other users, I recommend popular products.\n- In general, popular products are more likely to be bought, and those that don't will not. So I extract product which have been bought more than 50 times and use them to train my model.\n- For other products, I don’t recommend those at all. (Sorry sewing geniuses)\n\nThe map@12 is 0.148. My model learned 450,255 users' interactions. And there’re 1,371,980 users in the submission file. If I’m correct, the LB score will be about 0.0485( = 0.148 * 450255 / 1371980). I must have made a mistake somewhere.\n\nI’m wondering that there’re some problems in my “making recommendations” section. I found [this awesome notebook](https://www.kaggle.com/code/astrung?scriptVersionId=91596049&cellId=35) may improve my score. But still there will be a huge gap…\n\nAnd my another guess is that due to my strategy, the evaluation process was done in “super easy mode”, but submission process is “extremely hard mode”. \n\nThanks for reading and taking time!", "votes": null }, { "id": "1764839", "postDate": "04/22/2022 22:21:34", "content": "

Your train data is leaking into eval data. Your eval score should not be that high. This is only happening because you have somehow included data/info from validation spilt into your training data.

", "rawMarkdown": "Your train data is leaking into eval data. Your eval score should not be that high. This is only happening because you have somehow included data/info from validation spilt into your training data.", "votes": null }, { "id": "1764937", "postDate": "04/23/2022 03:00:01", "content": "

Thank you Mayukh Bhattacharyya.

\n

I'm wondering the data leakage too.
\nIn my config below, the data are ordered by timestamp, group by user, and split them with ratio of 95% : 3% : 2%. This config looks fine for me.
\nIn data split part, I'm (heavily) replying on RecBole API. It's seems that I need to deep dive into what's going on inside RecBole.

\n
    \"eval_args\": {\n        \"split\": {\"RS\": [0.95, 0.03, 0.02]},\n        \"group_by\": \"user\",\n        \"order\": \"TO\",\n        \"mode\": \"full\"\n    },\n
", "rawMarkdown": "Thank you Mayukh Bhattacharyya.\n\nI'm wondering the data leakage too. \nIn [my config](https://www.kaggle.com/code/peterpetrov826?scriptVersionId=93702403&cellId=38) below, the data are ordered by timestamp, group by user, and split them with ratio of 95% : 3% : 2%. This config looks fine for me. \nIn data split part, I'm (heavily) replying on RecBole API. It's seems that I need to deep dive into what's going on inside RecBole.\n\n```\n \"eval_args\": {\n \"split\": {\"RS\": [0.95, 0.03, 0.02]},\n \"group_by\": \"user\",\n \"order\": \"TO\",\n \"mode\": \"full\"\n },\n```", "votes": null }, { "id": "1765894", "postDate": "04/24/2022 02:41:46", "content": "

It seems that a few people using RecBole on this competion. And the competition deadline is approaching…
\nSo I opened an issue on the official GitHub. Here’s the link below.
\nhttps://github.com/RUCAIBox/RecBole/issues/1265

\n

Any ideas and suggestions are still welcome.🤗

", "rawMarkdown": "It seems that a few people using RecBole on this competion. And the competition deadline is approaching... \nSo I opened an issue on the official GitHub. Here’s the link below.\nhttps://github.com/RUCAIBox/RecBole/issues/1265\n\nAny ideas and suggestions are still welcome.🤗", "votes": null }, { "id": "1765991", "postDate": "04/24/2022 05:10:23", "content": "

This is because the MAP for this competition is different from the MAP implemented in Recbole.
\nYou can check the formula of the MAP implemented in Recbole at the link below.
\nhttps://recbole.io/docs/recbole/recbole.evaluator.metrics.html?highlight=map#recbole.evaluator.metrics.MAP

", "rawMarkdown": "This is because the MAP for this competition is different from the MAP implemented in Recbole.\nYou can check the formula of the MAP implemented in Recbole at the link below.\nhttps://recbole.io/docs/recbole/recbole.evaluator.metrics.html?highlight=map#recbole.evaluator.metrics.MAP", "votes": null }, { "id": "1766216", "postDate": "04/24/2022 10:04:46", "content": "

Thanks for your information zeno!

\n

I should have closer look at the formula…
\nIf you're using RecBole in this competition, please let me know.

\n\n

Huge thanks again😊

", "rawMarkdown": "Thanks for your information zeno!\n\nI should have closer look at the formula...\nIf you're using RecBole in this competition, please let me know.\n- Do you define custom MAP and use it?\n- The gap is small enough?\n\nHuge thanks again😊", "votes": null } ], "comments": [ { "id": 1764839, "author_name": "mayukh18", "author_url": "", "post_date": "04/22/2022 22:21:34", "content": "

Your train data is leaking into eval data. Your eval score should not be that high. This is only happening because you have somehow included data/info from validation spilt into your training data.

", "votes": null, "replies": [ { "id": 1764937, "author_name": "peterpetrov826", "author_url": "", "post_date": "04/23/2022 03:00:01", "content": "

Thank you Mayukh Bhattacharyya.

\n

I'm wondering the data leakage too.
\nIn my config below, the data are ordered by timestamp, group by user, and split them with ratio of 95% : 3% : 2%. This config looks fine for me.
\nIn data split part, I'm (heavily) replying on RecBole API. It's seems that I need to deep dive into what's going on inside RecBole.

\n
    \"eval_args\": {\n        \"split\": {\"RS\": [0.95, 0.03, 0.02]},\n        \"group_by\": \"user\",\n        \"order\": \"TO\",\n        \"mode\": \"full\"\n    },\n
", "votes": null, "replies": [] } ] }, { "id": 1765894, "author_name": "peterpetrov826", "author_url": "", "post_date": "04/24/2022 02:41:46", "content": "

It seems that a few people using RecBole on this competion. And the competition deadline is approaching…
\nSo I opened an issue on the official GitHub. Here’s the link below.
\nhttps://github.com/RUCAIBox/RecBole/issues/1265

\n

Any ideas and suggestions are still welcome.🤗

", "votes": null, "replies": [] }, { "id": 1765991, "author_name": "zenonn", "author_url": "", "post_date": "04/24/2022 05:10:23", "content": "

This is because the MAP for this competition is different from the MAP implemented in Recbole.
\nYou can check the formula of the MAP implemented in Recbole at the link below.
\nhttps://recbole.io/docs/recbole/recbole.evaluator.metrics.html?highlight=map#recbole.evaluator.metrics.MAP

", "votes": null, "replies": [ { "id": 1766216, "author_name": "peterpetrov826", "author_url": "", "post_date": "04/24/2022 10:04:46", "content": "

Thanks for your information zeno!

\n

I should have closer look at the formula…
\nIf you're using RecBole in this competition, please let me know.

\n\n

Huge thanks again😊

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1764510": "Hello everyone.\n\nI made a [notebook](https://www.kaggle.com/code/peterpetrov826/fork-of-using-recbole/notebook) which uses RecBole. Surprisingly, I got map@12 = 0.148 on eval set by running `trainer.evaluate(test_data)`. But the LB score was only 0.0124. Do you have an idea where does this huge gap come from?\n\nAs you may know, [RecBole](https://recbole.io/) is an open-source recommendation library. It’s a kind of wrapper of PyTorch and you can build about 80 models easily.\n\nLet me share my strategy.\n- It’s difficult to recommend for users who rarely shop. So I extracted users who have bought more than 2 times and use them to train my model.\n- For other users, I recommend popular products.\n- In general, popular products are more likely to be bought, and those that don't will not. So I extract product which have been bought more than 50 times and use them to train my model.\n- For other products, I don’t recommend those at all. (Sorry sewing geniuses)\n\nThe map@12 is 0.148. My model learned 450,255 users' interactions. And there’re 1,371,980 users in the submission file. If I’m correct, the LB score will be about 0.0485( = 0.148 * 450255 / 1371980). I must have made a mistake somewhere.\n\nI’m wondering that there’re some problems in my “making recommendations” section. I found [this awesome notebook](https://www.kaggle.com/code/astrung?scriptVersionId=91596049&cellId=35) may improve my score. But still there will be a huge gap…\n\nAnd my another guess is that due to my strategy, the evaluation process was done in “super easy mode”, but submission process is “extremely hard mode”. \n\nThanks for reading and taking time!", "1764839": "Your train data is leaking into eval data. Your eval score should not be that high. This is only happening because you have somehow included data/info from validation spilt into your training data.", "1764937": "Thank you Mayukh Bhattacharyya.\n\nI'm wondering the data leakage too. \nIn [my config](https://www.kaggle.com/code/peterpetrov826?scriptVersionId=93702403&cellId=38) below, the data are ordered by timestamp, group by user, and split them with ratio of 95% : 3% : 2%. This config looks fine for me. \nIn data split part, I'm (heavily) replying on RecBole API. It's seems that I need to deep dive into what's going on inside RecBole.\n\n```\n \"eval_args\": {\n \"split\": {\"RS\": [0.95, 0.03, 0.02]},\n \"group_by\": \"user\",\n \"order\": \"TO\",\n \"mode\": \"full\"\n },\n```", "1765894": "It seems that a few people using RecBole on this competion. And the competition deadline is approaching... \nSo I opened an issue on the official GitHub. Here’s the link below.\nhttps://github.com/RUCAIBox/RecBole/issues/1265\n\nAny ideas and suggestions are still welcome.🤗", "1765991": "This is because the MAP for this competition is different from the MAP implemented in Recbole.\nYou can check the formula of the MAP implemented in Recbole at the link below.\nhttps://recbole.io/docs/recbole/recbole.evaluator.metrics.html?highlight=map#recbole.evaluator.metrics.MAP", "1766216": "Thanks for your information zeno!\n\nI should have closer look at the formula...\nIf you're using RecBole in this competition, please let me know.\n- Do you define custom MAP and use it?\n- The gap is small enough?\n\nHuge thanks again😊" }, "source": "meta" }