![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Nov-2022/recall.png\")
I thought that type always appeared in the order: click -> cart -> order.
\nHowever, from my survey(notebook), you can see that cart or order appear without click, furthermore, it is possible to have a series of the same type for each aid.
This means that we need to analyze not only co-occurrence (as is currently used for baselines) but the items which are ordered multiple times.
", "messages": [ { "id": "2016878", "postDate": "11/04/2022 10:21:27", "content": "I thought that type always appeared in the order: click -> cart -> order.
\nHowever, from my survey(notebook), you can see that cart or order appear without click, furthermore, it is possible to have a series of the same type for each aid.
This means that we need to analyze not only co-occurrence (as is currently used for baselines) but the items which are ordered multiple times.
", "rawMarkdown": "I thought that `type` always appeared in the order: `click` -> `cart` -> `order`.\nHowever, from my survey([notebook](https://www.kaggle.com/code/tj0612/eda-understand-user-behavior-via-digraph?kernelSessionId=109959389)), you can see that `cart` or `order` appear without `click`, furthermore, it is possible to have a series of the same `type` for each `aid`. \n\nThis means that we need to analyze not only co-occurrence (as is currently used for baselines) but the items which are ordered multiple times.", "votes": null }, { "id": "2017161", "postDate": "11/04/2022 14:49:19", "content": "I noticed that too. Session 12391984 starts with 32 orders then do 2 clicks and 3 orders. Why there aren't any carts in that session?
", "rawMarkdown": "I noticed that too. Session 12391984 starts with 32 orders then do 2 clicks and 3 orders. Why there aren't any carts in that session?", "votes": null }, { "id": "2017166", "postDate": "11/04/2022 15:00:19", "content": "Thank you for comment.
\nThis is just my hypothesis: it may be that only the last action within a certain threshold time is recorded.
No problem. Another thing is what would be the ground-truth of those kind of sessions? For example; session 12839499 has 31 unique orders. Ground-truth of the first time step would have 31 items but recall@20 would penalize missed 11 remaining false negatives. @pnormann
", "rawMarkdown": "No problem. Another thing is what would be the ground-truth of those kind of sessions? For example; session 12839499 has 31 unique orders. Ground-truth of the first time step would have 31 items but recall@20 would penalize missed 11 remaining false negatives. @pnormann", "votes": null }, { "id": "2017181", "postDate": "11/04/2022 15:18:55", "content": "Certainly. If you idea is correct, the highest value of LB is not 1.0.
", "rawMarkdown": "Certainly. If you idea is correct, the highest value of LB is not 1.0.", "votes": null }, { "id": "2018475", "postDate": "11/05/2022 18:15:22", "content": "Hey @gunesevitan and @tj0612, I can reassure you that you will not be penalized for cases where the ground truth contains more than 20 items. In other words, if you make 20 correct predictions in such a case, you will still get a score of 1.0.
", "rawMarkdown": "Hey @gunesevitan and @tj0612, I can reassure you that you will not be penalized for cases where the ground truth contains more than 20 items. In other words, if you make 20 correct predictions in such a case, you will still get a score of 1.0.", "votes": null }, { "id": "2018509", "postDate": "11/05/2022 18:53:44", "content": "@pnormann Can you share the formula you are using to compute this competition's metric. According to the classic recall@K formula i think you would not get LB 1.0 when the number of ground truths is greater than 20.
In the formula above, A_i is the list of ground truth for user i. And P_i is the list of prediction for user i. When the cardinality of A_i is greater than k, we see that the fraction become less than 1.0 regardless of the list of predictions.
I am particularly interested to know what score do i get if i make 19 correction predictions when the ground truth has 30 items. Will that be 19/30 and then if i make one more correct prediction, my score jumps to 30/30! Or is the denominator of your fraction min(20, |A_i|) ?
I searched OTTO's GitHub, i think they modify the recall formula by adding min(20, |A_i|) to formula denominator. Philip, can you confirmed that this competition adds min to the denominator. Here is code from GitHub here
@cdeotte You are right. The denominator for the Recall@k we use for this competition is min(20, |A_i|), as you have already discovered in our code :) I'll try to add a LaTeX formula to the evaluation page to prevent any further misconceptions.
![\"sess\"](\"https://i.ibb.co/Wxp3Fbw/Screenshot-from-2022-11-06-15-13-58.png\")
What about sessions starting with carts or orders? How do you create ground-truth clicks for their first timesteps?
", "rawMarkdown": "\n\nWhat about sessions starting with carts or orders? How do you create ground-truth clicks for their first timesteps?", "votes": null }, { "id": "2019209", "postDate": "11/06/2022 12:24:36", "content": "@gunesevitan In that case, the first cart or order event would not be part of the ground truth and would only serve as an input feature to your model. Please have a look at the ground_truth function from our GitHub repository for more details.
Great converstion here 😊 Turns out I still am calculating the score incorrectly in my code 😄 The missing piece I believe is what @cdeotte pointed out, what happens when len(GT) > 20
Thanks for this exchange of comments 🙏
", "rawMarkdown": "Great converstion here 😊 Turns out I **still** am calculating the score incorrectly in my code 😄 The missing piece I believe is what @cdeotte pointed out, what happens when `len(GT) > 20`\n\nThanks for this exchange of comments 🙏", "votes": null }, { "id": "2020937", "postDate": "11/07/2022 21:34:09", "content": "A LaTeX formula of our metric was added to the evaluation page ✔️
", "rawMarkdown": "A LaTeX formula of our metric was added to the evaluation page ✔️", "votes": null } ], "comments": [ { "id": 2017161, "author_name": "gunesevitan", "author_url": "", "post_date": "11/04/2022 14:49:19", "content": "I noticed that too. Session 12391984 starts with 32 orders then do 2 clicks and 3 orders. Why there aren't any carts in that session?
", "votes": null, "replies": [ { "id": 2017166, "author_name": "tj0612", "author_url": "", "post_date": "11/04/2022 15:00:19", "content": "Thank you for comment.
\nThis is just my hypothesis: it may be that only the last action within a certain threshold time is recorded.
No problem. Another thing is what would be the ground-truth of those kind of sessions? For example; session 12839499 has 31 unique orders. Ground-truth of the first time step would have 31 items but recall@20 would penalize missed 11 remaining false negatives. @pnormann
", "votes": null, "replies": [] }, { "id": 2017181, "author_name": "tj0612", "author_url": "", "post_date": "11/04/2022 15:18:55", "content": "Certainly. If you idea is correct, the highest value of LB is not 1.0.
", "votes": null, "replies": [] }, { "id": 2018475, "author_name": "pnormann", "author_url": "", "post_date": "11/05/2022 18:15:22", "content": "Hey @gunesevitan and @tj0612, I can reassure you that you will not be penalized for cases where the ground truth contains more than 20 items. In other words, if you make 20 correct predictions in such a case, you will still get a score of 1.0.
", "votes": null, "replies": [] }, { "id": 2018509, "author_name": "cdeotte", "author_url": "", "post_date": "11/05/2022 18:53:44", "content": "@pnormann Can you share the formula you are using to compute this competition's metric. According to the classic recall@K formula i think you would not get LB 1.0 when the number of ground truths is greater than 20.
In the formula above, A_i is the list of ground truth for user i. And P_i is the list of prediction for user i. When the cardinality of A_i is greater than k, we see that the fraction become less than 1.0 regardless of the list of predictions.
I am particularly interested to know what score do i get if i make 19 correction predictions when the ground truth has 30 items. Will that be 19/30 and then if i make one more correct prediction, my score jumps to 30/30! Or is the denominator of your fraction min(20, |A_i|) ?
I searched OTTO's GitHub, i think they modify the recall formula by adding min(20, |A_i|) to formula denominator. Philip, can you confirmed that this competition adds min to the denominator. Here is code from GitHub here
@cdeotte You are right. The denominator for the Recall@k we use for this competition is min(20, |A_i|), as you have already discovered in our code :) I'll try to add a LaTeX formula to the evaluation page to prevent any further misconceptions.
What about sessions starting with carts or orders? How do you create ground-truth clicks for their first timesteps?
", "votes": null, "replies": [] }, { "id": 2019209, "author_name": "pnormann", "author_url": "", "post_date": "11/06/2022 12:24:36", "content": "@gunesevitan In that case, the first cart or order event would not be part of the ground truth and would only serve as an input feature to your model. Please have a look at the ground_truth function from our GitHub repository for more details.
Great converstion here 😊 Turns out I still am calculating the score incorrectly in my code 😄 The missing piece I believe is what @cdeotte pointed out, what happens when len(GT) > 20
Thanks for this exchange of comments 🙏
", "votes": null, "replies": [] }, { "id": 2020937, "author_name": "pnormann", "author_url": "", "post_date": "11/07/2022 21:34:09", "content": "A LaTeX formula of our metric was added to the evaluation page ✔️
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2016878": "I thought that `type` always appeared in the order: `click` -> `cart` -> `order`.\nHowever, from my survey([notebook](https://www.kaggle.com/code/tj0612/eda-understand-user-behavior-via-digraph?kernelSessionId=109959389)), you can see that `cart` or `order` appear without `click`, furthermore, it is possible to have a series of the same `type` for each `aid`. \n\nThis means that we need to analyze not only co-occurrence (as is currently used for baselines) but the items which are ordered multiple times.", "2017161": "I noticed that too. Session 12391984 starts with 32 orders then do 2 clicks and 3 orders. Why there aren't any carts in that session?", "2017166": "Thank you for comment.\nThis is just my hypothesis: it may be that only the last action within a certain threshold time is recorded.", "2017172": "No problem. Another thing is what would be the ground-truth of those kind of sessions? For example; session 12839499 has 31 unique orders. Ground-truth of the first time step would have 31 items but recall@20 would penalize missed 11 remaining false negatives. @pnormann", "2017181": "Certainly. If you idea is correct, the highest value of LB is not 1.0.", "2018475": "Hey @gunesevitan and @tj0612, I can reassure you that you will not be penalized for cases where the ground truth contains more than 20 items. In other words, if you make 20 correct predictions in such a case, you will still get a score of 1.0.", "2018509": "pnormann Can you share the formula you are using to compute this competition's metric. According to the classic `recall@K` formula i think you would not get LB 1.0 when the number of ground truths is greater than 20.\n\n\n\nIn the formula above, `A_i` is the list of ground truth for user `i`. And `P_i` is the list of prediction for user `i`. When the cardinality of `A_i` is greater than `k`, we see that the fraction become less than 1.0 regardless of the list of predictions.", "2018510": "I am particularly interested to know what score do i get if i make 19 correction predictions when the ground truth has 30 items. Will that be `19/30` and then if i make one more correct prediction, my score jumps to `30/30`! Or is the denominator of your fraction `min(20, |A_i|)` ?", "2018522": "I searched OTTO's GitHub, i think they modify the recall formula by adding `min(20, |A_i|)` to formula denominator. Philip, can you confirmed that this competition adds `min` to the denominator. Here is code from GitHub [here][1]\n\n[1]: https://github.com/otto-de/recsys-dataset/blob/17ed879b6da075eb39f918cf1c34e9c92296c13f/src/evaluate.py#L68", "2018530": "cdeotte You are right. The denominator for the Recall@k we use for this competition is `min(20, |A_i|)`, as you have already discovered in our code :) I'll try to add a LaTeX formula to the evaluation page to prevent any further misconceptions.", "2019198": "\n\nWhat about sessions starting with carts or orders? How do you create ground-truth clicks for their first timesteps?", "2019209": "gunesevitan In that case, the first cart or order event would not be part of the ground truth and would only serve as an input feature to your model. Please have a look at the [`ground_truth`](https://github.com/otto-de/recsys-dataset/blob/main/src/labels.py#L5) function from our GitHub repository for more details.", "2020002": "Great converstion here 😊 Turns out I **still** am calculating the score incorrectly in my code 😄 The missing piece I believe is what @cdeotte pointed out, what happens when `len(GT) > 20`\n\nThanks for this exchange of comments 🙏", "2020937": "A LaTeX formula of our metric was added to the evaluation page ✔️" }, "source": "meta" }