{
  "id": 369516,
  "title": "How is test-set extracted?",
  "url": "/competitions/otto-recommender-system/discussion/369516",
  "author_name": "",
  "post_date": "2022-11-30T10:12:52.971304300Z",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>During some experiments to check whether carts and orders always follow a click on the same <code>aid</code> (you can see the result <a href=\"https://www.kaggle.com/code/sbunzini/carts-and-orders-follow-clicks\" target=\"_blank\">here</a>), a couple of questions came to my mind.</p>\n<ol>\n<li><p>How the test-set is extracted? <br>\nI'm aware that the <strong>click</strong> we need to predict corresponds to the next click event of the session, but what for <strong>carts</strong> and <strong>orders</strong>? Do they represent all the <strong>carts/orders</strong> in the complete session (so both the seen and unseen parts) or just the ones in the unseen part of the session? </p></li>\n<li><p>Related to the previous question, how are the labels extracted? <br>\nLooking at the <code>ground_truth</code> method in the <a href=\"https://github.com/otto-de/recsys-dataset/blob/main/src/labels.py\" target=\"_blank\">official GitHub repo</a>, it seems that after each <strong>cart</strong> or <strong>order</strong> event, the corresponding item no longer belongs to <strong>carts/orders</strong> labels we need to predict</p></li>\n</ol>\n<p>Take a look at the example the organizers provide:</p>\n<pre><code>{\n        \"aid\": 2,\n        \"ts\": 1661200030000,\n        \"type\": \"clicks\",\n        \"labels\": {\n            \"clicks\": 3,\n            \"carts\": [2, 3],\n            \"orders\": [2, 3]\n        }\n    },\n    {\n        \"aid\": 2,\n        \"ts\": 1661200040000,\n        \"type\": \"carts\",\n        \"labels\": {\n            \"clicks\": 3,\n            \"carts\": [3],\n            \"orders\": [2, 3]\n        }\n    },\n</code></pre>\n<p>After the <strong>cart</strong> event on <code>aid = 2</code>, that item is removed from the <code>carts</code> labels.</p>\n<p>Am I missing something? Should we avoid predicting already-seen carts and orders?</p>",
  "messages": [
    {
      "id": "2049755",
      "postDate": "11/30/2022 10:12:52",
      "content": "<p>During some experiments to check whether carts and orders always follow a click on the same <code>aid</code> (you can see the result <a href=\"https://www.kaggle.com/code/sbunzini/carts-and-orders-follow-clicks\" target=\"_blank\">here</a>), a couple of questions came to my mind.</p>\n<ol>\n<li><p>How the test-set is extracted? <br>\nI'm aware that the <strong>click</strong> we need to predict corresponds to the next click event of the session, but what for <strong>carts</strong> and <strong>orders</strong>? Do they represent all the <strong>carts/orders</strong> in the complete session (so both the seen and unseen parts) or just the ones in the unseen part of the session? </p></li>\n<li><p>Related to the previous question, how are the labels extracted? <br>\nLooking at the <code>ground_truth</code> method in the <a href=\"https://github.com/otto-de/recsys-dataset/blob/main/src/labels.py\" target=\"_blank\">official GitHub repo</a>, it seems that after each <strong>cart</strong> or <strong>order</strong> event, the corresponding item no longer belongs to <strong>carts/orders</strong> labels we need to predict</p></li>\n</ol>\n<p>Take a look at the example the organizers provide:</p>\n<pre><code>{\n        \"aid\": 2,\n        \"ts\": 1661200030000,\n        \"type\": \"clicks\",\n        \"labels\": {\n            \"clicks\": 3,\n            \"carts\": [2, 3],\n            \"orders\": [2, 3]\n        }\n    },\n    {\n        \"aid\": 2,\n        \"ts\": 1661200040000,\n        \"type\": \"carts\",\n        \"labels\": {\n            \"clicks\": 3,\n            \"carts\": [3],\n            \"orders\": [2, 3]\n        }\n    },\n</code></pre>\n<p>After the <strong>cart</strong> event on <code>aid = 2</code>, that item is removed from the <code>carts</code> labels.</p>\n<p>Am I missing something? Should we avoid predicting already-seen carts and orders?</p>",
      "rawMarkdown": "During some experiments to check whether carts and orders always follow a click on the same `aid` (you can see the result [here](https://www.kaggle.com/code/sbunzini/carts-and-orders-follow-clicks)), a couple of questions came to my mind.\n\n1. How the test-set is extracted? \nI'm aware that the **click** we need to predict corresponds to the next click event of the session, but what for **carts** and **orders**? Do they represent all the **carts/orders** in the complete session (so both the seen and unseen parts) or just the ones in the unseen part of the session? \n\n2. Related to the previous question, how are the labels extracted? \nLooking at the `ground_truth` method in the [official GitHub repo](https://github.com/otto-de/recsys-dataset/blob/main/src/labels.py), it seems that after each **cart** or **order** event, the corresponding item no longer belongs to **carts/orders** labels we need to predict\n\nTake a look at the example the organizers provide:\n```\n{\n        \"aid\": 2,\n        \"ts\": 1661200030000,\n        \"type\": \"clicks\",\n        \"labels\": {\n            \"clicks\": 3,\n            \"carts\": [2, 3],\n            \"orders\": [2, 3]\n        }\n    },\n    {\n        \"aid\": 2,\n        \"ts\": 1661200040000,\n        \"type\": \"carts\",\n        \"labels\": {\n            \"clicks\": 3,\n            \"carts\": [3],\n            \"orders\": [2, 3]\n        }\n    },\n```\nAfter the **cart** event on `aid = 2`, that item is removed from the `carts` labels.\n\nAm I missing something? Should we avoid predicting already-seen carts and orders?",
      "votes": null
    },
    {
      "id": "2050674",
      "postDate": "11/30/2022 23:13:00",
      "content": "<p>Please see here: <a href=\"https://www.kaggle.com/competitions/otto-recommender-system/discussion/363862\" target=\"_blank\">What does the image from the evaluation section mean?!</a></p>\n<p>In the sequence that you mention, at timestep 2, <code>aid == 2</code> is no longer a label, is no longer the ground truth we are to predict. These are labels from the \"future\". So yes, as you go through the session, if there no subsequent <code>cart</code> action on <code>aid</code> <code>2</code>, that <code>2</code> will get removed from the labels.</p>",
      "rawMarkdown": "Please see here: [What does the image from the evaluation section mean?!](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363862)\n\nIn the sequence that you mention, at timestep 2, `aid == 2` is no longer a label, is no longer the ground truth we are to predict. These are labels from the \"future\". So yes, as you go through the session, if there no subsequent `cart` action on `aid` `2`, that `2` will get removed from the labels.",
      "votes": null
    },
    {
      "id": "2051297",
      "postDate": "12/01/2022 10:27:00",
      "content": "<p>Hey <a href=\"https://www.kaggle.com/radek1\" target=\"_blank\">@radek1</a> ! Thanks for pointing out that discussion and for your reply! Therefore, maybe a good approach would be to assign less weight to those <code>aid</code> already in carts/orders. I will try it as soon as possible</p>",
      "rawMarkdown": "Hey @radek1 ! Thanks for pointing out that discussion and for your reply! Therefore, maybe a good approach would be to assign less weight to those `aid` already in carts/orders. I will try it as soon as possible",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2050674,
      "author_name": "radek1",
      "author_url": "",
      "post_date": "11/30/2022 23:13:00",
      "content": "<p>Please see here: <a href=\"https://www.kaggle.com/competitions/otto-recommender-system/discussion/363862\" target=\"_blank\">What does the image from the evaluation section mean?!</a></p>\n<p>In the sequence that you mention, at timestep 2, <code>aid == 2</code> is no longer a label, is no longer the ground truth we are to predict. These are labels from the \"future\". So yes, as you go through the session, if there no subsequent <code>cart</code> action on <code>aid</code> <code>2</code>, that <code>2</code> will get removed from the labels.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2051297,
          "author_name": "sbunzini",
          "author_url": "",
          "post_date": "12/01/2022 10:27:00",
          "content": "<p>Hey <a href=\"https://www.kaggle.com/radek1\" target=\"_blank\">@radek1</a> ! Thanks for pointing out that discussion and for your reply! Therefore, maybe a good approach would be to assign less weight to those <code>aid</code> already in carts/orders. I will try it as soon as possible</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2049755": "During some experiments to check whether carts and orders always follow a click on the same `aid` (you can see the result [here](https://www.kaggle.com/code/sbunzini/carts-and-orders-follow-clicks)), a couple of questions came to my mind.\n\n1. How the test-set is extracted? \nI'm aware that the **click** we need to predict corresponds to the next click event of the session, but what for **carts** and **orders**? Do they represent all the **carts/orders** in the complete session (so both the seen and unseen parts) or just the ones in the unseen part of the session? \n\n2. Related to the previous question, how are the labels extracted? \nLooking at the `ground_truth` method in the [official GitHub repo](https://github.com/otto-de/recsys-dataset/blob/main/src/labels.py), it seems that after each **cart** or **order** event, the corresponding item no longer belongs to **carts/orders** labels we need to predict\n\nTake a look at the example the organizers provide:\n```\n{\n        \"aid\": 2,\n        \"ts\": 1661200030000,\n        \"type\": \"clicks\",\n        \"labels\": {\n            \"clicks\": 3,\n            \"carts\": [2, 3],\n            \"orders\": [2, 3]\n        }\n    },\n    {\n        \"aid\": 2,\n        \"ts\": 1661200040000,\n        \"type\": \"carts\",\n        \"labels\": {\n            \"clicks\": 3,\n            \"carts\": [3],\n            \"orders\": [2, 3]\n        }\n    },\n```\nAfter the **cart** event on `aid = 2`, that item is removed from the `carts` labels.\n\nAm I missing something? Should we avoid predicting already-seen carts and orders?",
    "2050674": "Please see here: [What does the image from the evaluation section mean?!](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363862)\n\nIn the sequence that you mention, at timestep 2, `aid == 2` is no longer a label, is no longer the ground truth we are to predict. These are labels from the \"future\". So yes, as you go through the session, if there no subsequent `cart` action on `aid` `2`, that `2` will get removed from the labels.",
    "2051297": "Hey @radek1 ! Thanks for pointing out that discussion and for your reply! Therefore, maybe a good approach would be to assign less weight to those `aid` already in carts/orders. I will try it as soon as possible"
  },
  "source": "meta"
}