{
  "id": 25471,
  "title": "Questions About Offline Evaluation",
  "url": "/competitions/outbrain-click-prediction/discussion/25471",
  "author_name": "",
  "post_date": "2016-11-15T08:22:53.320Z",
  "votes": 1,
  "comment_count": 5,
  "views": 423,
  "content": "<p>The evaluation is map@12, how can I test my result offline using train_test_split? Because in training set, one display_id have only one clicked ad_id.</p>",
  "messages": [
    {
      "id": "144741",
      "postDate": "11/15/2016 08:22:53",
      "content": "<p>The evaluation is map@12, how can I test my result offline using train_test_split? Because in training set, one display_id have only one clicked ad_id.</p>",
      "rawMarkdown": "The evaluation is map@12, how can I test my result offline using train_test_split? Because in training set, one display_id have only one clicked ad_id.",
      "votes": null
    },
    {
      "id": "144823",
      "postDate": "11/15/2016 15:40:25",
      "content": "<p>Yes. There is one clicked ad_id for each display_id though there are multiple ad_ids being shown. So when we split the data for validation, we need to make sure all the rows from the display_id are present in the same sample. Then we could compute the map@12 score. </p>\n\n<p>Let us take an example to see how map@12 is calculated. Suppose say 'd' is the display_id with 12 ad_ids 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad. Now based on the order of recommendation, the map@12 score will vary. </p>\n\n<ol>\n<li>If the prediction is ['a',1,2,3,4,5,6,7,8,9,10,11], 'a' is at 1st position and so map@12 will be 1/1 = 1</li>\n<li>If the prediction is [1,'a',2,3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/2. = 0.5</li>\n<li>If the prediction is [1,2,'a',3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/3. = 0.33</li>\n</ol>\n\n<p>......</p>\n\n<ol start=\"12\">\n<li>If the prediction is [1,2,3,4,5,6,7,8,9,10,11,'a'], 'a' is at 2nd position and so map@12 will be 1/12. = 0.083</li>\n</ol>\n\n<p>Hope this helps.!</p>",
      "rawMarkdown": "Yes. There is one clicked ad_id for each display_id though there are multiple ad_ids being shown. So when we split the data for validation, we need to make sure all the rows from the display_id are present in the same sample. Then we could compute the map@12 score. \r\n\r\nLet us take an example to see how map@12 is calculated. Suppose say 'd' is the display_id with 12 ad_ids 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad. Now based on the order of recommendation, the map@12 score will vary. \r\n\r\n1. If the prediction is ['a',1,2,3,4,5,6,7,8,9,10,11], 'a' is at 1st position and so map@12 will be 1/1 = 1\r\n2. If the prediction is [1,'a',2,3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/2. = 0.5\r\n3. If the prediction is [1,2,'a',3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/3. = 0.33\r\n\r\n......\r\n\r\n12. If the prediction is [1,2,3,4,5,6,7,8,9,10,11,'a'], 'a' is at 2nd position and so map@12 will be 1/12. = 0.083\r\n\r\nHope this helps.!",
      "votes": null
    },
    {
      "id": "144959",
      "postDate": "11/16/2016 05:28:13",
      "content": "<p>[quote=SRK;144823]</p>\n\n<p>Yes. There is one clicked ad_id for each display_id though there are multiple ad_ids being shown. So when we split the data for validation, we need to make sure all the rows from the display_id are present in the same sample. Then we could compute the map@12 score. </p>\n\n<p>Let us take an example to see how map@12 is calculated. Suppose say 'd' is the display_id with 12 ad_ids 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad. Now based on the order of recommendation, the map@12 score will vary. </p>\n\n<ol>\n<li>If the prediction is ['a',1,2,3,4,5,6,7,8,9,10,11], 'a' is at 1st position and so map@12 will be 1/1 = 1</li>\n<li>If the prediction is [1,'a',2,3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/2. = 0.5</li>\n<li>If the prediction is [1,2,'a',3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/3. = 0.33</li>\n</ol>\n\n<p>......</p>\n\n<ol start=\"12\">\n<li>If the prediction is [1,2,3,4,5,6,7,8,9,10,11,'a'], 'a' is at 2nd position and so map@12 will be 1/12. = 0.083</li>\n</ol>\n\n<p>Hope this helps.!</p>\n\n<p>[/quote]</p>\n\n<p>thx, I will try this method</p>",
      "rawMarkdown": "[quote=SRK;144823]\r\n\r\nYes. There is one clicked ad_id for each display_id though there are multiple ad_ids being shown. So when we split the data for validation, we need to make sure all the rows from the display_id are present in the same sample. Then we could compute the map@12 score. \r\n\r\nLet us take an example to see how map@12 is calculated. Suppose say 'd' is the display_id with 12 ad_ids 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad. Now based on the order of recommendation, the map@12 score will vary. \r\n\r\n1. If the prediction is ['a',1,2,3,4,5,6,7,8,9,10,11], 'a' is at 1st position and so map@12 will be 1/1 = 1\r\n2. If the prediction is [1,'a',2,3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/2. = 0.5\r\n3. If the prediction is [1,2,'a',3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/3. = 0.33\r\n\r\n......\r\n\r\n12. If the prediction is [1,2,3,4,5,6,7,8,9,10,11,'a'], 'a' is at 2nd position and so map@12 will be 1/12. = 0.083\r\n\r\nHope this helps.!\r\n\r\n\r\n\r\n\r\n\r\n[/quote]\r\n\r\nthx, I will try this method",
      "votes": null
    },
    {
      "id": "145011",
      "postDate": "11/16/2016 09:56:48",
      "content": "<p>[quote=SRK;144823]</p>\n\n<p>Yes. There is one clicked ad_id for each display_id though there are multiple ad_ids being shown. So when we split the data for validation, we need to make sure all the rows from the display_id are present in the same sample. Then we could compute the map@12 score. </p>\n\n<p>Let us take an example to see how map@12 is calculated. Suppose say 'd' is the display_id with 12 ad_ids 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad. Now based on the order of recommendation, the map@12 score will vary. </p>\n\n<ol>\n<li>If the prediction is ['a',1,2,3,4,5,6,7,8,9,10,11], 'a' is at 1st position and so map@12 will be 1/1 = 1</li>\n<li>If the prediction is [1,'a',2,3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/2. = 0.5</li>\n<li>If the prediction is [1,2,'a',3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/3. = 0.33</li>\n</ol>\n\n<p>......</p>\n\n<ol start=\"12\">\n<li>If the prediction is [1,2,3,4,5,6,7,8,9,10,11,'a'], 'a' is at 2nd position and so map@12 will be 1/12. = 0.083</li>\n</ol>\n\n<p>Hope this helps.!</p>\n\n<p>[/quote]</p>\n\n<p>@SRK, I use your method to compute map@12, but I feel uncertain of this method. I use below code to compute apk@12</p>\n\n<pre><code>def apk(actual, predicted, k=12):\nif len(predicted)&gt;k:\n    predicted = predicted[:k]\nscore = 0.0\nnum_hits = 0.0\nfor i,p in enumerate(predicted):\n    if p in actual and p not in predicted[:i]:\n        num_hits += 1.0\n        score += num_hits / (i+1.0)\nif not actual:\n    return 0.0\nreturn score / min(len(actual), k)\n</code></pre>\n\n<p>I think the actual list is clicked ad in training set. For example, 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad.\nthe actual list is ['a'], is it right?</p>",
      "rawMarkdown": "[quote=SRK;144823]\r\n\r\nYes. There is one clicked ad_id for each display_id though there are multiple ad_ids being shown. So when we split the data for validation, we need to make sure all the rows from the display_id are present in the same sample. Then we could compute the map@12 score. \r\n\r\nLet us take an example to see how map@12 is calculated. Suppose say 'd' is the display_id with 12 ad_ids 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad. Now based on the order of recommendation, the map@12 score will vary. \r\n\r\n1. If the prediction is ['a',1,2,3,4,5,6,7,8,9,10,11], 'a' is at 1st position and so map@12 will be 1/1 = 1\r\n2. If the prediction is [1,'a',2,3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/2. = 0.5\r\n3. If the prediction is [1,2,'a',3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/3. = 0.33\r\n\r\n......\r\n\r\n12. If the prediction is [1,2,3,4,5,6,7,8,9,10,11,'a'], 'a' is at 2nd position and so map@12 will be 1/12. = 0.083\r\n\r\nHope this helps.!\r\n\r\n[/quote]\r\n\r\n@SRK, I use your method to compute map@12, but I feel uncertain of this method. I use below code to compute apk@12\r\n\r\n    def apk(actual, predicted, k=12):\r\n    if len(predicted)>k:\r\n        predicted = predicted[:k]\r\n    score = 0.0\r\n    num_hits = 0.0\r\n    for i,p in enumerate(predicted):\r\n        if p in actual and p not in predicted[:i]:\r\n            num_hits += 1.0\r\n            score += num_hits / (i+1.0)\r\n    if not actual:\r\n        return 0.0\r\n    return score / min(len(actual), k)\r\n\r\nI think the actual list is clicked ad in training set. For example, 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad.\r\nthe actual list is ['a'], is it right?",
      "votes": null
    },
    {
      "id": "145014",
      "postDate": "11/16/2016 10:09:00",
      "content": "<p>@didiwai : Yes you are right. Actual list is ['a']. </p>\n\n<p>Using a predicted list ['a',1,2,3,4,5,6,7,8,9,10,11] will give a score 1.0 for the function you have mentioned. </p>",
      "rawMarkdown": "didiwai : Yes you are right. Actual list is ['a']. \r\n\r\nUsing a predicted list ['a',1,2,3,4,5,6,7,8,9,10,11] will give a score 1.0 for the function you have mentioned.",
      "votes": null
    },
    {
      "id": "145017",
      "postDate": "11/16/2016 10:13:16",
      "content": "<p>[quote=SRK;145014]</p>\n\n<p>@didiwai : Yes you are right. Actual list is ['a']. </p>\n\n<p>Using a predicted list ['a',1,2,3,4,5,6,7,8,9,10,11] will give a score 1.0 for the function you have mentioned. </p>\n\n<p>[/quote]\n@SRK, Thanks for your reply.</p>",
      "rawMarkdown": "[quote=SRK;145014]\r\n\r\n@didiwai : Yes you are right. Actual list is ['a']. \r\n\r\nUsing a predicted list ['a',1,2,3,4,5,6,7,8,9,10,11] will give a score 1.0 for the function you have mentioned. \r\n\r\n[/quote]\r\n@SRK, Thanks for your reply.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 144823,
      "author_name": "sudalairajkumar",
      "author_url": "",
      "post_date": "11/15/2016 15:40:25",
      "content": "<p>Yes. There is one clicked ad_id for each display_id though there are multiple ad_ids being shown. So when we split the data for validation, we need to make sure all the rows from the display_id are present in the same sample. Then we could compute the map@12 score. </p>\n\n<p>Let us take an example to see how map@12 is calculated. Suppose say 'd' is the display_id with 12 ad_ids 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad. Now based on the order of recommendation, the map@12 score will vary. </p>\n\n<ol>\n<li>If the prediction is ['a',1,2,3,4,5,6,7,8,9,10,11], 'a' is at 1st position and so map@12 will be 1/1 = 1</li>\n<li>If the prediction is [1,'a',2,3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/2. = 0.5</li>\n<li>If the prediction is [1,2,'a',3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/3. = 0.33</li>\n</ol>\n\n<p>......</p>\n\n<ol start=\"12\">\n<li>If the prediction is [1,2,3,4,5,6,7,8,9,10,11,'a'], 'a' is at 2nd position and so map@12 will be 1/12. = 0.083</li>\n</ol>\n\n<p>Hope this helps.!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 144959,
      "author_name": "wangheyu",
      "author_url": "",
      "post_date": "11/16/2016 05:28:13",
      "content": "<p>[quote=SRK;144823]</p>\n\n<p>Yes. There is one clicked ad_id for each display_id though there are multiple ad_ids being shown. So when we split the data for validation, we need to make sure all the rows from the display_id are present in the same sample. Then we could compute the map@12 score. </p>\n\n<p>Let us take an example to see how map@12 is calculated. Suppose say 'd' is the display_id with 12 ad_ids 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad. Now based on the order of recommendation, the map@12 score will vary. </p>\n\n<ol>\n<li>If the prediction is ['a',1,2,3,4,5,6,7,8,9,10,11], 'a' is at 1st position and so map@12 will be 1/1 = 1</li>\n<li>If the prediction is [1,'a',2,3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/2. = 0.5</li>\n<li>If the prediction is [1,2,'a',3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/3. = 0.33</li>\n</ol>\n\n<p>......</p>\n\n<ol start=\"12\">\n<li>If the prediction is [1,2,3,4,5,6,7,8,9,10,11,'a'], 'a' is at 2nd position and so map@12 will be 1/12. = 0.083</li>\n</ol>\n\n<p>Hope this helps.!</p>\n\n<p>[/quote]</p>\n\n<p>thx, I will try this method</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 145011,
      "author_name": "wangheyu",
      "author_url": "",
      "post_date": "11/16/2016 09:56:48",
      "content": "<p>[quote=SRK;144823]</p>\n\n<p>Yes. There is one clicked ad_id for each display_id though there are multiple ad_ids being shown. So when we split the data for validation, we need to make sure all the rows from the display_id are present in the same sample. Then we could compute the map@12 score. </p>\n\n<p>Let us take an example to see how map@12 is calculated. Suppose say 'd' is the display_id with 12 ad_ids 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad. Now based on the order of recommendation, the map@12 score will vary. </p>\n\n<ol>\n<li>If the prediction is ['a',1,2,3,4,5,6,7,8,9,10,11], 'a' is at 1st position and so map@12 will be 1/1 = 1</li>\n<li>If the prediction is [1,'a',2,3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/2. = 0.5</li>\n<li>If the prediction is [1,2,'a',3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/3. = 0.33</li>\n</ol>\n\n<p>......</p>\n\n<ol start=\"12\">\n<li>If the prediction is [1,2,3,4,5,6,7,8,9,10,11,'a'], 'a' is at 2nd position and so map@12 will be 1/12. = 0.083</li>\n</ol>\n\n<p>Hope this helps.!</p>\n\n<p>[/quote]</p>\n\n<p>@SRK, I use your method to compute map@12, but I feel uncertain of this method. I use below code to compute apk@12</p>\n\n<pre><code>def apk(actual, predicted, k=12):\nif len(predicted)&gt;k:\n    predicted = predicted[:k]\nscore = 0.0\nnum_hits = 0.0\nfor i,p in enumerate(predicted):\n    if p in actual and p not in predicted[:i]:\n        num_hits += 1.0\n        score += num_hits / (i+1.0)\nif not actual:\n    return 0.0\nreturn score / min(len(actual), k)\n</code></pre>\n\n<p>I think the actual list is clicked ad in training set. For example, 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad.\nthe actual list is ['a'], is it right?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 145014,
      "author_name": "sudalairajkumar",
      "author_url": "",
      "post_date": "11/16/2016 10:09:00",
      "content": "<p>@didiwai : Yes you are right. Actual list is ['a']. </p>\n\n<p>Using a predicted list ['a',1,2,3,4,5,6,7,8,9,10,11] will give a score 1.0 for the function you have mentioned. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 145017,
      "author_name": "wangheyu",
      "author_url": "",
      "post_date": "11/16/2016 10:13:16",
      "content": "<p>[quote=SRK;145014]</p>\n\n<p>@didiwai : Yes you are right. Actual list is ['a']. </p>\n\n<p>Using a predicted list ['a',1,2,3,4,5,6,7,8,9,10,11] will give a score 1.0 for the function you have mentioned. </p>\n\n<p>[/quote]\n@SRK, Thanks for your reply.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "144741": "The evaluation is map@12, how can I test my result offline using train_test_split? Because in training set, one display_id have only one clicked ad_id.",
    "144823": "Yes. There is one clicked ad_id for each display_id though there are multiple ad_ids being shown. So when we split the data for validation, we need to make sure all the rows from the display_id are present in the same sample. Then we could compute the map@12 score. \r\n\r\nLet us take an example to see how map@12 is calculated. Suppose say 'd' is the display_id with 12 ad_ids 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad. Now based on the order of recommendation, the map@12 score will vary. \r\n\r\n1. If the prediction is ['a',1,2,3,4,5,6,7,8,9,10,11], 'a' is at 1st position and so map@12 will be 1/1 = 1\r\n2. If the prediction is [1,'a',2,3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/2. = 0.5\r\n3. If the prediction is [1,2,'a',3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/3. = 0.33\r\n\r\n......\r\n\r\n12. If the prediction is [1,2,3,4,5,6,7,8,9,10,11,'a'], 'a' is at 2nd position and so map@12 will be 1/12. = 0.083\r\n\r\nHope this helps.!",
    "144959": "[quote=SRK;144823]\r\n\r\nYes. There is one clicked ad_id for each display_id though there are multiple ad_ids being shown. So when we split the data for validation, we need to make sure all the rows from the display_id are present in the same sample. Then we could compute the map@12 score. \r\n\r\nLet us take an example to see how map@12 is calculated. Suppose say 'd' is the display_id with 12 ad_ids 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad. Now based on the order of recommendation, the map@12 score will vary. \r\n\r\n1. If the prediction is ['a',1,2,3,4,5,6,7,8,9,10,11], 'a' is at 1st position and so map@12 will be 1/1 = 1\r\n2. If the prediction is [1,'a',2,3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/2. = 0.5\r\n3. If the prediction is [1,2,'a',3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/3. = 0.33\r\n\r\n......\r\n\r\n12. If the prediction is [1,2,3,4,5,6,7,8,9,10,11,'a'], 'a' is at 2nd position and so map@12 will be 1/12. = 0.083\r\n\r\nHope this helps.!\r\n\r\n\r\n\r\n\r\n\r\n[/quote]\r\n\r\nthx, I will try this method",
    "145011": "[quote=SRK;144823]\r\n\r\nYes. There is one clicked ad_id for each display_id though there are multiple ad_ids being shown. So when we split the data for validation, we need to make sure all the rows from the display_id are present in the same sample. Then we could compute the map@12 score. \r\n\r\nLet us take an example to see how map@12 is calculated. Suppose say 'd' is the display_id with 12 ad_ids 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad. Now based on the order of recommendation, the map@12 score will vary. \r\n\r\n1. If the prediction is ['a',1,2,3,4,5,6,7,8,9,10,11], 'a' is at 1st position and so map@12 will be 1/1 = 1\r\n2. If the prediction is [1,'a',2,3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/2. = 0.5\r\n3. If the prediction is [1,2,'a',3,4,5,6,7,8,9,10,11], 'a' is at 2nd position and so map@12 will be 1/3. = 0.33\r\n\r\n......\r\n\r\n12. If the prediction is [1,2,3,4,5,6,7,8,9,10,11,'a'], 'a' is at 2nd position and so map@12 will be 1/12. = 0.083\r\n\r\nHope this helps.!\r\n\r\n[/quote]\r\n\r\n@SRK, I use your method to compute map@12, but I feel uncertain of this method. I use below code to compute apk@12\r\n\r\n    def apk(actual, predicted, k=12):\r\n    if len(predicted)>k:\r\n        predicted = predicted[:k]\r\n    score = 0.0\r\n    num_hits = 0.0\r\n    for i,p in enumerate(predicted):\r\n        if p in actual and p not in predicted[:i]:\r\n            num_hits += 1.0\r\n            score += num_hits / (i+1.0)\r\n    if not actual:\r\n        return 0.0\r\n    return score / min(len(actual), k)\r\n\r\nI think the actual list is clicked ad in training set. For example, 'a',1,2,3,4,5,6,7,8,9,10,11 with 'a' being the clicked ad.\r\nthe actual list is ['a'], is it right?",
    "145014": "didiwai : Yes you are right. Actual list is ['a']. \r\n\r\nUsing a predicted list ['a',1,2,3,4,5,6,7,8,9,10,11] will give a score 1.0 for the function you have mentioned.",
    "145017": "[quote=SRK;145014]\r\n\r\n@didiwai : Yes you are right. Actual list is ['a']. \r\n\r\nUsing a predicted list ['a',1,2,3,4,5,6,7,8,9,10,11] will give a score 1.0 for the function you have mentioned. \r\n\r\n[/quote]\r\n@SRK, Thanks for your reply."
  },
  "source": "meta"
}