{
  "id": 69996,
  "title": "LB score calculation is wrong? ",
  "url": "/competitions/inclusive-images-challenge/discussion/69996",
  "author_name": "",
  "post_date": "2018-10-29T19:25:36.500846800Z",
  "votes": 4,
  "comment_count": 7,
  "views": 0,
  "content": "<p>We noticed that our recent submission was giving a score below 0.1 on LB, where we expected it to be much higher (at least should be at similar level to our last few submits). We suspected the LB score calculation is wrong. Can host team please help verify this? Thanks</p>",
  "messages": [
    {
      "id": "412204",
      "postDate": "10/29/2018 19:25:36",
      "content": "<p>We noticed that our recent submission was giving a score below 0.1 on LB, where we expected it to be much higher (at least should be at similar level to our last few submits). We suspected the LB score calculation is wrong. Can host team please help verify this? Thanks</p>",
      "rawMarkdown": "We noticed that our recent submission was giving a score below 0.1 on LB, where we expected it to be much higher (at least should be at similar level to our last few submits). We suspected the LB score calculation is wrong. Can host team please help verify this? Thanks",
      "votes": null
    },
    {
      "id": "412210",
      "postDate": "10/29/2018 19:36:07",
      "content": "<p>I confirm, it is broken now. The solution is simple: you should sort your predictions in the same way as it sorted in sample_submission (by image_id).</p>",
      "rawMarkdown": "I confirm, it is broken now. The solution is simple: you should sort your predictions in the same way as it sorted in sample_submission (by image_id).",
      "votes": null
    },
    {
      "id": "412213",
      "postDate": "10/29/2018 19:41:00",
      "content": "<p>Thanks for confirming. </p>",
      "rawMarkdown": "Thanks for confirming.",
      "votes": null
    },
    {
      "id": "412265",
      "postDate": "10/29/2018 22:01:00",
      "content": "<p>Thank you for bringing this to our attention. We are currently investigating. In the meantime, please follow @Pavel's recommendation to ensure your submission's sort order matches the sample submission file.</p>",
      "rawMarkdown": "Thank you for bringing this to our attention. We are currently investigating. In the meantime, please follow @Pavel's recommendation to ensure your submission's sort order matches the sample submission file.",
      "votes": null
    },
    {
      "id": "412308",
      "postDate": "10/30/2018 00:43:14",
      "content": "<p>Hi Julia, it seems that the LB score calculation is also broken in PLAsTiCC Competition. </p>\n\n<p><a href=\"https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70014\">https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70014</a></p>",
      "rawMarkdown": "Hi Julia, it seems that the LB score calculation is also broken in PLAsTiCC Competition. \n\nhttps://www.kaggle.com/c/PLAsTiCC-2018/discussion/70014",
      "votes": null
    },
    {
      "id": "412366",
      "postDate": "10/30/2018 04:34:34",
      "content": "<p>So, can you re-calculate the submit csv?</p>",
      "rawMarkdown": "So, can you re-calculate the submit csv?",
      "votes": null
    },
    {
      "id": "412380",
      "postDate": "10/30/2018 05:33:50",
      "content": "<p>Thanks for bringing it to our attention! I went ahead and fixed the underlying sorting issue and re-scored the affected submissions.</p>",
      "rawMarkdown": "Thanks for bringing it to our attention! I went ahead and fixed the underlying sorting issue and re-scored the affected submissions.",
      "votes": null
    },
    {
      "id": "412386",
      "postDate": "10/30/2018 05:48:36",
      "content": "<p>This is fixed now and I rescored all affected submissions. Thanks for the report!</p>",
      "rawMarkdown": "This is fixed now and I rescored all affected submissions. Thanks for the report!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 412210,
      "author_name": "pavelost",
      "author_url": "",
      "post_date": "10/29/2018 19:36:07",
      "content": "<p>I confirm, it is broken now. The solution is simple: you should sort your predictions in the same way as it sorted in sample_submission (by image_id).</p>",
      "votes": null,
      "replies": [
        {
          "id": 412213,
          "author_name": "weimin",
          "author_url": "",
          "post_date": "10/29/2018 19:41:00",
          "content": "<p>Thanks for confirming. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 412386,
          "author_name": "jeffmoser",
          "author_url": "",
          "post_date": "10/30/2018 05:48:36",
          "content": "<p>This is fixed now and I rescored all affected submissions. Thanks for the report!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 412265,
      "author_name": "juliaelliott",
      "author_url": "",
      "post_date": "10/29/2018 22:01:00",
      "content": "<p>Thank you for bringing this to our attention. We are currently investigating. In the meantime, please follow @Pavel's recommendation to ensure your submission's sort order matches the sample submission file.</p>",
      "votes": null,
      "replies": [
        {
          "id": 412308,
          "author_name": "lfcpeng17",
          "author_url": "",
          "post_date": "10/30/2018 00:43:14",
          "content": "<p>Hi Julia, it seems that the LB score calculation is also broken in PLAsTiCC Competition. </p>\n\n<p><a href=\"https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70014\">https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70014</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 412366,
          "author_name": "finlay",
          "author_url": "",
          "post_date": "10/30/2018 04:34:34",
          "content": "<p>So, can you re-calculate the submit csv?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 412380,
          "author_name": "jeffmoser",
          "author_url": "",
          "post_date": "10/30/2018 05:33:50",
          "content": "<p>Thanks for bringing it to our attention! I went ahead and fixed the underlying sorting issue and re-scored the affected submissions.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "412204": "We noticed that our recent submission was giving a score below 0.1 on LB, where we expected it to be much higher (at least should be at similar level to our last few submits). We suspected the LB score calculation is wrong. Can host team please help verify this? Thanks",
    "412210": "I confirm, it is broken now. The solution is simple: you should sort your predictions in the same way as it sorted in sample_submission (by image_id).",
    "412213": "Thanks for confirming.",
    "412265": "Thank you for bringing this to our attention. We are currently investigating. In the meantime, please follow @Pavel's recommendation to ensure your submission's sort order matches the sample submission file.",
    "412308": "Hi Julia, it seems that the LB score calculation is also broken in PLAsTiCC Competition. \n\nhttps://www.kaggle.com/c/PLAsTiCC-2018/discussion/70014",
    "412366": "So, can you re-calculate the submit csv?",
    "412380": "Thanks for bringing it to our attention! I went ahead and fixed the underlying sorting issue and re-scored the affected submissions.",
    "412386": "This is fixed now and I rescored all affected submissions. Thanks for the report!"
  },
  "source": "meta"
}