{
  "id": 98017,
  "title": "RESOLVED: Bug Investigation",
  "url": "/competitions/freesound-audio-tagging-2019/discussion/98017",
  "author_name": "",
  "post_date": "2019-06-30T17:37:42.231343500Z",
  "votes": 9,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hey All,</p>\n\n<p>Since releasing the Private Leaderboard, we believe we've uncovered a bug related to dataset versioning. Specifically, if a dataset version on a kernel was changed between it's submission and our re-run, it may have resulted in a lower than expected score, or potentially an error. </p>\n\n<p>We're in the process of investigating further and re-running impacted kernels.</p>\n\n<p><em>We expect this to result in leaderboard changes, including potentially in-the-money and medal positions</em> - Apologies for the inconvenience - I'll comment on this thread when the re-run has been completed.</p>\n\n<p>Addison</p>",
  "messages": [
    {
      "id": "565277",
      "postDate": "06/30/2019 17:37:42",
      "content": "<p>Hey All,</p>\n\n<p>Since releasing the Private Leaderboard, we believe we've uncovered a bug related to dataset versioning. Specifically, if a dataset version on a kernel was changed between it's submission and our re-run, it may have resulted in a lower than expected score, or potentially an error. </p>\n\n<p>We're in the process of investigating further and re-running impacted kernels.</p>\n\n<p><em>We expect this to result in leaderboard changes, including potentially in-the-money and medal positions</em> - Apologies for the inconvenience - I'll comment on this thread when the re-run has been completed.</p>\n\n<p>Addison</p>",
      "rawMarkdown": "Hey All,\n\nSince releasing the Private Leaderboard, we believe we've uncovered a bug related to dataset versioning. Specifically, if a dataset version on a kernel was changed between it's submission and our re-run, it may have resulted in a lower than expected score, or potentially an error. \n\nWe're in the process of investigating further and re-running impacted kernels.\n\n*We expect this to result in leaderboard changes, including potentially in-the-money and medal positions* - Apologies for the inconvenience - I'll comment on this thread when the re-run has been completed.\n\nAddison",
      "votes": null
    },
    {
      "id": "565418",
      "postDate": "06/30/2019 23:21:27",
      "content": "<p>Is that means you re-run kernels with old version of dataset? Do you check the version is same with the version that is used in the 1st stage submission?</p>",
      "rawMarkdown": "Is that means you re-run kernels with old version of dataset? Do you check the version is same with the version that is used in the 1st stage submission?",
      "votes": null
    },
    {
      "id": "567952",
      "postDate": "07/04/2019 07:30:56",
      "content": "<p>Could it also affect the total number of teams reflected in final ranking? That drop from 880 to 404 after the completion of the competition?</p>",
      "rawMarkdown": "Could it also affect the total number of teams reflected in final ranking? That drop from 880 to 404 after the completion of the competition?",
      "votes": null
    },
    {
      "id": "570669",
      "postDate": "07/08/2019 15:49:13",
      "content": "<p>This has been resolved. A few times will see their position change on the leaderboard. Specifically, we'll highlight that the top two teams have switched places.</p>\n\n<p>Once again, apologies for the inconvenience, and thanks for helping us continue to make this a great competition!</p>\n\n<p>Addison</p>",
      "rawMarkdown": "This has been resolved. A few times will see their position change on the leaderboard. Specifically, we'll highlight that the top two teams have switched places.\n\nOnce again, apologies for the inconvenience, and thanks for helping us continue to make this a great competition!\n\nAddison",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 565418,
      "author_name": "osciiart",
      "author_url": "",
      "post_date": "06/30/2019 23:21:27",
      "content": "<p>Is that means you re-run kernels with old version of dataset? Do you check the version is same with the version that is used in the 1st stage submission?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 567952,
      "author_name": "yurykry",
      "author_url": "",
      "post_date": "07/04/2019 07:30:56",
      "content": "<p>Could it also affect the total number of teams reflected in final ranking? That drop from 880 to 404 after the completion of the competition?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 570669,
      "author_name": "addisonhoward",
      "author_url": "",
      "post_date": "07/08/2019 15:49:13",
      "content": "<p>This has been resolved. A few times will see their position change on the leaderboard. Specifically, we'll highlight that the top two teams have switched places.</p>\n\n<p>Once again, apologies for the inconvenience, and thanks for helping us continue to make this a great competition!</p>\n\n<p>Addison</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "565277": "Hey All,\n\nSince releasing the Private Leaderboard, we believe we've uncovered a bug related to dataset versioning. Specifically, if a dataset version on a kernel was changed between it's submission and our re-run, it may have resulted in a lower than expected score, or potentially an error. \n\nWe're in the process of investigating further and re-running impacted kernels.\n\n*We expect this to result in leaderboard changes, including potentially in-the-money and medal positions* - Apologies for the inconvenience - I'll comment on this thread when the re-run has been completed.\n\nAddison",
    "565418": "Is that means you re-run kernels with old version of dataset? Do you check the version is same with the version that is used in the 1st stage submission?",
    "567952": "Could it also affect the total number of teams reflected in final ranking? That drop from 880 to 404 after the completion of the competition?",
    "570669": "This has been resolved. A few times will see their position change on the leaderboard. Specifically, we'll highlight that the top two teams have switched places.\n\nOnce again, apologies for the inconvenience, and thanks for helping us continue to make this a great competition!\n\nAddison"
  },
  "source": "meta"
}