{
  "id": 2510,
  "title": "Public Leaderboard Integrity",
  "url": "/competitions/predict-closed-questions-on-stack-overflow/discussion/2510",
  "author_name": "",
  "post_date": "2012-08-31T02:55:13.753Z",
  "votes": 1,
  "comment_count": 4,
  "views": 2139,
  "content": "<p>I have removed the current first place team's submission (score 0.11864, team stackUnderflow) since I'm suspicious that they simply looked up a large portion of the answers, and I got no response from the user when I reached out.</p>\r\n<p>While it is straigthforward to look up whether a question in the public leaderboard set is open or closed and submit, please don't do this. It makes the leaderboard less meaningful and the challenge less fun for the rest of the participants. Also, due to\r\n the two-staged structure of this contest, it won't help you win any prizes.</p>\r\n<p>If you are concerned about the validity of any participant's submission, let us know by posting in this forum thread and we can look into it further on our end.</p>",
  "messages": [
    {
      "id": "13681",
      "postDate": "08/31/2012 02:55:13",
      "content": "<p>I have removed the current first place team's submission (score 0.11864, team stackUnderflow) since I'm suspicious that they simply looked up a large portion of the answers, and I got no response from the user when I reached out.</p>\r\n<p>While it is straigthforward to look up whether a question in the public leaderboard set is open or closed and submit, please don't do this. It makes the leaderboard less meaningful and the challenge less fun for the rest of the participants. Also, due to\r\n the two-staged structure of this contest, it won't help you win any prizes.</p>\r\n<p>If you are concerned about the validity of any participant's submission, let us know by posting in this forum thread and we can look into it further on our end.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "13682",
      "postDate": "08/31/2012 03:18:28",
      "content": "<p>Hah, I was suspicious about that too. But I was so sure it's possible to significantly improve (and it probably still is) it really motivated me...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "13729",
      "postDate": "09/01/2012 00:27:29",
      "content": "<p>hi Ben, sorry I didn't get a chance to reply sooner.</p>\r\n<p>Indeed I've used the Stack Exchange API to query open/close status of questions in the public leaderborad data set.</p>\r\n<p>Obviously I realize this will not win any prizes (I did state it was &quot;just for fun&quot; in the submission comments). After all, the 2nd phase of the competition will be testing on data collected\r\n<strong>after</strong> the deadline to submit the final models..</p>\r\n<p>My goal was to get the ground truth of the current test data so that I can evaluate performance of my models offline, without being restricted by the two-submissions-per-day limit (I am aware of techniques like cross-validation, but they're so time-consuming)</p>\r\n<p>Interesting fact: there are about 500 questions that were deleted since the dataset was collected.<br>\r\nI should also mention that there is a wealth of additional information one can get from the API: like user activity on the site (has the user been asking too many poor questions), stats about tags (popular vs. niche tags), search for similar questions (duplicates),\r\n etc.. Even when deploying the final solution on the site, this info is available at the time a new question is created, and potentially can be used to make decision about whether it should be closed or not.</p>\r\n<p>Hope I didn't break many rules, I'm only using publicly available data after all :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "13730",
      "postDate": "09/01/2012 00:31:15",
      "content": "<p>That's a good idea; it would be awesome to extract more user information as extra data, though maybe it's already available in the extra data dump? I haven't looked at that yet.</p>\r\n<p>I'm just using 7/16-7/31 as a validation set. :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "13743",
      "postDate": "09/01/2012 04:17:40",
      "content": "<p>@Amro, did your actually run through a lookup for the entire public leaderboard data or was it only a partial run.\r\n</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 13682,
      "author_name": "andysloane",
      "author_url": "",
      "post_date": "08/31/2012 03:18:28",
      "content": "<p>Hah, I was suspicious about that too. But I was so sure it's possible to significantly improve (and it probably still is) it really motivated me...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 13729,
      "author_name": "stackunderflow",
      "author_url": "",
      "post_date": "09/01/2012 00:27:29",
      "content": "<p>hi Ben, sorry I didn't get a chance to reply sooner.</p>\r\n<p>Indeed I've used the Stack Exchange API to query open/close status of questions in the public leaderborad data set.</p>\r\n<p>Obviously I realize this will not win any prizes (I did state it was &quot;just for fun&quot; in the submission comments). After all, the 2nd phase of the competition will be testing on data collected\r\n<strong>after</strong> the deadline to submit the final models..</p>\r\n<p>My goal was to get the ground truth of the current test data so that I can evaluate performance of my models offline, without being restricted by the two-submissions-per-day limit (I am aware of techniques like cross-validation, but they're so time-consuming)</p>\r\n<p>Interesting fact: there are about 500 questions that were deleted since the dataset was collected.<br>\r\nI should also mention that there is a wealth of additional information one can get from the API: like user activity on the site (has the user been asking too many poor questions), stats about tags (popular vs. niche tags), search for similar questions (duplicates),\r\n etc.. Even when deploying the final solution on the site, this info is available at the time a new question is created, and potentially can be used to make decision about whether it should be closed or not.</p>\r\n<p>Hope I didn't break many rules, I'm only using publicly available data after all :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 13730,
      "author_name": "andysloane",
      "author_url": "",
      "post_date": "09/01/2012 00:31:15",
      "content": "<p>That's a good idea; it would be awesome to extract more user information as extra data, though maybe it's already available in the extra data dump? I haven't looked at that yet.</p>\r\n<p>I'm just using 7/16-7/31 as a validation set. :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 13743,
      "author_name": "hackinghabits",
      "author_url": "",
      "post_date": "09/01/2012 04:17:40",
      "content": "<p>@Amro, did your actually run through a lookup for the entire public leaderboard data or was it only a partial run.\r\n</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "13681": "",
    "13682": "",
    "13729": "",
    "13730": "",
    "13743": ""
  },
  "source": "meta"
}