{
  "id": 1693,
  "title": "Final evaluation phase",
  "url": "/competitions/GestureChallenge/discussion/1693",
  "author_name": "",
  "post_date": "2012-04-07T00:30:56.100Z",
  "votes": null,
  "comment_count": 2,
  "views": 3049,
  "content": "<p>The deadline for submitting code has expired and we are making available the decryption key for the final evaluation data, which can be downloaded from the data page:</p>\r\n<p>4lkc221:</p>\r\n<p>The data may be uncompressed and decrypted with Winzip (a free trial version may be downloaded from the WInzip website).</p>\r\n<p>We encourage every participant to submit final evaluation results. Don't be discouraged if your validation data results are not among the best. It is frequent that people overfit the validation data.</p>",
  "messages": [
    {
      "id": "9992",
      "postDate": "04/07/2012 00:30:56",
      "content": "<p>The deadline for submitting code has expired and we are making available the decryption key for the final evaluation data, which can be downloaded from the data page:</p>\r\n<p>4lkc221:</p>\r\n<p>The data may be uncompressed and decrypted with Winzip (a free trial version may be downloaded from the WInzip website).</p>\r\n<p>We encourage every participant to submit final evaluation results. Don't be discouraged if your validation data results are not among the best. It is frequent that people overfit the validation data.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "10004",
      "postDate": "04/07/2012 03:28:08",
      "content": "<p>Just a heads-up. I had to delete space characters from the csv files in the final dataset to use my program unchanged. The csv files in the validation set did not have spaces in them.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "10049",
      "postDate": "04/10/2012 05:19:00",
      "content": "<p>I may have completely misunderstood the purpose of this final phase but I'm very surprised to see people still improving their scores on the validation set. I thought that these last couple of days are only for running our systems on the final data without\r\n making any changes to the algorithms or parameters as they were at Friday midnight. If changes are still allowed then this is now a contest of how well we can tune our systems to fit the final data, which is surely not what the organizers wanted. I'm sure\r\n that each of our systems is complex enough so that everyone can gain a percent or two doing some creative parameter juggling exercise, and looking at the leaderboard a percent or two can mean a lot in rankings. I hope and suggest that the final rankings will\r\n be determined on a new set of data that are so far unseen by any competing system. (Even in this case I'm a bit worried about the fairness of comparing a system as it is now with a system shown at a conference two months from now, but let the organizers find\r\n out how these extra months show up in their scoring methods.)</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 10004,
      "author_name": "anlthms",
      "author_url": "",
      "post_date": "04/07/2012 03:28:08",
      "content": "<p>Just a heads-up. I had to delete space characters from the csv files in the final dataset to use my program unchanged. The csv files in the validation set did not have spaces in them.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 10049,
      "author_name": "balazsgodeny",
      "author_url": "",
      "post_date": "04/10/2012 05:19:00",
      "content": "<p>I may have completely misunderstood the purpose of this final phase but I'm very surprised to see people still improving their scores on the validation set. I thought that these last couple of days are only for running our systems on the final data without\r\n making any changes to the algorithms or parameters as they were at Friday midnight. If changes are still allowed then this is now a contest of how well we can tune our systems to fit the final data, which is surely not what the organizers wanted. I'm sure\r\n that each of our systems is complex enough so that everyone can gain a percent or two doing some creative parameter juggling exercise, and looking at the leaderboard a percent or two can mean a lot in rankings. I hope and suggest that the final rankings will\r\n be determined on a new set of data that are so far unseen by any competing system. (Even in this case I'm a bit worried about the fairness of comparing a system as it is now with a system shown at a conference two months from now, but let the organizers find\r\n out how these extra months show up in their scoring methods.)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "9992": "",
    "10004": "",
    "10049": ""
  },
  "source": "meta"
}