{
  "id": 3502,
  "title": "Evaluation noise",
  "url": "/competitions/flight/discussion/3502",
  "author_name": "",
  "post_date": "2012-12-30T12:28:12.140Z",
  "votes": null,
  "comment_count": 3,
  "views": 3060,
  "content": "<p>I've been trying to estimate the uncertainty in the evaluation with my current simplistic solution. Repeating the evaluation for 14 days 100 times, the stddev of the scores is ~0.35 with a mean of ~7.2. It's possible that the uncertainty is different with\r\n other kinds of predictors. What is yours?</p>\r\n<p>Public and eventually the final scores will have this kind of noise on them.</p>",
  "messages": [
    {
      "id": "18814",
      "postDate": "12/30/2012 12:28:12",
      "content": "<p>I've been trying to estimate the uncertainty in the evaluation with my current simplistic solution. Repeating the evaluation for 14 days 100 times, the stddev of the scores is ~0.35 with a mean of ~7.2. It's possible that the uncertainty is different with\r\n other kinds of predictors. What is yours?</p>\r\n<p>Public and eventually the final scores will have this kind of noise on them.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18815",
      "postDate": "12/30/2012 14:54:57",
      "content": "<p>is 7.2 the actual rmse you get<br>\r\nwhen you do:<br>\r\n0.75 * rmse (gate<em>actual, gate</em>pred) &#43; 0.25 * rmse (runway<em>actual, runway</em>pred)</p>\r\n<p>While my submission gets 9 odd on leaderboard it gets 21 on CV. How can this be possible? Is the above formula I am using correct for evaluation?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18816",
      "postDate": "12/30/2012 14:57:03",
      "content": "<p>[quote=Black Magic;18815]</p>\r\n<p>is 7.2 the actual rmse you get<br>\r\nwhen you do:<br>\r\n0.75 * rmse (gate<em>actual, gate</em>pred) &#43; 0.25 * rmse (runway<em>actual, runway</em>pred)</p>\r\n<p>While my submission gets 9 odd on leaderboard it gets 21 on CV. How can this be possible? Is the above formula I am using correct for evaluation?</p>\r\n<p>[/quote]</p>\r\n<p>Yes, 7.2 is the 0.75/0.25 weighted rmse on the training set. I think you have a bug, I can't really help.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18817",
      "postDate": "12/30/2012 15:20:21",
      "content": "<p>Thanks Mate.</p>\r\n<p>Looks like I have messed up some features in CV. Good to know that they match.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 18815,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "12/30/2012 14:54:57",
      "content": "<p>is 7.2 the actual rmse you get<br>\r\nwhen you do:<br>\r\n0.75 * rmse (gate<em>actual, gate</em>pred) &#43; 0.25 * rmse (runway<em>actual, runway</em>pred)</p>\r\n<p>While my submission gets 9 odd on leaderboard it gets 21 on CV. How can this be possible? Is the above formula I am using correct for evaluation?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18816,
      "author_name": "melisgl",
      "author_url": "",
      "post_date": "12/30/2012 14:57:03",
      "content": "<p>[quote=Black Magic;18815]</p>\r\n<p>is 7.2 the actual rmse you get<br>\r\nwhen you do:<br>\r\n0.75 * rmse (gate<em>actual, gate</em>pred) &#43; 0.25 * rmse (runway<em>actual, runway</em>pred)</p>\r\n<p>While my submission gets 9 odd on leaderboard it gets 21 on CV. How can this be possible? Is the above formula I am using correct for evaluation?</p>\r\n<p>[/quote]</p>\r\n<p>Yes, 7.2 is the 0.75/0.25 weighted rmse on the training set. I think you have a bug, I can't really help.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18817,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "12/30/2012 15:20:21",
      "content": "<p>Thanks Mate.</p>\r\n<p>Looks like I have messed up some features in CV. Good to know that they match.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "18814": "",
    "18815": "",
    "18816": "",
    "18817": ""
  },
  "source": "meta"
}