{
  "id": 8247,
  "title": "Classification accuracy of 0.16837",
  "url": "/competitions/decoding-the-human-brain/discussion/8247",
  "author_name": "",
  "post_date": "2014-05-22T07:23:28.973Z",
  "votes": -1,
  "comment_count": 3,
  "views": 1394,
  "content": "<p>I was looking at the leaderboard and noticed the entry at the very bottom by S.K, obtaining a leaderboard score of&nbsp;0.16837.</p>\n<p>Unless I am misunderstanding something, this should mean that by flipping all 1's to zeroes and 0's to ones, S.K would obtain a leaderboard score of&nbsp;0.83163 due to&nbsp;this being&nbsp;a two-classification problem.</p>\n<p>Is there something here I am not seeing?</p>",
  "messages": [
    {
      "id": "46041",
      "postDate": "05/22/2014 07:23:28",
      "content": "<p>I was looking at the leaderboard and noticed the entry at the very bottom by S.K, obtaining a leaderboard score of&nbsp;0.16837.</p>\n<p>Unless I am misunderstanding something, this should mean that by flipping all 1's to zeroes and 0's to ones, S.K would obtain a leaderboard score of&nbsp;0.83163 due to&nbsp;this being&nbsp;a two-classification problem.</p>\n<p>Is there something here I am not seeing?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "46046",
      "postDate": "05/22/2014 08:04:39",
      "content": "<p>Well, it's possible but also, if you misslabeled the trials you could obtain this accuracy having a wrong method. For example,</p>\n<p>actual labels = [0 0 0 0 1 1 1 1]</p>\n<p>predicted labels = [1 2 1 2 1 2 1 2] (misslabeled)</p>\n<p>predicted labels = [0 1 0 1 0 1 0 1] or [1 0 1 0 1 0 1 0] (with correct labels)</p>\n<p>if the accuracy is computed as mean(actual_labels == predicted_labels)</p>\n<p>Wrong accuracy -&gt; 0.25. Quite good!</p>\n<p>Actual accuracy: 0.50</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "46058",
      "postDate": "05/22/2014 13:15:45",
      "content": "<p>Hi,</p>\n<p>It seems that the Kaggle automatic scoring system accepts submissions where some of the predictions are missing. Basically you can create a file submission.csv like this one (I am slightly modifying the example in the&nbsp;<a href=\"https://www.kaggle.com/c/decoding-the-human-brain/details/evaluation\">evaluation</a> page):</p>\n<p>Id,Prediction<br>22001,1<br>22002,<br>22003,<br>22004,0<br>etc.</p>\n<p>Then (no surprise) the public score will be again the number of the correct predictions divided by the size of the public part of the test set. This means that a missing prediction always decreases the score. In this way you can get whatever score you want below 0.5, like the one mentioned in this thread.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "48662",
      "postDate": "06/04/2014 21:31:16",
      "content": "<p>Using some combinatorics and optimization this could lead the evaluation system to be&nbsp;prone to hack. Isn't it?</p>\n<p>I can just try 3 times (22001,1 and all others empty), if it randomly evaluates fortyish percent of the data?</p>\n<p>If it is not random, then it is even easier to find those&nbsp;40%.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 46046,
      "author_name": "lnicalo",
      "author_url": "",
      "post_date": "05/22/2014 08:04:39",
      "content": "<p>Well, it's possible but also, if you misslabeled the trials you could obtain this accuracy having a wrong method. For example,</p>\n<p>actual labels = [0 0 0 0 1 1 1 1]</p>\n<p>predicted labels = [1 2 1 2 1 2 1 2] (misslabeled)</p>\n<p>predicted labels = [0 1 0 1 0 1 0 1] or [1 0 1 0 1 0 1 0] (with correct labels)</p>\n<p>if the accuracy is computed as mean(actual_labels == predicted_labels)</p>\n<p>Wrong accuracy -&gt; 0.25. Quite good!</p>\n<p>Actual accuracy: 0.50</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 46058,
      "author_name": "emanuele",
      "author_url": "",
      "post_date": "05/22/2014 13:15:45",
      "content": "<p>Hi,</p>\n<p>It seems that the Kaggle automatic scoring system accepts submissions where some of the predictions are missing. Basically you can create a file submission.csv like this one (I am slightly modifying the example in the&nbsp;<a href=\"https://www.kaggle.com/c/decoding-the-human-brain/details/evaluation\">evaluation</a> page):</p>\n<p>Id,Prediction<br>22001,1<br>22002,<br>22003,<br>22004,0<br>etc.</p>\n<p>Then (no surprise) the public score will be again the number of the correct predictions divided by the size of the public part of the test set. This means that a missing prediction always decreases the score. In this way you can get whatever score you want below 0.5, like the one mentioned in this thread.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 48662,
      "author_name": "bozhkov",
      "author_url": "",
      "post_date": "06/04/2014 21:31:16",
      "content": "<p>Using some combinatorics and optimization this could lead the evaluation system to be&nbsp;prone to hack. Isn't it?</p>\n<p>I can just try 3 times (22001,1 and all others empty), if it randomly evaluates fortyish percent of the data?</p>\n<p>If it is not random, then it is even easier to find those&nbsp;40%.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "46041": "",
    "46046": "",
    "46058": "",
    "48662": ""
  },
  "source": "meta"
}