{
  "id": 7805,
  "title": "Beating the Benchmark with Hinge Loss (~0.66100)",
  "url": "/competitions/decoding-the-human-brain/discussion/7805",
  "author_name": "",
  "post_date": "2014-04-22T01:04:52.830Z",
  "votes": 25,
  "comment_count": 17,
  "views": 9103,
  "content": "<p>Another spectacular competition with amazing data sets! Using the <a href=\"https://github.com/FBK-NILab/DecMeg2014\">provided benchmark code</a>&nbsp;we repeat a similar process, but we use hinge loss in Vowpal Wabbit instead of logistic regression in sklearn. This improves the benchmark score and has an added benefit of using no more than 80MB of memory during training (vs. 10GB for in-memory logistic regression).</p>\n<p>Scripts are provided to:</p>\n<ul>\n<li><span style=\"line-height: 1.4\">M</span><span style=\"line-height: 1.4\">unge the data from .mat files to .vw (vowpal wabbit) files.</span></li>\n<li><span style=\"line-height: 1.4\">Generate a Kaggle submission from the prediction files made by VW.</span></li>\n<li><span style=\"line-height: 1.4\">Plot brain activities on a graph</span></li>\n</ul>\n<p><span style=\"line-height: 1.4\"><img src=\"http://i.imgur.com/nBw4O6l.png\" alt=\"Activity plot\" width=\"600\" height=\"450\"></span></p>\n<p><span style=\"line-height: 1.4\">The Vowpal Wabbit train command:</span></p>\n<p><code><span style=\"line-height: 1.4\">./vw face.train.vw -c -k --passes 60 --loss_function hinge --binary -f face.model.vw</span></code></p>\n<p><span style=\"line-height: 1.4\">All up-to-date code available in&nbsp;my <a href=\"https://github.com/MLWave/Kaggle-decoding-the-human-brain\">Github repo</a>. In-depth tutorial and code description on&nbsp;<a href=\"http://mlwave.com/predict-visual-stimuli-from-human-brain-activity/\">MLWave.com</a>.</span></p>\n<p><span style=\"line-height: 1.4\">Happy competition!</span></p>",
  "messages": [
    {
      "id": "42659",
      "postDate": "04/22/2014 01:04:52",
      "content": "<p>Another spectacular competition with amazing data sets! Using the <a href=\"https://github.com/FBK-NILab/DecMeg2014\">provided benchmark code</a>&nbsp;we repeat a similar process, but we use hinge loss in Vowpal Wabbit instead of logistic regression in sklearn. This improves the benchmark score and has an added benefit of using no more than 80MB of memory during training (vs. 10GB for in-memory logistic regression).</p>\n<p>Scripts are provided to:</p>\n<ul>\n<li><span style=\"line-height: 1.4\">M</span><span style=\"line-height: 1.4\">unge the data from .mat files to .vw (vowpal wabbit) files.</span></li>\n<li><span style=\"line-height: 1.4\">Generate a Kaggle submission from the prediction files made by VW.</span></li>\n<li><span style=\"line-height: 1.4\">Plot brain activities on a graph</span></li>\n</ul>\n<p><span style=\"line-height: 1.4\"><img src=\"http://i.imgur.com/nBw4O6l.png\" alt=\"Activity plot\" width=\"600\" height=\"450\"></span></p>\n<p><span style=\"line-height: 1.4\">The Vowpal Wabbit train command:</span></p>\n<p><code><span style=\"line-height: 1.4\">./vw face.train.vw -c -k --passes 60 --loss_function hinge --binary -f face.model.vw</span></code></p>\n<p><span style=\"line-height: 1.4\">All up-to-date code available in&nbsp;my <a href=\"https://github.com/MLWave/Kaggle-decoding-the-human-brain\">Github repo</a>. In-depth tutorial and code description on&nbsp;<a href=\"http://mlwave.com/predict-visual-stimuli-from-human-brain-activity/\">MLWave.com</a>.</span></p>\n<p><span style=\"line-height: 1.4\">Happy competition!</span></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42776",
      "postDate": "04/23/2014 02:15:03",
      "content": "<p>Great! Thanks @Triskelion</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42868",
      "postDate": "04/23/2014 20:19:57",
      "content": "<p><em>&quot;Each trial consists of 1.5 seconds of MEG recording (starting 0.5sec before the stimulus starts) &quot; </em></p>\n<p>Should this be understood as, the first 0.5 s nothing happens and than the stimulus starts.&nbsp; would it not then be better to use from ex. 0.25 - 0.75 ? Or have I misunderstood somthing</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42873",
      "postDate": "04/23/2014 21:26:01",
      "content": "<p>Actually yes, nothing happens before the start of stimulus, but someone may want to use pre-stimulus data for noise estimation, baseline correction, or any other intelligent idea for normalization to improve the decoding accuracy.&nbsp;About the post-stimulus data, it is up to you to decide...&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42967",
      "postDate": "04/24/2014 18:54:16",
      "content": "<p>And how long does the stimulus last? I can't seem to find this detail anywhere.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42991",
      "postDate": "04/25/2014 07:06:48",
      "content": "<p>Hi,</p>\n<p>According to the article of the study from which the dataset is taken, the stimulus is presented for a random duration between 0.8sec and 1.0sec.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "43219",
      "postDate": "04/28/2014 14:38:24",
      "content": "<p>Thank you for the code and directions. Just for the sake of curiosity I took your code and played around with feature generation phase and got these results so far (based on my submissions using vw):</p>\n<p>0.66610 when I did 120:305 features</p>\n<p>0.66327 with 100:305 features</p>\n<p>0.66100 with 1:305 features</p>\n<p>0.65193 with 140:305 features</p>\n<p>0.64626 with 0-0.4 seconds all features</p>\n<p>0.63889 with 0.1 - 0.3 seconds all features</p>\n<p>0.58503 without standardization</p>\n<p>0.55215 with single subject trained models with majority voting</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "44361",
      "postDate": "05/11/2014 06:51:23",
      "content": "<p><del>Triskelion, what ID (subject, trial) is shown on your plots?</del></p>\n<p>..I reproduced these plots. ID=01000.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "44482",
      "postDate": "05/13/2014 19:22:28",
      "content": "<p>EIGSI, When you say 0-.4 seconds, do you mean you trained on the data 0-.4 seconds after the stimulus, or do you mean the data that happened before the stimulus (that started at .5 seconds)?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "44769",
      "postDate": "05/18/2014 07:10:15",
      "content": "<p>Triskelion,</p>\n<p>Thanks for providing the starter code!. I ran the code as is, on a 64 bit windows machine but cannot see the 0.661 loss figure you mention-i see 0.5731. Any clues as to what i should be looking at? I mention the machine configuration because&nbsp;the VW default windows instruction do not work for latest VW &nbsp;and i had to patch things up with different versions of component libraries. Given the very small fixed point values, i am suspicious if my&nbsp;a mismatch in libraires version may be inducing the diffrence?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "44877",
      "postDate": "05/20/2014 01:30:26",
      "content": "<p>You should be looking at (this is an example output screen from another dataset):</p>\n<p><img src=\"http://i.imgur.com/xRt1n20.png\" alt=\"Average loss\" width=\"514\" height=\"164\"></p>\n<p>I too have a 64-bit machine, running Vowpal Wabbit on Windows in Cygwin gives me that average loss. Do note that different versions of Vowpal Wabbit may give different results (check with ./vw --version, I think mine was 7.6.1).</p>\n<p>Have you tried submitting and getting your&nbsp;score? If it is similar then you don't need to worry about this, focus on getting it down.</p>\n\n<p><sub>P.S.: I made a tutorial to install <a href=\"http://mlwave.com/install-vowpal-wabbit-on-windows-and-cygwin/\">Vowpal Wabbit on Windows with Cygwin</a>&nbsp;if you are interested.</sub></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "44978",
      "postDate": "05/20/2014 20:36:59",
      "content": "<p>Triskelion,</p>\n<p>Thank a ton for the pointer and the link to your blog-very useful. My training average loss is same as yours @ 0.252922. But the loss on test set and eventually the leader board is way to large @1.4 and 0.57 respectively. This is with a single pass. VW version is same-7.6.1.</p>\n<p>I will take a closer look at whats going on and &nbsp;re-post back.</p>\n<p>Thanks also for the detailed cygwin instructions. I wish I had fought my battle against Visual studio build a couple of days after you posed your instruction-would have save me lot of heartburn. I did manage a build on VS, will post the mods on your blog.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "44979",
      "postDate": "05/20/2014 20:47:36",
      "content": "<p>[quote=maveric;44978]</p>\n<p>But the loss on test set and eventually the leader board is way to large @1.4 and 0.57 respectively.</p>\n<p>[/quote]</p>\n<p>Loss on the test set you can ignore. Most of the time you are running this in &quot;-t&quot; (test only) mode, so there is no learning. Also since you use dummy (fake) labels for the test set, the loss can be huge (VW is correctly predicting many labels as &quot;-1&quot;, while all dummy labels are set to &quot;1&quot;, resulting in a huge loss).</p>\n<p>[quote=maveric;44978]</p>\n<p>&nbsp;I did manage a build on VS, will post the mods on your blog.</p>\n<p>[/quote]</p>\n<p>Wow! Very interested&nbsp;in that! I once managed to build on VS, but that was version 7.1, never quite figured it out with newer versions. Thank you for struggling with VS and taking one for the team :) Await your comments!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "44982",
      "postDate": "05/20/2014 21:56:12",
      "content": "<p>ofcource, why would i look at the test data loss, what-was-i-thinking :). Found the bug-i was using your gen_submission.py. In there, you are deciding as a 1, &nbsp;when the decision metric is equal to 1. I changed it to &gt;0 and i get the 0.661 loss.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49818",
      "postDate": "06/30/2014 03:09:27",
      "content": "<p>[quote=maveric;44982]</p>\n<p>ofcource, why would i look at the test data loss, what-was-i-thinking :). Found the bug-i was using your gen_submission.py. In there, you are deciding as a 1, &nbsp;when the decision metric is equal to 1. I changed it to &gt;0 and i get the 0.661 loss.</p>\n<p>[/quote]</p>\n\n<p>Maveric, you changed this line &nbsp;if float(row[0]) == 1: to&nbsp;if float(row[0]) &gt; 0: ?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49870",
      "postDate": "06/30/2014 17:19:49",
      "content": "<p>Yes. float(row[0]) &gt; =0 to be precise.&nbsp;</p>\n<p>Also, read through&nbsp;https://groups.yahoo.com/neo/groups/vowpal_wabbit/conversations/topics/2889</p>\n<p>to see how to derive probalility values from VW-something that is required for stacked generalisation and covariate shift.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "50512",
      "postDate": "07/15/2014 01:04:18",
      "content": "<p>Is it possible to replicate this .py code in R?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "50513",
      "postDate": "07/15/2014 01:17:58",
      "content": "<p>[quote=Maverick;50512]</p>\n<p>Is it possible to replicate this .py code in R?</p>\n<p>[/quote]</p>\n<p>Not the VW part I don't think (or maybe <a href=\"http://cran.r-project.org/web/packages/RVowpalWabbit/index.html\">package</a>). But you could output a .CSV with it and load that into R.</p>\n<p>See also:&nbsp;https://github.com/FBK-NILab/DecMeg2014/</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 42776,
      "author_name": "chenglongchen",
      "author_url": "",
      "post_date": "04/23/2014 02:15:03",
      "content": "<p>Great! Thanks @Triskelion</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42868,
      "author_name": "",
      "author_url": "",
      "post_date": "04/23/2014 20:19:57",
      "content": "<p><em>&quot;Each trial consists of 1.5 seconds of MEG recording (starting 0.5sec before the stimulus starts) &quot; </em></p>\n<p>Should this be understood as, the first 0.5 s nothing happens and than the stimulus starts.&nbsp; would it not then be better to use from ex. 0.25 - 0.75 ? Or have I misunderstood somthing</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42873,
      "author_name": "mkia83",
      "author_url": "",
      "post_date": "04/23/2014 21:26:01",
      "content": "<p>Actually yes, nothing happens before the start of stimulus, but someone may want to use pre-stimulus data for noise estimation, baseline correction, or any other intelligent idea for normalization to improve the decoding accuracy.&nbsp;About the post-stimulus data, it is up to you to decide...&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42967,
      "author_name": "cargon",
      "author_url": "",
      "post_date": "04/24/2014 18:54:16",
      "content": "<p>And how long does the stimulus last? I can't seem to find this detail anywhere.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42991,
      "author_name": "emanuele",
      "author_url": "",
      "post_date": "04/25/2014 07:06:48",
      "content": "<p>Hi,</p>\n<p>According to the article of the study from which the dataset is taken, the stimulus is presented for a random duration between 0.8sec and 1.0sec.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 43219,
      "author_name": "",
      "author_url": "",
      "post_date": "04/28/2014 14:38:24",
      "content": "<p>Thank you for the code and directions. Just for the sake of curiosity I took your code and played around with feature generation phase and got these results so far (based on my submissions using vw):</p>\n<p>0.66610 when I did 120:305 features</p>\n<p>0.66327 with 100:305 features</p>\n<p>0.66100 with 1:305 features</p>\n<p>0.65193 with 140:305 features</p>\n<p>0.64626 with 0-0.4 seconds all features</p>\n<p>0.63889 with 0.1 - 0.3 seconds all features</p>\n<p>0.58503 without standardization</p>\n<p>0.55215 with single subject trained models with majority voting</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 44361,
      "author_name": "nedelko",
      "author_url": "",
      "post_date": "05/11/2014 06:51:23",
      "content": "<p><del>Triskelion, what ID (subject, trial) is shown on your plots?</del></p>\n<p>..I reproduced these plots. ID=01000.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 44482,
      "author_name": "andrewmatteson",
      "author_url": "",
      "post_date": "05/13/2014 19:22:28",
      "content": "<p>EIGSI, When you say 0-.4 seconds, do you mean you trained on the data 0-.4 seconds after the stimulus, or do you mean the data that happened before the stimulus (that started at .5 seconds)?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 44769,
      "author_name": "wabbit",
      "author_url": "",
      "post_date": "05/18/2014 07:10:15",
      "content": "<p>Triskelion,</p>\n<p>Thanks for providing the starter code!. I ran the code as is, on a 64 bit windows machine but cannot see the 0.661 loss figure you mention-i see 0.5731. Any clues as to what i should be looking at? I mention the machine configuration because&nbsp;the VW default windows instruction do not work for latest VW &nbsp;and i had to patch things up with different versions of component libraries. Given the very small fixed point values, i am suspicious if my&nbsp;a mismatch in libraires version may be inducing the diffrence?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 44877,
      "author_name": "triskelion",
      "author_url": "",
      "post_date": "05/20/2014 01:30:26",
      "content": "<p>You should be looking at (this is an example output screen from another dataset):</p>\n<p><img src=\"http://i.imgur.com/xRt1n20.png\" alt=\"Average loss\" width=\"514\" height=\"164\"></p>\n<p>I too have a 64-bit machine, running Vowpal Wabbit on Windows in Cygwin gives me that average loss. Do note that different versions of Vowpal Wabbit may give different results (check with ./vw --version, I think mine was 7.6.1).</p>\n<p>Have you tried submitting and getting your&nbsp;score? If it is similar then you don't need to worry about this, focus on getting it down.</p>\n\n<p><sub>P.S.: I made a tutorial to install <a href=\"http://mlwave.com/install-vowpal-wabbit-on-windows-and-cygwin/\">Vowpal Wabbit on Windows with Cygwin</a>&nbsp;if you are interested.</sub></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 44978,
      "author_name": "wabbit",
      "author_url": "",
      "post_date": "05/20/2014 20:36:59",
      "content": "<p>Triskelion,</p>\n<p>Thank a ton for the pointer and the link to your blog-very useful. My training average loss is same as yours @ 0.252922. But the loss on test set and eventually the leader board is way to large @1.4 and 0.57 respectively. This is with a single pass. VW version is same-7.6.1.</p>\n<p>I will take a closer look at whats going on and &nbsp;re-post back.</p>\n<p>Thanks also for the detailed cygwin instructions. I wish I had fought my battle against Visual studio build a couple of days after you posed your instruction-would have save me lot of heartburn. I did manage a build on VS, will post the mods on your blog.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 44979,
      "author_name": "triskelion",
      "author_url": "",
      "post_date": "05/20/2014 20:47:36",
      "content": "<p>[quote=maveric;44978]</p>\n<p>But the loss on test set and eventually the leader board is way to large @1.4 and 0.57 respectively.</p>\n<p>[/quote]</p>\n<p>Loss on the test set you can ignore. Most of the time you are running this in &quot;-t&quot; (test only) mode, so there is no learning. Also since you use dummy (fake) labels for the test set, the loss can be huge (VW is correctly predicting many labels as &quot;-1&quot;, while all dummy labels are set to &quot;1&quot;, resulting in a huge loss).</p>\n<p>[quote=maveric;44978]</p>\n<p>&nbsp;I did manage a build on VS, will post the mods on your blog.</p>\n<p>[/quote]</p>\n<p>Wow! Very interested&nbsp;in that! I once managed to build on VS, but that was version 7.1, never quite figured it out with newer versions. Thank you for struggling with VS and taking one for the team :) Await your comments!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 44982,
      "author_name": "wabbit",
      "author_url": "",
      "post_date": "05/20/2014 21:56:12",
      "content": "<p>ofcource, why would i look at the test data loss, what-was-i-thinking :). Found the bug-i was using your gen_submission.py. In there, you are deciding as a 1, &nbsp;when the decision metric is equal to 1. I changed it to &gt;0 and i get the 0.661 loss.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49818,
      "author_name": "odessa",
      "author_url": "",
      "post_date": "06/30/2014 03:09:27",
      "content": "<p>[quote=maveric;44982]</p>\n<p>ofcource, why would i look at the test data loss, what-was-i-thinking :). Found the bug-i was using your gen_submission.py. In there, you are deciding as a 1, &nbsp;when the decision metric is equal to 1. I changed it to &gt;0 and i get the 0.661 loss.</p>\n<p>[/quote]</p>\n\n<p>Maveric, you changed this line &nbsp;if float(row[0]) == 1: to&nbsp;if float(row[0]) &gt; 0: ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49870,
      "author_name": "wabbit",
      "author_url": "",
      "post_date": "06/30/2014 17:19:49",
      "content": "<p>Yes. float(row[0]) &gt; =0 to be precise.&nbsp;</p>\n<p>Also, read through&nbsp;https://groups.yahoo.com/neo/groups/vowpal_wabbit/conversations/topics/2889</p>\n<p>to see how to derive probalility values from VW-something that is required for stacked generalisation and covariate shift.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 50512,
      "author_name": "anshul",
      "author_url": "",
      "post_date": "07/15/2014 01:04:18",
      "content": "<p>Is it possible to replicate this .py code in R?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 50513,
      "author_name": "triskelion",
      "author_url": "",
      "post_date": "07/15/2014 01:17:58",
      "content": "<p>[quote=Maverick;50512]</p>\n<p>Is it possible to replicate this .py code in R?</p>\n<p>[/quote]</p>\n<p>Not the VW part I don't think (or maybe <a href=\"http://cran.r-project.org/web/packages/RVowpalWabbit/index.html\">package</a>). But you could output a .CSV with it and load that into R.</p>\n<p>See also:&nbsp;https://github.com/FBK-NILab/DecMeg2014/</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "42659": "",
    "42776": "",
    "42868": "",
    "42873": "",
    "42967": "",
    "42991": "",
    "43219": "",
    "44361": "",
    "44482": "",
    "44769": "",
    "44877": "",
    "44978": "",
    "44979": "",
    "44982": "",
    "49818": "",
    "49870": "",
    "50512": "",
    "50513": ""
  },
  "source": "meta"
}