{
  "id": 11009,
  "title": "Beating the Benchmark ;)",
  "url": "/competitions/inria-bci-challenge/discussion/11009",
  "author_name": "",
  "post_date": "2014-11-21T16:50:40.640Z",
  "votes": 29,
  "comment_count": 17,
  "views": 6610,
  "content": "<p>Hello All,</p>\n<p>I'm back with a quick script to beat the Random Forest benchmark using Random Forests :)&nbsp;</p>\n<p>The script is attached. If you dont understand anything, feel free to ask.&nbsp;</p>\n<p>VOTE UP if it helped you in any way :)&nbsp;</p>\n<p>Thanks!</p>\n<p>comment with what score you get on LB</p>",
  "messages": [
    {
      "id": "58622",
      "postDate": "11/21/2014 16:50:40",
      "content": "<p>Hello All,</p>\n<p>I'm back with a quick script to beat the Random Forest benchmark using Random Forests :)&nbsp;</p>\n<p>The script is attached. If you dont understand anything, feel free to ask.&nbsp;</p>\n<p>VOTE UP if it helped you in any way :)&nbsp;</p>\n<p>Thanks!</p>\n<p>comment with what score you get on LB</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58624",
      "postDate": "11/21/2014 16:57:23",
      "content": "<p>updated and fixed some typos&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58627",
      "postDate": "11/21/2014 18:13:08",
      "content": "<p>To the one who gave a - 1, please explain why!&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58628",
      "postDate": "11/21/2014 18:27:58",
      "content": "<p>[quote=Abhishek;58627]</p>\n<p>To the one who gave a - 1, please explain why!&nbsp;</p>\n<p>[/quote]</p>\n<p>Probably someone who thinks they'd win one of these contests if it weren't for all the BTB codes being provided.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58634",
      "postDate": "11/21/2014 20:17:49",
      "content": "<p>[quote=inversion;58628]</p>\n<p>[quote=Abhishek;58627]</p>\n<p>To the one who gave a - 1, please explain why!&nbsp;</p>\n<p>[/quote]</p>\n<p>Probably someone who thinks they'd win one of these contests if it weren't for all the BTB codes being provided.</p>\n<p>[/quote]</p>\n\n<p>As someone who was slightly peeved about a good BTB released in a previous competition near finishing, even I have to say being mad about one this early and this simple is a little silly. &nbsp;</p>\n\n<p>But in general I think it'd be good for people to remember that not everyone is a top 10 finisher, and releasing very solid BTBs towards the end of a comp does undercut some people...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58665",
      "postDate": "11/22/2014 02:36:10",
      "content": "<p>[quote=mmyers;58634]</p>\n<p>[quote=inversion;58628]</p>\n<p>[quote=Abhishek;58627]</p>\n<p>To the one who gave a - 1, please explain why!&nbsp;</p>\n<p>[/quote]</p>\n<p>Probably someone who thinks they'd win one of these contests if it weren't for all the BTB codes being provided.</p>\n<p>[/quote]</p>\n<p>As someone who was slightly peeved about a good BTB released in a previous competition near finishing, even I have to say being mad about one this early and this simple is a little silly. &nbsp;</p>\n<p>But in general I think it'd be good for people to remember that not everyone is a top 10 finisher, and releasing very solid BTBs towards the end of a comp does undercut some people...</p>\n<p>[/quote]</p>\n<p>I'm risky here to get downvoted, but here is one not so pleasent thing about this. I just want to be it here for everybody, I don't really think that this how things should be.</p>\n<p>Kaggle organizers sad that their whole system is tuned for choosing best, tuned for Top-10 or about this. Why should Abhishek and others care about what Kaggle don't want to?</p>\n<p>Clarifiyng this point:</p>\n<p>Organizers don't care much about cheaters not in Top 10%</p>\n<p>No prizes for anybody besides top-3 even for compos where pure chance is forming some of that top-3 winners.</p>\n<p>Rating for 50-th place and 100-th place not differs really by their evaluation formula. However 6-th and 7-th place is differs much, but I don't think that real difference is same sign. You can't be at overall top rating just by getting consistent 'really good' result, you need to be at Top 10 for some of the competitions.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58669",
      "postDate": "11/22/2014 06:35:04",
      "content": "<p>I am using the following commands to extract training and testing data before training RandomForest:</p>\n<p>unzip -p train.zip | grep &quot;,1$&quot; &gt; train.csv<br>unzip -p test.zip | grep &quot;,1$&quot; &gt; test.csv</p>\n<p>It saves your time while running Abhishek's code.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58713",
      "postDate": "11/23/2014 05:58:44",
      "content": "",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58715",
      "postDate": "11/23/2014 06:08:09",
      "content": "<p>Javier,</p>\n<p>BTB = Beat the Benchmark</p>\n<p>what is meant is code for a model (released here in the forum) which scores better than the benchmark</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58953",
      "postDate": "11/25/2014 22:07:03",
      "content": "<p>Thanks for the benchmark,&nbsp;</p>\n<p>it gaves me a public LB score of =~ 0.55</p>\n<p>But my cv score with this benchmark is much lower:</p>\n<p>By using</p>\n<p>scores = cross_validation.cross_val_score(clf, X,y,cv=8, scoring='roc_auc', n_jobs=-1)</p>\n<p>my cv score is around 0.3. The difference with the public score is very important and i find it curious. Can you confirm that you notice the same thing?&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58954",
      "postDate": "11/25/2014 22:27:32",
      "content": "<p>I would not worry too much about this benchmark.</p>\n<p>The public leaderboard is based on only two subjects, and given that the proportion of errors is different for each subject, you can get an AUC in this ballpark merely by guessing a constant value for each of the subjects&nbsp;(with the correct ordering).</p>\n<p>If you look at BtB script, you can see it is using a single time point as the features, synchronized with the feedback signal.&nbsp;Given the nonzero time required for neural signals to be generated and propagated, I think it is unlikely that this feature is very informative.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59015",
      "postDate": "11/26/2014 16:30:51",
      "content": "<p>[quote=Ali Ziat;58953]</p>\n<p>Thanks for the benchmark,&nbsp;</p>\n<p>it gaves me a public LB score of =~ 0.55</p>\n<p>But my cv score with this benchmark is much lower:</p>\n<p>By using</p>\n<p>scores = cross_validation.cross_val_score(clf, X,y,cv=8, scoring='roc_auc', n_jobs=-1)</p>\n<p>my cv score is around 0.3. The difference with the public score is very important and i find it curious. Can you confirm that you notice the same thing?&nbsp;</p>\n<p>[/quote]</p>\n<p>The dataset (5440 training exemples) is too small to do a 8 fold cross-validation, if you try with cv=2 or 3 you'll get something closer to 0.5. Which is just as good as random guessing ;) Yay!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59031",
      "postDate": "11/26/2014 19:33:03",
      "content": "<p>[quote=npetitclerc;59015]</p>\n<p>The dataset (5440 training exemples) is too small to do a 8 fold cross-validation, if you try with cv=2 or 3 you'll get something closer to 0.5. Which is just as good as random guessing ;) Yay!</p>\n<p>[/quote]</p>\n\n<p>I don't understand why my 8 folds cv don't give me something around 0.5, i'm not convinced that it's because 5440 is too small for 8 folds cv; it still should'nt give me a score that far from random guessing even with 20 folds in my opinion&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59041",
      "postDate": "11/26/2014 20:06:43",
      "content": "<p>@Ali, First your AUC should be larger then 0.5, I am guessing sklearn is using the negative labels to calculate AUC which is giving the low score.&nbsp; Check this <a href=\"http://stackoverflow.com/questions/21587639/sklearn-svm-area-under-roc-less-than-0-5-for-training-data\">stackoverflow</a> thread for some ideas on how to get sklearn to know where your positive labels are. </p>\n<p>I think you can try 1-AUC to get what your AUC should be. in this case about thats about .7. For this competition it is probably better to perform CV by holding out whole test subjects. If you include data, about subject ID in your train set and perform CV on a random shuffle of the data the AUC score will likely be inflated, and not match what the algorithm will do on the test set. </p>\n<p>I wrote my own CV procedure, but you could also try <a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.LeavePLabelOut.html\">LeavePLabelsOut</a> from sklearn and create a column for your groups of subjects.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59048",
      "postDate": "11/26/2014 20:39:15",
      "content": "<p>Thank you very much for your answer @Phalaris.&nbsp;</p>\n<p>I'm going to write my own CV procedure too,it'll allow me more flexibility&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59099",
      "postDate": "11/27/2014 15:09:15",
      "content": "<p>[quote=Ali Ziat;59031]</p>\n<p>I don't understand why my 8 folds cv don't give me something around 0.5, i'm not convinced that it's because 5440 is too small for 8 folds cv; it still should'nt give me a score that far from random guessing even with 20 folds in my opinion&nbsp;</p>\n<p>[/quote]</p>\n<p>The same here. But I noticed that when I changed the random_state value, I got better score. So I tried different random_state values and chose the best one.</p>\n<p>Regarding the number of folds, IMO, 8 folds should be fine since if&nbsp;you are doing k-fold cross validation, you will use (k-1) folds to train your model and 1 fold to calculate the score.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63350",
      "postDate": "01/30/2015 09:16:43",
      "content": "<p>Hi Abhishek,</p>\n\n<p>Could you also post one in R? I'm new to data science and am learning R so it would be helpful for me to understand the script in R. Thanks in advance!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63351",
      "postDate": "01/30/2015 09:23:04",
      "content": "<p>Sorry, im not fond of R. :(</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 58624,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "11/21/2014 16:57:23",
      "content": "<p>updated and fixed some typos&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58627,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "11/21/2014 18:13:08",
      "content": "<p>To the one who gave a - 1, please explain why!&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58628,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "11/21/2014 18:27:58",
      "content": "<p>[quote=Abhishek;58627]</p>\n<p>To the one who gave a - 1, please explain why!&nbsp;</p>\n<p>[/quote]</p>\n<p>Probably someone who thinks they'd win one of these contests if it weren't for all the BTB codes being provided.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58634,
      "author_name": "mmyers",
      "author_url": "",
      "post_date": "11/21/2014 20:17:49",
      "content": "<p>[quote=inversion;58628]</p>\n<p>[quote=Abhishek;58627]</p>\n<p>To the one who gave a - 1, please explain why!&nbsp;</p>\n<p>[/quote]</p>\n<p>Probably someone who thinks they'd win one of these contests if it weren't for all the BTB codes being provided.</p>\n<p>[/quote]</p>\n\n<p>As someone who was slightly peeved about a good BTB released in a previous competition near finishing, even I have to say being mad about one this early and this simple is a little silly. &nbsp;</p>\n\n<p>But in general I think it'd be good for people to remember that not everyone is a top 10 finisher, and releasing very solid BTBs towards the end of a comp does undercut some people...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58665,
      "author_name": "dremovd",
      "author_url": "",
      "post_date": "11/22/2014 02:36:10",
      "content": "<p>[quote=mmyers;58634]</p>\n<p>[quote=inversion;58628]</p>\n<p>[quote=Abhishek;58627]</p>\n<p>To the one who gave a - 1, please explain why!&nbsp;</p>\n<p>[/quote]</p>\n<p>Probably someone who thinks they'd win one of these contests if it weren't for all the BTB codes being provided.</p>\n<p>[/quote]</p>\n<p>As someone who was slightly peeved about a good BTB released in a previous competition near finishing, even I have to say being mad about one this early and this simple is a little silly. &nbsp;</p>\n<p>But in general I think it'd be good for people to remember that not everyone is a top 10 finisher, and releasing very solid BTBs towards the end of a comp does undercut some people...</p>\n<p>[/quote]</p>\n<p>I'm risky here to get downvoted, but here is one not so pleasent thing about this. I just want to be it here for everybody, I don't really think that this how things should be.</p>\n<p>Kaggle organizers sad that their whole system is tuned for choosing best, tuned for Top-10 or about this. Why should Abhishek and others care about what Kaggle don't want to?</p>\n<p>Clarifiyng this point:</p>\n<p>Organizers don't care much about cheaters not in Top 10%</p>\n<p>No prizes for anybody besides top-3 even for compos where pure chance is forming some of that top-3 winners.</p>\n<p>Rating for 50-th place and 100-th place not differs really by their evaluation formula. However 6-th and 7-th place is differs much, but I don't think that real difference is same sign. You can't be at overall top rating just by getting consistent 'really good' result, you need to be at Top 10 for some of the competitions.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58669,
      "author_name": "nthanhtam",
      "author_url": "",
      "post_date": "11/22/2014 06:35:04",
      "content": "<p>I am using the following commands to extract training and testing data before training RandomForest:</p>\n<p>unzip -p train.zip | grep &quot;,1$&quot; &gt; train.csv<br>unzip -p test.zip | grep &quot;,1$&quot; &gt; test.csv</p>\n<p>It saves your time while running Abhishek's code.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58713,
      "author_name": "javiarsandra",
      "author_url": "",
      "post_date": "11/23/2014 05:58:44",
      "content": "",
      "votes": null,
      "replies": []
    },
    {
      "id": 58715,
      "author_name": "cdjbee",
      "author_url": "",
      "post_date": "11/23/2014 06:08:09",
      "content": "<p>Javier,</p>\n<p>BTB = Beat the Benchmark</p>\n<p>what is meant is code for a model (released here in the forum) which scores better than the benchmark</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58953,
      "author_name": "aliziat",
      "author_url": "",
      "post_date": "11/25/2014 22:07:03",
      "content": "<p>Thanks for the benchmark,&nbsp;</p>\n<p>it gaves me a public LB score of =~ 0.55</p>\n<p>But my cv score with this benchmark is much lower:</p>\n<p>By using</p>\n<p>scores = cross_validation.cross_val_score(clf, X,y,cv=8, scoring='roc_auc', n_jobs=-1)</p>\n<p>my cv score is around 0.3. The difference with the public score is very important and i find it curious. Can you confirm that you notice the same thing?&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58954,
      "author_name": "emolson",
      "author_url": "",
      "post_date": "11/25/2014 22:27:32",
      "content": "<p>I would not worry too much about this benchmark.</p>\n<p>The public leaderboard is based on only two subjects, and given that the proportion of errors is different for each subject, you can get an AUC in this ballpark merely by guessing a constant value for each of the subjects&nbsp;(with the correct ordering).</p>\n<p>If you look at BtB script, you can see it is using a single time point as the features, synchronized with the feedback signal.&nbsp;Given the nonzero time required for neural signals to be generated and propagated, I think it is unlikely that this feature is very informative.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59015,
      "author_name": "npetitclerc",
      "author_url": "",
      "post_date": "11/26/2014 16:30:51",
      "content": "<p>[quote=Ali Ziat;58953]</p>\n<p>Thanks for the benchmark,&nbsp;</p>\n<p>it gaves me a public LB score of =~ 0.55</p>\n<p>But my cv score with this benchmark is much lower:</p>\n<p>By using</p>\n<p>scores = cross_validation.cross_val_score(clf, X,y,cv=8, scoring='roc_auc', n_jobs=-1)</p>\n<p>my cv score is around 0.3. The difference with the public score is very important and i find it curious. Can you confirm that you notice the same thing?&nbsp;</p>\n<p>[/quote]</p>\n<p>The dataset (5440 training exemples) is too small to do a 8 fold cross-validation, if you try with cv=2 or 3 you'll get something closer to 0.5. Which is just as good as random guessing ;) Yay!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59031,
      "author_name": "aliziat",
      "author_url": "",
      "post_date": "11/26/2014 19:33:03",
      "content": "<p>[quote=npetitclerc;59015]</p>\n<p>The dataset (5440 training exemples) is too small to do a 8 fold cross-validation, if you try with cv=2 or 3 you'll get something closer to 0.5. Which is just as good as random guessing ;) Yay!</p>\n<p>[/quote]</p>\n\n<p>I don't understand why my 8 folds cv don't give me something around 0.5, i'm not convinced that it's because 5440 is too small for 8 folds cv; it still should'nt give me a score that far from random guessing even with 20 folds in my opinion&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59041,
      "author_name": "devinanzelmo",
      "author_url": "",
      "post_date": "11/26/2014 20:06:43",
      "content": "<p>@Ali, First your AUC should be larger then 0.5, I am guessing sklearn is using the negative labels to calculate AUC which is giving the low score.&nbsp; Check this <a href=\"http://stackoverflow.com/questions/21587639/sklearn-svm-area-under-roc-less-than-0-5-for-training-data\">stackoverflow</a> thread for some ideas on how to get sklearn to know where your positive labels are. </p>\n<p>I think you can try 1-AUC to get what your AUC should be. in this case about thats about .7. For this competition it is probably better to perform CV by holding out whole test subjects. If you include data, about subject ID in your train set and perform CV on a random shuffle of the data the AUC score will likely be inflated, and not match what the algorithm will do on the test set. </p>\n<p>I wrote my own CV procedure, but you could also try <a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.LeavePLabelOut.html\">LeavePLabelsOut</a> from sklearn and create a column for your groups of subjects.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59048,
      "author_name": "aliziat",
      "author_url": "",
      "post_date": "11/26/2014 20:39:15",
      "content": "<p>Thank you very much for your answer @Phalaris.&nbsp;</p>\n<p>I'm going to write my own CV procedure too,it'll allow me more flexibility&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59099,
      "author_name": "nthanhtam",
      "author_url": "",
      "post_date": "11/27/2014 15:09:15",
      "content": "<p>[quote=Ali Ziat;59031]</p>\n<p>I don't understand why my 8 folds cv don't give me something around 0.5, i'm not convinced that it's because 5440 is too small for 8 folds cv; it still should'nt give me a score that far from random guessing even with 20 folds in my opinion&nbsp;</p>\n<p>[/quote]</p>\n<p>The same here. But I noticed that when I changed the random_state value, I got better score. So I tried different random_state values and chose the best one.</p>\n<p>Regarding the number of folds, IMO, 8 folds should be fine since if&nbsp;you are doing k-fold cross validation, you will use (k-1) folds to train your model and 1 fold to calculate the score.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63350,
      "author_name": "hmboxwala",
      "author_url": "",
      "post_date": "01/30/2015 09:16:43",
      "content": "<p>Hi Abhishek,</p>\n\n<p>Could you also post one in R? I'm new to data science and am learning R so it would be helpful for me to understand the script in R. Thanks in advance!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63351,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "01/30/2015 09:23:04",
      "content": "<p>Sorry, im not fond of R. :(</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "58622": "",
    "58624": "",
    "58627": "",
    "58628": "",
    "58634": "",
    "58665": "",
    "58669": "",
    "58713": "",
    "58715": "",
    "58953": "",
    "58954": "",
    "59015": "",
    "59031": "",
    "59041": "",
    "59048": "",
    "59099": "",
    "63350": "",
    "63351": ""
  },
  "source": "meta"
}