{ "id": 12819, "title": " The Overfitting Avengers Solution & Code ", "url": "/competitions/inria-bci-challenge/writeups/the-overfitting-avengers-the-overfitting-avengers-", "author_name": "", "post_date": "2015-03-15T17:39:58.370Z", "votes": 14, "comment_count": 17, "views": 7315, "content": "

Hi,

\n

You can find our code and documentation on github.

\n

In short, we used a special form covariance matrices as feature and tools from Riemannian Geometry to manipulate them. A channel selection algorithm was involved, and the classification was achieved by an ElasticNet classifier. 

\n

The above pipeline was applied on a number of random subsets of subjects, and predictions are averaged across bagged models.

\n

The complete explanation of our model is provided here.

\n\n

We made two different submission. The first one does not make any use of the leakage information and satisfies an "online processing" constraint, which means that any trial performed by a subject can be classified without the need for future complementary data or information. The second model uses the leak, thus it is not online-compatible. The two models are built upon the same classification pipeline, but with parameters tuned independently to achieve the highest performance.

\n\n\n

Any comments are welcomes !

\n\n

Cheers,

\n

The overfitting Avengers :)

", "messages": [ { "id": "66251", "postDate": "03/15/2015 17:39:58", "content": "

Hi,

\n

You can find our code and documentation on github.

\n

In short, we used a special form covariance matrices as feature and tools from Riemannian Geometry to manipulate them. A channel selection algorithm was involved, and the classification was achieved by an ElasticNet classifier. 

\n

The above pipeline was applied on a number of random subsets of subjects, and predictions are averaged across bagged models.

\n

The complete explanation of our model is provided here.

\n\n

We made two different submission. The first one does not make any use of the leakage information and satisfies an "online processing" constraint, which means that any trial performed by a subject can be classified without the need for future complementary data or information. The second model uses the leak, thus it is not online-compatible. The two models are built upon the same classification pipeline, but with parameters tuned independently to achieve the highest performance.

\n\n\n

Any comments are welcomes !

\n\n

Cheers,

\n

The overfitting Avengers :)

", "rawMarkdown": "", "votes": null }, { "id": "66291", "postDate": "03/15/2015 21:55:09", "content": "

Thanks for posting your teams solutions! I have been looking forward seeing your methodology since the competition ended. The python code looks quite usable. I was considering converting your matlab code from decMeg to experiment with but now I don't have to. I will post some questions when I have time to closely read the solution.

", "rawMarkdown": "", "votes": null }, { "id": "66295", "postDate": "03/15/2015 22:48:57", "content": "

Great job! Are you finding Riemannian Geometry to be an indispensable tool for signal processing and BCI-related work?

", "rawMarkdown": "", "votes": null }, { "id": "66296", "postDate": "03/15/2015 22:49:37", "content": "

You're welcome :)

\n

For me, it was a good opportunity to switch to python, and i have to say that i will never go back to matlab.

\n

Now that i have an equivalent of my covariance toolbox in python, I plan to release a package as soon as possible.

\n

We took our time before publishing the code, but we wanted to make a deeper analysis of our solution. The final solution is a bit computationally intensive, but you can speed of the process by turning off the bagging and the electrode selection.

\n

It turns out that by doing this you can achieve a score ~0.843 (for the model without the leak) and it will takes around 2 minutes to process.

", "rawMarkdown": "", "votes": null }, { "id": "66301", "postDate": "03/15/2015 23:43:06", "content": "

[quote=TF;66295]

\n

Great job! Are you finding Riemannian Geometry to be an indispensable tool for signal processing and BCI-related work?

\n

[/quote]

\n

I would say definitely yes but it will be a bit unfair for all the other people who have done amazing work in the BCI community.

\n

At first, i developed these methods to get rid of the source separation step (i.e. spatial filtering) during online BCI experiments. While algorithm such Common Spatial Patterns (CSP) or XDAWN are really powerful, they lack of robustness and introduce extra parameters to tune. And if they are not used correctly, they can lead to a loss of spatial information.

\n

By using covariance matrices as features, you make sure you get all the spatial information in your classifier. Then, tools from Riemannian geometry provide you a convenient and robust way to manipulate them. More specifically, the tangent space mapping (which can be see as a kernel operation) is a very powerful tool. It allows to transform these matrices in euclidean vectors without breaking their particular structure  (Symmetric and positive definite). Then, you can do whatever you want with these vectors.

\n

Now, i have to say that you can probably get similar results with other methods, but thanks to a bunch of properties of the Riemannian metric, it is very well suited for cross-subject classification (which was the case for DecMeg and this challenge).

\n

Covariance matrices are involved in wide range of signal processing problem (well, let say every time you have a multivariate problem), so riemannian geometry has a lot of potential. It is also used in Radar signal processing and Image classification.

\n

Finally, I wanted to say that the riemannian metric i use is a special case of something bigger called Information Geometry, a theory that study the geometric properties of probability distributions. In my case, the distribution of interest is the multivariate normal distribution with zero mean (only parametrized by it covariance matrix), but such metric can be extracted for any other distribution (but the math behind can be hardcore).

", "rawMarkdown": "", "votes": null }, { "id": "66327", "postDate": "03/16/2015 07:06:43", "content": "

Alexandre, 

\n

many thanks for sharing your approach and your code!

\n

I have a few questions

\n

1) In DecMeg competition you used CSP filters in order to reduce dimensionality of the problem (DR seems to be an important preprocessing step for Riemannian geometry tools). Here you used a channel selection procedure instead of CSP. Could you please explain a little more about the motivation of this decision?

\n

2) Why did you obtain such a huge difference between Fold-wise CV (0.7294) and Private/Public LB (0.84585/0.85083)? As far as I understand you used the same criterion (joint AUC) for Fold-Wise CV calculation. The only difference I see is that you used 150 bagged models for CV and 500 bagged models for the final submission. But according to your post-competition analysis even 10 models give ~0.84 on the LB datasets.

\n

3) Could you please recommend useful books on Information Geometry with all necessary math concepts (even hardcore ones)? 

\n

Thanks in advance, 

\n

Mike

", "rawMarkdown": "", "votes": null }, { "id": "66469", "postDate": "03/16/2015 19:52:09", "content": "

Hi Mike,

\n

these are very relevant questions.

\n

1) There is multiple reason we took this decision, but first lets got back to the dimensionality reduction for Riemannian geometry (RG). Dealing with High dimensional data is the main problem when using RG. The main reasons are :

\n\n

My experience with EEG/MEG data is that if you keep everything bellow 64 channel, you will have no trouble.

\n

For the Decmeg Challenge, the dimensionality was simply to high to use any of my methods (including the channel selection) out of the box. So i had no choice and i decided to use spatial filtering.

\n

The problem with spatial filtering is that is is not very well suited for a cross-subject design. The basic hypothesis of source separation is that the training data and the test data are mixed by the same operator and are stationary. When you train on a subject, and apply on another, you take the risk to discard relevant information in the process. Again, when you aggregates data from different subject for training, you source separation becomes less effective because your algorithm try to estimate a unique unmixing matrix. 

\n

For decMeg, i found a workaround. My classification was in two stage, the first was a cross-subject classifier, using stacked generalization (so spatial filters were not train on aggregated data), the second was an iterative retraining using the data on the test subject only (the case where spatial filtering is really efficient). As you see, the key is to know what are the advantages and the flaws of you methods in order to use them in the best possible conditions.

\n

Now for this challenge, the dimensionality was manageable, so spatial filtering was not necessary. Still, it is a good thing to reduce dimensionality, so we apply the channel selection procedure to discard irrelevant channels. The number of channels we kept is not very tight (35). it makes everything running more smoothly without having much risk to discard relevant information. EEG has a low spatial resolution, so close channels are highly correlated and often carry the same information. Discarding a few of them doesn't hurt

\n

So the next question is why we didn't apply the same retraining process for this challenge. Well, first because we wanted to keep thing as 'online' as possible. The second is simply because the convergence were more tricky. The method i used in Decmeg was based on the assumption that the classes were balanced (which was the case by design), in order to set the optimal threshold for the re-labeling between iteration. Here the classes balance were unknown (if you don't use the leak).

\n\n

2) Okay, now for the second question, i will let my teammate answer.

\n

3) I think the best book you can find is "Methods of Information Geometry" from S. Amari. The basic idea is that each probability distribution is a point of a manifold with the parameters of the distribution as coordinates. For example a normal distribution of mean u and variance s is a point of coordinate (u,s) in the manifold. The trick is, the Information geometry make the assumption that the natural metric for this manifold is the Fisher information. After a bunch of equation manipulation, it allow you to define a true distance between two distribution, and therefore unlock a lot of issue (for example, the mean of a distribution is not the distribution with the mean parameters :)  )

\n\n

Alex

", "rawMarkdown": "", "votes": null }, { "id": "66509", "postDate": "03/16/2015 21:21:25", "content": "

Hey Mike, let me answer your 2nd question. The gap in performance between CV and privateLB arises from two factors:

\n

1) As also noted in the Results section of the readme, the model itself quite accurately inferred the class balance of subjects in privateLB, i.e. predictions of each subject got noticeably shifted w.r.t. one another, as according to their overall performances in the experiment. This caused a significant boost in global AUC. We think the effect was not (or vaguely) seen in CV due to a smaller test set size.

\n

2) The bias-variance tradeoff of CV. Concretely, the less folds you use in CV the more biased the performance will be (worse than the actual performance). In 4-fold CV you have 12 training subjects, whereas when preparing a submission you have 16 subjects, this is an addition of significant amount of information. You can get a "boost" in CV score by simply using 8 folds etc. On a side note, there were certain settings that brought a significant improvement in CV, but more rigorous tests (explained in the last paragraph here) revealed that they could potentially lead us to severe overfitting, despite promising publicLB scores.

\n

Cheers,

\n

Rafal

", "rawMarkdown": "", "votes": null }, { "id": "67689", "postDate": "03/23/2015 10:29:21", "content": "

Alexander,

\n

I'd like to congratulate you and your partner for this victory.

\n

Sometimes it looks like there's gap between academics and real world (i.e. Kaggle problems), and many times data engineers win competitions just using some 20 years-old techniques (such as Random Forest) and a sensible feature choice, while researchers publish complex papers in top machine-learning conferences claiming to improve the state-of-the-art.

\n

In you case, however, academic excellence meets performance excellence. Winning two different competitions using your Riemann-geometric stuff is definitely an incredibly outstanding achievement.

", "rawMarkdown": "", "votes": null }, { "id": "67810", "postDate": "03/23/2015 22:54:26", "content": "

Hi Alexandre & Rafal.

\n

Many thanks for your very detailed answers! I absolutely agree with Jose that your solution is fascinating. It combines both extremely interesting mathematical ideas and outstanding performance (and this combination as rare as unicorns:) ). Moreover you allows us to catch some internal details of your algorithms, so we should be (and we are!) much obliged to you.

\n

Alexandre, 

\n

Thanks, I understand your motivation of using channel selection here (in fact you have to use CSP in DecMeg due to huge number of MEG sensors). We've played a little with your DecMeg code and noticed two issues you mentioned (off-line analysis only and a requirement of balanced classes). Not so evident result (for me) is that performance of retraining strongly depends on distance type. The best choice is riemann (loo accuracy is ~0.756); logeuclid also shows competitive results (~0.744). Euclid doesn't work at all (~0.625). The most surprising result is that kullback slightly improves (~0.728) result of the generic classifier, but still shows worse accuracy than riemann. If anyone had asked me how to measure distance between two distributions I would have answered 'certainly kullback leibler'. Now I'm not so confident :) 

\n

I understand that performance of different metrics depends on a particular dataset, so kullback can get own back and exceed riemann in another problem. But I have no idea why logeuclid works at all :) Both riemann and kullback have some understandable mathematical basis (thank you for the IG book, I'll definitely read it!), but what about logeuclid? Could you please give a short comment why does it work?

\n

Rafal, 

\n

I agree, global AUC effect depends on sample size, so it boosts performance on the large Private set in comparison with single-subject AUC. However, the Public set is rather small, but your approach shows almost the same performance as for the Private set. Probably the reason is that there are "easy" subjects in the Public set.

\n

Thanks again,

\n

Mike

", "rawMarkdown": "", "votes": null }, { "id": "72858", "postDate": "04/21/2015 06:17:08", "content": "

hi sir, is there anyway we can have your features(not the meta features),it would be great help.we need to examine how that works in our classifier,and the accuracy it could manage with your features..

\n

thanking you in advance,

\n

Sunu A

", "rawMarkdown": "", "votes": null }, { "id": "72966", "postDate": "04/21/2015 17:02:43", "content": "

Hi Mike,

\n

Thanks for looking into the decMeg code and pointing the issue of the metric. 

\n\n

if i have to rank the metric by their efficiency on EEG data, i would do it like this :

\n

Riemann > Log-det > Log-euclid > kullback > Euclidean

\n

It may be different for other problem, as you said. The log-euclidean seems very popular in image processing and computer vision. 

\n

As a side note, i just released a python package, pyRiemann. It is an extended version of the code developed for this challenge, and it support the different metrics I mention in this post (except the kullblack, but i accept pull request :) )

", "rawMarkdown": "", "votes": null }, { "id": "72969", "postDate": "04/21/2015 17:10:44", "content": "

[quote=Sunu A;72858]

\n

hi sir, is there anyway we can have your features(not the meta features),it would be great help.we need to examine how that works in our classifier,and the accuracy it could manage with your features..

\n

thanking you in advance,

\n

Sunu A

\n

[/quote]

\n

Hi,

\n

The feature extraction we used include numerous supervised steps, like Xdawn spatial filtering, special form covariance estimation or channel selection. Giving you the raw feature will lead to biased results.

\n

However, we published the code with a easy mechanism to test your ideas. All of this is explained here. All you have to do is to build a python class that inherit from the sklearn base API (basically, two method, fit() and predict() ) and then declare it in the parameter file.

\n

This way, you will evaluates the performance of your classifier in the same scheme as our, and you will get comparable results.

\n\n

Alex

", "rawMarkdown": "", "votes": null }, { "id": "77258", "postDate": "05/06/2015 06:23:59", "content": "

Hello  sir...

\n

                 i have tried your code but i keep on ending up with memory error.As you have noted, i reduced my no of cores but still...In between,my system has only 4GB ram with four cores.Does your code works with my system spec?  i'm very new to python

\n\n\n\n

thanking you in advance

", "rawMarkdown": "", "votes": null }, { "id": "77277", "postDate": "05/06/2015 08:30:51", "content": "

Hi Alexandre! 

\n

As usual, many thanks for your very detailed answer which helps us to better understand your ideas!

\n

1) About metrics: I totally agree with you :) Some comments are below.

\n\n

2) I reckon there is a huge problem in algorithmic development for BCI: it's hard to compare results from different publications. There are three main issues:

\n\n

(I've started exploring BCI data science problems only few months ago, so I may be wrong. Please correct me in that case.)

\n

So in my opinion pyRiemann release is a great news (as well as your Matlab covariance Toolbox)!

\n

I plan to finish my current project in aerospace in 1-2 months and to dive into analysis of BCI data. So I hope I will have something to commit in pyRiemann :)

\n\n

Best regards, 

\n

Mike

", "rawMarkdown": "", "votes": null }, { "id": "77278", "postDate": "05/06/2015 08:33:50", "content": "

[quote=Sunu A;77258]

\n

Hello sir...

\n

i have tried your code but i keep on ending up with memory error.As you have noted, i reduced my no of cores but still...In between,my system has only 4GB ram with four cores.Does your code works with my system spec? i'm very new to python

\n

thanking you in advance

\n

[/quote]

\n

According to specification you need at least 8GB of RAM. It's rather typical situation for EEG analysis due to huge size of raw data.

\n

Best regards,

\n

Mike

", "rawMarkdown": "", "votes": null }, { "id": "77342", "postDate": "05/06/2015 15:46:13", "content": "

Hi Mike,

\n

Thank you for carrying this interesting discussion.

\n

I could not agree more with you :)

\n

Information geometry is gaining  momentum in a lot of different domain. We can see a lot of stuff in computer vision  for example. I would explain that by the facts that :

\n\n\n

About BCI, i totally share you point of view. There is a lack of reproducible research. The problem does not only comes from the methods implementation, but also from the environment it is applied. Results are so bad and datasets are so small that even changing the order of the frequential filter you apply as preprocessing step can change significantly the outcome of you classification.

\n

In addition, parameters are usually over-optimized for a specific dataset, and when you tried a new method on a new set of data, it require an high amount of domain specific knowledge to make it works properly. This partially explain the popularity of the 'old' methods like CSP. They does not give optimal results, but they are stable across a wide range of applications.

\n

The other problem is that EEG data are considered as medical data and are really hard to release publicly. You need consents to release from the subject, and it is rarely done before the experiment. There is a lot of data that are just sitting on the labs hard drive without any chance to be released even if you want to (i have tons of data like this).

\n

What we really need is a platform where you can submit your code and your method will be blindly evaluated on a wide range of dataset (you can even add realistic stress test, like adding noise on data or on labels, etc).

\n

Finally, thanks for your interest for pyRiemann. your commits will be greatly appreciated :)

\n

Best,

\n

Alexandre

", "rawMarkdown": "", "votes": null }, { "id": "91984", "postDate": "09/10/2015 06:53:54", "content": "

from where can i download data sets?

", "rawMarkdown": "from where can i download data sets?", "votes": null } ], "comments": [ { "id": 66291, "author_name": "devinanzelmo", "author_url": "", "post_date": "03/15/2015 21:55:09", "content": "

Thanks for posting your teams solutions! I have been looking forward seeing your methodology since the competition ended. The python code looks quite usable. I was considering converting your matlab code from decMeg to experiment with but now I don't have to. I will post some questions when I have time to closely read the solution.

", "votes": null, "replies": [] }, { "id": 66295, "author_name": "tf6452", "author_url": "", "post_date": "03/15/2015 22:48:57", "content": "

Great job! Are you finding Riemannian Geometry to be an indispensable tool for signal processing and BCI-related work?

", "votes": null, "replies": [] }, { "id": 66296, "author_name": "alexandrebarachant", "author_url": "", "post_date": "03/15/2015 22:49:37", "content": "

You're welcome :)

\n

For me, it was a good opportunity to switch to python, and i have to say that i will never go back to matlab.

\n

Now that i have an equivalent of my covariance toolbox in python, I plan to release a package as soon as possible.

\n

We took our time before publishing the code, but we wanted to make a deeper analysis of our solution. The final solution is a bit computationally intensive, but you can speed of the process by turning off the bagging and the electrode selection.

\n

It turns out that by doing this you can achieve a score ~0.843 (for the model without the leak) and it will takes around 2 minutes to process.

", "votes": null, "replies": [] }, { "id": 66301, "author_name": "alexandrebarachant", "author_url": "", "post_date": "03/15/2015 23:43:06", "content": "

[quote=TF;66295]

\n

Great job! Are you finding Riemannian Geometry to be an indispensable tool for signal processing and BCI-related work?

\n

[/quote]

\n

I would say definitely yes but it will be a bit unfair for all the other people who have done amazing work in the BCI community.

\n

At first, i developed these methods to get rid of the source separation step (i.e. spatial filtering) during online BCI experiments. While algorithm such Common Spatial Patterns (CSP) or XDAWN are really powerful, they lack of robustness and introduce extra parameters to tune. And if they are not used correctly, they can lead to a loss of spatial information.

\n

By using covariance matrices as features, you make sure you get all the spatial information in your classifier. Then, tools from Riemannian geometry provide you a convenient and robust way to manipulate them. More specifically, the tangent space mapping (which can be see as a kernel operation) is a very powerful tool. It allows to transform these matrices in euclidean vectors without breaking their particular structure  (Symmetric and positive definite). Then, you can do whatever you want with these vectors.

\n

Now, i have to say that you can probably get similar results with other methods, but thanks to a bunch of properties of the Riemannian metric, it is very well suited for cross-subject classification (which was the case for DecMeg and this challenge).

\n

Covariance matrices are involved in wide range of signal processing problem (well, let say every time you have a multivariate problem), so riemannian geometry has a lot of potential. It is also used in Radar signal processing and Image classification.

\n

Finally, I wanted to say that the riemannian metric i use is a special case of something bigger called Information Geometry, a theory that study the geometric properties of probability distributions. In my case, the distribution of interest is the multivariate normal distribution with zero mean (only parametrized by it covariance matrix), but such metric can be extracted for any other distribution (but the math behind can be hardcore).

", "votes": null, "replies": [] }, { "id": 66327, "author_name": "mibell", "author_url": "", "post_date": "03/16/2015 07:06:43", "content": "

Alexandre, 

\n

many thanks for sharing your approach and your code!

\n

I have a few questions

\n

1) In DecMeg competition you used CSP filters in order to reduce dimensionality of the problem (DR seems to be an important preprocessing step for Riemannian geometry tools). Here you used a channel selection procedure instead of CSP. Could you please explain a little more about the motivation of this decision?

\n

2) Why did you obtain such a huge difference between Fold-wise CV (0.7294) and Private/Public LB (0.84585/0.85083)? As far as I understand you used the same criterion (joint AUC) for Fold-Wise CV calculation. The only difference I see is that you used 150 bagged models for CV and 500 bagged models for the final submission. But according to your post-competition analysis even 10 models give ~0.84 on the LB datasets.

\n

3) Could you please recommend useful books on Information Geometry with all necessary math concepts (even hardcore ones)? 

\n

Thanks in advance, 

\n

Mike

", "votes": null, "replies": [] }, { "id": 66469, "author_name": "alexandrebarachant", "author_url": "", "post_date": "03/16/2015 19:52:09", "content": "

Hi Mike,

\n

these are very relevant questions.

\n

1) There is multiple reason we took this decision, but first lets got back to the dimensionality reduction for Riemannian geometry (RG). Dealing with High dimensional data is the main problem when using RG. The main reasons are :

\n\n

My experience with EEG/MEG data is that if you keep everything bellow 64 channel, you will have no trouble.

\n

For the Decmeg Challenge, the dimensionality was simply to high to use any of my methods (including the channel selection) out of the box. So i had no choice and i decided to use spatial filtering.

\n

The problem with spatial filtering is that is is not very well suited for a cross-subject design. The basic hypothesis of source separation is that the training data and the test data are mixed by the same operator and are stationary. When you train on a subject, and apply on another, you take the risk to discard relevant information in the process. Again, when you aggregates data from different subject for training, you source separation becomes less effective because your algorithm try to estimate a unique unmixing matrix. 

\n

For decMeg, i found a workaround. My classification was in two stage, the first was a cross-subject classifier, using stacked generalization (so spatial filters were not train on aggregated data), the second was an iterative retraining using the data on the test subject only (the case where spatial filtering is really efficient). As you see, the key is to know what are the advantages and the flaws of you methods in order to use them in the best possible conditions.

\n

Now for this challenge, the dimensionality was manageable, so spatial filtering was not necessary. Still, it is a good thing to reduce dimensionality, so we apply the channel selection procedure to discard irrelevant channels. The number of channels we kept is not very tight (35). it makes everything running more smoothly without having much risk to discard relevant information. EEG has a low spatial resolution, so close channels are highly correlated and often carry the same information. Discarding a few of them doesn't hurt

\n

So the next question is why we didn't apply the same retraining process for this challenge. Well, first because we wanted to keep thing as 'online' as possible. The second is simply because the convergence were more tricky. The method i used in Decmeg was based on the assumption that the classes were balanced (which was the case by design), in order to set the optimal threshold for the re-labeling between iteration. Here the classes balance were unknown (if you don't use the leak).

\n\n

2) Okay, now for the second question, i will let my teammate answer.

\n

3) I think the best book you can find is "Methods of Information Geometry" from S. Amari. The basic idea is that each probability distribution is a point of a manifold with the parameters of the distribution as coordinates. For example a normal distribution of mean u and variance s is a point of coordinate (u,s) in the manifold. The trick is, the Information geometry make the assumption that the natural metric for this manifold is the Fisher information. After a bunch of equation manipulation, it allow you to define a true distance between two distribution, and therefore unlock a lot of issue (for example, the mean of a distribution is not the distribution with the mean parameters :)  )

\n\n

Alex

", "votes": null, "replies": [] }, { "id": 66509, "author_name": "rafalcycon", "author_url": "", "post_date": "03/16/2015 21:21:25", "content": "

Hey Mike, let me answer your 2nd question. The gap in performance between CV and privateLB arises from two factors:

\n

1) As also noted in the Results section of the readme, the model itself quite accurately inferred the class balance of subjects in privateLB, i.e. predictions of each subject got noticeably shifted w.r.t. one another, as according to their overall performances in the experiment. This caused a significant boost in global AUC. We think the effect was not (or vaguely) seen in CV due to a smaller test set size.

\n

2) The bias-variance tradeoff of CV. Concretely, the less folds you use in CV the more biased the performance will be (worse than the actual performance). In 4-fold CV you have 12 training subjects, whereas when preparing a submission you have 16 subjects, this is an addition of significant amount of information. You can get a "boost" in CV score by simply using 8 folds etc. On a side note, there were certain settings that brought a significant improvement in CV, but more rigorous tests (explained in the last paragraph here) revealed that they could potentially lead us to severe overfitting, despite promising publicLB scores.

\n

Cheers,

\n

Rafal

", "votes": null, "replies": [] }, { "id": 67689, "author_name": "joseleiva", "author_url": "", "post_date": "03/23/2015 10:29:21", "content": "

Alexander,

\n

I'd like to congratulate you and your partner for this victory.

\n

Sometimes it looks like there's gap between academics and real world (i.e. Kaggle problems), and many times data engineers win competitions just using some 20 years-old techniques (such as Random Forest) and a sensible feature choice, while researchers publish complex papers in top machine-learning conferences claiming to improve the state-of-the-art.

\n

In you case, however, academic excellence meets performance excellence. Winning two different competitions using your Riemann-geometric stuff is definitely an incredibly outstanding achievement.

", "votes": null, "replies": [] }, { "id": 67810, "author_name": "mibell", "author_url": "", "post_date": "03/23/2015 22:54:26", "content": "

Hi Alexandre & Rafal.

\n

Many thanks for your very detailed answers! I absolutely agree with Jose that your solution is fascinating. It combines both extremely interesting mathematical ideas and outstanding performance (and this combination as rare as unicorns:) ). Moreover you allows us to catch some internal details of your algorithms, so we should be (and we are!) much obliged to you.

\n

Alexandre, 

\n

Thanks, I understand your motivation of using channel selection here (in fact you have to use CSP in DecMeg due to huge number of MEG sensors). We've played a little with your DecMeg code and noticed two issues you mentioned (off-line analysis only and a requirement of balanced classes). Not so evident result (for me) is that performance of retraining strongly depends on distance type. The best choice is riemann (loo accuracy is ~0.756); logeuclid also shows competitive results (~0.744). Euclid doesn't work at all (~0.625). The most surprising result is that kullback slightly improves (~0.728) result of the generic classifier, but still shows worse accuracy than riemann. If anyone had asked me how to measure distance between two distributions I would have answered 'certainly kullback leibler'. Now I'm not so confident :) 

\n

I understand that performance of different metrics depends on a particular dataset, so kullback can get own back and exceed riemann in another problem. But I have no idea why logeuclid works at all :) Both riemann and kullback have some understandable mathematical basis (thank you for the IG book, I'll definitely read it!), but what about logeuclid? Could you please give a short comment why does it work?

\n

Rafal, 

\n

I agree, global AUC effect depends on sample size, so it boosts performance on the large Private set in comparison with single-subject AUC. However, the Public set is rather small, but your approach shows almost the same performance as for the Private set. Probably the reason is that there are "easy" subjects in the Public set.

\n

Thanks again,

\n

Mike

", "votes": null, "replies": [] }, { "id": 72858, "author_name": "sunu3848", "author_url": "", "post_date": "04/21/2015 06:17:08", "content": "

hi sir, is there anyway we can have your features(not the meta features),it would be great help.we need to examine how that works in our classifier,and the accuracy it could manage with your features..

\n

thanking you in advance,

\n

Sunu A

", "votes": null, "replies": [] }, { "id": 72966, "author_name": "alexandrebarachant", "author_url": "", "post_date": "04/21/2015 17:02:43", "content": "

Hi Mike,

\n

Thanks for looking into the decMeg code and pointing the issue of the metric. 

\n\n

if i have to rank the metric by their efficiency on EEG data, i would do it like this :

\n

Riemann > Log-det > Log-euclid > kullback > Euclidean

\n

It may be different for other problem, as you said. The log-euclidean seems very popular in image processing and computer vision. 

\n

As a side note, i just released a python package, pyRiemann. It is an extended version of the code developed for this challenge, and it support the different metrics I mention in this post (except the kullblack, but i accept pull request :) )

", "votes": null, "replies": [] }, { "id": 72969, "author_name": "alexandrebarachant", "author_url": "", "post_date": "04/21/2015 17:10:44", "content": "

[quote=Sunu A;72858]

\n

hi sir, is there anyway we can have your features(not the meta features),it would be great help.we need to examine how that works in our classifier,and the accuracy it could manage with your features..

\n

thanking you in advance,

\n

Sunu A

\n

[/quote]

\n

Hi,

\n

The feature extraction we used include numerous supervised steps, like Xdawn spatial filtering, special form covariance estimation or channel selection. Giving you the raw feature will lead to biased results.

\n

However, we published the code with a easy mechanism to test your ideas. All of this is explained here. All you have to do is to build a python class that inherit from the sklearn base API (basically, two method, fit() and predict() ) and then declare it in the parameter file.

\n

This way, you will evaluates the performance of your classifier in the same scheme as our, and you will get comparable results.

\n\n

Alex

", "votes": null, "replies": [] }, { "id": 77258, "author_name": "sunu3848", "author_url": "", "post_date": "05/06/2015 06:23:59", "content": "

Hello  sir...

\n

                 i have tried your code but i keep on ending up with memory error.As you have noted, i reduced my no of cores but still...In between,my system has only 4GB ram with four cores.Does your code works with my system spec?  i'm very new to python

\n\n\n\n

thanking you in advance

", "votes": null, "replies": [] }, { "id": 77277, "author_name": "mibell", "author_url": "", "post_date": "05/06/2015 08:30:51", "content": "

Hi Alexandre! 

\n

As usual, many thanks for your very detailed answer which helps us to better understand your ideas!

\n

1) About metrics: I totally agree with you :) Some comments are below.

\n\n

2) I reckon there is a huge problem in algorithmic development for BCI: it's hard to compare results from different publications. There are three main issues:

\n\n

(I've started exploring BCI data science problems only few months ago, so I may be wrong. Please correct me in that case.)

\n

So in my opinion pyRiemann release is a great news (as well as your Matlab covariance Toolbox)!

\n

I plan to finish my current project in aerospace in 1-2 months and to dive into analysis of BCI data. So I hope I will have something to commit in pyRiemann :)

\n\n

Best regards, 

\n

Mike

", "votes": null, "replies": [] }, { "id": 77278, "author_name": "mibell", "author_url": "", "post_date": "05/06/2015 08:33:50", "content": "

[quote=Sunu A;77258]

\n

Hello sir...

\n

i have tried your code but i keep on ending up with memory error.As you have noted, i reduced my no of cores but still...In between,my system has only 4GB ram with four cores.Does your code works with my system spec? i'm very new to python

\n

thanking you in advance

\n

[/quote]

\n

According to specification you need at least 8GB of RAM. It's rather typical situation for EEG analysis due to huge size of raw data.

\n

Best regards,

\n

Mike

", "votes": null, "replies": [] }, { "id": 77342, "author_name": "alexandrebarachant", "author_url": "", "post_date": "05/06/2015 15:46:13", "content": "

Hi Mike,

\n

Thank you for carrying this interesting discussion.

\n

I could not agree more with you :)

\n

Information geometry is gaining  momentum in a lot of different domain. We can see a lot of stuff in computer vision  for example. I would explain that by the facts that :

\n\n\n

About BCI, i totally share you point of view. There is a lack of reproducible research. The problem does not only comes from the methods implementation, but also from the environment it is applied. Results are so bad and datasets are so small that even changing the order of the frequential filter you apply as preprocessing step can change significantly the outcome of you classification.

\n

In addition, parameters are usually over-optimized for a specific dataset, and when you tried a new method on a new set of data, it require an high amount of domain specific knowledge to make it works properly. This partially explain the popularity of the 'old' methods like CSP. They does not give optimal results, but they are stable across a wide range of applications.

\n

The other problem is that EEG data are considered as medical data and are really hard to release publicly. You need consents to release from the subject, and it is rarely done before the experiment. There is a lot of data that are just sitting on the labs hard drive without any chance to be released even if you want to (i have tons of data like this).

\n

What we really need is a platform where you can submit your code and your method will be blindly evaluated on a wide range of dataset (you can even add realistic stress test, like adding noise on data or on labels, etc).

\n

Finally, thanks for your interest for pyRiemann. your commits will be greatly appreciated :)

\n

Best,

\n

Alexandre

", "votes": null, "replies": [] }, { "id": 91984, "author_name": "qasimgilani", "author_url": "", "post_date": "09/10/2015 06:53:54", "content": "

from where can i download data sets?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "66251": "", "66291": "", "66295": "", "66296": "", "66301": "", "66327": "", "66469": "", "66509": "", "67689": "", "67810": "", "72858": "", "72966": "", "72969": "", "77258": "", "77277": "", "77278": "", "77342": "", "91984": "from where can i download data sets?" }, "source": "meta" }