{ "id": 7919, "title": "Useful transformations", "url": "/competitions/decoding-the-human-brain/discussion/7919", "author_name": "", "post_date": "2014-04-29T09:17:45.283Z", "votes": null, "comment_count": 6, "views": 1933, "content": "

Here you can discuss about useful transformation on MEG data to extract higher level patterns.

", "messages": [ { "id": "43279", "postDate": "04/29/2014 09:17:45", "content": "

Here you can discuss about useful transformation on MEG data to extract higher level patterns.

", "rawMarkdown": "", "votes": null }, { "id": "43352", "postDate": "04/30/2014 09:39:50", "content": "

Are we allowed to do source reconstruction? Presumably it would involve using the full sensor layout, with 3D positions, which can be easily found in Fieldtrip. So would this violate the terms of the competition?

", "rawMarkdown": "", "votes": null }, { "id": "43427", "postDate": "05/01/2014 06:17:16", "content": "

Hi,

\n

We are discussing this point. An answer will follow soon. 

", "rawMarkdown": "", "votes": null }, { "id": "43442", "postDate": "05/01/2014 12:43:24", "content": "

Hi! I would assume that, in the spirit of good data competitions, any transformation of the data is allowed, right? All that should count is the final predicted result :)

", "rawMarkdown": "", "votes": null }, { "id": "43514", "postDate": "05/02/2014 14:42:33", "content": "

Dear All,

\n

After some discussion among the people organizing this competition, we decided to accept all possible transformations that the participants will propose, including source reconstruction, i.e. approximating the currents/dipoles of the brain that generated the observed magnetic field measured by the MEG.

\n

A potential problem with source reconstruction is the use of external information not provided in the data page of this competition - which is forbidden by the rules. For example: the detailed sensor layout available from some MEG data analysis libraries, or even more detailed information (available only for the training subjects) from the original dataset from which the data of this competition is extracted. Since it is difficult to define a clear boundary here, we decided to accept everything. In some sense, this decision weakens the rule of not using external information.

\n

We kindly invite the participants expert on source reconstruction to share code with all the other participants. We - competition hosts - do not have plan to release example code for source reconstruction.

\n

A final note: decoding from the reconstructed sources instead of the MEG data, is a recent and almost unexplored trend in MEG data analysis. Our preliminary practical experience is that there is no clear gain in that direction, at least for the problem of this competition. But again, this is almost unknown territory.

", "rawMarkdown": "", "votes": null }, { "id": "44054", "postDate": "05/06/2014 15:36:58", "content": "

Another question on a possible transformation. It is of course standard practice to transform the input features before classification such that they are expressed in the same unit, e.g. z-transform or a scaling between 0 and 1. Usually in machine learning one would do this transform based *only* on quantities estimated from the training set (i.e. estimate mean and sd of training set, and compute z(Xtest) as z(Xtest) = (Xtest - meanTrain) / sdTrain). However, in this particular competition we have access to all the testing data beforehand, i.e. there is no requirement to classify entirely unseen data. Therefore we could do a z-scoring using all data, test and train combined.

\n

I believe this should not be allowed, as in effect we are then using test data to transform the training set (it has been shown that this can significantly improve classifier accuracy). Could an admin confirm that this is indeed not allowed? Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "44152", "postDate": "05/07/2014 09:10:05", "content": "

Hi Eelke,

\n

I disagree with you. In this competition you have MEG measurements (X) and the associated categories of stimulus (y). As long as you don't use information about "y" of the test set, there is no issue or unwanted bias in the analysis that generates the submission. So feel free to use X of the test set. If you are concerned about circularity / double-dipping in the estimate of the score then you shouldn't.

\n

This competition is about predicting y on new subjects and it is a known fact that the X of different subjects is systematically different. For this reason, one possible approach to the create an effective prediction is exactly to use the X of the test subject in order to adapt (or to "transfer", as some literature names it) the knowledge that you get from (X,y) of the train subject(s).

\n

In a standard classification setting, to which you refer in your message, you want to create a classifier that is not specific of a single test set. For this reason you don't want use X of the test set in the process. But this is valid as long as you assume that both the train and test sets come from the same distribution. And in that case you are interested in the expected score over future examples (here trials). If that is not the case - as it is here - then there are other approaches which might be more effective than the standard one - and they use X of the test set. For example semi-supervised learning and transfer learning do that.

\n

Notice that the scientific questions that we are addressing with this competition is: "what is the prediction accuracy that we should expect on new subjects?". So it is fine if participants want to adapt the same method for each subject in the test set. Because we are interested in the expected accuracy over subjects.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 43352, "author_name": "mirceastoica", "author_url": "", "post_date": "04/30/2014 09:39:50", "content": "

Are we allowed to do source reconstruction? Presumably it would involve using the full sensor layout, with 3D positions, which can be easily found in Fieldtrip. So would this violate the terms of the competition?

", "votes": null, "replies": [] }, { "id": 43427, "author_name": "emanuele", "author_url": "", "post_date": "05/01/2014 06:17:16", "content": "

Hi,

\n

We are discussing this point. An answer will follow soon. 

", "votes": null, "replies": [] }, { "id": 43442, "author_name": "eelkespaak", "author_url": "", "post_date": "05/01/2014 12:43:24", "content": "

Hi! I would assume that, in the spirit of good data competitions, any transformation of the data is allowed, right? All that should count is the final predicted result :)

", "votes": null, "replies": [] }, { "id": 43514, "author_name": "emanuele", "author_url": "", "post_date": "05/02/2014 14:42:33", "content": "

Dear All,

\n

After some discussion among the people organizing this competition, we decided to accept all possible transformations that the participants will propose, including source reconstruction, i.e. approximating the currents/dipoles of the brain that generated the observed magnetic field measured by the MEG.

\n

A potential problem with source reconstruction is the use of external information not provided in the data page of this competition - which is forbidden by the rules. For example: the detailed sensor layout available from some MEG data analysis libraries, or even more detailed information (available only for the training subjects) from the original dataset from which the data of this competition is extracted. Since it is difficult to define a clear boundary here, we decided to accept everything. In some sense, this decision weakens the rule of not using external information.

\n

We kindly invite the participants expert on source reconstruction to share code with all the other participants. We - competition hosts - do not have plan to release example code for source reconstruction.

\n

A final note: decoding from the reconstructed sources instead of the MEG data, is a recent and almost unexplored trend in MEG data analysis. Our preliminary practical experience is that there is no clear gain in that direction, at least for the problem of this competition. But again, this is almost unknown territory.

", "votes": null, "replies": [] }, { "id": 44054, "author_name": "eelkespaak", "author_url": "", "post_date": "05/06/2014 15:36:58", "content": "

Another question on a possible transformation. It is of course standard practice to transform the input features before classification such that they are expressed in the same unit, e.g. z-transform or a scaling between 0 and 1. Usually in machine learning one would do this transform based *only* on quantities estimated from the training set (i.e. estimate mean and sd of training set, and compute z(Xtest) as z(Xtest) = (Xtest - meanTrain) / sdTrain). However, in this particular competition we have access to all the testing data beforehand, i.e. there is no requirement to classify entirely unseen data. Therefore we could do a z-scoring using all data, test and train combined.

\n

I believe this should not be allowed, as in effect we are then using test data to transform the training set (it has been shown that this can significantly improve classifier accuracy). Could an admin confirm that this is indeed not allowed? Thanks!

", "votes": null, "replies": [] }, { "id": 44152, "author_name": "emanuele", "author_url": "", "post_date": "05/07/2014 09:10:05", "content": "

Hi Eelke,

\n

I disagree with you. In this competition you have MEG measurements (X) and the associated categories of stimulus (y). As long as you don't use information about "y" of the test set, there is no issue or unwanted bias in the analysis that generates the submission. So feel free to use X of the test set. If you are concerned about circularity / double-dipping in the estimate of the score then you shouldn't.

\n

This competition is about predicting y on new subjects and it is a known fact that the X of different subjects is systematically different. For this reason, one possible approach to the create an effective prediction is exactly to use the X of the test subject in order to adapt (or to "transfer", as some literature names it) the knowledge that you get from (X,y) of the train subject(s).

\n

In a standard classification setting, to which you refer in your message, you want to create a classifier that is not specific of a single test set. For this reason you don't want use X of the test set in the process. But this is valid as long as you assume that both the train and test sets come from the same distribution. And in that case you are interested in the expected score over future examples (here trials). If that is not the case - as it is here - then there are other approaches which might be more effective than the standard one - and they use X of the test set. For example semi-supervised learning and transfer learning do that.

\n

Notice that the scientific questions that we are addressing with this competition is: "what is the prediction accuracy that we should expect on new subjects?". So it is fine if participants want to adapt the same method for each subject in the test set. Because we are interested in the expected accuracy over subjects.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "43279": "", "43352": "", "43427": "", "43442": "", "43514": "", "44054": "", "44152": "" }, "source": "meta" }