{ "id": 7882, "title": "Python preprocess of MAT Files", "url": "/competitions/decoding-the-human-brain/discussion/7882", "author_name": "", "post_date": "2014-04-26T14:40:06.143Z", "votes": null, "comment_count": 8, "views": 2837, "content": "

Hey,

\n\n

I'm C/C++ programmer and need want to have a look in to the data.

\n

With youre Example-Code on Kaggle ive tried to convert the train MAT files to one large ascii file with the format

\n\n

ID    CHANNEL  TARGET   MEASURE#1     MEASURE#2   ... MEASURE#375

\n

this is the code :

\n

#!/usr/bin/python

\n

import numpy as np
from scipy.io import loadmat

\n

subjects_train = range(1, 17)
fp = open('train.dat', 'w')

\n

for subject in subjects_train:

filename = 'train_subject%02d.mat' % subject
data = loadmat(filename, squeeze_me=True)

\n

X = data['X' ]
Y = data['y' ]

#X -= X.mean(0)
#X = np.nan_to_num(X / X.std(0))

\n

trails=len( X )
for trail in range( trails ):
id='%02d%03d' % (subject, trail)
print id
channels=len(X[0])
for channel in range( channels ):
times=len( X[0][0] )
fp.write( id + "\\t" + str(channel) + "\\t"+str(Y[ trail]) )
for time in range( times ):
fp.write( "\\t" + "%20.17f " % ( X[ trail ][ channel ][ time ]) )
fp.write("\\n")
fp.close()

\n\n

is there anything wrong with it ? Couze the values lookes really small ? :-/

", "messages": [ { "id": "43080", "postDate": "04/26/2014 14:40:06", "content": "

Hey,

\n\n

I'm C/C++ programmer and need want to have a look in to the data.

\n

With youre Example-Code on Kaggle ive tried to convert the train MAT files to one large ascii file with the format

\n\n

ID    CHANNEL  TARGET   MEASURE#1     MEASURE#2   ... MEASURE#375

\n

this is the code :

\n

#!/usr/bin/python

\n

import numpy as np
from scipy.io import loadmat

\n

subjects_train = range(1, 17)
fp = open('train.dat', 'w')

\n

for subject in subjects_train:

filename = 'train_subject%02d.mat' % subject
data = loadmat(filename, squeeze_me=True)

\n

X = data['X' ]
Y = data['y' ]

#X -= X.mean(0)
#X = np.nan_to_num(X / X.std(0))

\n

trails=len( X )
for trail in range( trails ):
id='%02d%03d' % (subject, trail)
print id
channels=len(X[0])
for channel in range( channels ):
times=len( X[0][0] )
fp.write( id + "\\t" + str(channel) + "\\t"+str(Y[ trail]) )
for time in range( times ):
fp.write( "\\t" + "%20.17f " % ( X[ trail ][ channel ][ time ]) )
fp.write("\\n")
fp.close()

\n\n

is there anything wrong with it ? Couze the values lookes really small ? :-/

", "rawMarkdown": "", "votes": null }, { "id": "43088", "postDate": "04/26/2014 17:01:20", "content": "

Hi,

\n

The magnetic field induced by the currents in the brain is very very low. Consider that the unit of measure is Tesla (T) and the actual values are in the order of 10^-15 T, called femtoTesla (fT). More precisely, each magnetometer - i.e. the sensor measuring the z component of the magnetic field in one location - returns Tesla. Each gradiometer - i.e. the sensor measuring the x or y component of the gradient of the magnetic field in one location - returns Tesla/cm. In each of the 102 locations there is one magnetometer and two gradiometers.

\n

Having said that, it may be convenient for you to save the data in floating point exponential format, with "%e" (or "%g"), instead of  the decimal format "%20.17f" that you use. I guess "%1.20e" should be enough. See here for additional details:

\n

  https://docs.python.org/2/library/stdtypes.html#string-formatting

\n

  https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting

\n\n

Note: the setup of gradiometers/magnetometers and number of locations changes across different MEG devices. The one used for acquiring the data of this competition, i.e. Elekta Neuromag VectorView, is as described above.

", "rawMarkdown": "", "votes": null }, { "id": "43131", "postDate": "04/27/2014 08:51:27", "content": "

Hey,

\n

thank you really mutch for youre time to answer the question. You make mutch things clear for me.

\n

But in one lines, the challenge is to predict 2 classes ( Face and Scramble ) from the measure. There is no importans of analyset this https://github.com/mne-tools/mne-python/blob/master/mne/layouts/Vectorview-all.lout file ?

", "rawMarkdown": "", "votes": null }, { "id": "43134", "postDate": "04/27/2014 10:16:59", "content": "

Hi,

\n

The Vectorview-all.lout is a text file that describes the approximate 2D position of the sensors with respect to the head of the subject. I used that file and the data from subject 1 to create the figure in the description page of the competition. It might be useful to know these approximate positions in order to account for spatial correlations between the timeseries. But I have no strong opinion if this information can affect the score of the competition.

\n

I plan to release a bit more code in the next days, in the Github repository. For example the one I used to create the figure.

", "rawMarkdown": "", "votes": null }, { "id": "43157", "postDate": "04/27/2014 15:28:41", "content": "

I'm new to this sort of data, so my apologies if these questions can be inferred from the existing data description. 

\n", "rawMarkdown": "", "votes": null }, { "id": "43162", "postDate": "04/27/2014 16:09:08", "content": "

Hi,

\n

Thank you for your questions.

\n

The sensors are in the same order over all 23 subjects. The order of the channels in the dataset is the same as in Vectorview-all.lout that is in the Github repository of the competition. The other two files that you mention, which are available from a different repository (specifically here), provide the same information of Vectorview-all.lout, but split in two parts: Vectorview-mag for magnetometers and Vectorview-grad for gradiometers. So, in my opinion, the right way to proceed, is to use Vectorview-all.lout in order to retrieve the name and position of each sensor. For example, sensor "MEG 0113" has x=-73.416206 and y=33.416687 (see the 2nd line of the file). Notice that if the number in the sensor name ends with "1" then it is a magnetometer. If it ends with "2" or "3", then it is a gradiometer. For example, "MEG 0113" is a gradiometer and "MEG 0111" is a magnetometer.

", "rawMarkdown": "", "votes": null }, { "id": "43601", "postDate": "05/03/2014 12:07:55", "content": "

Hi Emanuele,

\n

I would suggest that you add this:

\n

[quote=Emanuele;43162]

\n

Notice that if the number in the sensor name ends with "1" then it is a magnetometer. If it ends with "2" or "3", then it is a gradiometer. For example, "MEG 0113" is a gradiometer and "MEG 0111" is a magnetometer.

\n

[/quote]

\n

to the description of the data. It may be useful for everyone (not just people reading "Python preprocess of MAT files").

\n

Thanks

", "rawMarkdown": "", "votes": null }, { "id": "43617", "postDate": "05/03/2014 15:40:25", "content": "

Good point. I'll ask to the Kaggle people to change the data description page. We don't have permission to do that anymore.

", "rawMarkdown": "", "votes": null }, { "id": "47345", "postDate": "05/30/2014 16:09:10", "content": "

Hello,

\n

What does the first column in Vectorview-all.lout specify?

\n

Thanks.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 43088, "author_name": "emanuele", "author_url": "", "post_date": "04/26/2014 17:01:20", "content": "

Hi,

\n

The magnetic field induced by the currents in the brain is very very low. Consider that the unit of measure is Tesla (T) and the actual values are in the order of 10^-15 T, called femtoTesla (fT). More precisely, each magnetometer - i.e. the sensor measuring the z component of the magnetic field in one location - returns Tesla. Each gradiometer - i.e. the sensor measuring the x or y component of the gradient of the magnetic field in one location - returns Tesla/cm. In each of the 102 locations there is one magnetometer and two gradiometers.

\n

Having said that, it may be convenient for you to save the data in floating point exponential format, with "%e" (or "%g"), instead of  the decimal format "%20.17f" that you use. I guess "%1.20e" should be enough. See here for additional details:

\n

  https://docs.python.org/2/library/stdtypes.html#string-formatting

\n

  https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting

\n\n

Note: the setup of gradiometers/magnetometers and number of locations changes across different MEG devices. The one used for acquiring the data of this competition, i.e. Elekta Neuromag VectorView, is as described above.

", "votes": null, "replies": [] }, { "id": 43131, "author_name": "robstr", "author_url": "", "post_date": "04/27/2014 08:51:27", "content": "

Hey,

\n

thank you really mutch for youre time to answer the question. You make mutch things clear for me.

\n

But in one lines, the challenge is to predict 2 classes ( Face and Scramble ) from the measure. There is no importans of analyset this https://github.com/mne-tools/mne-python/blob/master/mne/layouts/Vectorview-all.lout file ?

", "votes": null, "replies": [] }, { "id": 43134, "author_name": "emanuele", "author_url": "", "post_date": "04/27/2014 10:16:59", "content": "

Hi,

\n

The Vectorview-all.lout is a text file that describes the approximate 2D position of the sensors with respect to the head of the subject. I used that file and the data from subject 1 to create the figure in the description page of the competition. It might be useful to know these approximate positions in order to account for spatial correlations between the timeseries. But I have no strong opinion if this information can affect the score of the competition.

\n

I plan to release a bit more code in the next days, in the Github repository. For example the one I used to create the figure.

", "votes": null, "replies": [] }, { "id": 43157, "author_name": "alexstephens", "author_url": "", "post_date": "04/27/2014 15:28:41", "content": "

I'm new to this sort of data, so my apologies if these questions can be inferred from the existing data description. 

\n", "votes": null, "replies": [] }, { "id": 43162, "author_name": "emanuele", "author_url": "", "post_date": "04/27/2014 16:09:08", "content": "

Hi,

\n

Thank you for your questions.

\n

The sensors are in the same order over all 23 subjects. The order of the channels in the dataset is the same as in Vectorview-all.lout that is in the Github repository of the competition. The other two files that you mention, which are available from a different repository (specifically here), provide the same information of Vectorview-all.lout, but split in two parts: Vectorview-mag for magnetometers and Vectorview-grad for gradiometers. So, in my opinion, the right way to proceed, is to use Vectorview-all.lout in order to retrieve the name and position of each sensor. For example, sensor "MEG 0113" has x=-73.416206 and y=33.416687 (see the 2nd line of the file). Notice that if the number in the sensor name ends with "1" then it is a magnetometer. If it ends with "2" or "3", then it is a gradiometer. For example, "MEG 0113" is a gradiometer and "MEG 0111" is a magnetometer.

", "votes": null, "replies": [] }, { "id": 43601, "author_name": "gbrdata", "author_url": "", "post_date": "05/03/2014 12:07:55", "content": "

Hi Emanuele,

\n

I would suggest that you add this:

\n

[quote=Emanuele;43162]

\n

Notice that if the number in the sensor name ends with "1" then it is a magnetometer. If it ends with "2" or "3", then it is a gradiometer. For example, "MEG 0113" is a gradiometer and "MEG 0111" is a magnetometer.

\n

[/quote]

\n

to the description of the data. It may be useful for everyone (not just people reading "Python preprocess of MAT files").

\n

Thanks

", "votes": null, "replies": [] }, { "id": 43617, "author_name": "emanuele", "author_url": "", "post_date": "05/03/2014 15:40:25", "content": "

Good point. I'll ask to the Kaggle people to change the data description page. We don't have permission to do that anymore.

", "votes": null, "replies": [] }, { "id": 47345, "author_name": "jaidevd", "author_url": "", "post_date": "05/30/2014 16:09:10", "content": "

Hello,

\n

What does the first column in Vectorview-all.lout specify?

\n

Thanks.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "43080": "", "43088": "", "43131": "", "43134": "", "43157": "", "43162": "", "43601": "", "43617": "", "47345": "" }, "source": "meta" }