{ "id": 7899, "title": "MAT to CSV converter", "url": "/competitions/decoding-the-human-brain/discussion/7899", "author_name": "", "post_date": "2014-04-27T17:28:44.650Z", "votes": 9, "comment_count": 4, "views": 6297, "content": "

Hi,

\n

I prepared a converter to create a comma-separated value (CSV) file from each of the original Matlab v5 (MAT) files of the competition.The converter is in Python (v2.7.x) and requires only SciPy to be installed in order to work:

\n

https://github.com/FBK-NILab/DecMeg2014/blob/master/python/mat2csv.py

\n

Just issuing "python mat2csv.py", or executing mat2csv.py, is enough to convert all files, provided that the variables mat_dir and csv_dir are correct in your setting. Please have a look to the code - it should be pretty easy to understand and to adapt.

\n

The format of the CSV file is straightforward: each line of the CSV file is a trial. For train data, the first value is the class-label of the trial (y), i.e. the category of the stimulus presented to the subject (1=Face, 0=Scramble). For test data, the first value is the Id of the trial. In both cases, the remaining values in the line are the MEG values (X) of the multiple timeseries of that trial stored sequentially. Each timeseries consists of 375 timepoints, i.e. 1.5sec recorded at 250Hz. The groups of 375 values, one for each of the 306 channel/sensor, are stored sequentially, with the channels/sensors in the same order as in the MAT file. For this reason, each line consists of (1 + 375 x 306) = 114751 values.

\n

I think that it is not possible to upload the CSV (or compressed CSV) files in Kaggle, because they are too large. Consider that each CSV file is approximately 2Gb. In my preliminary tests, the gzip compressor shrinks each CSV file to approximately 900Mb. For comparison, the corresponding MAT file is 260Mb only. Part of the reason for this notable difference is the txt vs. binary format. Another reason is that, in MAT files, values are stored as float32, while in CSV files I used put 20-digits IEEE exponential format - which may be too conservative.

\n

So, in my opinion, if you need CSV data, it is better to download the MAT files distributed with the competition, and then to transform them in CSV format on your local disk with this converter. The only difficulty is to run the converter, i.e. to have Python and SciPy installed.

\n

Note: the converter is not extensively tested, so if you find a bug or want to propose an enhancement, you are warmly invited to contact me or to send pull requests through Gihub.

", "messages": [ { "id": "43169", "postDate": "04/27/2014 17:28:44", "content": "

Hi,

\n

I prepared a converter to create a comma-separated value (CSV) file from each of the original Matlab v5 (MAT) files of the competition.The converter is in Python (v2.7.x) and requires only SciPy to be installed in order to work:

\n

https://github.com/FBK-NILab/DecMeg2014/blob/master/python/mat2csv.py

\n

Just issuing "python mat2csv.py", or executing mat2csv.py, is enough to convert all files, provided that the variables mat_dir and csv_dir are correct in your setting. Please have a look to the code - it should be pretty easy to understand and to adapt.

\n

The format of the CSV file is straightforward: each line of the CSV file is a trial. For train data, the first value is the class-label of the trial (y), i.e. the category of the stimulus presented to the subject (1=Face, 0=Scramble). For test data, the first value is the Id of the trial. In both cases, the remaining values in the line are the MEG values (X) of the multiple timeseries of that trial stored sequentially. Each timeseries consists of 375 timepoints, i.e. 1.5sec recorded at 250Hz. The groups of 375 values, one for each of the 306 channel/sensor, are stored sequentially, with the channels/sensors in the same order as in the MAT file. For this reason, each line consists of (1 + 375 x 306) = 114751 values.

\n

I think that it is not possible to upload the CSV (or compressed CSV) files in Kaggle, because they are too large. Consider that each CSV file is approximately 2Gb. In my preliminary tests, the gzip compressor shrinks each CSV file to approximately 900Mb. For comparison, the corresponding MAT file is 260Mb only. Part of the reason for this notable difference is the txt vs. binary format. Another reason is that, in MAT files, values are stored as float32, while in CSV files I used put 20-digits IEEE exponential format - which may be too conservative.

\n

So, in my opinion, if you need CSV data, it is better to download the MAT files distributed with the competition, and then to transform them in CSV format on your local disk with this converter. The only difficulty is to run the converter, i.e. to have Python and SciPy installed.

\n

Note: the converter is not extensively tested, so if you find a bug or want to propose an enhancement, you are warmly invited to contact me or to send pull requests through Gihub.

", "rawMarkdown": "", "votes": null }, { "id": "44343", "postDate": "05/10/2014 14:17:35", "content": "

Since I spent some essential time to process mat-file in pure C++, I put here potentially useful information.

\n

The main hint is that Matlab uses unusual (for C users) order for walking through indices: first index is incremented first.

\n

First two attached plots are wrong because of mistake on indices walking order, but the last two plots look right.

\n

For subject=01, trial=100, sensor=MEG1731 (195-th sensor)  I got the values: 5.82E-13, 5.39E-13, 1.66E-13, 8.88E-14, 3.22E-13, etc...

", "rawMarkdown": "", "votes": null }, { "id": "44359", "postDate": "05/11/2014 05:29:14", "content": "

Since I have only 2 Gb RAM on my PC I unable to reproduce the benchmark. Emanuele, could you upload the array of coefficients obtained by logistic regression?

\n

The last plot (with Z-transformation) in previous post looks to be essentially different from the plot given in description. Has anybody reproduced such plots?

", "rawMarkdown": "", "votes": null }, { "id": "44399", "postDate": "05/12/2014 10:24:17", "content": "

Hi,

\n

If you have only 2Gb of RAM you may want to have a look to this thread about using very efficient online learning.

\n

Notice that the timeseries shown in the figure of the description page, are averages over all trials of each class (blue = Face, red = Scrambled Face). This is a common way to show the recordings in MEG data analysis. Moreover I picked the sensors for which this average difference is clearly visible, for didactical purpose. If you look at single trials, as you did, the noise covers almost everything of the signal of the mental process of interest. Moreover, in many locations the signal is too low to be measured (= there is no signal at all).

", "rawMarkdown": "", "votes": null }, { "id": "875904", "postDate": "06/06/2020 08:52:44", "content": "

https://www.kaggle.com/vinayak123tyagi/mat-to-csv-code

", "rawMarkdown": "https://www.kaggle.com/vinayak123tyagi/mat-to-csv-code", "votes": null } ], "comments": [ { "id": 44343, "author_name": "nedelko", "author_url": "", "post_date": "05/10/2014 14:17:35", "content": "

Since I spent some essential time to process mat-file in pure C++, I put here potentially useful information.

\n

The main hint is that Matlab uses unusual (for C users) order for walking through indices: first index is incremented first.

\n

First two attached plots are wrong because of mistake on indices walking order, but the last two plots look right.

\n

For subject=01, trial=100, sensor=MEG1731 (195-th sensor)  I got the values: 5.82E-13, 5.39E-13, 1.66E-13, 8.88E-14, 3.22E-13, etc...

", "votes": null, "replies": [] }, { "id": 44359, "author_name": "nedelko", "author_url": "", "post_date": "05/11/2014 05:29:14", "content": "

Since I have only 2 Gb RAM on my PC I unable to reproduce the benchmark. Emanuele, could you upload the array of coefficients obtained by logistic regression?

\n

The last plot (with Z-transformation) in previous post looks to be essentially different from the plot given in description. Has anybody reproduced such plots?

", "votes": null, "replies": [] }, { "id": 44399, "author_name": "emanuele", "author_url": "", "post_date": "05/12/2014 10:24:17", "content": "

Hi,

\n

If you have only 2Gb of RAM you may want to have a look to this thread about using very efficient online learning.

\n

Notice that the timeseries shown in the figure of the description page, are averages over all trials of each class (blue = Face, red = Scrambled Face). This is a common way to show the recordings in MEG data analysis. Moreover I picked the sensors for which this average difference is clearly visible, for didactical purpose. If you look at single trials, as you did, the noise covers almost everything of the signal of the mental process of interest. Moreover, in many locations the signal is too low to be measured (= there is no signal at all).

", "votes": null, "replies": [] }, { "id": 875904, "author_name": "vinayak123tyagi", "author_url": "", "post_date": "06/06/2020 08:52:44", "content": "

https://www.kaggle.com/vinayak123tyagi/mat-to-csv-code

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "43169": "", "44343": "", "44359": "", "44399": "", "875904": "https://www.kaggle.com/vinayak123tyagi/mat-to-csv-code" }, "source": "meta" }