{ "id": 10319, "title": ".mat to .csv", "url": "/competitions/seizure-prediction/discussion/10319", "author_name": "", "post_date": "2014-09-13T13:07:48.227Z", "votes": 1, "comment_count": 20, "views": 7221, "content": "

Is there a way to convert the .mat files to .csv files? I am not using matlab so I can't read the data.

", "messages": [ { "id": "53638", "postDate": "09/13/2014 13:07:48", "content": "

Is there a way to convert the .mat files to .csv files? I am not using matlab so I can't read the data.

", "rawMarkdown": "", "votes": null }, { "id": "53658", "postDate": "09/13/2014 18:50:39", "content": "

What are you using?

", "rawMarkdown": "", "votes": null }, { "id": "53659", "postDate": "09/13/2014 19:03:09", "content": "

scipy has a loadmat function that will allow you to load the file, from there you could export to csv if you're not comfortable with python.

\n

But to be honest you're better off using .mat or .hkl (hickle) because they are binary file types, i think the CSV will be even bigger.

", "rawMarkdown": "", "votes": null }, { "id": "53669", "postDate": "09/14/2014 00:37:17", "content": "

@elyase I am using c++.

\n\n

@franklyn I know a little bit of python and I'll look into it. Thanks

", "rawMarkdown": "", "votes": null }, { "id": "53670", "postDate": "09/14/2014 01:37:45", "content": "

You might find this library helpful

\n\n

http://sourceforge.net/projects/matio/

", "rawMarkdown": "", "votes": null }, { "id": "53676", "postDate": "09/14/2014 03:41:37", "content": "

Thanks franklyn. It made my life a lot easier. I've done a ton of programming in c++ but I barely know the basics of any other language.

", "rawMarkdown": "", "votes": null }, { "id": "53678", "postDate": "09/14/2014 03:43:12", "content": "

what machine learning library do you use in c++ ?

", "rawMarkdown": "", "votes": null }, { "id": "53682", "postDate": "09/14/2014 08:05:30", "content": "

I am not using any machine learning library. I am planning on coding everything myself.

", "rawMarkdown": "", "votes": null }, { "id": "53688", "postDate": "09/14/2014 12:30:25", "content": "

You can use a C-library called 'matio' to read the mat file and then, if you want, save it it in usual binary format.

", "rawMarkdown": "", "votes": null }, { "id": "53745", "postDate": "09/15/2014 02:04:53", "content": "

In the previous competition I was using hickle (wraps h5py), this time around with a much bigger dataset I'm using h5py directly as hickle seemed to behave strangely slow for me when dealing with 1000s of files. FYI h5py uses the hdf5 file format.

\n

During my testing the original mat format seems to perform the same as hdf5 with regards to speed of loading data. So there seems to be no harm in sticking with mat.

\n

Based on my experience with csv in the MNIST dataset, you really don't want to use csv. It's going to be bigger and much much slower to load.

", "rawMarkdown": "", "votes": null }, { "id": "54037", "postDate": "09/16/2014 16:56:10", "content": "

Hi everyone,

\n

I downloaded the matio library, but the documentation is nearly inexistant.

\n

If someone using this library could give an example of code for the transformation of the .mat into a .csv it would be nice.

\n

I think I should use the mat_varRead function, but it doesn't work with the "data" variable.

\n

Which variable name should I give to mat_varRead to get the real data?

\n

Thanks

", "rawMarkdown": "", "votes": null }, { "id": "54058", "postDate": "09/16/2014 18:46:03", "content": "

Hi

\n

In the example I include as atacchment, the data matrix  from "Patient_2_interictal_segment_0001.mat" is read into an array of double.

\n

Then you can put it into  a file in any format (csv or binary) or just use it in your C/C++ functions.

\n

you can compile it with :

\n

gcc readdatamatrixfrom_matfile.c -lmatio

\n\n

I'm using matio version 1.3 . If you use the new version (1.5) the instructions:

\n

field = Mat_VarGetStructField(structname, &index,BY_INDEX,0);

\n

have to be replaced by 

\n

field = Mat_VarGetStructFieldByIndex(structname, index,0);

\n

There an example  that helped me here 

\n

http://sourceforge.net/p/matio/discussion/609377/thread/b703ce7a/

", "rawMarkdown": "", "votes": null }, { "id": "54150", "postDate": "09/17/2014 14:12:37", "content": "

Thank you very much, it's very helpful.

", "rawMarkdown": "", "votes": null }, { "id": "54261", "postDate": "09/18/2014 20:35:58", "content": "

can someone give me an example of data format from the mat file ? I am trying to improve on the example given ? 

", "rawMarkdown": "", "votes": null }, { "id": "54304", "postDate": "09/19/2014 15:30:19", "content": "

Why is everything stored as double if they are just short ints?

", "rawMarkdown": "", "votes": null }, { "id": "54306", "postDate": "09/19/2014 16:03:30", "content": "

'double' is the default type for matlab (*.mat are matlabfiles). So if you want to read the binary mat file you get the original data as 'double'. In 'Dog* ' data the sampling frequency  is not an int, it's value is is 399.66 . So I would no risk to convert all the data to 'int' after reading it to 'double'.

", "rawMarkdown": "", "votes": null }, { "id": "54573", "postDate": "09/24/2014 04:35:54", "content": "

[quote=Apple314;53682]

\n

I am not using any machine learning library. I am planning on coding everything myself.

\n

[/quote]

\n\n

Thank you for this.

", "rawMarkdown": "", "votes": null }, { "id": "54574", "postDate": "09/24/2014 04:38:56", "content": "

be nice.

", "rawMarkdown": "", "votes": null }, { "id": "56251", "postDate": "10/19/2014 07:36:12", "content": "

Has anyone used matio with visual C+ 2010 express? I cant even get the thing to compile.

\n

I was able to use one of the project files that came with it. but then I needed a library called "hdf5" which i downloaded from somewhere, but it didn't come with the header file. This is costing me way too much time, and life would be a lot simpler if kaggle just gave the files in zipped csv format which would be the same size anyway.

\n

if anyone has a ready to use recipe for me to convert the files to csv, either with matio or another way, I would be really greatful.

", "rawMarkdown": "", "votes": null }, { "id": "106593", "postDate": "02/02/2016 09:30:10", "content": "
import scipy.io\nimport numpy as np\ndata = scipy.io.loadmat("file.mat")\n\nfor i in data:\n     if '__' not in i and 'readme' not in i:\n           np.savetxt(("file.csv"),data[i],delimiter=',')\n
", "rawMarkdown": "import scipy.io\r\n import numpy as np\r\n data = scipy.io.loadmat(\"file.mat\")\r\n\r\n for i in data:\r\n if '__' not in i and 'readme' not in i:\r\n np.savetxt((\"file.csv\"),data[i],delimiter=',')", "votes": null }, { "id": "748710", "postDate": "02/17/2020 21:59:09", "content": "

This gives the TypeError: only size-1 arrays can be converted to Python scalars

", "rawMarkdown": "This gives the TypeError: only size-1 arrays can be converted to Python scalars", "votes": null } ], "comments": [ { "id": 53658, "author_name": "elyase", "author_url": "", "post_date": "09/13/2014 18:50:39", "content": "

What are you using?

", "votes": null, "replies": [] }, { "id": 53659, "author_name": "franklyn", "author_url": "", "post_date": "09/13/2014 19:03:09", "content": "

scipy has a loadmat function that will allow you to load the file, from there you could export to csv if you're not comfortable with python.

\n

But to be honest you're better off using .mat or .hkl (hickle) because they are binary file types, i think the CSV will be even bigger.

", "votes": null, "replies": [] }, { "id": 53669, "author_name": "", "author_url": "", "post_date": "09/14/2014 00:37:17", "content": "

@elyase I am using c++.

\n\n

@franklyn I know a little bit of python and I'll look into it. Thanks

", "votes": null, "replies": [] }, { "id": 53670, "author_name": "franklyn", "author_url": "", "post_date": "09/14/2014 01:37:45", "content": "

You might find this library helpful

\n\n

http://sourceforge.net/projects/matio/

", "votes": null, "replies": [] }, { "id": 53676, "author_name": "", "author_url": "", "post_date": "09/14/2014 03:41:37", "content": "

Thanks franklyn. It made my life a lot easier. I've done a ton of programming in c++ but I barely know the basics of any other language.

", "votes": null, "replies": [] }, { "id": 53678, "author_name": "franklyn", "author_url": "", "post_date": "09/14/2014 03:43:12", "content": "

what machine learning library do you use in c++ ?

", "votes": null, "replies": [] }, { "id": 53682, "author_name": "", "author_url": "", "post_date": "09/14/2014 08:05:30", "content": "

I am not using any machine learning library. I am planning on coding everything myself.

", "votes": null, "replies": [] }, { "id": 53688, "author_name": "ruiprodrigues", "author_url": "", "post_date": "09/14/2014 12:30:25", "content": "

You can use a C-library called 'matio' to read the mat file and then, if you want, save it it in usual binary format.

", "votes": null, "replies": [] }, { "id": 53745, "author_name": "michaelhills", "author_url": "", "post_date": "09/15/2014 02:04:53", "content": "

In the previous competition I was using hickle (wraps h5py), this time around with a much bigger dataset I'm using h5py directly as hickle seemed to behave strangely slow for me when dealing with 1000s of files. FYI h5py uses the hdf5 file format.

\n

During my testing the original mat format seems to perform the same as hdf5 with regards to speed of loading data. So there seems to be no harm in sticking with mat.

\n

Based on my experience with csv in the MNIST dataset, you really don't want to use csv. It's going to be bigger and much much slower to load.

", "votes": null, "replies": [] }, { "id": 54037, "author_name": "ws45100529", "author_url": "", "post_date": "09/16/2014 16:56:10", "content": "

Hi everyone,

\n

I downloaded the matio library, but the documentation is nearly inexistant.

\n

If someone using this library could give an example of code for the transformation of the .mat into a .csv it would be nice.

\n

I think I should use the mat_varRead function, but it doesn't work with the "data" variable.

\n

Which variable name should I give to mat_varRead to get the real data?

\n

Thanks

", "votes": null, "replies": [] }, { "id": 54058, "author_name": "ruiprodrigues", "author_url": "", "post_date": "09/16/2014 18:46:03", "content": "

Hi

\n

In the example I include as atacchment, the data matrix  from "Patient_2_interictal_segment_0001.mat" is read into an array of double.

\n

Then you can put it into  a file in any format (csv or binary) or just use it in your C/C++ functions.

\n

you can compile it with :

\n

gcc readdatamatrixfrom_matfile.c -lmatio

\n\n

I'm using matio version 1.3 . If you use the new version (1.5) the instructions:

\n

field = Mat_VarGetStructField(structname, &index,BY_INDEX,0);

\n

have to be replaced by 

\n

field = Mat_VarGetStructFieldByIndex(structname, index,0);

\n

There an example  that helped me here 

\n

http://sourceforge.net/p/matio/discussion/609377/thread/b703ce7a/

", "votes": null, "replies": [] }, { "id": 54150, "author_name": "ws45100529", "author_url": "", "post_date": "09/17/2014 14:12:37", "content": "

Thank you very much, it's very helpful.

", "votes": null, "replies": [] }, { "id": 54261, "author_name": "chamarts", "author_url": "", "post_date": "09/18/2014 20:35:58", "content": "

can someone give me an example of data format from the mat file ? I am trying to improve on the example given ? 

", "votes": null, "replies": [] }, { "id": 54304, "author_name": "", "author_url": "", "post_date": "09/19/2014 15:30:19", "content": "

Why is everything stored as double if they are just short ints?

", "votes": null, "replies": [] }, { "id": 54306, "author_name": "ruiprodrigues", "author_url": "", "post_date": "09/19/2014 16:03:30", "content": "

'double' is the default type for matlab (*.mat are matlabfiles). So if you want to read the binary mat file you get the original data as 'double'. In 'Dog* ' data the sampling frequency  is not an int, it's value is is 399.66 . So I would no risk to convert all the data to 'int' after reading it to 'double'.

", "votes": null, "replies": [] }, { "id": 54573, "author_name": "frederickhadley", "author_url": "", "post_date": "09/24/2014 04:35:54", "content": "

[quote=Apple314;53682]

\n

I am not using any machine learning library. I am planning on coding everything myself.

\n

[/quote]

\n\n

Thank you for this.

", "votes": null, "replies": [] }, { "id": 54574, "author_name": "franklyn", "author_url": "", "post_date": "09/24/2014 04:38:56", "content": "

be nice.

", "votes": null, "replies": [] }, { "id": 56251, "author_name": "jossmit", "author_url": "", "post_date": "10/19/2014 07:36:12", "content": "

Has anyone used matio with visual C+ 2010 express? I cant even get the thing to compile.

\n

I was able to use one of the project files that came with it. but then I needed a library called "hdf5" which i downloaded from somewhere, but it didn't come with the header file. This is costing me way too much time, and life would be a lot simpler if kaggle just gave the files in zipped csv format which would be the same size anyway.

\n

if anyone has a ready to use recipe for me to convert the files to csv, either with matio or another way, I would be really greatful.

", "votes": null, "replies": [] }, { "id": 106593, "author_name": "kaushalaman", "author_url": "", "post_date": "02/02/2016 09:30:10", "content": "
import scipy.io\nimport numpy as np\ndata = scipy.io.loadmat("file.mat")\n\nfor i in data:\n     if '__' not in i and 'readme' not in i:\n           np.savetxt(("file.csv"),data[i],delimiter=',')\n
", "votes": null, "replies": [ { "id": 748710, "author_name": "shrutikaggle", "author_url": "", "post_date": "02/17/2020 21:59:09", "content": "

This gives the TypeError: only size-1 arrays can be converted to Python scalars

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "53638": "", "53658": "", "53659": "", "53669": "", "53670": "", "53676": "", "53678": "", "53682": "", "53688": "", "53745": "", "54037": "", "54058": "", "54150": "", "54261": "", "54304": "", "54306": "", "54573": "", "54574": "", "56251": "", "106593": "import scipy.io\r\n import numpy as np\r\n data = scipy.io.loadmat(\"file.mat\")\r\n\r\n for i in data:\r\n if '__' not in i and 'readme' not in i:\r\n np.savetxt((\"file.csv\"),data[i],delimiter=',')", "748710": "This gives the TypeError: only size-1 arrays can be converted to Python scalars" }, "source": "meta" }