{ "id": 10677, "title": "Using the .mat files", "url": "/competitions/seizure-prediction/discussion/10677", "author_name": "", "post_date": "2014-10-22T05:46:05.460Z", "votes": null, "comment_count": 3, "views": 1387, "content": "

Hi everybody, I was able to open the mat files given in the previous competition on this topic using the R package "R.matlab". Now even trying to open a single file results in an error

\n

library("R.matlab")

\n

p1 <-readMat("Patient_1/Patient_1_preictal_segment_0001.mat")R(1050,0xac126a28) malloc: *** mmap(size=1440002048) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
R(1050,0xac126a28) malloc: *** mmap(size=1440002048) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Error: cannot allocate vector of size 1.3 Gb
Execution halted

\n

I am using R version 2 15.1 and have 4 Gb of RAM. The file is 172 Mb, but I guess it expands a lot when turned into an R object. I was thinking maybe I should use some package like bigmemory or ff? Anybody had any luck with that?

\n

No luck trying to convert the file to csv from within Matlab:

\n

csvwrite('prova.csv', interictal_segment_1)
Undefined function 'real' for input arguments of type 'struct'.Error in dlmwrite (line 195)
str = sprintf('%.*g%+.*gi',precn,real(m(i,j)),precn,imag(m(i,j)));Error in csvwrite (line 43)
dlmwrite(filename, m, ',', r, c);

\n\n

Thank you for any suggestion!

", "messages": [ { "id": "56437", "postDate": "10/22/2014 05:46:05", "content": "

Hi everybody, I was able to open the mat files given in the previous competition on this topic using the R package "R.matlab". Now even trying to open a single file results in an error

\n

library("R.matlab")

\n

p1 <-readMat("Patient_1/Patient_1_preictal_segment_0001.mat")R(1050,0xac126a28) malloc: *** mmap(size=1440002048) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
R(1050,0xac126a28) malloc: *** mmap(size=1440002048) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Error: cannot allocate vector of size 1.3 Gb
Execution halted

\n

I am using R version 2 15.1 and have 4 Gb of RAM. The file is 172 Mb, but I guess it expands a lot when turned into an R object. I was thinking maybe I should use some package like bigmemory or ff? Anybody had any luck with that?

\n

No luck trying to convert the file to csv from within Matlab:

\n

csvwrite('prova.csv', interictal_segment_1)
Undefined function 'real' for input arguments of type 'struct'.Error in dlmwrite (line 195)
str = sprintf('%.*g%+.*gi',precn,real(m(i,j)),precn,imag(m(i,j)));Error in csvwrite (line 43)
dlmwrite(filename, m, ',', r, c);

\n\n

Thank you for any suggestion!

", "rawMarkdown": "", "votes": null }, { "id": "56438", "postDate": "10/22/2014 06:04:54", "content": "

csvwrite('prova.csv', interictal_segment_1.data) does work.

", "rawMarkdown": "", "votes": null }, { "id": "56439", "postDate": "10/22/2014 06:08:32", "content": "

Hard to say what to tell you. I use R too, but to get it into ff, you still need to load it into memory first. Are you on a 32bit or 64bit system? That can drastically reduce your ability to create large data sets too. Otherwise, it is just a big data set and you may need to invest in more ram (surprisingly cheap these days).  The last data sets were 1 sec long iirc (with patients up to 5khz frequency). These are 10min so (10*60)x as long.

\n\n

Of course you could ask a friend to load it for you and send it to you as csv, which you could parse into smaller chunks on your own too, but still with 15 rows and 3M columns you still might run into trouble.

", "rawMarkdown": "", "votes": null }, { "id": "57135", "postDate": "10/31/2014 21:15:07", "content": "

@mario 

\n

Here's a matlab script for converting a batch of files to csv. You can store sampling_frequency and data_length_sec data separately for each subject inside variables in your program.

\n

for i = 1:195
    file = sprintf('Patient_1_test_segment_%04d.mat', i);
    load(file);
    str = eval(sprintf('test_segment_%d.data', i));
    csv = sprintf('Patient_1_test_segment_%04d.csv', i);
    dlmwrite(csv, str,'delimiter',',');
    clear;
end

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 56438, "author_name": "mariopasquato", "author_url": "", "post_date": "10/22/2014 06:04:54", "content": "

csvwrite('prova.csv', interictal_segment_1.data) does work.

", "votes": null, "replies": [] }, { "id": 56439, "author_name": "piinformatics", "author_url": "", "post_date": "10/22/2014 06:08:32", "content": "

Hard to say what to tell you. I use R too, but to get it into ff, you still need to load it into memory first. Are you on a 32bit or 64bit system? That can drastically reduce your ability to create large data sets too. Otherwise, it is just a big data set and you may need to invest in more ram (surprisingly cheap these days).  The last data sets were 1 sec long iirc (with patients up to 5khz frequency). These are 10min so (10*60)x as long.

\n\n

Of course you could ask a friend to load it for you and send it to you as csv, which you could parse into smaller chunks on your own too, but still with 15 rows and 3M columns you still might run into trouble.

", "votes": null, "replies": [] }, { "id": 57135, "author_name": "adamsoffer", "author_url": "", "post_date": "10/31/2014 21:15:07", "content": "

@mario 

\n

Here's a matlab script for converting a batch of files to csv. You can store sampling_frequency and data_length_sec data separately for each subject inside variables in your program.

\n

for i = 1:195
    file = sprintf('Patient_1_test_segment_%04d.mat', i);
    load(file);
    str = eval(sprintf('test_segment_%d.data', i));
    csv = sprintf('Patient_1_test_segment_%04d.csv', i);
    dlmwrite(csv, str,'delimiter',',');
    clear;
end

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "56437": "", "56438": "", "56439": "", "57135": "" }, "source": "meta" }