{ "id": 4392, "title": "extra_unsupervised_data.csv file", "url": "/competitions/challenges-in-representation-learning-the-black-box-learning-challenge/discussion/4392", "author_name": "", "post_date": "2013-04-20T19:22:40.560Z", "votes": null, "comment_count": 7, "views": 2202, "content": "

The extra_unsupervised_data.csv file is large enough that I am not able to load it in my 12 GB of RAM (at least in R and Excel).

\r\n

My workaround would be to take a random subset of that file, like 50%, but I am not sure how to do it since I am not able to open it.  Any ideas?

", "messages": [ { "id": "23240", "postDate": "04/20/2013 19:22:40", "content": "

The extra_unsupervised_data.csv file is large enough that I am not able to load it in my 12 GB of RAM (at least in R and Excel).

\r\n

My workaround would be to take a random subset of that file, like 50%, but I am not sure how to do it since I am not able to open it.  Any ideas?

", "rawMarkdown": "", "votes": null }, { "id": "23241", "postDate": "04/20/2013 19:30:39", "content": "

You could go through the file one line at a time. In R:

\r\n

infile <- file("in.csv", open='r')

\r\n

outfile <- file("out.csv", open="w")

\r\n

while (length(line <- readLines(fileConnection, n=1)) > 0) {

\r\n

        if (rbinom(1,1,0.2)==1) writeLines(line, outfile)

\r\n

}

\r\n

This will make a new file with about 20% of the lines. (Oh, probably be more careful so you still have a header row.)

", "rawMarkdown": "", "votes": null }, { "id": "23242", "postDate": "04/20/2013 19:33:11", "content": "

Thanks a lot for the help!

", "rawMarkdown": "", "votes": null }, { "id": "23245", "postDate": "04/20/2013 19:51:34", "content": "

It should not take up that much memory. I don't know what R and Excel are doing, but here is the memory consumption for a few data structures:

\r\n

dense 32 bit matrix: 0.9 GB

\r\n

dense 64 bit matrix: 1.9 GB

\r\n

sparse matrix, using 64 bits to store the row, 64 bits to store the column, and 64 bits to store the value of each element: 5.7 GB

\r\n

To take up 12 GB, it'd have to somehow use an average of 400 bits per entry in the matrix.

", "rawMarkdown": "", "votes": null }, { "id": "23855", "postDate": "05/03/2013 04:11:54", "content": "

Ian is correct in that in R the data is exactly 1.9 GB (you can get that info with object.size)

\r\n

However R's read.csv performs terribly on large datasets and consumes a lot of unnecessary memory and time in the process.

\r\n

For a data set like this you can get much better results with 'scan':

\r\n

    extra.data <- scan(file="extra_unsupervised_data.csv",sep=',')

\r\n

And to make this data into a matrix you simply do this:

\r\n

    extra.data.m <- matrix(extra.data,ncol=1875)

\r\n

This should definitely work with 12GB of ram and very likely work with less. It's also a very useful trick for working with reasonably large data sets in R

\r\n

 

", "rawMarkdown": "", "votes": null }, { "id": "23883", "postDate": "05/04/2013 00:12:34", "content": "

I'm pretty sure that you'll want to include byrow=TRUE in your conversation to a matrix.

", "rawMarkdown": "", "votes": null }, { "id": "23884", "postDate": "05/04/2013 01:42:21", "content": "

A couple other options from here:

\r\n

http://stackoverflow.com/questions/1727772/quickly-reading-very-large-tables-as-dataframes-in-r

\r\n

library(sqldf)
\r\nf <- file('extra_unsupervised_data/extra_unsupervised_data.csv')
\r\nsystem.time(bigdf <- sqldf("select * from f", dbname = tempfile(), file.format = list(header = T, row.names = F)))
\r\n
\r\n  user  system elapsed
\r\n320.92   28.30  350.48

\r\n

> require(data.table)
\r\nLoading required package: data.table
\r\ndata.table 1.8.8  For help type: help("data.table")
\r\n> system.time(DT <- fread('extra_unsupervised_data/extra_unsupervised_data.csv'))
\r\n   user  system elapsed
\r\n 204.51    0.99  206.61

\r\n

Plus data.table gives you a progress indicator.

\r\n

 

", "rawMarkdown": "", "votes": null }, { "id": "23886", "postDate": "05/04/2013 04:22:03", "content": "

David normally you would be correct but it looks as though the .csv is actually in column order. Since it's so large I just did this quick python sanity check:

\r\n
f = open('/extra_unsupervised_data.csv')
b = f.readline()
bs = b.split(',')
len(bs)
>> 135735
\r\n
So each line in the .csv file seems to represent an entire column of data.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 23241, "author_name": "ajschumacher", "author_url": "", "post_date": "04/20/2013 19:30:39", "content": "

You could go through the file one line at a time. In R:

\r\n

infile <- file("in.csv", open='r')

\r\n

outfile <- file("out.csv", open="w")

\r\n

while (length(line <- readLines(fileConnection, n=1)) > 0) {

\r\n

        if (rbinom(1,1,0.2)==1) writeLines(line, outfile)

\r\n

}

\r\n

This will make a new file with about 20% of the lines. (Oh, probably be more careful so you still have a header row.)

", "votes": null, "replies": [] }, { "id": 23242, "author_name": "benoitplante", "author_url": "", "post_date": "04/20/2013 19:33:11", "content": "

Thanks a lot for the help!

", "votes": null, "replies": [] }, { "id": 23245, "author_name": "iangoodfellow", "author_url": "", "post_date": "04/20/2013 19:51:34", "content": "

It should not take up that much memory. I don't know what R and Excel are doing, but here is the memory consumption for a few data structures:

\r\n

dense 32 bit matrix: 0.9 GB

\r\n

dense 64 bit matrix: 1.9 GB

\r\n

sparse matrix, using 64 bits to store the row, 64 bits to store the column, and 64 bits to store the value of each element: 5.7 GB

\r\n

To take up 12 GB, it'd have to somehow use an average of 400 bits per entry in the matrix.

", "votes": null, "replies": [] }, { "id": 23855, "author_name": "willkurt", "author_url": "", "post_date": "05/03/2013 04:11:54", "content": "

Ian is correct in that in R the data is exactly 1.9 GB (you can get that info with object.size)

\r\n

However R's read.csv performs terribly on large datasets and consumes a lot of unnecessary memory and time in the process.

\r\n

For a data set like this you can get much better results with 'scan':

\r\n

    extra.data <- scan(file="extra_unsupervised_data.csv",sep=',')

\r\n

And to make this data into a matrix you simply do this:

\r\n

    extra.data.m <- matrix(extra.data,ncol=1875)

\r\n

This should definitely work with 12GB of ram and very likely work with less. It's also a very useful trick for working with reasonably large data sets in R

\r\n

 

", "votes": null, "replies": [] }, { "id": 23883, "author_name": "dmcgarry", "author_url": "", "post_date": "05/04/2013 00:12:34", "content": "

I'm pretty sure that you'll want to include byrow=TRUE in your conversation to a matrix.

", "votes": null, "replies": [] }, { "id": 23884, "author_name": "barrenwuffet", "author_url": "", "post_date": "05/04/2013 01:42:21", "content": "

A couple other options from here:

\r\n

http://stackoverflow.com/questions/1727772/quickly-reading-very-large-tables-as-dataframes-in-r

\r\n

library(sqldf)
\r\nf <- file('extra_unsupervised_data/extra_unsupervised_data.csv')
\r\nsystem.time(bigdf <- sqldf("select * from f", dbname = tempfile(), file.format = list(header = T, row.names = F)))
\r\n
\r\n  user  system elapsed
\r\n320.92   28.30  350.48

\r\n

> require(data.table)
\r\nLoading required package: data.table
\r\ndata.table 1.8.8  For help type: help("data.table")
\r\n> system.time(DT <- fread('extra_unsupervised_data/extra_unsupervised_data.csv'))
\r\n   user  system elapsed
\r\n 204.51    0.99  206.61

\r\n

Plus data.table gives you a progress indicator.

\r\n

 

", "votes": null, "replies": [] }, { "id": 23886, "author_name": "willkurt", "author_url": "", "post_date": "05/04/2013 04:22:03", "content": "

David normally you would be correct but it looks as though the .csv is actually in column order. Since it's so large I just did this quick python sanity check:

\r\n
f = open('/extra_unsupervised_data.csv')
b = f.readline()
bs = b.split(',')
len(bs)
>> 135735
\r\n
So each line in the .csv file seems to represent an entire column of data.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "23240": "", "23241": "", "23242": "", "23245": "", "23855": "", "23883": "", "23884": "", "23886": "" }, "source": "meta" }