{
  "id": 9506,
  "title": "Memory limit in R reached when reading .mat files",
  "url": "/competitions/decoding-the-human-brain/discussion/9506",
  "author_name": "",
  "post_date": "2014-06-20T17:50:58.457Z",
  "votes": null,
  "comment_count": 6,
  "views": 2879,
  "content": "<p>I'm trying to open the training cases and whatever I seem to try I get a memory error. I've increased to max memory on my machine, memory.limit(7962) but I still get an error when trying to read the files:</p>\n\n<p><code>rm(list=ls())<br>library(R.matlab)<br>fname &lt;- paste0(getwd(), &quot;/data/train_subject01.mat&quot;)<br>data &lt;- readMat(fname)<br></code><code></code></p>\n<p>Gives the error:</p>\n<p><code>Error: cannot allocate vector of size 2.0 Gb<br><br></code></p>\n<p>The memory info:</p>\n<p><code>&gt; memory.size()<br>[1] 2332.85<br>&gt; memory.limit()<br>[1] 7962</code><code></code></p>\n\n<p>I'm using R on a Windows 8.1 machine with 8 Gb of memory.&nbsp;</p>",
  "messages": [
    {
      "id": "49344",
      "postDate": "06/20/2014 17:50:58",
      "content": "<p>I'm trying to open the training cases and whatever I seem to try I get a memory error. I've increased to max memory on my machine, memory.limit(7962) but I still get an error when trying to read the files:</p>\n\n<p><code>rm(list=ls())<br>library(R.matlab)<br>fname &lt;- paste0(getwd(), &quot;/data/train_subject01.mat&quot;)<br>data &lt;- readMat(fname)<br></code><code></code></p>\n<p>Gives the error:</p>\n<p><code>Error: cannot allocate vector of size 2.0 Gb<br><br></code></p>\n<p>The memory info:</p>\n<p><code>&gt; memory.size()<br>[1] 2332.85<br>&gt; memory.limit()<br>[1] 7962</code><code></code></p>\n\n<p>I'm using R on a Windows 8.1 machine with 8 Gb of memory.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49356",
      "postDate": "06/21/2014 12:40:34",
      "content": "<p>Using latest R version? When I read the files in R it peaked around 5-6gb and dropped to around 1-2gb, so if Windows is already using 2+gb without R then you might have an issue because of it.&nbsp;</p>\n\n<p>My macbook with 8gb ram reads them just fine, unfortunately I'm not close to a windows computer, midsummer vacation whooray :)</p>\n\n<p>I think I could setup a google drive with those individual parts. But the internet in the middle of nowhere isn't up for uploading ~27gb</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49359",
      "postDate": "06/21/2014 13:37:19",
      "content": "<p>Thank you for your reply. My bad though, I forgot that my laptop actually only had 4 Gb :-P Nothing with 8 Gb was on the market when I buyed it...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49393",
      "postDate": "06/23/2014 06:27:30",
      "content": "<p>ah, it happens. You probably won't be needing the google drive machination ?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49394",
      "postDate": "06/23/2014 06:34:44",
      "content": "<p>Recently got a stationary with 16 Gb so I should be fine. Not familiar with Google drive machination, anything that implies cloud computations sounds interesting, care to elaborate?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49540",
      "postDate": "06/25/2014 06:57:41",
      "content": "<p>Google drive is just cloud storage, 15gb of it for free. So I could have smashed those mats to csv's or tsv's and shared it with you through the drive.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49541",
      "postDate": "06/25/2014 07:06:34",
      "content": "<p>Ahh, that's very kind of you :-) Fortunately it seems to work now, just need to figure out how to win this thing...</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 49356,
      "author_name": "xachriel",
      "author_url": "",
      "post_date": "06/21/2014 12:40:34",
      "content": "<p>Using latest R version? When I read the files in R it peaked around 5-6gb and dropped to around 1-2gb, so if Windows is already using 2+gb without R then you might have an issue because of it.&nbsp;</p>\n\n<p>My macbook with 8gb ram reads them just fine, unfortunately I'm not close to a windows computer, midsummer vacation whooray :)</p>\n\n<p>I think I could setup a google drive with those individual parts. But the internet in the middle of nowhere isn't up for uploading ~27gb</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49359,
      "author_name": "maxgordon0",
      "author_url": "",
      "post_date": "06/21/2014 13:37:19",
      "content": "<p>Thank you for your reply. My bad though, I forgot that my laptop actually only had 4 Gb :-P Nothing with 8 Gb was on the market when I buyed it...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49393,
      "author_name": "xachriel",
      "author_url": "",
      "post_date": "06/23/2014 06:27:30",
      "content": "<p>ah, it happens. You probably won't be needing the google drive machination ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49394,
      "author_name": "maxgordon0",
      "author_url": "",
      "post_date": "06/23/2014 06:34:44",
      "content": "<p>Recently got a stationary with 16 Gb so I should be fine. Not familiar with Google drive machination, anything that implies cloud computations sounds interesting, care to elaborate?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49540,
      "author_name": "xachriel",
      "author_url": "",
      "post_date": "06/25/2014 06:57:41",
      "content": "<p>Google drive is just cloud storage, 15gb of it for free. So I could have smashed those mats to csv's or tsv's and shared it with you through the drive.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49541,
      "author_name": "maxgordon0",
      "author_url": "",
      "post_date": "06/25/2014 07:06:34",
      "content": "<p>Ahh, that's very kind of you :-) Fortunately it seems to work now, just need to figure out how to win this thing...</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "49344": "",
    "49356": "",
    "49359": "",
    "49393": "",
    "49394": "",
    "49540": "",
    "49541": ""
  },
  "source": "meta"
}