{
  "id": 4087,
  "title": "Running out of memory",
  "url": "/competitions/icdar2013-gender-prediction-from-handwriting/discussion/4087",
  "author_name": "",
  "post_date": "2013-03-24T19:58:43.387Z",
  "votes": 1,
  "comment_count": 3,
  "views": 2364,
  "content": "<p>Hi,</p>\r\n<p>I'm a bit new to these activities and programming in R in general. I'm running into the issue of running out of memory when trying to create the benchmark models provided by the administrators (e.g. linear regression and random forest examples). I know eventually\r\n I may want to focus on identifying a more useful subset of features with which to build my models and may not run into this problem then. Until then, does anyone have some advice (or can point me to a solution) for getting around the memory issue? I'd like\r\n to verify that I can create the benchmark models correctly.</p>\r\n<p>I am running on a 32bit Windows 7 machine with 3GB of RAM. The specific error I am getting is the following:</p>\r\n<p>Error: cannot allocate vector of size&nbsp;XX Mb</p>\r\n<p>Where XX is different depending upon the model I am trying to build.</p>\r\n<p>Thanks.</p>",
  "messages": [
    {
      "id": "21597",
      "postDate": "03/24/2013 19:58:43",
      "content": "<p>Hi,</p>\r\n<p>I'm a bit new to these activities and programming in R in general. I'm running into the issue of running out of memory when trying to create the benchmark models provided by the administrators (e.g. linear regression and random forest examples). I know eventually\r\n I may want to focus on identifying a more useful subset of features with which to build my models and may not run into this problem then. Until then, does anyone have some advice (or can point me to a solution) for getting around the memory issue? I'd like\r\n to verify that I can create the benchmark models correctly.</p>\r\n<p>I am running on a 32bit Windows 7 machine with 3GB of RAM. The specific error I am getting is the following:</p>\r\n<p>Error: cannot allocate vector of size&nbsp;XX Mb</p>\r\n<p>Where XX is different depending upon the model I am trying to build.</p>\r\n<p>Thanks.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "21598",
      "postDate": "03/24/2013 20:41:21",
      "content": "<p>Check out the big memory (bigmemory) package in R.</p>\r\n<p>http://cran.r-project.org/web/packages/bigmemory/index.html</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "21644",
      "postDate": "03/25/2013 21:02:46",
      "content": "<p>[quote=Dash-N-Mash;21597]</p>\r\n<p>Hi,</p>\r\n<p>I'm a bit new to these activities and programming in R in general. I'm running into the issue of running out of memory when trying to create the benchmark models provided by the administrators (e.g. linear regression and random forest examples). I know eventually\r\n I may want to focus on identifying a more useful subset of features with which to build my models and may not run into this problem then. Until then, does anyone have some advice (or can point me to a solution) for getting around the memory issue? I'd like\r\n to verify that I can create the benchmark models correctly.</p>\r\n<p>I am running on a 32bit Windows 7 machine with 3GB of RAM. The specific error I am getting is the following:</p>\r\n<p>Error: cannot allocate vector of size&nbsp;XX Mb</p>\r\n<p>Where XX is different depending upon the model I am trying to build.</p>\r\n<p>Thanks.</p>\r\n<p>[/quote]</p>\r\n<p>3GB&nbsp; &amp; 32-bit are not sufficient. Random forest consumes a lot of memory(memory to keep a copy of the training dataset and memory to keep the gigantic forest). Not sure how easy it is for you to move to a 64-bit computer with as much memory ( north of 8gb)\r\n you can get. You can also try Amazon EC2 ( try searching the forums - loads of threads on this topic).</p>\r\n<p>BigGLM [http://www.r-bloggers.com/bigglm-on-your-big-data-set-in-open-source-r-it-just-works-similar-as-in-sas/ ] package may also be useful to you ( although I've never tried it- I have a 32gb ram laptop).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "21653",
      "postDate": "03/26/2013 00:09:21",
      "content": "<p>Sashi, thanks for the tip on Amazon EC2. That looks like a great resource, especially for those of us with slightly underpowered machines.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 21598,
      "author_name": "ryanjsmith",
      "author_url": "",
      "post_date": "03/24/2013 20:41:21",
      "content": "<p>Check out the big memory (bigmemory) package in R.</p>\r\n<p>http://cran.r-project.org/web/packages/bigmemory/index.html</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 21644,
      "author_name": "sashikanthdareddy",
      "author_url": "",
      "post_date": "03/25/2013 21:02:46",
      "content": "<p>[quote=Dash-N-Mash;21597]</p>\r\n<p>Hi,</p>\r\n<p>I'm a bit new to these activities and programming in R in general. I'm running into the issue of running out of memory when trying to create the benchmark models provided by the administrators (e.g. linear regression and random forest examples). I know eventually\r\n I may want to focus on identifying a more useful subset of features with which to build my models and may not run into this problem then. Until then, does anyone have some advice (or can point me to a solution) for getting around the memory issue? I'd like\r\n to verify that I can create the benchmark models correctly.</p>\r\n<p>I am running on a 32bit Windows 7 machine with 3GB of RAM. The specific error I am getting is the following:</p>\r\n<p>Error: cannot allocate vector of size&nbsp;XX Mb</p>\r\n<p>Where XX is different depending upon the model I am trying to build.</p>\r\n<p>Thanks.</p>\r\n<p>[/quote]</p>\r\n<p>3GB&nbsp; &amp; 32-bit are not sufficient. Random forest consumes a lot of memory(memory to keep a copy of the training dataset and memory to keep the gigantic forest). Not sure how easy it is for you to move to a 64-bit computer with as much memory ( north of 8gb)\r\n you can get. You can also try Amazon EC2 ( try searching the forums - loads of threads on this topic).</p>\r\n<p>BigGLM [http://www.r-bloggers.com/bigglm-on-your-big-data-set-in-open-source-r-it-just-works-similar-as-in-sas/ ] package may also be useful to you ( although I've never tried it- I have a 32gb ram laptop).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 21653,
      "author_name": "dashnmash",
      "author_url": "",
      "post_date": "03/26/2013 00:09:21",
      "content": "<p>Sashi, thanks for the tip on Amazon EC2. That looks like a great resource, especially for those of us with slightly underpowered machines.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "21597": "",
    "21598": "",
    "21644": "",
    "21653": ""
  },
  "source": "meta"
}