{
  "id": 20175,
  "title": "iPython notebook and runtime problem",
  "url": "/competitions/expedia-hotel-recommendations/discussion/20175",
  "author_name": "",
  "post_date": "2016-04-16T16:45:16.257Z",
  "votes": null,
  "comment_count": 8,
  "views": 759,
  "content": "<p>Hello, </p>\n\n<p>I have a problem with iPython Notebook when I want to display the train data (~27 000 000 lines) or do a simple calculations above. The notebook crashes when I want to do some calculations. I have a MacBook Pro, 2.4 Ghz intel Core i5, 4Go 1500 Mhz DDR3. </p>\n\n<p>Any solution please. </p>",
  "messages": [
    {
      "id": "115136",
      "postDate": "04/16/2016 16:45:16",
      "content": "<p>Hello, </p>\n\n<p>I have a problem with iPython Notebook when I want to display the train data (~27 000 000 lines) or do a simple calculations above. The notebook crashes when I want to do some calculations. I have a MacBook Pro, 2.4 Ghz intel Core i5, 4Go 1500 Mhz DDR3. </p>\n\n<p>Any solution please. </p>",
      "rawMarkdown": "Hello, \r\n\r\nI have a problem with iPython Notebook when I want to display the train data (~27 000 000 lines) or do a simple calculations above. The notebook crashes when I want to do some calculations. I have a MacBook Pro, 2.4 Ghz intel Core i5, 4Go 1500 Mhz DDR3. \r\n\r\nAny solution please.",
      "votes": null
    },
    {
      "id": "115139",
      "postDate": "04/16/2016 16:49:30",
      "content": "<p>The train set itself is 3.79 GB after extraction.  Using it will obviously create problems on a 4GB machine</p>",
      "rawMarkdown": "The train set itself is 3.79 GB after extraction.  Using it will obviously create problems on a 4GB machine",
      "votes": null
    },
    {
      "id": "115140",
      "postDate": "04/16/2016 16:54:55",
      "content": "<p>Exactly. There are no other reasonably priced alternative using Amazon Service ?</p>",
      "rawMarkdown": "Exactly. There are no other reasonably priced alternative using Amazon Service ?",
      "votes": null
    },
    {
      "id": "115144",
      "postDate": "04/16/2016 17:13:30",
      "content": "<p>Just found out that my 4GB PC couldn't even load the train set. Thinking of upgrading my machine to 12 GB(the data set should work, hopefully) as it would be a cheaper than a cloud instance :P . Until then I'll stick to Santander.</p>",
      "rawMarkdown": "Just found out that my 4GB PC couldn't even load the train set. Thinking of upgrading my machine to 12 GB(the data set should work, hopefully) as it would be a cheaper than a cloud instance :P . Until then I'll stick to Santander.",
      "votes": null
    },
    {
      "id": "115154",
      "postDate": "04/16/2016 18:40:35",
      "content": "<p>Thank you, I can't upgrade my machine to 12 GB I will try to create a Spark cluster with 2 nodes (4GB machine and 8GB machine)</p>",
      "rawMarkdown": "Thank you, I can't upgrade my machine to 12 GB I will try to create a Spark cluster with 2 nodes (4GB machine and 8GB machine)",
      "votes": null
    },
    {
      "id": "115164",
      "postDate": "04/16/2016 19:18:44",
      "content": "<p>Can I for example load a part of data (~100000 rows) and generalize the analysis of all rows ? </p>",
      "rawMarkdown": "Can I for example load a part of data (~100000 rows) and generalize the analysis of all rows ?",
      "votes": null
    },
    {
      "id": "115167",
      "postDate": "04/16/2016 19:52:18",
      "content": "<p>Just trying to load everything for the first time in R. 16GB seems not enough!</p>",
      "rawMarkdown": "Just trying to load everything for the first time in R. 16GB seems not enough!",
      "votes": null
    },
    {
      "id": "115169",
      "postDate": "04/16/2016 19:59:15",
      "content": "<p>What  do you mean by load everything for the first time in R ? </p>",
      "rawMarkdown": "What  do you mean by load everything for the first time in R ?",
      "votes": null
    },
    {
      "id": "115348",
      "postDate": "04/18/2016 09:13:42",
      "content": "<p>I hope this <a href=\"http://stackoverflow.com/questions/9352887/strategies-for-reading-in-csv-files-in-pieces\">post</a> on stackoverflow might help you in partially loading a .csv into R.</p>",
      "rawMarkdown": "I hope this [post][1] on stackoverflow might help you in partially loading a .csv into R.\r\n\r\n  [1]: http://stackoverflow.com/questions/9352887/strategies-for-reading-in-csv-files-in-pieces",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 115139,
      "author_name": "nageshk",
      "author_url": "",
      "post_date": "04/16/2016 16:49:30",
      "content": "<p>The train set itself is 3.79 GB after extraction.  Using it will obviously create problems on a 4GB machine</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 115140,
      "author_name": "aissaelouafi",
      "author_url": "",
      "post_date": "04/16/2016 16:54:55",
      "content": "<p>Exactly. There are no other reasonably priced alternative using Amazon Service ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 115144,
      "author_name": "nageshk",
      "author_url": "",
      "post_date": "04/16/2016 17:13:30",
      "content": "<p>Just found out that my 4GB PC couldn't even load the train set. Thinking of upgrading my machine to 12 GB(the data set should work, hopefully) as it would be a cheaper than a cloud instance :P . Until then I'll stick to Santander.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 115154,
      "author_name": "aissaelouafi",
      "author_url": "",
      "post_date": "04/16/2016 18:40:35",
      "content": "<p>Thank you, I can't upgrade my machine to 12 GB I will try to create a Spark cluster with 2 nodes (4GB machine and 8GB machine)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 115164,
      "author_name": "aissaelouafi",
      "author_url": "",
      "post_date": "04/16/2016 19:18:44",
      "content": "<p>Can I for example load a part of data (~100000 rows) and generalize the analysis of all rows ? </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 115167,
      "author_name": "mightybird",
      "author_url": "",
      "post_date": "04/16/2016 19:52:18",
      "content": "<p>Just trying to load everything for the first time in R. 16GB seems not enough!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 115169,
      "author_name": "aissaelouafi",
      "author_url": "",
      "post_date": "04/16/2016 19:59:15",
      "content": "<p>What  do you mean by load everything for the first time in R ? </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 115348,
      "author_name": "nageshk",
      "author_url": "",
      "post_date": "04/18/2016 09:13:42",
      "content": "<p>I hope this <a href=\"http://stackoverflow.com/questions/9352887/strategies-for-reading-in-csv-files-in-pieces\">post</a> on stackoverflow might help you in partially loading a .csv into R.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "115136": "Hello, \r\n\r\nI have a problem with iPython Notebook when I want to display the train data (~27 000 000 lines) or do a simple calculations above. The notebook crashes when I want to do some calculations. I have a MacBook Pro, 2.4 Ghz intel Core i5, 4Go 1500 Mhz DDR3. \r\n\r\nAny solution please.",
    "115139": "The train set itself is 3.79 GB after extraction.  Using it will obviously create problems on a 4GB machine",
    "115140": "Exactly. There are no other reasonably priced alternative using Amazon Service ?",
    "115144": "Just found out that my 4GB PC couldn't even load the train set. Thinking of upgrading my machine to 12 GB(the data set should work, hopefully) as it would be a cheaper than a cloud instance :P . Until then I'll stick to Santander.",
    "115154": "Thank you, I can't upgrade my machine to 12 GB I will try to create a Spark cluster with 2 nodes (4GB machine and 8GB machine)",
    "115164": "Can I for example load a part of data (~100000 rows) and generalize the analysis of all rows ?",
    "115167": "Just trying to load everything for the first time in R. 16GB seems not enough!",
    "115169": "What  do you mean by load everything for the first time in R ?",
    "115348": "I hope this [post][1] on stackoverflow might help you in partially loading a .csv into R.\r\n\r\n  [1]: http://stackoverflow.com/questions/9352887/strategies-for-reading-in-csv-files-in-pieces"
  },
  "source": "meta"
}