Anyone tried loading the whole training set and working magic on it? I have a 16gb ram macbook, taking ages to do some data wrangling.
", "messages": [ { "id": "121290", "postDate": "05/25/2016 12:05:26", "content": "Anyone tried loading the whole training set and working magic on it? I have a 16gb ram macbook, taking ages to do some data wrangling.
", "rawMarkdown": "Anyone tried loading the whole training set and working magic on it? I have a 16gb ram macbook, taking ages to do some data wrangling.", "votes": null }, { "id": "121316", "postDate": "05/25/2016 15:32:20", "content": "I am using Apache Spark and it is working well. It takes me 1.5 min to read the dataset from parquet file and 10 seconds to 2 min to do an aggregate over the whole dataset.
", "rawMarkdown": "I am using Apache Spark and it is working well. It takes me 1.5 min to read the dataset from parquet file and 10 seconds to 2 min to do an aggregate over the whole dataset.", "votes": null }, { "id": "121338", "postDate": "05/25/2016 18:35:46", "content": "I was in doubt about using Spark on this problem (never used it before), but was afraid that it wouldn't work well on my notebook (i5 with 12Gb of RAM). I was assuming that it required lots of RAM to work well.
\n\nAre you running it on your laptop? What are your specs?
", "rawMarkdown": "I was in doubt about using Spark on this problem (never used it before), but was afraid that it wouldn't work well on my notebook (i5 with 12Gb of RAM). I was assuming that it required lots of RAM to work well.\r\n\r\nAre you running it on your laptop? What are your specs?", "votes": null }, { "id": "121341", "postDate": "05/25/2016 18:46:25", "content": "well right now I am running it on a VM with 8 cores and 32GB RAM just because I have it. But it is actually not that demanding and fairly easy to set up. Look up my earlier posts, I have posted a notebook reproducing some basic solution/EDA with Spark.
", "rawMarkdown": "well right now I am running it on a VM with 8 cores and 32GB RAM just because I have it. But it is actually not that demanding and fairly easy to set up. Look up my earlier posts, I have posted a notebook reproducing some basic solution/EDA with Spark.", "votes": null }, { "id": "121343", "postDate": "05/25/2016 18:50:08", "content": "Wow, 8 cores and 32GB ram. I am thinking whether building a customized computer with super specs or setting up a cloud environment with super specs on AWS would be better for all these data fun?
", "rawMarkdown": "Wow, 8 cores and 32GB ram. I am thinking whether building a customized computer with super specs or setting up a cloud environment with super specs on AWS would be better for all these data fun?", "votes": null }, { "id": "121349", "postDate": "05/25/2016 19:18:34", "content": "[quote=zyazzy;121343]
\n\nWow, 8 cores and 32GB ram. I am thinking whether building a customized computer with super specs or setting up a cloud environment with super specs on AWS would be better for all these data fun?
\n\n[/quote]
\n\nThats for sure :)
\n\nThis data set is relativelly big, so our modest notebooks can't handle it all at once :(
", "rawMarkdown": "[quote=zyazzy;121343]\r\n\r\nWow, 8 cores and 32GB ram. I am thinking whether building a customized computer with super specs or setting up a cloud environment with super specs on AWS would be better for all these data fun?\r\n\r\n[/quote]\r\n\r\nThats for sure :)\r\n\r\nThis data set is relativelly big, so our modest notebooks can't handle it all at once :(", "votes": null }, { "id": "121361", "postDate": "05/25/2016 20:41:08", "content": "Depends on what you intend to do. I keep a 8 core w/ 72 gb ram and an 8 core w/32 gb of ram in my attic data center. I paid $450 for the first (it's a decomissioned facebook server) and got the latter off a neighbor for $25 at a yard sale. They're both probably 7 years old now.
\n\nI run Xenserver on the big box and keep an analytics vm on it as well as some other vm's for other things I do for work. However the smaller box is a dedicated ubuntu 16.04 server, and it does what I need.
\n\nIf you're just doing periodic Kaggle analysis, I'd say go AWS. If you will really use a dedicated test lab, there's lots of relatively inexpensive decom'd servers on ebay. Heads up - you'll want an attic or other place to hide them. They're LOUD, especially when you get them processing heavy loads. My attic always sounds like a large swarm of angry hornets has taken over.
", "rawMarkdown": "Depends on what you intend to do. I keep a 8 core w/ 72 gb ram and an 8 core w/32 gb of ram in my attic data center. I paid $450 for the first (it's a decomissioned facebook server) and got the latter off a neighbor for $25 at a yard sale. They're both probably 7 years old now.\r\n\r\nI run Xenserver on the big box and keep an analytics vm on it as well as some other vm's for other things I do for work. However the smaller box is a dedicated ubuntu 16.04 server, and it does what I need.\r\n\r\nIf you're just doing periodic Kaggle analysis, I'd say go AWS. If you will really use a dedicated test lab, there's lots of relatively inexpensive decom'd servers on ebay. Heads up - you'll want an attic or other place to hide them. They're LOUD, especially when you get them processing heavy loads. My attic always sounds like a large swarm of angry hornets has taken over.", "votes": null } ], "comments": [ { "id": 121316, "author_name": "vykhand", "author_url": "", "post_date": "05/25/2016 15:32:20", "content": "I am using Apache Spark and it is working well. It takes me 1.5 min to read the dataset from parquet file and 10 seconds to 2 min to do an aggregate over the whole dataset.
", "votes": null, "replies": [] }, { "id": 121338, "author_name": "khaoticmind", "author_url": "", "post_date": "05/25/2016 18:35:46", "content": "I was in doubt about using Spark on this problem (never used it before), but was afraid that it wouldn't work well on my notebook (i5 with 12Gb of RAM). I was assuming that it required lots of RAM to work well.
\n\nAre you running it on your laptop? What are your specs?
", "votes": null, "replies": [] }, { "id": 121341, "author_name": "vykhand", "author_url": "", "post_date": "05/25/2016 18:46:25", "content": "well right now I am running it on a VM with 8 cores and 32GB RAM just because I have it. But it is actually not that demanding and fairly easy to set up. Look up my earlier posts, I have posted a notebook reproducing some basic solution/EDA with Spark.
", "votes": null, "replies": [] }, { "id": 121343, "author_name": "zyazzy", "author_url": "", "post_date": "05/25/2016 18:50:08", "content": "Wow, 8 cores and 32GB ram. I am thinking whether building a customized computer with super specs or setting up a cloud environment with super specs on AWS would be better for all these data fun?
", "votes": null, "replies": [] }, { "id": 121349, "author_name": "khaoticmind", "author_url": "", "post_date": "05/25/2016 19:18:34", "content": "[quote=zyazzy;121343]
\n\nWow, 8 cores and 32GB ram. I am thinking whether building a customized computer with super specs or setting up a cloud environment with super specs on AWS would be better for all these data fun?
\n\n[/quote]
\n\nThats for sure :)
\n\nThis data set is relativelly big, so our modest notebooks can't handle it all at once :(
", "votes": null, "replies": [] }, { "id": 121361, "author_name": "theamazingdrew", "author_url": "", "post_date": "05/25/2016 20:41:08", "content": "Depends on what you intend to do. I keep a 8 core w/ 72 gb ram and an 8 core w/32 gb of ram in my attic data center. I paid $450 for the first (it's a decomissioned facebook server) and got the latter off a neighbor for $25 at a yard sale. They're both probably 7 years old now.
\n\nI run Xenserver on the big box and keep an analytics vm on it as well as some other vm's for other things I do for work. However the smaller box is a dedicated ubuntu 16.04 server, and it does what I need.
\n\nIf you're just doing periodic Kaggle analysis, I'd say go AWS. If you will really use a dedicated test lab, there's lots of relatively inexpensive decom'd servers on ebay. Heads up - you'll want an attic or other place to hide them. They're LOUD, especially when you get them processing heavy loads. My attic always sounds like a large swarm of angry hornets has taken over.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "121290": "Anyone tried loading the whole training set and working magic on it? I have a 16gb ram macbook, taking ages to do some data wrangling.", "121316": "I am using Apache Spark and it is working well. It takes me 1.5 min to read the dataset from parquet file and 10 seconds to 2 min to do an aggregate over the whole dataset.", "121338": "I was in doubt about using Spark on this problem (never used it before), but was afraid that it wouldn't work well on my notebook (i5 with 12Gb of RAM). I was assuming that it required lots of RAM to work well.\r\n\r\nAre you running it on your laptop? What are your specs?", "121341": "well right now I am running it on a VM with 8 cores and 32GB RAM just because I have it. But it is actually not that demanding and fairly easy to set up. Look up my earlier posts, I have posted a notebook reproducing some basic solution/EDA with Spark.", "121343": "Wow, 8 cores and 32GB ram. I am thinking whether building a customized computer with super specs or setting up a cloud environment with super specs on AWS would be better for all these data fun?", "121349": "[quote=zyazzy;121343]\r\n\r\nWow, 8 cores and 32GB ram. I am thinking whether building a customized computer with super specs or setting up a cloud environment with super specs on AWS would be better for all these data fun?\r\n\r\n[/quote]\r\n\r\nThats for sure :)\r\n\r\nThis data set is relativelly big, so our modest notebooks can't handle it all at once :(", "121361": "Depends on what you intend to do. I keep a 8 core w/ 72 gb ram and an 8 core w/32 gb of ram in my attic data center. I paid $450 for the first (it's a decomissioned facebook server) and got the latter off a neighbor for $25 at a yard sale. They're both probably 7 years old now.\r\n\r\nI run Xenserver on the big box and keep an analytics vm on it as well as some other vm's for other things I do for work. However the smaller box is a dedicated ubuntu 16.04 server, and it does what I need.\r\n\r\nIf you're just doing periodic Kaggle analysis, I'd say go AWS. If you will really use a dedicated test lab, there's lots of relatively inexpensive decom'd servers on ebay. Heads up - you'll want an attic or other place to hide them. They're LOUD, especially when you get them processing heavy loads. My attic always sounds like a large swarm of angry hornets has taken over." }, "source": "meta" }