HI, I'm relatively new to problems that actually comes from real world, I have done classification problems in class but all the data set in class is pretty small (I remember it's roughly around 10MB for training data). For this task, the train data given is 33.7GB which I don't think my laptop can handle at all, plus we have to use GPUs.
\n\nI checked AWS but I assume the training for this task would probably take a long time and I don't think I can afford the cost. (I do have a friend's gaming desktop I can borrow) Does anyone have any suggestions on what computer I should use for this task, in terms of performance, possible cost and availability? or it doesn't really matter? Thanks!
", "messages": [ { "id": "103139", "postDate": "12/29/2015 08:49:38", "content": "HI, I'm relatively new to problems that actually comes from real world, I have done classification problems in class but all the data set in class is pretty small (I remember it's roughly around 10MB for training data). For this task, the train data given is 33.7GB which I don't think my laptop can handle at all, plus we have to use GPUs.
\n\nI checked AWS but I assume the training for this task would probably take a long time and I don't think I can afford the cost. (I do have a friend's gaming desktop I can borrow) Does anyone have any suggestions on what computer I should use for this task, in terms of performance, possible cost and availability? or it doesn't really matter? Thanks!
", "rawMarkdown": "HI, I'm relatively new to problems that actually comes from real world, I have done classification problems in class but all the data set in class is pretty small (I remember it's roughly around 10MB for training data). For this task, the train data given is 33.7GB which I don't think my laptop can handle at all, plus we have to use GPUs. \r\n\r\nI checked AWS but I assume the training for this task would probably take a long time and I don't think I can afford the cost. (I do have a friend's gaming desktop I can borrow) Does anyone have any suggestions on what computer I should use for this task, in terms of performance, possible cost and availability? or it doesn't really matter? Thanks!", "votes": null }, { "id": "103157", "postDate": "12/29/2015 14:12:55", "content": "If you want to use AWS, check into spot instances. They can generally be gotten for fairly cheap: I think it was about $0.10 an hour last time I checked. Of course they may be killed on you at any time, but if you checkpoint your data that can still be workable.
\n\nIn general faster is better because you can try out ideas faster, but there's a tradeoff between speed and cost if you don't already have an appropriate machine lying around.
\n\nMy suggestion would be to start with a small model that you can run on your own laptop or your friends gaming rig. Once you have an approach that you like, then you can consider moving it over to AWS or finding some other fast computer.
\n\nAs a point of reference, I'm currently running my models on a late-2013 MacbookPro. It has a GPU, but it's fairly underpowered for this task. However, there's a decent chance that I'll eventually move things over to AWS spot instance as my models get larger.
", "rawMarkdown": "If you want to use AWS, check into spot instances. They can generally be gotten for fairly cheap: I think it was about $0.10 an hour last time I checked. Of course they may be killed on you at any time, but if you checkpoint your data that can still be workable.\r\n\r\nIn general faster is better because you can try out ideas faster, but there's a tradeoff between speed and cost if you don't already have an appropriate machine lying around. \r\n\r\nMy suggestion would be to start with a small model that you can run on your own laptop or your friends gaming rig. Once you have an approach that you like, then you can consider moving it over to AWS or finding some other fast computer.\r\n\r\nAs a point of reference, I'm currently running my models on a late-2013 MacbookPro. It has a GPU, but it's fairly underpowered for this task. However, there's a decent chance that I'll eventually move things over to AWS spot instance as my models get larger.", "votes": null }, { "id": "103173", "postDate": "12/29/2015 20:52:46", "content": "Thanks for the suggestion and sharing! Okay, I will just start first and see how much computation this needs. And also good luck.
", "rawMarkdown": "Thanks for the suggestion and sharing! Okay, I will just start first and see how much computation this needs. And also good luck.", "votes": null }, { "id": "103175", "postDate": "12/29/2015 21:32:18", "content": "[quote=Samshipengs;103173]
\n\nThanks for the suggestion and sharing! Okay, I will just start first and see how much computation this needs. And also good luck.
\n\n[/quote]
\n\nI can tell you that I generated my score (0.0356) in 4-5 hours which include setting up EC2 GPU and installing mxnet, preprocessing from benchmark script and running models 4-5 times. It shouldn't cost more than 2$ using spot instance to get my score.
\n\nHope it was helpful. Let me know if you need any further information.
", "rawMarkdown": "[quote=Samshipengs;103173]\r\n\r\nThanks for the suggestion and sharing! Okay, I will just start first and see how much computation this needs. And also good luck. \r\n\r\n[/quote]\r\n\r\nI can tell you that I generated my score (0.0356) in 4-5 hours which include setting up EC2 GPU and installing mxnet, preprocessing from benchmark script and running models 4-5 times. It shouldn't cost more than 2$ using spot instance to get my score.\r\n\r\nHope it was helpful. Let me know if you need any further information.", "votes": null }, { "id": "103237", "postDate": "12/30/2015 15:27:40", "content": "Hi,
\n\nI run the mxnet R code on a desktop with i5 + GTX960 + 8GB memory and I didn't meet any problems.
\n\nJust a reference.
\n\nQiang Kou
", "rawMarkdown": "Hi, \r\n\r\nI run the mxnet R code on a desktop with i5 + GTX960 + 8GB memory and I didn't meet any problems.\r\n\r\nJust a reference.\r\n\r\nQiang Kou", "votes": null }, { "id": "103251", "postDate": "12/30/2015 18:52:40", "content": "I have an outdated AMD 4 core CPU + 16 GB memory + GTX 960 4GB, and I can run MXnet on this dataset in Python with no problem either. The MXnet tutorial score can be easily reproduced with this configuration.
", "rawMarkdown": "I have an outdated AMD 4 core CPU + 16 GB memory + GTX 960 4GB, and I can run MXnet on this dataset in Python with no problem either. The MXnet tutorial score can be easily reproduced with this configuration.", "votes": null }, { "id": "103300", "postDate": "12/31/2015 15:11:49", "content": "I only use a 4GB RAM machine and it runs well. This data doesn't require big RAM, but you need a good GPU
", "rawMarkdown": "I only use a 4GB RAM machine and it runs well. This data doesn't require big RAM, but you need a good GPU", "votes": null } ], "comments": [ { "id": 103157, "author_name": "bitsofbits", "author_url": "", "post_date": "12/29/2015 14:12:55", "content": "If you want to use AWS, check into spot instances. They can generally be gotten for fairly cheap: I think it was about $0.10 an hour last time I checked. Of course they may be killed on you at any time, but if you checkpoint your data that can still be workable.
\n\nIn general faster is better because you can try out ideas faster, but there's a tradeoff between speed and cost if you don't already have an appropriate machine lying around.
\n\nMy suggestion would be to start with a small model that you can run on your own laptop or your friends gaming rig. Once you have an approach that you like, then you can consider moving it over to AWS or finding some other fast computer.
\n\nAs a point of reference, I'm currently running my models on a late-2013 MacbookPro. It has a GPU, but it's fairly underpowered for this task. However, there's a decent chance that I'll eventually move things over to AWS spot instance as my models get larger.
", "votes": null, "replies": [] }, { "id": 103173, "author_name": "", "author_url": "", "post_date": "12/29/2015 20:52:46", "content": "Thanks for the suggestion and sharing! Okay, I will just start first and see how much computation this needs. And also good luck.
", "votes": null, "replies": [] }, { "id": 103175, "author_name": "thakurrajanand", "author_url": "", "post_date": "12/29/2015 21:32:18", "content": "[quote=Samshipengs;103173]
\n\nThanks for the suggestion and sharing! Okay, I will just start first and see how much computation this needs. And also good luck.
\n\n[/quote]
\n\nI can tell you that I generated my score (0.0356) in 4-5 hours which include setting up EC2 GPU and installing mxnet, preprocessing from benchmark script and running models 4-5 times. It shouldn't cost more than 2$ using spot instance to get my score.
\n\nHope it was helpful. Let me know if you need any further information.
", "votes": null, "replies": [] }, { "id": 103237, "author_name": "thirdwing", "author_url": "", "post_date": "12/30/2015 15:27:40", "content": "Hi,
\n\nI run the mxnet R code on a desktop with i5 + GTX960 + 8GB memory and I didn't meet any problems.
\n\nJust a reference.
\n\nQiang Kou
", "votes": null, "replies": [] }, { "id": 103251, "author_name": "phunter", "author_url": "", "post_date": "12/30/2015 18:52:40", "content": "I have an outdated AMD 4 core CPU + 16 GB memory + GTX 960 4GB, and I can run MXnet on this dataset in Python with no problem either. The MXnet tutorial score can be easily reproduced with this configuration.
", "votes": null, "replies": [] }, { "id": 103300, "author_name": "yejiming", "author_url": "", "post_date": "12/31/2015 15:11:49", "content": "I only use a 4GB RAM machine and it runs well. This data doesn't require big RAM, but you need a good GPU
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "103139": "HI, I'm relatively new to problems that actually comes from real world, I have done classification problems in class but all the data set in class is pretty small (I remember it's roughly around 10MB for training data). For this task, the train data given is 33.7GB which I don't think my laptop can handle at all, plus we have to use GPUs. \r\n\r\nI checked AWS but I assume the training for this task would probably take a long time and I don't think I can afford the cost. (I do have a friend's gaming desktop I can borrow) Does anyone have any suggestions on what computer I should use for this task, in terms of performance, possible cost and availability? or it doesn't really matter? Thanks!", "103157": "If you want to use AWS, check into spot instances. They can generally be gotten for fairly cheap: I think it was about $0.10 an hour last time I checked. Of course they may be killed on you at any time, but if you checkpoint your data that can still be workable.\r\n\r\nIn general faster is better because you can try out ideas faster, but there's a tradeoff between speed and cost if you don't already have an appropriate machine lying around. \r\n\r\nMy suggestion would be to start with a small model that you can run on your own laptop or your friends gaming rig. Once you have an approach that you like, then you can consider moving it over to AWS or finding some other fast computer.\r\n\r\nAs a point of reference, I'm currently running my models on a late-2013 MacbookPro. It has a GPU, but it's fairly underpowered for this task. However, there's a decent chance that I'll eventually move things over to AWS spot instance as my models get larger.", "103173": "Thanks for the suggestion and sharing! Okay, I will just start first and see how much computation this needs. And also good luck.", "103175": "[quote=Samshipengs;103173]\r\n\r\nThanks for the suggestion and sharing! Okay, I will just start first and see how much computation this needs. And also good luck. \r\n\r\n[/quote]\r\n\r\nI can tell you that I generated my score (0.0356) in 4-5 hours which include setting up EC2 GPU and installing mxnet, preprocessing from benchmark script and running models 4-5 times. It shouldn't cost more than 2$ using spot instance to get my score.\r\n\r\nHope it was helpful. Let me know if you need any further information.", "103237": "Hi, \r\n\r\nI run the mxnet R code on a desktop with i5 + GTX960 + 8GB memory and I didn't meet any problems.\r\n\r\nJust a reference.\r\n\r\nQiang Kou", "103251": "I have an outdated AMD 4 core CPU + 16 GB memory + GTX 960 4GB, and I can run MXnet on this dataset in Python with no problem either. The MXnet tutorial score can be easily reproduced with this configuration.", "103300": "I only use a 4GB RAM machine and it runs well. This data doesn't require big RAM, but you need a good GPU" }, "source": "meta" }