{ "id": 18396, "title": "Learning CNN and AWS", "url": "/competitions/second-annual-data-science-bowl/discussion/18396", "author_name": "", "post_date": "2016-01-15T00:30:30.093Z", "votes": null, "comment_count": 17, "views": 4404, "content": "

Hello everyone!

\n\n

I have been following the Deep Learning Tutorial provided by Booz Allen Hamilton. \nSetting up environment was a pain but I think everything is working now.

\n\n

I have seen that people use AWS so that their personal computer don't have to be running full power all the time. I looked up AWS and I was swamped by all the features on AWS and tutorial there are online. (for example, whats the difference between using the AWS machine learning service, and running programs on EC2?)

\n\n

Does anyone have a suggested tutorial to follow, given that I have successfully trained neural networks with both Caffe and Theano/Lasagne/nolearn?

\n\n

many thanks

", "messages": [ { "id": "104656", "postDate": "01/15/2016 00:30:30", "content": "

Hello everyone!

\n\n

I have been following the Deep Learning Tutorial provided by Booz Allen Hamilton. \nSetting up environment was a pain but I think everything is working now.

\n\n

I have seen that people use AWS so that their personal computer don't have to be running full power all the time. I looked up AWS and I was swamped by all the features on AWS and tutorial there are online. (for example, whats the difference between using the AWS machine learning service, and running programs on EC2?)

\n\n

Does anyone have a suggested tutorial to follow, given that I have successfully trained neural networks with both Caffe and Theano/Lasagne/nolearn?

\n\n

many thanks

", "rawMarkdown": "Hello everyone!\r\n\r\nI have been following the Deep Learning Tutorial provided by Booz Allen Hamilton. \r\nSetting up environment was a pain but I think everything is working now.\r\n\r\nI have seen that people use AWS so that their personal computer don't have to be running full power all the time. I looked up AWS and I was swamped by all the features on AWS and tutorial there are online. (for example, whats the difference between using the AWS machine learning service, and running programs on EC2?)\r\n\r\nDoes anyone have a suggested tutorial to follow, given that I have successfully trained neural networks with both Caffe and Theano/Lasagne/nolearn? \r\n\r\nmany thanks", "votes": null }, { "id": "104662", "postDate": "01/15/2016 03:24:50", "content": "

@I Love Solder,

\n\n

When I got started on using AWS for ConvNets I used the instruction on this page. I suspect that they are pretty dated by this point, for instance I doubt the referenced AMI is still around, but perhaps they'll help get you started.

", "rawMarkdown": "I Love Solder,\r\n\r\nWhen I got started on using AWS for ConvNets I used the instruction on this [page][1]. I suspect that they are pretty dated by this point, for instance I doubt the referenced AMI is still around, but perhaps they'll help get you started.\r\n\r\n\r\n [1]: https://www.kaggle.com/c/facial-keypoints-detection/details/deep-learning-tutorial", "votes": null }, { "id": "104667", "postDate": "01/15/2016 04:53:29", "content": "

The AWS machine learning service seemed, the last time I checked, to likely be a wrapper around a machine learning tool called Vowpal Wabbit, and not a fully featured machine learning environment. Amazon EC2 will be your bread and butter; just think of it as a remote computer.

\n\n

The best reference I've found for Convnets are Andrej Karpathy's CS-231n course notes.

", "rawMarkdown": "The AWS machine learning service seemed, the last time I checked, to likely be a wrapper around a machine learning tool called Vowpal Wabbit, and not a fully featured machine learning environment. Amazon EC2 will be your bread and butter; just think of it as a remote computer.\r\n\r\nThe best reference I've found for Convnets are Andrej Karpathy's CS-231n course notes.", "votes": null }, { "id": "104755", "postDate": "01/16/2016 09:45:28", "content": "

I just wrote a blog talking about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)

\n\n

The CUDA installation on AWS is the tricky part, since the kernel and nouveau have conflicts with CUDA driver. Please follow every single steps in the blog for installing on AWS. After having CUDA on AWS, one can continue to install MXnet (with S3 support and some trouble shooting) by following the blog.

", "rawMarkdown": "I just wrote a blog talking about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)\r\n\r\nThe CUDA installation on AWS is the tricky part, since the kernel and nouveau have conflicts with CUDA driver. Please follow every single steps in the blog for installing on AWS. After having CUDA on AWS, one can continue to install MXnet (with S3 support and some trouble shooting) by following the blog.", "votes": null }, { "id": "104764", "postDate": "01/16/2016 10:30:58", "content": "

an interesting complement would be to show how to use S3 and spot instances (and ideally, provide code to do it programmatically)

\n\n

[quote=phunter;104755]

\n\n

I just wrote a blog talking about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)

\n\n

The CUDA installation on AWS is the tricky part, since the kernel and nouveau have conflicts with CUDA driver. Please follow every single steps in the blog for installing on AWS. After having CUDA on AWS, one can continue to install MXnet (with S3 support and some trouble shooting) by following the blog.

\n\n

[/quote]

", "rawMarkdown": "an interesting complement would be to show how to use S3 and spot instances (and ideally, provide code to do it programmatically)\r\n\r\n[quote=phunter;104755]\r\n\r\nI just wrote a blog talking about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)\r\n\r\nThe CUDA installation on AWS is the tricky part, since the kernel and nouveau have conflicts with CUDA driver. Please follow every single steps in the blog for installing on AWS. After having CUDA on AWS, one can continue to install MXnet (with S3 support and some trouble shooting) by following the blog.\r\n\r\n[/quote]", "votes": null }, { "id": "104791", "postDate": "01/16/2016 18:21:39", "content": "

AWS has its official spot instance tutorial (with video) https://aws.amazon.com/ec2/spot/spot-and-science/ and S3 as well http://docs.aws.amazon.com/AmazonS3/latest/gsg/GetStartedWithS3.html

\n\n

Using S3 in mxnet is super simple as in its official document (with example)\nhttp://mxnet.readthedocs.org/en/latest/aws.html#use-amazon-s3-to-host-data

", "rawMarkdown": "AWS has its official spot instance tutorial (with video) https://aws.amazon.com/ec2/spot/spot-and-science/ and S3 as well http://docs.aws.amazon.com/AmazonS3/latest/gsg/GetStartedWithS3.html \r\n\r\nUsing S3 in mxnet is super simple as in its official document (with example)\r\nhttp://mxnet.readthedocs.org/en/latest/aws.html#use-amazon-s3-to-host-data", "votes": null }, { "id": "104980", "postDate": "01/18/2016 15:08:51", "content": "

@I Love Solder. I managed to get my AWS EC2 up and running this weekend. let me know if you have any remaining questions.

", "rawMarkdown": "I Love Solder. I managed to get my AWS EC2 up and running this weekend. let me know if you have any remaining questions.", "votes": null }, { "id": "106131", "postDate": "01/28/2016 18:19:24", "content": "

[quote=WD;104980]

\n\n

@I Love Solder. I managed to get my AWS EC2 up and running this weekend. let me know if you have any remaining questions.

\n\n

[/quote]

\n\n

@WD What tutorial or combination of tutorials did you use to get yourself setup and running? THanks in advance!

", "rawMarkdown": "[quote=WD;104980]\r\n\r\n@I Love Solder. I managed to get my AWS EC2 up and running this weekend. let me know if you have any remaining questions. \r\n\r\n[/quote]\r\n\r\n@WD What tutorial or combination of tutorials did you use to get yourself setup and running? THanks in advance!", "votes": null }, { "id": "106173", "postDate": "01/28/2016 22:43:52", "content": "

i basically used https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN) to setup my installation on a AWS G2x2. i added and mounted a seperate EBS drive for the data, and made sure that the instance is backed up by saving the instance to an image upon installation of all packages. Then i added the MXNET installation - as per the Mxnet tutoral. For the latter, make sure that you comment out those instalation lines that might not be relevant (e.g. the S3 or the CUDNN lines). This might not all be clear, or you might run into issues, so definitely fire me other questions.

", "rawMarkdown": "i basically used https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN) to setup my installation on a AWS G2x2. i added and mounted a seperate EBS drive for the data, and made sure that the instance is backed up by saving the instance to an image upon installation of all packages. Then i added the MXNET installation - as per the Mxnet tutoral. For the latter, make sure that you comment out those instalation lines that might not be relevant (e.g. the S3 or the CUDNN lines). This might not all be clear, or you might run into issues, so definitely fire me other questions.", "votes": null }, { "id": "106666", "postDate": "02/03/2016 03:46:07", "content": "

@WD Did you install cuDNN and Caffe? I'm reviewing @Phunter's blog and noticed that cuDNN was optional install and no mention of installing Caffe. I'm guessing that MXNET is used in lieu of Caffe.

", "rawMarkdown": "WD Did you install cuDNN and Caffe? I'm reviewing @Phunter's blog and noticed that cuDNN was optional install and no mention of installing Caffe. I'm guessing that MXNET is used in lieu of Caffe.", "votes": null }, { "id": "106768", "postDate": "02/03/2016 19:45:47", "content": "

[quote=Mostafa;104764]

\n\n

an interesting complement would be to show how to use S3 and spot instances (and ideally, provide code to do it programmatically)

\n\n

[quote=phunter;104755]

\n\n

I just wrote a blog talking about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)

\n\n

The CUDA installation on AWS is the tricky part, since the kernel and nouveau have conflicts with CUDA driver. Please follow every single steps in the blog for installing on AWS. After having CUDA on AWS, one can continue to install MXnet (with S3 support and some trouble shooting) by following the blog.

\n\n

[/quote]

\n\n

[/quote]\nData science bowl has an official step-by-step guide of starting AWS spot instance with MXnet.jl http://www.datasciencebowl.com/aws_guide/ one can feel free to continue with my tutorial blog about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)

", "rawMarkdown": "[quote=Mostafa;104764]\r\n\r\n\r\nan interesting complement would be to show how to use S3 and spot instances (and ideally, provide code to do it programmatically)\r\n\r\n[quote=phunter;104755]\r\n\r\nI just wrote a blog talking about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)\r\n\r\nThe CUDA installation on AWS is the tricky part, since the kernel and nouveau have conflicts with CUDA driver. Please follow every single steps in the blog for installing on AWS. After having CUDA on AWS, one can continue to install MXnet (with S3 support and some trouble shooting) by following the blog.\r\n\r\n[/quote]\r\n\r\n\r\n[/quote]\r\nData science bowl has an official step-by-step guide of starting AWS spot instance with MXnet.jl http://www.datasciencebowl.com/aws_guide/ one can feel free to continue with my tutorial blog about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)", "votes": null }, { "id": "108136", "postDate": "02/15/2016 19:27:06", "content": "

Have AWS up and running now. I followed phunter's blog (the one with the cat) but needed to add a couple configuration settings from the install caffe blog as well.

\n\n

My new hurdle is uploading the data. Does anyone have any tips or tricks?

\n\n

Thanks!\nJon

", "rawMarkdown": "Have AWS up and running now. I followed phunter's blog (the one with the cat) but needed to add a couple configuration settings from the install caffe blog as well.\r\n\r\nMy new hurdle is uploading the data. Does anyone have any tips or tricks?\r\n\r\nThanks!\r\nJon", "votes": null }, { "id": "110197", "postDate": "03/03/2016 18:34:05", "content": "

@WD,

\n\n

how do you back up an instance by saving it to image? do you have a link

\n\n

EDIT: so I created an image from the ec2 management console and I believe that the image worked.

\n\n

I still haven't tried to download and store data onto AWS so I might have further questions later on!

", "rawMarkdown": "WD, \r\n\r\nhow do you back up an instance by saving it to image? do you have a link \r\n\r\nEDIT: so I created an image from the ec2 management console and I believe that the image worked. \r\n\r\nI still haven't tried to download and store data onto AWS so I might have further questions later on!", "votes": null }, { "id": "110225", "postDate": "03/03/2016 21:45:27", "content": "

in the instance menu, if you go to actions, there should be an option that says "save instance to image". Does that work?

", "rawMarkdown": "in the instance menu, if you go to actions, there should be an option that says \"save instance to image\". Does that work?", "votes": null }, { "id": "110233", "postDate": "03/04/2016 00:09:20", "content": "

@WD,

\n\n

yeah I was able to save an image that way, thanks!

\n\n

Now for the data part, do you download all the data onto the root disk(using wget)? Or do you upload the data onto S3 or something?

\n\n

If you do load the data onto S3, does your root disk need to be big enough to hold all of the data or is it possible to query the S3 database as the program goes? \nI wasn't able to find very clear instructions on how to use S3 database with EC2.

", "rawMarkdown": "WD,\r\n\r\nyeah I was able to save an image that way, thanks!\r\n\r\nNow for the data part, do you download all the data onto the root disk(using wget)? Or do you upload the data onto S3 or something? \r\n\r\nIf you do load the data onto S3, does your root disk need to be big enough to hold all of the data or is it possible to query the S3 database as the program goes? \r\nI wasn't able to find very clear instructions on how to use S3 database with EC2.", "votes": null }, { "id": "110238", "postDate": "03/04/2016 00:25:42", "content": "

@I Love Solder I just unzipped everything to a local disk on AWS: it is not a big deal.

", "rawMarkdown": "I Love Solder I just unzipped everything to a local disk on AWS: it is not a big deal.", "votes": null }, { "id": "110241", "postDate": "03/04/2016 00:47:53", "content": "

@phunter

\n\n

Thanks for your response. So after you do your computations and stuff do you save instance to an image again? so that you can keep your data files and results? Or are you able to save those data some other ways?

", "rawMarkdown": "phunter\r\n\r\nThanks for your response. So after you do your computations and stuff do you save instance to an image again? so that you can keep your data files and results? Or are you able to save those data some other ways?", "votes": null }, { "id": "110246", "postDate": "03/04/2016 01:26:22", "content": "

It depends how much money you want to put to AWS. A lucky GPU AWS spot instance can cost as low as 0.07$ per hour, so I just don't turn it off :-)

", "rawMarkdown": "It depends how much money you want to put to AWS. A lucky GPU AWS spot instance can cost as low as 0.07$ per hour, so I just don't turn it off :-)", "votes": null } ], "comments": [ { "id": 104662, "author_name": "bitsofbits", "author_url": "", "post_date": "01/15/2016 03:24:50", "content": "

@I Love Solder,

\n\n

When I got started on using AWS for ConvNets I used the instruction on this page. I suspect that they are pretty dated by this point, for instance I doubt the referenced AMI is still around, but perhaps they'll help get you started.

", "votes": null, "replies": [] }, { "id": 104667, "author_name": "telser", "author_url": "", "post_date": "01/15/2016 04:53:29", "content": "

The AWS machine learning service seemed, the last time I checked, to likely be a wrapper around a machine learning tool called Vowpal Wabbit, and not a fully featured machine learning environment. Amazon EC2 will be your bread and butter; just think of it as a remote computer.

\n\n

The best reference I've found for Convnets are Andrej Karpathy's CS-231n course notes.

", "votes": null, "replies": [] }, { "id": 104755, "author_name": "phunter", "author_url": "", "post_date": "01/16/2016 09:45:28", "content": "

I just wrote a blog talking about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)

\n\n

The CUDA installation on AWS is the tricky part, since the kernel and nouveau have conflicts with CUDA driver. Please follow every single steps in the blog for installing on AWS. After having CUDA on AWS, one can continue to install MXnet (with S3 support and some trouble shooting) by following the blog.

", "votes": null, "replies": [] }, { "id": 104764, "author_name": "mostafafr", "author_url": "", "post_date": "01/16/2016 10:30:58", "content": "

an interesting complement would be to show how to use S3 and spot instances (and ideally, provide code to do it programmatically)

\n\n

[quote=phunter;104755]

\n\n

I just wrote a blog talking about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)

\n\n

The CUDA installation on AWS is the tricky part, since the kernel and nouveau have conflicts with CUDA driver. Please follow every single steps in the blog for installing on AWS. After having CUDA on AWS, one can continue to install MXnet (with S3 support and some trouble shooting) by following the blog.

\n\n

[/quote]

", "votes": null, "replies": [] }, { "id": 104791, "author_name": "phunter", "author_url": "", "post_date": "01/16/2016 18:21:39", "content": "

AWS has its official spot instance tutorial (with video) https://aws.amazon.com/ec2/spot/spot-and-science/ and S3 as well http://docs.aws.amazon.com/AmazonS3/latest/gsg/GetStartedWithS3.html

\n\n

Using S3 in mxnet is super simple as in its official document (with example)\nhttp://mxnet.readthedocs.org/en/latest/aws.html#use-amazon-s3-to-host-data

", "votes": null, "replies": [] }, { "id": 104980, "author_name": "wouterd1", "author_url": "", "post_date": "01/18/2016 15:08:51", "content": "

@I Love Solder. I managed to get my AWS EC2 up and running this weekend. let me know if you have any remaining questions.

", "votes": null, "replies": [] }, { "id": 106131, "author_name": "joncarlisle", "author_url": "", "post_date": "01/28/2016 18:19:24", "content": "

[quote=WD;104980]

\n\n

@I Love Solder. I managed to get my AWS EC2 up and running this weekend. let me know if you have any remaining questions.

\n\n

[/quote]

\n\n

@WD What tutorial or combination of tutorials did you use to get yourself setup and running? THanks in advance!

", "votes": null, "replies": [] }, { "id": 106173, "author_name": "wouterd1", "author_url": "", "post_date": "01/28/2016 22:43:52", "content": "

i basically used https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN) to setup my installation on a AWS G2x2. i added and mounted a seperate EBS drive for the data, and made sure that the instance is backed up by saving the instance to an image upon installation of all packages. Then i added the MXNET installation - as per the Mxnet tutoral. For the latter, make sure that you comment out those instalation lines that might not be relevant (e.g. the S3 or the CUDNN lines). This might not all be clear, or you might run into issues, so definitely fire me other questions.

", "votes": null, "replies": [] }, { "id": 106666, "author_name": "joncarlisle", "author_url": "", "post_date": "02/03/2016 03:46:07", "content": "

@WD Did you install cuDNN and Caffe? I'm reviewing @Phunter's blog and noticed that cuDNN was optional install and no mention of installing Caffe. I'm guessing that MXNET is used in lieu of Caffe.

", "votes": null, "replies": [] }, { "id": 106768, "author_name": "phunter", "author_url": "", "post_date": "02/03/2016 19:45:47", "content": "

[quote=Mostafa;104764]

\n\n

an interesting complement would be to show how to use S3 and spot instances (and ideally, provide code to do it programmatically)

\n\n

[quote=phunter;104755]

\n\n

I just wrote a blog talking about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)

\n\n

The CUDA installation on AWS is the tricky part, since the kernel and nouveau have conflicts with CUDA driver. Please follow every single steps in the blog for installing on AWS. After having CUDA on AWS, one can continue to install MXnet (with S3 support and some trouble shooting) by following the blog.

\n\n

[/quote]

\n\n

[/quote]\nData science bowl has an official step-by-step guide of starting AWS spot instance with MXnet.jl http://www.datasciencebowl.com/aws_guide/ one can feel free to continue with my tutorial blog about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)

", "votes": null, "replies": [] }, { "id": 108136, "author_name": "joncarlisle", "author_url": "", "post_date": "02/15/2016 19:27:06", "content": "

Have AWS up and running now. I followed phunter's blog (the one with the cat) but needed to add a couple configuration settings from the install caffe blog as well.

\n\n

My new hurdle is uploading the data. Does anyone have any tips or tricks?

\n\n

Thanks!\nJon

", "votes": null, "replies": [] }, { "id": 110197, "author_name": "ilovesolder", "author_url": "", "post_date": "03/03/2016 18:34:05", "content": "

@WD,

\n\n

how do you back up an instance by saving it to image? do you have a link

\n\n

EDIT: so I created an image from the ec2 management console and I believe that the image worked.

\n\n

I still haven't tried to download and store data onto AWS so I might have further questions later on!

", "votes": null, "replies": [] }, { "id": 110225, "author_name": "wouterd1", "author_url": "", "post_date": "03/03/2016 21:45:27", "content": "

in the instance menu, if you go to actions, there should be an option that says "save instance to image". Does that work?

", "votes": null, "replies": [] }, { "id": 110233, "author_name": "ilovesolder", "author_url": "", "post_date": "03/04/2016 00:09:20", "content": "

@WD,

\n\n

yeah I was able to save an image that way, thanks!

\n\n

Now for the data part, do you download all the data onto the root disk(using wget)? Or do you upload the data onto S3 or something?

\n\n

If you do load the data onto S3, does your root disk need to be big enough to hold all of the data or is it possible to query the S3 database as the program goes? \nI wasn't able to find very clear instructions on how to use S3 database with EC2.

", "votes": null, "replies": [] }, { "id": 110238, "author_name": "phunter", "author_url": "", "post_date": "03/04/2016 00:25:42", "content": "

@I Love Solder I just unzipped everything to a local disk on AWS: it is not a big deal.

", "votes": null, "replies": [] }, { "id": 110241, "author_name": "ilovesolder", "author_url": "", "post_date": "03/04/2016 00:47:53", "content": "

@phunter

\n\n

Thanks for your response. So after you do your computations and stuff do you save instance to an image again? so that you can keep your data files and results? Or are you able to save those data some other ways?

", "votes": null, "replies": [] }, { "id": 110246, "author_name": "phunter", "author_url": "", "post_date": "03/04/2016 01:26:22", "content": "

It depends how much money you want to put to AWS. A lucky GPU AWS spot instance can cost as low as 0.07$ per hour, so I just don't turn it off :-)

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "104656": "Hello everyone!\r\n\r\nI have been following the Deep Learning Tutorial provided by Booz Allen Hamilton. \r\nSetting up environment was a pain but I think everything is working now.\r\n\r\nI have seen that people use AWS so that their personal computer don't have to be running full power all the time. I looked up AWS and I was swamped by all the features on AWS and tutorial there are online. (for example, whats the difference between using the AWS machine learning service, and running programs on EC2?)\r\n\r\nDoes anyone have a suggested tutorial to follow, given that I have successfully trained neural networks with both Caffe and Theano/Lasagne/nolearn? \r\n\r\nmany thanks", "104662": "I Love Solder,\r\n\r\nWhen I got started on using AWS for ConvNets I used the instruction on this [page][1]. I suspect that they are pretty dated by this point, for instance I doubt the referenced AMI is still around, but perhaps they'll help get you started.\r\n\r\n\r\n [1]: https://www.kaggle.com/c/facial-keypoints-detection/details/deep-learning-tutorial", "104667": "The AWS machine learning service seemed, the last time I checked, to likely be a wrapper around a machine learning tool called Vowpal Wabbit, and not a fully featured machine learning environment. Amazon EC2 will be your bread and butter; just think of it as a remote computer.\r\n\r\nThe best reference I've found for Convnets are Andrej Karpathy's CS-231n course notes.", "104755": "I just wrote a blog talking about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)\r\n\r\nThe CUDA installation on AWS is the tricky part, since the kernel and nouveau have conflicts with CUDA driver. Please follow every single steps in the blog for installing on AWS. After having CUDA on AWS, one can continue to install MXnet (with S3 support and some trouble shooting) by following the blog.", "104764": "an interesting complement would be to show how to use S3 and spot instances (and ideally, provide code to do it programmatically)\r\n\r\n[quote=phunter;104755]\r\n\r\nI just wrote a blog talking about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)\r\n\r\nThe CUDA installation on AWS is the tricky part, since the kernel and nouveau have conflicts with CUDA driver. Please follow every single steps in the blog for installing on AWS. After having CUDA on AWS, one can continue to install MXnet (with S3 support and some trouble shooting) by following the blog.\r\n\r\n[/quote]", "104791": "AWS has its official spot instance tutorial (with video) https://aws.amazon.com/ec2/spot/spot-and-science/ and S3 as well http://docs.aws.amazon.com/AmazonS3/latest/gsg/GetStartedWithS3.html \r\n\r\nUsing S3 in mxnet is super simple as in its official document (with example)\r\nhttp://mxnet.readthedocs.org/en/latest/aws.html#use-amazon-s3-to-host-data", "104980": "I Love Solder. I managed to get my AWS EC2 up and running this weekend. let me know if you have any remaining questions.", "106131": "[quote=WD;104980]\r\n\r\n@I Love Solder. I managed to get my AWS EC2 up and running this weekend. let me know if you have any remaining questions. \r\n\r\n[/quote]\r\n\r\n@WD What tutorial or combination of tutorials did you use to get yourself setup and running? THanks in advance!", "106173": "i basically used https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN) to setup my installation on a AWS G2x2. i added and mounted a seperate EBS drive for the data, and made sure that the instance is backed up by saving the instance to an image upon installation of all packages. Then i added the MXNET installation - as per the Mxnet tutoral. For the latter, make sure that you comment out those instalation lines that might not be relevant (e.g. the S3 or the CUDNN lines). This might not all be clear, or you might run into issues, so definitely fire me other questions.", "106666": "WD Did you install cuDNN and Caffe? I'm reviewing @Phunter's blog and noticed that cuDNN was optional install and no mention of installing Caffe. I'm guessing that MXNET is used in lieu of Caffe.", "106768": "[quote=Mostafa;104764]\r\n\r\n\r\nan interesting complement would be to show how to use S3 and spot instances (and ideally, provide code to do it programmatically)\r\n\r\n[quote=phunter;104755]\r\n\r\nI just wrote a blog talking about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)\r\n\r\nThe CUDA installation on AWS is the tricky part, since the kernel and nouveau have conflicts with CUDA driver. Please follow every single steps in the blog for installing on AWS. After having CUDA on AWS, one can continue to install MXnet (with S3 support and some trouble shooting) by following the blog.\r\n\r\n[/quote]\r\n\r\n\r\n[/quote]\r\nData science bowl has an official step-by-step guide of starting AWS spot instance with MXnet.jl http://www.datasciencebowl.com/aws_guide/ one can feel free to continue with my tutorial blog about installing MXnet on AWS GPU instance. https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ (this link has a cat inside)", "108136": "Have AWS up and running now. I followed phunter's blog (the one with the cat) but needed to add a couple configuration settings from the install caffe blog as well.\r\n\r\nMy new hurdle is uploading the data. Does anyone have any tips or tricks?\r\n\r\nThanks!\r\nJon", "110197": "WD, \r\n\r\nhow do you back up an instance by saving it to image? do you have a link \r\n\r\nEDIT: so I created an image from the ec2 management console and I believe that the image worked. \r\n\r\nI still haven't tried to download and store data onto AWS so I might have further questions later on!", "110225": "in the instance menu, if you go to actions, there should be an option that says \"save instance to image\". Does that work?", "110233": "WD,\r\n\r\nyeah I was able to save an image that way, thanks!\r\n\r\nNow for the data part, do you download all the data onto the root disk(using wget)? Or do you upload the data onto S3 or something? \r\n\r\nIf you do load the data onto S3, does your root disk need to be big enough to hold all of the data or is it possible to query the S3 database as the program goes? \r\nI wasn't able to find very clear instructions on how to use S3 database with EC2.", "110238": "I Love Solder I just unzipped everything to a local disk on AWS: it is not a big deal.", "110241": "phunter\r\n\r\nThanks for your response. So after you do your computations and stuff do you save instance to an image again? so that you can keep your data files and results? Or are you able to save those data some other ways?", "110246": "It depends how much money you want to put to AWS. A lucky GPU AWS spot instance can cost as low as 0.07$ per hour, so I just don't turn it off :-)" }, "source": "meta" }