Kagglers,\nI'm currently making a tutorial on using AWS cost effectively on the DSB2016. I've used AWS with spot pricing many times across many Kaggles (via frequently scp backing up work). However, I'd like to solicit your feedback on using AWS for this particular challenge so I can build a better tutorial.\nThanks,\nMike (admin)
", "messages": [ { "id": "103649", "postDate": "01/05/2016 06:02:20", "content": "Kagglers,\nI'm currently making a tutorial on using AWS cost effectively on the DSB2016. I've used AWS with spot pricing many times across many Kaggles (via frequently scp backing up work). However, I'd like to solicit your feedback on using AWS for this particular challenge so I can build a better tutorial.\nThanks,\nMike (admin)
", "rawMarkdown": "Kagglers,\r\nI'm currently making a tutorial on using AWS cost effectively on the DSB2016. I've used AWS with spot pricing many times across many Kaggles (via frequently scp backing up work). However, I'd like to solicit your feedback on using AWS for this particular challenge so I can build a better tutorial.\r\nThanks,\r\nMike (admin)", "votes": null }, { "id": "103651", "postDate": "01/05/2016 06:13:55", "content": "Maybe a tutorial of starting a MXnet docker image with CUDA can help? http://mxnt.ml/en/latest/build.html#docker-images Since there is a MXnet tutorial, with this preinstalled docker, people can quickly pickup the tutorial.
", "rawMarkdown": "Maybe a tutorial of starting a MXnet docker image with CUDA can help? http://mxnt.ml/en/latest/build.html#docker-images Since there is a MXnet tutorial, with this preinstalled docker, people can quickly pickup the tutorial.", "votes": null }, { "id": "103696", "postDate": "01/05/2016 16:13:33", "content": "@Mike Kim, That sounds like a great idea.
\n\nThe last time I used AWS for this sort of thing was a while ago, so I forget the details, but my general strategy was:
\n\nI don't recall the details of #1, but I don't think it's too difficult. For #2, I have code setup to do that automatically in Lasagne, but in this contest most people are using mxNet, so I'm not sure if the code itself would be useful. However, I'm happy to supply it if you like.
", "rawMarkdown": "Mike Kim, That sounds like a great idea. \r\n\r\nThe last time I used AWS for this sort of thing was a while ago, so I forget the details, but my general strategy was:\r\n\r\n1. Mount a non-root volume with the do not delete flag set (I don't think root volumes can be preserved, although it's possible that has changed).\r\n2. Setup my neural net code so that it dumped out the weights every 20 epochs or so to the mounted volume.\r\n\r\nI don't recall the details of #1, but I don't think it's too difficult. For #2, I have code setup to do that automatically in Lasagne, but in this contest most people are using mxNet, so I'm not sure if the code itself would be useful. However, I'm happy to supply it if you like.", "votes": null }, { "id": "103729", "postDate": "01/05/2016 22:11:29", "content": "If you are using mxnet, one thing you can do is directly work with S3 as data source and path to save data, so you won't miss anything when machine goes off, and do not need to re-process the data when you restart the machine.
\n\nWe think this can be quite helpful, but would love to hear your feedback on this
", "rawMarkdown": "If you are using mxnet, one thing you can do is directly work with S3 as data source and path to save data, so you won't miss anything when machine goes off, and do not need to re-process the data when you restart the machine.\r\n\r\nWe think this can be quite helpful, but would love to hear your feedback on this", "votes": null }, { "id": "103842", "postDate": "01/07/2016 00:06:40", "content": "So should I follow this: http://mxnet.readthedocs.org/en/latest/aws.html and see if I run into any problems? I also see there's a public AMI already on AWS here: https://github.com/dmlc/MXNet.jl/issues/43\nAm I missing anything important here?
", "rawMarkdown": "So should I follow this: http://mxnet.readthedocs.org/en/latest/aws.html and see if I run into any problems? I also see there's a public AMI already on AWS here: https://github.com/dmlc/MXNet.jl/issues/43\r\nAm I missing anything important here?", "votes": null }, { "id": "103847", "postDate": "01/07/2016 00:40:41", "content": "Yes, the public AMI seems did not compile with s3 enabled, but you can easily recompile it
", "rawMarkdown": "Yes, the public AMI seems did not compile with s3 enabled, but you can easily recompile it", "votes": null }, { "id": "105250", "postDate": "01/21/2016 12:45:40", "content": "[quote=Tim Hochberg;103696]\n2. Setup my neural net code so that it dumped out the weights every 20 epochs or so to the mounted volume.\nFor #2, I have code setup to do that automatically in Lasagne, but in this contest most people are using mxNet, so I'm not sure if the code itself would be useful. However, I'm happy to supply it if you like.
\n\n[/quote]
\n\nyes, please supply the code
", "rawMarkdown": "[quote=Tim Hochberg;103696]\r\n2. Setup my neural net code so that it dumped out the weights every 20 epochs or so to the mounted volume.\r\nFor #2, I have code setup to do that automatically in Lasagne, but in this contest most people are using mxNet, so I'm not sure if the code itself would be useful. However, I'm happy to supply it if you like.\r\n\r\n[/quote]\r\n\r\nyes, please supply the code", "votes": null }, { "id": "105275", "postDate": "01/21/2016 18:01:05", "content": "once the weights are dumped on the mounted volume you can sync to an s3 bucket with the boto python library http://boto.cloudhackers.com/en/latest/ I use this cause I run spot instances that get terminated abruptly
\n\ndef write_model_data_s3(filename):
\n\n s3 = boto.connect_s3()\n key = s3.get_bucket('weights').new_key(filename)\n key.set_contents_from_filename(filename)\ndef read_model_data_s3(model, filename):
\n\n s3 = boto.connect_s3()\n key = s3.get_bucket('weights').get_key(filename)\n key.get_contents_to_filename(filename)\nmy current setup - after my instance got killed a few times - and had to reinstall packages (fingers crossed that this works):
\n\nTo be honest - this might not be a failproof / smart setup. so please let me know where i can improve. I have not yet fully understood why S3 is superior to EBS. It seems slightly more complex to setup than to stay only in EC2, but i might be missing something.
\n\n(maybe - as a community - we should create a (set of) public AMIs that allows other to leapfrog over the hassle of installing CUDA and other packages and drop straight into a well-functioning AMI that can be rebooted as needed)
", "rawMarkdown": "my current setup - after my instance got killed a few times - and had to reinstall packages (fingers crossed that this works): \r\n\r\n* Launch spot instance. Uncheck the checkmark of \"delete on termination\" of the root drive. \r\n* Regardless, create a seperate EBS volume to store the data, and mount this drive seperately to the instance. \r\n* Download the data on the seperate EBS volume. so at least you never have to download that data-again \r\n* (install all packages) \r\n* Make an image out of your current instance (in EC2 dashboard - Instances - Actions - create image). I believe that will allow one to restore the instance with all settings if it gets killed. it automatically creates a snapshot of all EBS drives as well. (the AWS language on the difference between snapshots and images is a bit confusing - but i believe (correct me if wrong) that image creation is the safest bet.\r\n\r\nTo be honest - this might not be a failproof / smart setup. so please let me know where i can improve. I have not yet fully understood why S3 is superior to EBS. It seems slightly more complex to setup than to stay only in EC2, but i might be missing something.\r\n\r\n(maybe - as a community - we should create a (set of) public AMIs that allows other to leapfrog over the hassle of installing CUDA and other packages and drop straight into a well-functioning AMI that can be rebooted as needed)", "votes": null }, { "id": "105282", "postDate": "01/21/2016 18:49:17", "content": "@WD you can setup dropbox on your AWS and auto sync the output data directory with local directory. It is one of couple of solutions. You can also ssh-mount a local directory to AWS, and all output are sync-ed to local.
", "rawMarkdown": "WD you can setup dropbox on your AWS and auto sync the output data directory with local directory. It is one of couple of solutions. You can also ssh-mount a local directory to AWS, and all output are sync-ed to local.", "votes": null }, { "id": "105326", "postDate": "01/22/2016 04:08:12", "content": "@Mike Kim, such a tutorial would be helpful. Do you plan to create a public AMI that we can leverage. Thanks!
", "rawMarkdown": "Mike Kim, such a tutorial would be helpful. Do you plan to create a public AMI that we can leverage. Thanks!", "votes": null }, { "id": "105328", "postDate": "01/22/2016 04:37:31", "content": "@Mostafa, this assumes that you are using nolearn as well as lasagne. The dumper itself is very simple:
\n\nclass WeightDumper(object):\n\n def __init__(self, name, interval=20):\n self.name = name\n self.interval = interval\n\n def __call__(self, nn, train_history):\n current_epoch = train_history[-1]['epoch']\n if (current_epoch % self.interval) == 0:\n nn.save_params_to("{0}-{1}.pickle".format(self.name, current_epoch))\nThen one would also include the following when creating your net:
\n\nnet = NeuralNet(\n # ...\n on_epoch_finished=[\n WeightDumper("PATH_TO_DUMP_TO"),\n ],\n # ...\n )\nThat's it. Hope that helps.
", "rawMarkdown": "Mostafa, this assumes that you are using nolearn as well as lasagne. The dumper itself is very simple:\r\n\r\n class WeightDumper(object):\r\n\r\n def __init__(self, name, interval=20):\r\n self.name = name\r\n self.interval = interval\r\n\r\n def __call__(self, nn, train_history):\r\n current_epoch = train_history[-1]['epoch']\r\n if (current_epoch % self.interval) == 0:\r\n nn.save_params_to(\"{0}-{1}.pickle\".format(self.name, current_epoch))\r\n\r\nThen one would also include the following when creating your net:\r\n\r\n net = NeuralNet(\r\n # ...\r\n on_epoch_finished=[\r\n WeightDumper(\"PATH_TO_DUMP_TO\"),\r\n ],\r\n # ...\r\n )\r\n\r\nThat's it. Hope that helps.", "votes": null }, { "id": "105532", "postDate": "01/24/2016 08:26:24", "content": "Noob question. let's assume one has an AWS instance with 8 cpus and 1 gpu. I believe that one can open multiple putty/terminal windows/sessions on the same instance. Is there any benefit in doing so? Or is it the case that if I open another putty session to do some file indexing while the main sesssion is e.g. proprocessing, then the result is that they run in parallel, but that the preprocessing will now go a bit slower?
", "rawMarkdown": "Noob question. let's assume one has an AWS instance with 8 cpus and 1 gpu. I believe that one can open multiple putty/terminal windows/sessions on the same instance. Is there any benefit in doing so? Or is it the case that if I open another putty session to do some file indexing while the main sesssion is e.g. proprocessing, then the result is that they run in parallel, but that the preprocessing will now go a bit slower?", "votes": null }, { "id": "105533", "postDate": "01/24/2016 10:22:04", "content": "Hi Mike,\nI would be very interested in this, but please could it be more platform-independent than just Mxnet? Thanks,
", "rawMarkdown": "Hi Mike,\r\nI would be very interested in this, but please could it be more platform-independent than just Mxnet? Thanks,", "votes": null } ], "comments": [ { "id": 103651, "author_name": "phunter", "author_url": "", "post_date": "01/05/2016 06:13:55", "content": "Maybe a tutorial of starting a MXnet docker image with CUDA can help? http://mxnt.ml/en/latest/build.html#docker-images Since there is a MXnet tutorial, with this preinstalled docker, people can quickly pickup the tutorial.
", "votes": null, "replies": [] }, { "id": 103696, "author_name": "bitsofbits", "author_url": "", "post_date": "01/05/2016 16:13:33", "content": "@Mike Kim, That sounds like a great idea.
\n\nThe last time I used AWS for this sort of thing was a while ago, so I forget the details, but my general strategy was:
\n\nI don't recall the details of #1, but I don't think it's too difficult. For #2, I have code setup to do that automatically in Lasagne, but in this contest most people are using mxNet, so I'm not sure if the code itself would be useful. However, I'm happy to supply it if you like.
", "votes": null, "replies": [] }, { "id": 103729, "author_name": "tqchen", "author_url": "", "post_date": "01/05/2016 22:11:29", "content": "If you are using mxnet, one thing you can do is directly work with S3 as data source and path to save data, so you won't miss anything when machine goes off, and do not need to re-process the data when you restart the machine.
\n\nWe think this can be quite helpful, but would love to hear your feedback on this
", "votes": null, "replies": [] }, { "id": 103842, "author_name": "mikeskim", "author_url": "", "post_date": "01/07/2016 00:06:40", "content": "So should I follow this: http://mxnet.readthedocs.org/en/latest/aws.html and see if I run into any problems? I also see there's a public AMI already on AWS here: https://github.com/dmlc/MXNet.jl/issues/43\nAm I missing anything important here?
", "votes": null, "replies": [] }, { "id": 103847, "author_name": "tqchen", "author_url": "", "post_date": "01/07/2016 00:40:41", "content": "Yes, the public AMI seems did not compile with s3 enabled, but you can easily recompile it
", "votes": null, "replies": [] }, { "id": 105250, "author_name": "mostafafr", "author_url": "", "post_date": "01/21/2016 12:45:40", "content": "[quote=Tim Hochberg;103696]\n2. Setup my neural net code so that it dumped out the weights every 20 epochs or so to the mounted volume.\nFor #2, I have code setup to do that automatically in Lasagne, but in this contest most people are using mxNet, so I'm not sure if the code itself would be useful. However, I'm happy to supply it if you like.
\n\n[/quote]
\n\nyes, please supply the code
", "votes": null, "replies": [] }, { "id": 105275, "author_name": "godaibo", "author_url": "", "post_date": "01/21/2016 18:01:05", "content": "once the weights are dumped on the mounted volume you can sync to an s3 bucket with the boto python library http://boto.cloudhackers.com/en/latest/ I use this cause I run spot instances that get terminated abruptly
\n\ndef write_model_data_s3(filename):
\n\n s3 = boto.connect_s3()\n key = s3.get_bucket('weights').new_key(filename)\n key.set_contents_from_filename(filename)\ndef read_model_data_s3(model, filename):
\n\n s3 = boto.connect_s3()\n key = s3.get_bucket('weights').get_key(filename)\n key.get_contents_to_filename(filename)\nmy current setup - after my instance got killed a few times - and had to reinstall packages (fingers crossed that this works):
\n\nTo be honest - this might not be a failproof / smart setup. so please let me know where i can improve. I have not yet fully understood why S3 is superior to EBS. It seems slightly more complex to setup than to stay only in EC2, but i might be missing something.
\n\n(maybe - as a community - we should create a (set of) public AMIs that allows other to leapfrog over the hassle of installing CUDA and other packages and drop straight into a well-functioning AMI that can be rebooted as needed)
", "votes": null, "replies": [] }, { "id": 105282, "author_name": "phunter", "author_url": "", "post_date": "01/21/2016 18:49:17", "content": "@WD you can setup dropbox on your AWS and auto sync the output data directory with local directory. It is one of couple of solutions. You can also ssh-mount a local directory to AWS, and all output are sync-ed to local.
", "votes": null, "replies": [] }, { "id": 105326, "author_name": "opraveen", "author_url": "", "post_date": "01/22/2016 04:08:12", "content": "@Mike Kim, such a tutorial would be helpful. Do you plan to create a public AMI that we can leverage. Thanks!
", "votes": null, "replies": [] }, { "id": 105328, "author_name": "bitsofbits", "author_url": "", "post_date": "01/22/2016 04:37:31", "content": "@Mostafa, this assumes that you are using nolearn as well as lasagne. The dumper itself is very simple:
\n\nclass WeightDumper(object):\n\n def __init__(self, name, interval=20):\n self.name = name\n self.interval = interval\n\n def __call__(self, nn, train_history):\n current_epoch = train_history[-1]['epoch']\n if (current_epoch % self.interval) == 0:\n nn.save_params_to("{0}-{1}.pickle".format(self.name, current_epoch))\nThen one would also include the following when creating your net:
\n\nnet = NeuralNet(\n # ...\n on_epoch_finished=[\n WeightDumper("PATH_TO_DUMP_TO"),\n ],\n # ...\n )\nThat's it. Hope that helps.
", "votes": null, "replies": [] }, { "id": 105532, "author_name": "wouterd1", "author_url": "", "post_date": "01/24/2016 08:26:24", "content": "Noob question. let's assume one has an AWS instance with 8 cpus and 1 gpu. I believe that one can open multiple putty/terminal windows/sessions on the same instance. Is there any benefit in doing so? Or is it the case that if I open another putty session to do some file indexing while the main sesssion is e.g. proprocessing, then the result is that they run in parallel, but that the preprocessing will now go a bit slower?
", "votes": null, "replies": [] }, { "id": 105533, "author_name": "eoghanf", "author_url": "", "post_date": "01/24/2016 10:22:04", "content": "Hi Mike,\nI would be very interested in this, but please could it be more platform-independent than just Mxnet? Thanks,
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "103649": "Kagglers,\r\nI'm currently making a tutorial on using AWS cost effectively on the DSB2016. I've used AWS with spot pricing many times across many Kaggles (via frequently scp backing up work). However, I'd like to solicit your feedback on using AWS for this particular challenge so I can build a better tutorial.\r\nThanks,\r\nMike (admin)", "103651": "Maybe a tutorial of starting a MXnet docker image with CUDA can help? http://mxnt.ml/en/latest/build.html#docker-images Since there is a MXnet tutorial, with this preinstalled docker, people can quickly pickup the tutorial.", "103696": "Mike Kim, That sounds like a great idea. \r\n\r\nThe last time I used AWS for this sort of thing was a while ago, so I forget the details, but my general strategy was:\r\n\r\n1. Mount a non-root volume with the do not delete flag set (I don't think root volumes can be preserved, although it's possible that has changed).\r\n2. Setup my neural net code so that it dumped out the weights every 20 epochs or so to the mounted volume.\r\n\r\nI don't recall the details of #1, but I don't think it's too difficult. For #2, I have code setup to do that automatically in Lasagne, but in this contest most people are using mxNet, so I'm not sure if the code itself would be useful. However, I'm happy to supply it if you like.", "103729": "If you are using mxnet, one thing you can do is directly work with S3 as data source and path to save data, so you won't miss anything when machine goes off, and do not need to re-process the data when you restart the machine.\r\n\r\nWe think this can be quite helpful, but would love to hear your feedback on this", "103842": "So should I follow this: http://mxnet.readthedocs.org/en/latest/aws.html and see if I run into any problems? I also see there's a public AMI already on AWS here: https://github.com/dmlc/MXNet.jl/issues/43\r\nAm I missing anything important here?", "103847": "Yes, the public AMI seems did not compile with s3 enabled, but you can easily recompile it", "105250": "[quote=Tim Hochberg;103696]\r\n2. Setup my neural net code so that it dumped out the weights every 20 epochs or so to the mounted volume.\r\nFor #2, I have code setup to do that automatically in Lasagne, but in this contest most people are using mxNet, so I'm not sure if the code itself would be useful. However, I'm happy to supply it if you like.\r\n\r\n[/quote]\r\n\r\nyes, please supply the code", "105275": "once the weights are dumped on the mounted volume you can sync to an s3 bucket with the boto python library http://boto.cloudhackers.com/en/latest/ I use this cause I run spot instances that get terminated abruptly\r\n\r\n\r\ndef write_model_data_s3(filename):\r\n\r\n s3 = boto.connect_s3()\r\n key = s3.get_bucket('weights').new_key(filename)\r\n key.set_contents_from_filename(filename)\r\n\r\ndef read_model_data_s3(model, filename):\r\n\r\n s3 = boto.connect_s3()\r\n key = s3.get_bucket('weights').get_key(filename)\r\n key.get_contents_to_filename(filename)", "105278": "my current setup - after my instance got killed a few times - and had to reinstall packages (fingers crossed that this works): \r\n\r\n* Launch spot instance. Uncheck the checkmark of \"delete on termination\" of the root drive. \r\n* Regardless, create a seperate EBS volume to store the data, and mount this drive seperately to the instance. \r\n* Download the data on the seperate EBS volume. so at least you never have to download that data-again \r\n* (install all packages) \r\n* Make an image out of your current instance (in EC2 dashboard - Instances - Actions - create image). I believe that will allow one to restore the instance with all settings if it gets killed. it automatically creates a snapshot of all EBS drives as well. (the AWS language on the difference between snapshots and images is a bit confusing - but i believe (correct me if wrong) that image creation is the safest bet.\r\n\r\nTo be honest - this might not be a failproof / smart setup. so please let me know where i can improve. I have not yet fully understood why S3 is superior to EBS. It seems slightly more complex to setup than to stay only in EC2, but i might be missing something.\r\n\r\n(maybe - as a community - we should create a (set of) public AMIs that allows other to leapfrog over the hassle of installing CUDA and other packages and drop straight into a well-functioning AMI that can be rebooted as needed)", "105282": "WD you can setup dropbox on your AWS and auto sync the output data directory with local directory. It is one of couple of solutions. You can also ssh-mount a local directory to AWS, and all output are sync-ed to local.", "105326": "Mike Kim, such a tutorial would be helpful. Do you plan to create a public AMI that we can leverage. Thanks!", "105328": "Mostafa, this assumes that you are using nolearn as well as lasagne. The dumper itself is very simple:\r\n\r\n class WeightDumper(object):\r\n\r\n def __init__(self, name, interval=20):\r\n self.name = name\r\n self.interval = interval\r\n\r\n def __call__(self, nn, train_history):\r\n current_epoch = train_history[-1]['epoch']\r\n if (current_epoch % self.interval) == 0:\r\n nn.save_params_to(\"{0}-{1}.pickle\".format(self.name, current_epoch))\r\n\r\nThen one would also include the following when creating your net:\r\n\r\n net = NeuralNet(\r\n # ...\r\n on_epoch_finished=[\r\n WeightDumper(\"PATH_TO_DUMP_TO\"),\r\n ],\r\n # ...\r\n )\r\n\r\nThat's it. Hope that helps.", "105532": "Noob question. let's assume one has an AWS instance with 8 cpus and 1 gpu. I believe that one can open multiple putty/terminal windows/sessions on the same instance. Is there any benefit in doing so? Or is it the case that if I open another putty session to do some file indexing while the main sesssion is e.g. proprocessing, then the result is that they run in parallel, but that the preprocessing will now go a bit slower?", "105533": "Hi Mike,\r\nI would be very interested in this, but please could it be more platform-independent than just Mxnet? Thanks," }, "source": "meta" }