The first 7 minutes of this video explains a very simple method to set up a GPU instance on Google Cloud using Jetware pre-made instances.
\nhttps://www.youtube.com/watch?v=W-FqRBoyTgw
Just two things to keep in mind:
\n1. The video mentions the need to request a gpu quota - I didn't need to do this so this requirement may no longer be applicable.
\n2. Keras is pre-installed but to get it to work the backend first needs to be manually set as follows:
import os\nos.environ['KERAS_BACKEND'] = 'tensorflow'\nimport keras\nUpdate:
\nThese are the steps I followed to set up the environment for this competition - just in case someone finds this useful.
On the jupyter notebook click the 'new' button and select 'terminal'. Then enter the following:
\n\n $ pip install sklearn\n\n $ pip install scikit-image\n\n $ pip install pydicom\n\n $ pip install opencv-python \n\n# install kaggle cli\n $ pip install kaggle-cli\n\n# download competition files from kaggle\n# for 'competition' enter: rsna-pneumonia-detection-challenge\n# Note that it should be a kaggle password and not a google or facebook password.\n\n $ kg download -u yourkaggleusername -p yourkagglepassword -c competition\n\n# unzip the zipped image files to a new destination folder\n\n $ unzip stage_1_test_images.zip -d stage_1_test_images\n $ unzip stage_1_train_images.zip -d stage_1_train_images\nI ran into errors reading the csv files using pandas read_csv(). To solve this problem I downloaded the csv files from kaggle to my local pc and then uploaded them to the instance using the 'upload' button on the jupyter notebook. Then read_csv() worked.
", "messages": [ { "id": "387868", "postDate": "09/15/2018 20:24:22", "content": "The first 7 minutes of this video explains a very simple method to set up a GPU instance on Google Cloud using Jetware pre-made instances.
\nhttps://www.youtube.com/watch?v=W-FqRBoyTgw
Just two things to keep in mind:
\n1. The video mentions the need to request a gpu quota - I didn't need to do this so this requirement may no longer be applicable.
\n2. Keras is pre-installed but to get it to work the backend first needs to be manually set as follows:
import os\nos.environ['KERAS_BACKEND'] = 'tensorflow'\nimport keras\nUpdate:
\nThese are the steps I followed to set up the environment for this competition - just in case someone finds this useful.
On the jupyter notebook click the 'new' button and select 'terminal'. Then enter the following:
\n\n $ pip install sklearn\n\n $ pip install scikit-image\n\n $ pip install pydicom\n\n $ pip install opencv-python \n\n# install kaggle cli\n $ pip install kaggle-cli\n\n# download competition files from kaggle\n# for 'competition' enter: rsna-pneumonia-detection-challenge\n# Note that it should be a kaggle password and not a google or facebook password.\n\n $ kg download -u yourkaggleusername -p yourkagglepassword -c competition\n\n# unzip the zipped image files to a new destination folder\n\n $ unzip stage_1_test_images.zip -d stage_1_test_images\n $ unzip stage_1_train_images.zip -d stage_1_train_images\nI ran into errors reading the csv files using pandas read_csv(). To solve this problem I downloaded the csv files from kaggle to my local pc and then uploaded them to the instance using the 'upload' button on the jupyter notebook. Then read_csv() worked.
", "rawMarkdown": "The first 7 minutes of this video explains a very simple method to set up a GPU instance on Google Cloud using Jetware pre-made instances.here is my quick setup for gcp instance, the whole process takes about 15 min:
\n\nsudo add-apt-repository ppa:graphics-drivers/ppa -y && sudo apt update && sudo apt-get install -y nvidia-396 nvidia- \nmodprobe\nsudo reboot\nnvidia-smi\n\nwget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.0.176-1_amd64.deb\nwget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb\nwget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libcudnn7-dev_7.0.5.15-1+cuda9.0_amd64.deb\nwget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libnccl2_2.1.4-1+cuda9.0_amd64.deb\nwget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libnccl-dev_2.1.4-1+cuda9.0_amd64.deb\nsudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub\n\nsudo dpkg -i cuda-repo-ubuntu1604_9.0.176-1_amd64.deb\nsudo dpkg -i libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb\nsudo dpkg -i libcudnn7-dev_7.0.5.15-1+cuda9.0_amd64.deb\nsudo dpkg -i libnccl2_2.1.4-1+cuda9.0_amd64.deb\nsudo dpkg -i libnccl-dev_2.1.4-1+cuda9.0_amd64.deb\n\nsudo apt-get update\nsudo apt-get install -y --allow-unauthenticated cuda=9.0.176-1\nsudo apt-get install -y libcudnn7-dev\nsudo apt-get install -y libnccl-dev\nreboot\n\n\nsudo apt install -y python-pip python3-pip\n\npip install kaggle\npip install cython\npip install numpy\npip install tensorflow-gpu\npip install matplotlib\npip install pillow\n\nsudo apt install -y libopencv-dev\npip install opencv-python \nOr use a preconfigured image, per this blog post, e.g. (assuming you have gcloud installed):
\n\n export IMAGE_FAMILY=\"tf-latest-cu92\"\n export ZONE=\"us-east1-b\"\n export INSTANCE_NAME=\"gpu-instance\"\n export INSTANCE_TYPE=\"n1-highmem-4\"\n gcloud compute instances create $INSTANCE_NAME \\\n --zone=$ZONE \\\n --image-family=$IMAGE_FAMILY \\\n --image-project=deeplearning-platform-release \\\n --maintenance-policy=TERMINATE \\\n --accelerator='type=nvidia-tesla-p100,count=1' \\\n --machine-type=$INSTANCE_TYPE \\\n --boot-disk-size=240GB \\\n --metadata='install-nvidia-driver=True' \\\n --preemptible\n(I opted for a single preemptible p100. If you want more than one GPU, you probably do need to request a quota increase.) Then you don't have to worry about CUDA and Tensorflow versions, driver compatibility, and such. Though some things (like the Kaggle API) you might still have to install yourself.
", "rawMarkdown": "Or use a preconfigured image, per [this blog post][1], e.g. (assuming you have gcloud installed):\n export IMAGE_FAMILY=\"tf-latest-cu92\"\n export ZONE=\"us-east1-b\"\n export INSTANCE_NAME=\"gpu-instance\"\n export INSTANCE_TYPE=\"n1-highmem-4\"\n gcloud compute instances create $INSTANCE_NAME \\\n --zone=$ZONE \\\n --image-family=$IMAGE_FAMILY \\\n --image-project=deeplearning-platform-release \\\n --maintenance-policy=TERMINATE \\\n --accelerator='type=nvidia-tesla-p100,count=1' \\\n --machine-type=$INSTANCE_TYPE \\\n --boot-disk-size=240GB \\\n --metadata='install-nvidia-driver=True' \\\n --preemptible\nYes; I think this is the most straightforward solution. It even has JupyterLab running already. You can just run the above code in the cloud console without Gcloud Cli. However, I do have a problem using virtual environments with JupyterLab. Does anyone have any pointers for this?
", "rawMarkdown": "Yes; I think this is the most straightforward solution. It even has JupyterLab running already. You can just run the above code in the cloud console without Gcloud Cli. However, I do have a problem using virtual environments with JupyterLab. Does anyone have any pointers for this?", "votes": null }, { "id": "388290", "postDate": "09/16/2018 17:56:53", "content": "I use GCP as well. I started with the tensorflow deep learning GCP image with ubuntu. However, I HATE not having a GUI. So I installed xfce and VNCserver and some xrdp server. I RDP to the instance, which brings me to the VNC server where I login. Right now I'm the only one accessing it, but this should allow me to run this instance with others in the future if I choose.
\n\nI then installed anaconda navigator and I use that to launch jupyter notebooks and whatever else. I use the notebooks through the RDP session as opposed to through the web. Once you've built your instance out the way you like, just image it. Then you can delete everything but the image when you're done for the day, saving a ton of money and giving you the opportunity to re-launch the same image on more (or less) robust hardware, as the task demands.
\n\nFor example, I use the same image in GCP but sometimes I run it on a 16cpu P100 box and other times I'll just run something light like a K80. Obviously this flexibility results in significant cost savings. I've also found GCP far easier to deal with than AWS, not to mention far less expensive.
", "rawMarkdown": "I use GCP as well. I started with the tensorflow deep learning GCP image with ubuntu. However, I HATE not having a GUI. So I installed xfce and VNCserver and some xrdp server. I RDP to the instance, which brings me to the VNC server where I login. Right now I'm the only one accessing it, but this should allow me to run this instance with others in the future if I choose.\n\nI then installed anaconda navigator and I use that to launch jupyter notebooks and whatever else. I use the notebooks through the RDP session as opposed to through the web. Once you've built your instance out the way you like, just image it. Then you can delete everything but the image when you're done for the day, saving a ton of money and giving you the opportunity to re-launch the same image on more (or less) robust hardware, as the task demands.\n\nFor example, I use the same image in GCP but sometimes I run it on a 16cpu P100 box and other times I'll just run something light like a K80. Obviously this flexibility results in significant cost savings. I've also found GCP far easier to deal with than AWS, not to mention far less expensive.", "votes": null }, { "id": "397004", "postDate": "10/01/2018 18:37:54", "content": "Hi, I would like to seek an advice from you guys! Given that I already requested to increase my qouta and it was already approved and reflected on my qouta list ( increase my qouta limit from 1 to 2 in preemptible P100), why I was always encountered this issue \"Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 1.0 globally\".. when I'm creating my VM instance with 2 preemptible P100? Do you already encountered this same issue? If yes, how did you address it? Thank you! :-).. I found that this competition requires a lot of computational power! :-(
", "rawMarkdown": "Hi, I would like to seek an advice from you guys! Given that I already requested to increase my qouta and it was already approved and reflected on my qouta list ( increase my qouta limit from 1 to 2 in preemptible P100), why I was always encountered this issue \"Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 1.0 globally\".. when I'm creating my VM instance with 2 preemptible P100? Do you already encountered this same issue? If yes, how did you address it? Thank you! :-).. I found that this competition requires a lot of computational power! :-(", "votes": null }, { "id": "397016", "postDate": "10/01/2018 19:00:20", "content": "Note that preemptible gpus can only be attached to preemptive instances as far as I know, so you need to set the preemptive attribute on the instance
", "rawMarkdown": "Note that preemptible gpus can only be attached to preemptive instances as far as I know, so you need to set the preemptive attribute on the instance", "votes": null }, { "id": "408107", "postDate": "10/22/2018 10:25:48", "content": "Was this Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 1.0 globally issue solved?
", "rawMarkdown": "Was this Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 1.0 globally issue solved?", "votes": null }, { "id": "408273", "postDate": "10/22/2018 15:26:14", "content": "Thanks a lot for the link and setup example, it was really helpful. I decided to move from AWS to GCP as my cloud gpu solution and everything is so smooth and slick, not to mention cheaper.
", "rawMarkdown": "Thanks a lot for the link and setup example, it was really helpful. I decided to move from AWS to GCP as my cloud gpu solution and everything is so smooth and slick, not to mention cheaper.", "votes": null }, { "id": "411941", "postDate": "10/29/2018 09:03:46", "content": "How did the issue Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 1.0 globally got solved. I am getting the same error. Please help, google help is not of much help.\nRegards,\nAnirban
", "rawMarkdown": "How did the issue Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 1.0 globally got solved. I am getting the same error. Please help, google help is not of much help.\nRegards,\nAnirban", "votes": null }, { "id": "413178", "postDate": "10/31/2018 12:23:19", "content": "I got my issue resolved. Felt would share with others. In my case I had requested for 4 gpus. However , google only alloted one gpu to me. If you are getting error like GPUS_ALL_REGIONS' exceeded. Limit: 1.0 please change the number of gpus to 1 and see if it works or not. If it works get back with google cloud support with a screenshot and they would resolve the issue. In my case prior to i changed my gpu to 1 instead of the requested 4 , the google cloud support kept saying my service was normal and there is no need for tech support. After I did the above , I found them accepting this being a tech problem and would get it resolved.
\n\nHope this helps anyone in my situation.
", "rawMarkdown": "I got my issue resolved. Felt would share with others. In my case I had requested for 4 gpus. However , google only alloted one gpu to me. If you are getting error like GPUS_ALL_REGIONS' exceeded. Limit: 1.0 please change the number of gpus to 1 and see if it works or not. If it works get back with google cloud support with a screenshot and they would resolve the issue. In my case prior to i changed my gpu to 1 instead of the requested 4 , the google cloud support kept saying my service was normal and there is no need for tech support. After I did the above , I found them accepting this being a tech problem and would get it resolved.\n\nHope this helps anyone in my situation.", "votes": null }, { "id": "439614", "postDate": "12/15/2018 23:54:30", "content": "Initially, I had the same problem with reading csv files. \nthen the problem was solved after I used another approach below to download the Kaggle dataset (although it was meant for google colab, worked in this scenario too).
\n\n", "rawMarkdown": "Initially, I had the same problem with reading csv files. \nthen the problem was solved after I used another approach below to download the Kaggle dataset (although it was meant for google colab, worked in this scenario too). \n\nhttps://stackoverflow.com/questions/49310470/using-kaggle-datasets-into-google-colab/53574984#53574984", "votes": null } ], "comments": [ { "id": 387889, "author_name": "moshel", "author_url": "", "post_date": "09/15/2018 21:42:21", "content": "here is my quick setup for gcp instance, the whole process takes about 15 min:
\n\nsudo add-apt-repository ppa:graphics-drivers/ppa -y && sudo apt update && sudo apt-get install -y nvidia-396 nvidia- \nmodprobe\nsudo reboot\nnvidia-smi\n\nwget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.0.176-1_amd64.deb\nwget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb\nwget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libcudnn7-dev_7.0.5.15-1+cuda9.0_amd64.deb\nwget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libnccl2_2.1.4-1+cuda9.0_amd64.deb\nwget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libnccl-dev_2.1.4-1+cuda9.0_amd64.deb\nsudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub\n\nsudo dpkg -i cuda-repo-ubuntu1604_9.0.176-1_amd64.deb\nsudo dpkg -i libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb\nsudo dpkg -i libcudnn7-dev_7.0.5.15-1+cuda9.0_amd64.deb\nsudo dpkg -i libnccl2_2.1.4-1+cuda9.0_amd64.deb\nsudo dpkg -i libnccl-dev_2.1.4-1+cuda9.0_amd64.deb\n\nsudo apt-get update\nsudo apt-get install -y --allow-unauthenticated cuda=9.0.176-1\nsudo apt-get install -y libcudnn7-dev\nsudo apt-get install -y libnccl-dev\nreboot\n\n\nsudo apt install -y python-pip python3-pip\n\npip install kaggle\npip install cython\npip install numpy\npip install tensorflow-gpu\npip install matplotlib\npip install pillow\n\nsudo apt install -y libopencv-dev\npip install opencv-python \nOr use a preconfigured image, per this blog post, e.g. (assuming you have gcloud installed):
\n\n export IMAGE_FAMILY=\"tf-latest-cu92\"\n export ZONE=\"us-east1-b\"\n export INSTANCE_NAME=\"gpu-instance\"\n export INSTANCE_TYPE=\"n1-highmem-4\"\n gcloud compute instances create $INSTANCE_NAME \\\n --zone=$ZONE \\\n --image-family=$IMAGE_FAMILY \\\n --image-project=deeplearning-platform-release \\\n --maintenance-policy=TERMINATE \\\n --accelerator='type=nvidia-tesla-p100,count=1' \\\n --machine-type=$INSTANCE_TYPE \\\n --boot-disk-size=240GB \\\n --metadata='install-nvidia-driver=True' \\\n --preemptible\n(I opted for a single preemptible p100. If you want more than one GPU, you probably do need to request a quota increase.) Then you don't have to worry about CUDA and Tensorflow versions, driver compatibility, and such. Though some things (like the Kaggle API) you might still have to install yourself.
", "votes": null, "replies": [ { "id": 388234, "author_name": "arpandhatt", "author_url": "", "post_date": "09/16/2018 15:53:09", "content": "Yes; I think this is the most straightforward solution. It even has JupyterLab running already. You can just run the above code in the cloud console without Gcloud Cli. However, I do have a problem using virtual environments with JupyterLab. Does anyone have any pointers for this?
", "votes": null, "replies": [] }, { "id": 408273, "author_name": "taindow", "author_url": "", "post_date": "10/22/2018 15:26:14", "content": "Thanks a lot for the link and setup example, it was really helpful. I decided to move from AWS to GCP as my cloud gpu solution and everything is so smooth and slick, not to mention cheaper.
", "votes": null, "replies": [] } ] }, { "id": 388290, "author_name": "flagshipdynamics", "author_url": "", "post_date": "09/16/2018 17:56:53", "content": "I use GCP as well. I started with the tensorflow deep learning GCP image with ubuntu. However, I HATE not having a GUI. So I installed xfce and VNCserver and some xrdp server. I RDP to the instance, which brings me to the VNC server where I login. Right now I'm the only one accessing it, but this should allow me to run this instance with others in the future if I choose.
\n\nI then installed anaconda navigator and I use that to launch jupyter notebooks and whatever else. I use the notebooks through the RDP session as opposed to through the web. Once you've built your instance out the way you like, just image it. Then you can delete everything but the image when you're done for the day, saving a ton of money and giving you the opportunity to re-launch the same image on more (or less) robust hardware, as the task demands.
\n\nFor example, I use the same image in GCP but sometimes I run it on a 16cpu P100 box and other times I'll just run something light like a K80. Obviously this flexibility results in significant cost savings. I've also found GCP far easier to deal with than AWS, not to mention far less expensive.
", "votes": null, "replies": [] }, { "id": 397004, "author_name": "", "author_url": "", "post_date": "10/01/2018 18:37:54", "content": "Hi, I would like to seek an advice from you guys! Given that I already requested to increase my qouta and it was already approved and reflected on my qouta list ( increase my qouta limit from 1 to 2 in preemptible P100), why I was always encountered this issue \"Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 1.0 globally\".. when I'm creating my VM instance with 2 preemptible P100? Do you already encountered this same issue? If yes, how did you address it? Thank you! :-).. I found that this competition requires a lot of computational power! :-(
", "votes": null, "replies": [ { "id": 397016, "author_name": "moshel", "author_url": "", "post_date": "10/01/2018 19:00:20", "content": "Note that preemptible gpus can only be attached to preemptive instances as far as I know, so you need to set the preemptive attribute on the instance
", "votes": null, "replies": [] }, { "id": 408107, "author_name": "rituraj17", "author_url": "", "post_date": "10/22/2018 10:25:48", "content": "Was this Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 1.0 globally issue solved?
", "votes": null, "replies": [] }, { "id": 411941, "author_name": "ag12345", "author_url": "", "post_date": "10/29/2018 09:03:46", "content": "How did the issue Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 1.0 globally got solved. I am getting the same error. Please help, google help is not of much help.\nRegards,\nAnirban
", "votes": null, "replies": [] }, { "id": 413178, "author_name": "ag12345", "author_url": "", "post_date": "10/31/2018 12:23:19", "content": "I got my issue resolved. Felt would share with others. In my case I had requested for 4 gpus. However , google only alloted one gpu to me. If you are getting error like GPUS_ALL_REGIONS' exceeded. Limit: 1.0 please change the number of gpus to 1 and see if it works or not. If it works get back with google cloud support with a screenshot and they would resolve the issue. In my case prior to i changed my gpu to 1 instead of the requested 4 , the google cloud support kept saying my service was normal and there is no need for tech support. After I did the above , I found them accepting this being a tech problem and would get it resolved.
\n\nHope this helps anyone in my situation.
", "votes": null, "replies": [] } ] }, { "id": 439614, "author_name": "avocano2018", "author_url": "", "post_date": "12/15/2018 23:54:30", "content": "Initially, I had the same problem with reading csv files. \nthen the problem was solved after I used another approach below to download the Kaggle dataset (although it was meant for google colab, worked in this scenario too).
\n\n", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "387868": "The first 7 minutes of this video explains a very simple method to set up a GPU instance on Google Cloud using Jetware pre-made instances. export IMAGE_FAMILY=\"tf-latest-cu92\"\n export ZONE=\"us-east1-b\"\n export INSTANCE_NAME=\"gpu-instance\"\n export INSTANCE_TYPE=\"n1-highmem-4\"\n gcloud compute instances create $INSTANCE_NAME \\\n --zone=$ZONE \\\n --image-family=$IMAGE_FAMILY \\\n --image-project=deeplearning-platform-release \\\n --maintenance-policy=TERMINATE \\\n --accelerator='type=nvidia-tesla-p100,count=1' \\\n --machine-type=$INSTANCE_TYPE \\\n --boot-disk-size=240GB \\\n --metadata='install-nvidia-driver=True' \\\n --preemptible\n