Is anyone using GCP to train their models? With the limited amount of Kaggle GPU, training externally is the way to go. Colab is really good, but has the 12 hour runtime constraint, and the GPU keeps changing every runtime so often gets slower the more you use it. I wanna use GCP to train, but is it feasible? If you're using it, please let me know, and any tips/instructions on how to setup everything for training would be really helpful.
", "messages": [ { "id": "743734", "postDate": "02/12/2020 08:43:28", "content": "Is anyone using GCP to train their models? With the limited amount of Kaggle GPU, training externally is the way to go. Colab is really good, but has the 12 hour runtime constraint, and the GPU keeps changing every runtime so often gets slower the more you use it. I wanna use GCP to train, but is it feasible? If you're using it, please let me know, and any tips/instructions on how to setup everything for training would be really helpful.
", "rawMarkdown": "Is anyone using GCP to train their models? With the limited amount of Kaggle GPU, training externally is the way to go. Colab is really good, but has the 12 hour runtime constraint, and the GPU keeps changing every runtime so often gets slower the more you use it. I wanna use GCP to train, but is it feasible? If you're using it, please let me know, and any tips/instructions on how to setup everything for training would be really helpful.", "votes": null }, { "id": "743980", "postDate": "02/12/2020 13:04:28", "content": "I don’t have that much experience with GCP, but I hope this might help:\n- You get $300 dollar when signing up, that’s a pretty good amount to get started.\n- I’d suggest getting a VM with Ubuntu, and setting up Miniconda to get Tensorflow 2 (with you’re using PyTorch, I’m guessing it’s the same) because all those Cuda and Cudnn installs will be much simpler.\n- While you do your installs and send your data, do it with the smallest CPU / RAM config possible, to save money. Only once you’ve got everything up and ready, put a bigger RAM + GPU on the instance. Just like Kaggle, optimize your GPU time. On GCP it’s not 30h you’re limited to, but your money.\n- You can also hook up Jupyter notebooks to your instance. There are plenty of tutorials on how to do that. This can be a nice option too.
\n\nHope this helps!
", "rawMarkdown": "I don’t have _that_ much experience with GCP, but I hope this might help:\n- You get $300 dollar when signing up, that’s a pretty good amount to get started.\n- I’d suggest getting a VM with Ubuntu, and setting up Miniconda to get Tensorflow 2 (with you’re using PyTorch, I’m guessing it’s the same) because all those Cuda and Cudnn installs will be much simpler.\n- While you do your installs and send your data, do it with the smallest CPU / RAM config possible, to save money. Only once you’ve got everything up and ready, put a bigger RAM + GPU on the instance. Just like Kaggle, optimize your GPU time. On GCP it’s not 30h you’re limited to, but your money.\n- You can also hook up Jupyter notebooks to your instance. There are plenty of tutorials on how to do that. This can be a nice option too.\n\nHope this helps!", "votes": null }, { "id": "744013", "postDate": "02/12/2020 13:29:04", "content": "Yeah i did sign-up and set up a machine (installed the Deep Learning with Pytorch package) which they already had. However, the issue I'm having right now is I'm not able to connect to my remote instance using the IP. Do you have any idea how to fix that?\nAlso a question -\nOnce I run my notebook on the instance, can i turn off my PC and will the instance keep running my notebook?\nThanks for replying :)
", "rawMarkdown": "Yeah i did sign-up and set up a machine (installed the Deep Learning with Pytorch package) which they already had. However, the issue I'm having right now is I'm not able to connect to my remote instance using the IP. Do you have any idea how to fix that?\nAlso a question -\nOnce I run my notebook on the instance, can i turn off my PC and will the instance keep running my notebook?\nThanks for replying :)", "votes": null }, { "id": "744053", "postDate": "02/12/2020 14:15:25", "content": "I think the easiest is to set up a ssh connection, that way from a terminal you can simply connect. I don't think I've ever tried through an IP connection, so I couldn't tell. (Unless it's just me and they actually just are the same thing haha).
\n\nAnd no, you have to manually shut down your instance! Your local machine and the instance are 2 distinct entities, that have nothing to do with each other apart from communicating. If you don't close the instance, Google has no way of knowing that you want to close your instance.
\n\nBy the way, look into tmux, it's great for not loosing connection and interrupting the connection when your internet connection suddenly restarts, or your laptop dies. It's a go to-tool IMHO for working fluidly and frustration-free on GCP.
", "rawMarkdown": "I think the easiest is to set up a ssh connection, that way from a terminal you can simply connect. I don't think I've ever tried through an IP connection, so I couldn't tell. (Unless it's just me and they actually just are the same thing haha).\n\nAnd no, you have to manually shut down your instance! Your local machine and the instance are 2 distinct entities, that have nothing to do with each other apart from communicating. If you don't close the instance, Google has no way of knowing that you want to close your instance. \n\nBy the way, look into tmux, it's great for not loosing connection and interrupting the connection when your internet connection suddenly restarts, or your laptop dies. It's a go to-tool IMHO for working fluidly and frustration-free on GCP.", "votes": null }, { "id": "744168", "postDate": "02/12/2020 16:26:48", "content": "Thank you. I will try it out!
", "rawMarkdown": "Thank you. I will try it out!", "votes": null } ], "comments": [ { "id": 743980, "author_name": "maxlenormand", "author_url": "", "post_date": "02/12/2020 13:04:28", "content": "I don’t have that much experience with GCP, but I hope this might help:\n- You get $300 dollar when signing up, that’s a pretty good amount to get started.\n- I’d suggest getting a VM with Ubuntu, and setting up Miniconda to get Tensorflow 2 (with you’re using PyTorch, I’m guessing it’s the same) because all those Cuda and Cudnn installs will be much simpler.\n- While you do your installs and send your data, do it with the smallest CPU / RAM config possible, to save money. Only once you’ve got everything up and ready, put a bigger RAM + GPU on the instance. Just like Kaggle, optimize your GPU time. On GCP it’s not 30h you’re limited to, but your money.\n- You can also hook up Jupyter notebooks to your instance. There are plenty of tutorials on how to do that. This can be a nice option too.
\n\nHope this helps!
", "votes": null, "replies": [ { "id": 744013, "author_name": "p4rallax", "author_url": "", "post_date": "02/12/2020 13:29:04", "content": "Yeah i did sign-up and set up a machine (installed the Deep Learning with Pytorch package) which they already had. However, the issue I'm having right now is I'm not able to connect to my remote instance using the IP. Do you have any idea how to fix that?\nAlso a question -\nOnce I run my notebook on the instance, can i turn off my PC and will the instance keep running my notebook?\nThanks for replying :)
", "votes": null, "replies": [] }, { "id": 744053, "author_name": "maxlenormand", "author_url": "", "post_date": "02/12/2020 14:15:25", "content": "I think the easiest is to set up a ssh connection, that way from a terminal you can simply connect. I don't think I've ever tried through an IP connection, so I couldn't tell. (Unless it's just me and they actually just are the same thing haha).
\n\nAnd no, you have to manually shut down your instance! Your local machine and the instance are 2 distinct entities, that have nothing to do with each other apart from communicating. If you don't close the instance, Google has no way of knowing that you want to close your instance.
\n\nBy the way, look into tmux, it's great for not loosing connection and interrupting the connection when your internet connection suddenly restarts, or your laptop dies. It's a go to-tool IMHO for working fluidly and frustration-free on GCP.
", "votes": null, "replies": [] }, { "id": 744168, "author_name": "sohier", "author_url": "", "post_date": "02/12/2020 16:26:48", "content": "Thank you. I will try it out!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "743734": "Is anyone using GCP to train their models? With the limited amount of Kaggle GPU, training externally is the way to go. Colab is really good, but has the 12 hour runtime constraint, and the GPU keeps changing every runtime so often gets slower the more you use it. I wanna use GCP to train, but is it feasible? If you're using it, please let me know, and any tips/instructions on how to setup everything for training would be really helpful.", "743980": "I don’t have _that_ much experience with GCP, but I hope this might help:\n- You get $300 dollar when signing up, that’s a pretty good amount to get started.\n- I’d suggest getting a VM with Ubuntu, and setting up Miniconda to get Tensorflow 2 (with you’re using PyTorch, I’m guessing it’s the same) because all those Cuda and Cudnn installs will be much simpler.\n- While you do your installs and send your data, do it with the smallest CPU / RAM config possible, to save money. Only once you’ve got everything up and ready, put a bigger RAM + GPU on the instance. Just like Kaggle, optimize your GPU time. On GCP it’s not 30h you’re limited to, but your money.\n- You can also hook up Jupyter notebooks to your instance. There are plenty of tutorials on how to do that. This can be a nice option too.\n\nHope this helps!", "744013": "Yeah i did sign-up and set up a machine (installed the Deep Learning with Pytorch package) which they already had. However, the issue I'm having right now is I'm not able to connect to my remote instance using the IP. Do you have any idea how to fix that?\nAlso a question -\nOnce I run my notebook on the instance, can i turn off my PC and will the instance keep running my notebook?\nThanks for replying :)", "744053": "I think the easiest is to set up a ssh connection, that way from a terminal you can simply connect. I don't think I've ever tried through an IP connection, so I couldn't tell. (Unless it's just me and they actually just are the same thing haha).\n\nAnd no, you have to manually shut down your instance! Your local machine and the instance are 2 distinct entities, that have nothing to do with each other apart from communicating. If you don't close the instance, Google has no way of knowing that you want to close your instance. \n\nBy the way, look into tmux, it's great for not loosing connection and interrupting the connection when your internet connection suddenly restarts, or your laptop dies. It's a go to-tool IMHO for working fluidly and frustration-free on GCP.", "744168": "[gsutil ssh](https://cloud.google.com/sdk/gcloud/reference/compute/ssh) handles the ssh setup for you; I would use that with GCP. \n- I'll second Maxime; tmux is invaluable. Just remember that your VM will stay on until you turn it off! When I need to conserve, I run my scripts from bash and append a \"&& sudo poweroff\" call.", "744206": "Thank you. I will try it out!" }, "source": "meta" }