![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F761268%2F27146bf7826af0450ace2751415732e9%2FScreenshot%202020-10-19%20at%202.27.13%20PM.png?generation=1603098187407696&alt=media\")
Which GPU is best choice? Because we need to train for 1 to 3 days or more based on training data of our choice
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F761268%2F153229d04368b722db2826e9190947f6%2FScreenshot%202020-10-19%20at%202.27.40%20PM.png?generation=1603098039177569&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F761268%2Fb5d1c2fd78309a186290328c731196ef%2FScreenshot%202020-10-19%20at%202.27.23%20PM.png?generation=1603098016973319&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F761268%2Face3f0c8cae4d263467b7b7158f473b1%2FScreenshot%202020-10-19%20at%202.29.10%20PM.png?generation=1603098061943123&alt=media\")
Google Cloud Price Calculator
\nhttps://cloud.google.com/products/calculator
\nGoogle GPU Pricing
\nhttps://cloud.google.com/compute/gpus-pricing
Thanks to Kaggle team and Lyft team to supporting with $300 for handling these experiments
", "messages": [ { "id": "1053438", "postDate": "10/19/2020 02:25:44", "content": "Which GPU is best choice? Because we need to train for 1 to 3 days or more based on training data of our choice
\n
Google Cloud Price Calculator
\nhttps://cloud.google.com/products/calculator
\nGoogle GPU Pricing
\nhttps://cloud.google.com/compute/gpus-pricing
Thanks to Kaggle team and Lyft team to supporting with $300 for handling these experiments
", "rawMarkdown": "Which GPU is best choice? Because we need to train for 1 to 3 days or more based on training data of our choice\n\n\n\n\n\n\n\n\n\n**Google Cloud Price Calculator**\nhttps://cloud.google.com/products/calculator\n**Google GPU Pricing**\nhttps://cloud.google.com/compute/gpus-pricing\n\nThanks to Kaggle team and Lyft team to supporting with $300 for handling these experiments", "votes": null }, { "id": "1054153", "postDate": "10/19/2020 18:13:14", "content": "Did you manage to Increase GPU quota?? Do you mind sharing which zone works good for GPU
\nThanks!
You can play with Zones @ https://cloud.google.com/compute/gpus-pricing but default zone i feel is best option already. so need to select cpu + gpu vm
", "rawMarkdown": "You can play with Zones @ https://cloud.google.com/compute/gpus-pricing but default zone i feel is best option already. so need to select cpu + gpu vm", "votes": null }, { "id": "1054184", "postDate": "10/19/2020 18:51:13", "content": "Thanks! I got confused between gcloud's AI notebooks and VM.
", "rawMarkdown": "Thanks! I got confused between gcloud's AI notebooks and VM.", "votes": null }, { "id": "1054275", "postDate": "10/19/2020 19:40:46", "content": "For the rasterizer, it's mostly the CPU that is the bottleneck. So, I would choose a cheaper GPU and check the load on both GPU (bash: nvidia-smi) and CPU (bash: htop).
Try to get both loads as high as possible to get the maximum out of the credits.
\nAlso, there are more GPU types such as V100 and P100. Not all regions have GPUs and some GPUs are region specific.
", "rawMarkdown": "For the rasterizer, it's mostly the CPU that is the bottleneck. So, I would choose a cheaper GPU and check the load on both GPU (bash: `nvidia-smi`) and CPU (bash: `htop`).\n\nTry to get both loads as high as possible to get the maximum out of the credits.\n\nAlso, there are more GPU types such as V100 and P100. Not all regions have GPUs and some GPUs are region specific.", "votes": null }, { "id": "1054425", "postDate": "10/19/2020 23:24:44", "content": "i am using V100, it's faster than P100 and K80, with T4 i don't know but i think V100 is best option
", "rawMarkdown": "i am using V100, it's faster than P100 and K80, with T4 i don't know but i think V100 is best option", "votes": null }, { "id": "1054487", "postDate": "10/20/2020 01:38:27", "content": "what combination of cpu you are using. how long it taking to train train.zarr
", "rawMarkdown": "what combination of cpu you are using. how long it taking to train train.zarr", "votes": null }, { "id": "1054924", "postDate": "10/20/2020 09:58:10", "content": "one EPOCH???
", "rawMarkdown": "one EPOCH???", "votes": null }, { "id": "1054936", "postDate": "10/20/2020 10:15:56", "content": "i mean for to train full train.zarr :)
", "rawMarkdown": "i mean for to train full train.zarr :)", "votes": null }, { "id": "1055060", "postDate": "10/20/2020 12:55:29", "content": "When I tried to upgrade to Google Cloud AI Platform Notebook, I always stuck at \"Opening notebook with JupyterLab\". Any possible reasons? Thanks!
", "rawMarkdown": "When I tried to upgrade to Google Cloud AI Platform Notebook, I always stuck at \"Opening notebook with JupyterLab\". Any possible reasons? Thanks!", "votes": null }, { "id": "1055068", "postDate": "10/20/2020 13:04:17", "content": "If you are using GPU you might wanna check your Global GPU quota, If your quota limit is 0 you'll have to request for a increase in quota
\n![\"You](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5964694%2Ff167681de4668c8c3fe795df18946043%2FScreenshot%202020-10-20%20183607.png?generation=1603199202988736&alt=media\")
\nwhich probably will be declined if you are using a Free tier account, you will have to upgrade to billing if you want to deploy GPU
That was my point ;). I guess nobody is training for a full epoch of train_full.zarr. It would take weeks on a decent computer.
", "rawMarkdown": "That was my point ;). I guess nobody is training for a full epoch of train_full.zarr. It would take weeks on a decent computer.", "votes": null }, { "id": "1055099", "postDate": "10/20/2020 13:31:02", "content": "On my quota page, I have all the items above except NVIDA GPUs. Why is that?
\nI have upgraded billing account, which is required if you want to redeem the $300 credits.
", "rawMarkdown": "On my quota page, I have all the items above except NVIDA GPUs. Why is that?\n\nI have upgraded billing account, which is required if you want to redeem the $300 credits.", "votes": null }, { "id": "1055107", "postDate": "10/20/2020 13:38:15", "content": "On the filter table search for 'gpu' ![\"as](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5964694%2F9947126bbe88dd3aa1ffaa44a23b7bf5%2FScreenshot%202020-10-20%20190721.png?generation=1603201081971498&alt=media\")
Thanks. You are right. My quota for any kind of NVIDA GPUs is all zero.
", "rawMarkdown": "Thanks. You are right. My quota for any kind of NVIDA GPUs is all zero.", "votes": null }, { "id": "1055195", "postDate": "10/20/2020 15:07:02", "content": "Some people aren't even training one epoch on train.zarr ;)
\nAnd I use the T4 on GC. Speed limitations will mostly come from the number of cores on GC.
:)) i am use Google COlab pro and use v100 to train. Have 4 core cpu. Training with high RAM is quickly than Kaggle x2-3 times.
", "rawMarkdown": ":)) i am use Google COlab pro and use v100 to train. Have 4 core cpu. Training with high RAM is quickly than Kaggle x2-3 times.", "votes": null }, { "id": "1056421", "postDate": "10/21/2020 17:25:19", "content": "Perhaps Google could open up the TPU Research Cloud to the contest? https://www.tensorflow.org/tfrc It's definitely not easy to develop against the TPU (you pretty much need tfrecords for best performance, and XLA compile errors can take 10-15 minutes to get back to you sometimes) but it makes your gcloud credits go farther.
", "rawMarkdown": "Perhaps Google could open up the TPU Research Cloud to the contest? https://www.tensorflow.org/tfrc It's definitely not easy to develop against the TPU (you pretty much need tfrecords for best performance, and XLA compile errors can take 10-15 minutes to get back to you sometimes) but it makes your gcloud credits go farther.", "votes": null }, { "id": "1061611", "postDate": "10/27/2020 06:21:13", "content": "Hi @madtitan0210 I am trying to get my \"Global GPU quota\" to be increased to 1. But each time I request, I get back an email saying - \"Unfortunately, we are unable to grant you additional quota at this time\". I have also upgraded to billing so that is not the problem. Request your help.
", "rawMarkdown": "Hi @madtitan0210 I am trying to get my \"Global GPU quota\" to be increased to 1. But each time I request, I get back an email saying - \"Unfortunately, we are unable to grant you additional quota at this time\". I have also upgraded to billing so that is not the problem. Request your help.", "votes": null }, { "id": "1061676", "postDate": "10/27/2020 07:45:19", "content": "Try to request in other zone @arpitrf
", "rawMarkdown": "Try to request in other zone @arpitrf", "votes": null }, { "id": "1061763", "postDate": "10/27/2020 10:00:22", "content": "Hi @arpitrf you might wanna check this thread on GCP here and also check the restrictions on GPU here - The restrictions section
\nWere you able to deploy CPU only notebook or VM??
", "rawMarkdown": "Hi @arpitrf you might wanna check this thread on GCP [here](https://stackoverflow.com/a/48292871) and also check the restrictions on GPU [here - The restrictions section](https://cloud.google.com/compute/docs/gpus/#restrictions) \n\nWere you able to deploy CPU only notebook or VM??", "votes": null }, { "id": "1061779", "postDate": "10/27/2020 10:15:20", "content": "Yes I am able to deploy CPU only notebooks
", "rawMarkdown": "Yes I am able to deploy CPU only notebooks", "votes": null }, { "id": "1062096", "postDate": "10/27/2020 15:09:26", "content": "Try to request quota increases in only Iowa (and +1 for global GPUs) all in one request. Yes the quota system is awful, especially the high number of quotas. I’ve talked directly with a PM at GCloud once about it and they won’t fix it :(
", "rawMarkdown": "Try to request quota increases in only Iowa (and +1 for global GPUs) all in one request. Yes the quota system is awful, especially the high number of quotas. I’ve talked directly with a PM at GCloud once about it and they won’t fix it :(", "votes": null } ], "comments": [ { "id": 1054153, "author_name": "madtitan0210", "author_url": "", "post_date": "10/19/2020 18:13:14", "content": "Did you manage to Increase GPU quota?? Do you mind sharing which zone works good for GPU
\nThanks!
You can play with Zones @ https://cloud.google.com/compute/gpus-pricing but default zone i feel is best option already. so need to select cpu + gpu vm
", "votes": null, "replies": [] }, { "id": 1054184, "author_name": "madtitan0210", "author_url": "", "post_date": "10/19/2020 18:51:13", "content": "Thanks! I got confused between gcloud's AI notebooks and VM.
", "votes": null, "replies": [] } ] }, { "id": 1054275, "author_name": "ilu000", "author_url": "", "post_date": "10/19/2020 19:40:46", "content": "For the rasterizer, it's mostly the CPU that is the bottleneck. So, I would choose a cheaper GPU and check the load on both GPU (bash: nvidia-smi) and CPU (bash: htop).
Try to get both loads as high as possible to get the maximum out of the credits.
\nAlso, there are more GPU types such as V100 and P100. Not all regions have GPUs and some GPUs are region specific.
", "votes": null, "replies": [] }, { "id": 1054425, "author_name": "doanquanvietnamca", "author_url": "", "post_date": "10/19/2020 23:24:44", "content": "i am using V100, it's faster than P100 and K80, with T4 i don't know but i think V100 is best option
", "votes": null, "replies": [ { "id": 1054487, "author_name": "seshurajup", "author_url": "", "post_date": "10/20/2020 01:38:27", "content": "what combination of cpu you are using. how long it taking to train train.zarr
", "votes": null, "replies": [] }, { "id": 1054924, "author_name": "ilu000", "author_url": "", "post_date": "10/20/2020 09:58:10", "content": "one EPOCH???
", "votes": null, "replies": [] }, { "id": 1054936, "author_name": "seshurajup", "author_url": "", "post_date": "10/20/2020 10:15:56", "content": "i mean for to train full train.zarr :)
", "votes": null, "replies": [] }, { "id": 1055089, "author_name": "ilu000", "author_url": "", "post_date": "10/20/2020 13:26:12", "content": "That was my point ;). I guess nobody is training for a full epoch of train_full.zarr. It would take weeks on a decent computer.
", "votes": null, "replies": [] }, { "id": 1055195, "author_name": "indswetrust", "author_url": "", "post_date": "10/20/2020 15:07:02", "content": "Some people aren't even training one epoch on train.zarr ;)
\nAnd I use the T4 on GC. Speed limitations will mostly come from the number of cores on GC.
:)) i am use Google COlab pro and use v100 to train. Have 4 core cpu. Training with high RAM is quickly than Kaggle x2-3 times.
", "votes": null, "replies": [] } ] }, { "id": 1055060, "author_name": "frankpanxj", "author_url": "", "post_date": "10/20/2020 12:55:29", "content": "When I tried to upgrade to Google Cloud AI Platform Notebook, I always stuck at \"Opening notebook with JupyterLab\". Any possible reasons? Thanks!
", "votes": null, "replies": [ { "id": 1055068, "author_name": "madtitan0210", "author_url": "", "post_date": "10/20/2020 13:04:17", "content": "If you are using GPU you might wanna check your Global GPU quota, If your quota limit is 0 you'll have to request for a increase in quota
\n
\nwhich probably will be declined if you are using a Free tier account, you will have to upgrade to billing if you want to deploy GPU
On my quota page, I have all the items above except NVIDA GPUs. Why is that?
\nI have upgraded billing account, which is required if you want to redeem the $300 credits.
", "votes": null, "replies": [] }, { "id": 1055107, "author_name": "madtitan0210", "author_url": "", "post_date": "10/20/2020 13:38:15", "content": "On the filter table search for 'gpu'
Thanks. You are right. My quota for any kind of NVIDA GPUs is all zero.
", "votes": null, "replies": [] }, { "id": 1061611, "author_name": "arpitrf", "author_url": "", "post_date": "10/27/2020 06:21:13", "content": "Hi @madtitan0210 I am trying to get my \"Global GPU quota\" to be increased to 1. But each time I request, I get back an email saying - \"Unfortunately, we are unable to grant you additional quota at this time\". I have also upgraded to billing so that is not the problem. Request your help.
", "votes": null, "replies": [] }, { "id": 1061676, "author_name": "seshurajup", "author_url": "", "post_date": "10/27/2020 07:45:19", "content": "Try to request in other zone @arpitrf
", "votes": null, "replies": [] }, { "id": 1061763, "author_name": "madtitan0210", "author_url": "", "post_date": "10/27/2020 10:00:22", "content": "Hi @arpitrf you might wanna check this thread on GCP here and also check the restrictions on GPU here - The restrictions section
\nWere you able to deploy CPU only notebook or VM??
", "votes": null, "replies": [] }, { "id": 1061779, "author_name": "arpitrf", "author_url": "", "post_date": "10/27/2020 10:15:20", "content": "Yes I am able to deploy CPU only notebooks
", "votes": null, "replies": [] }, { "id": 1062096, "author_name": "oarphme", "author_url": "", "post_date": "10/27/2020 15:09:26", "content": "Try to request quota increases in only Iowa (and +1 for global GPUs) all in one request. Yes the quota system is awful, especially the high number of quotas. I’ve talked directly with a PM at GCloud once about it and they won’t fix it :(
", "votes": null, "replies": [] } ] }, { "id": 1056421, "author_name": "oarphme", "author_url": "", "post_date": "10/21/2020 17:25:19", "content": "Perhaps Google could open up the TPU Research Cloud to the contest? https://www.tensorflow.org/tfrc It's definitely not easy to develop against the TPU (you pretty much need tfrecords for best performance, and XLA compile errors can take 10-15 minutes to get back to you sometimes) but it makes your gcloud credits go farther.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1053438": "Which GPU is best choice? Because we need to train for 1 to 3 days or more based on training data of our choice\n\n\n\n\n\n\n\n\n\n**Google Cloud Price Calculator**\nhttps://cloud.google.com/products/calculator\n**Google GPU Pricing**\nhttps://cloud.google.com/compute/gpus-pricing\n\nThanks to Kaggle team and Lyft team to supporting with $300 for handling these experiments", "1054153": "Did you manage to Increase GPU quota?? Do you mind sharing which zone works good for GPU\nThanks!", "1054166": "You can play with Zones @ https://cloud.google.com/compute/gpus-pricing but default zone i feel is best option already. so need to select cpu + gpu vm", "1054184": "Thanks! I got confused between gcloud's AI notebooks and VM.", "1054275": "For the rasterizer, it's mostly the CPU that is the bottleneck. So, I would choose a cheaper GPU and check the load on both GPU (bash: `nvidia-smi`) and CPU (bash: `htop`).\n\nTry to get both loads as high as possible to get the maximum out of the credits.\n\nAlso, there are more GPU types such as V100 and P100. Not all regions have GPUs and some GPUs are region specific.", "1054425": "i am using V100, it's faster than P100 and K80, with T4 i don't know but i think V100 is best option", "1054487": "what combination of cpu you are using. how long it taking to train train.zarr", "1054924": "one EPOCH???", "1054936": "i mean for to train full train.zarr :)", "1055060": "When I tried to upgrade to Google Cloud AI Platform Notebook, I always stuck at \"Opening notebook with JupyterLab\". Any possible reasons? Thanks!", "1055068": "If you are using GPU you might wanna check your Global GPU quota, If your quota limit is 0 you'll have to request for a increase in [quota](https://console.cloud.google.com/iam-admin/quotas)\n\nwhich probably will be declined if you are using a Free tier account, you will have to upgrade to billing if you want to deploy GPU", "1055089": "That was my point ;). I guess nobody is training for a full epoch of train_full.zarr. It would take weeks on a decent computer.", "1055099": "On my quota page, I have all the items above except NVIDA GPUs. Why is that?\n\nI have upgraded billing account, which is required if you want to redeem the $300 credits.", "1055107": "On the filter table search for 'gpu' ", "1055119": "Thanks. You are right. My quota for any kind of NVIDA GPUs is all zero.", "1055195": "Some people aren't even training one epoch on train.zarr ;) \nAnd I use the T4 on GC. Speed limitations will mostly come from the number of cores on GC.", "1055514": ":)) i am use Google COlab pro and use v100 to train. Have 4 core cpu. Training with high RAM is quickly than Kaggle x2-3 times.", "1056421": "Perhaps Google could open up the TPU Research Cloud to the contest? https://www.tensorflow.org/tfrc It's definitely not easy to develop against the TPU (you pretty much need tfrecords for best performance, and XLA compile errors can take 10-15 minutes to get back to you sometimes) but it makes your gcloud credits go farther.", "1061611": "Hi @madtitan0210 I am trying to get my \"Global GPU quota\" to be increased to 1. But each time I request, I get back an email saying - \"Unfortunately, we are unable to grant you additional quota at this time\". I have also upgraded to billing so that is not the problem. Request your help.", "1061676": "Try to request in other zone @arpitrf", "1061763": "Hi @arpitrf you might wanna check this thread on GCP [here](https://stackoverflow.com/a/48292871) and also check the restrictions on GPU [here - The restrictions section](https://cloud.google.com/compute/docs/gpus/#restrictions) \n\nWere you able to deploy CPU only notebook or VM??", "1061779": "Yes I am able to deploy CPU only notebooks", "1062096": "Try to request quota increases in only Iowa (and +1 for global GPUs) all in one request. Yes the quota system is awful, especially the high number of quotas. I’ve talked directly with a PM at GCloud once about it and they won’t fix it :(" }, "source": "meta" }