I'm working with my own dataset in a gcp storage. The problem is that i donth have the storage in the same zone as the notebook is. So they charge me a lot more of expected when i use tpus because the tpus call the dataset a lot of times.
\n\nThis is the list of prices from GCS https://cloud.google.com/storage/pricing
", "messages": [ { "id": "758176", "postDate": "02/27/2020 14:18:27", "content": "I'm working with my own dataset in a gcp storage. The problem is that i donth have the storage in the same zone as the notebook is. So they charge me a lot more of expected when i use tpus because the tpus call the dataset a lot of times.
\n\nThis is the list of prices from GCS https://cloud.google.com/storage/pricing
", "rawMarkdown": "I'm working with my own dataset in a gcp storage. The problem is that i donth have the storage in the same zone as the notebook is. So they charge me a lot more of expected when i use tpus because the tpus call the dataset a lot of times.\n\nThis is the list of prices from GCS https://cloud.google.com/storage/pricing", "votes": null }, { "id": "758300", "postDate": "02/27/2020 16:13:01", "content": "I think that's because current TPU's can only be used with GCS. For this dataset already it's hosted in GCS as a TF Record.
\n\nAlso Kaggle notebooks are usually a docker instance to best of my knowledge
", "rawMarkdown": "I think that's because current TPU's can only be used with GCS. For this dataset already it's hosted in GCS as a TF Record.\n\nAlso Kaggle notebooks are usually a docker instance to best of my knowledge", "votes": null }, { "id": "758328", "postDate": "02/27/2020 16:35:16", "content": "Yes, i know that. I created my own TFRecord dataset and store in my own GCP Storage. After that im using Kaggle TPUs and works fine.\nThe problem is that I am suffering charges for the transmission of storage data to kaggle tpus for being in different areas.
", "rawMarkdown": "Yes, i know that. I created my own TFRecord dataset and store in my own GCP Storage. After that im using Kaggle TPUs and works fine.\nThe problem is that I am suffering charges for the transmission of storage data to kaggle tpus for being in different areas.", "votes": null }, { "id": "758348", "postDate": "02/27/2020 16:54:46", "content": "Hi, please can you explain how you have used your own GCP storage with Kaggle TPUs, I am struggling to get it to work.
", "rawMarkdown": "Hi, please can you explain how you have used your own GCP storage with Kaggle TPUs, I am struggling to get it to work.", "votes": null }, { "id": "758583", "postDate": "02/28/2020 00:16:36", "content": "Not a direct answer but the info might be useful:
\n\nKaggle notebook VMs can be anywhere in the world. When you request GCS access yo a Kaggle dataset using KaggleDatasets().get_gcs_path(), the dataset is copied to a GCS bucket in the same region as your VM and TPU for optimal data access speed.
Yes. I had to pass the kaggle dataset to TFRecord in my own gcs bucket. I use something similar to this code https://github.com/Ian-O-W/Computer-Vision/blob/master/Pass%20dataset%20to%20gcs%20in%20tfRecords%20format%20for%20TPU%20use.ipynb \nAfter that i can connect kaggle with my GCP project with Google credentials variable enviroment to give kaggle notebook access to my GCS bucket.
\n\nAt the end of the code you can see how to execute tpus model in kaggle with this format.
", "rawMarkdown": "Yes. I had to pass the kaggle dataset to TFRecord in my own gcs bucket. I use something similar to this code https://github.com/Ian-O-W/Computer-Vision/blob/master/Pass%20dataset%20to%20gcs%20in%20tfRecords%20format%20for%20TPU%20use.ipynb \nAfter that i can connect kaggle with my GCP project with Google credentials variable enviroment to give kaggle notebook access to my GCS bucket.\n\nAt the end of the code you can see how to execute tpus model in kaggle with this format.", "votes": null }, { "id": "763612", "postDate": "03/04/2020 16:58:12", "content": "Okey. Then is that copy wich generate me charges from transfer between regions.\nThanks!
", "rawMarkdown": "Okey. Then is that copy wich generate me charges from transfer between regions.\nThanks!", "votes": null } ], "comments": [ { "id": 758300, "author_name": "kurianbenoy", "author_url": "", "post_date": "02/27/2020 16:13:01", "content": "I think that's because current TPU's can only be used with GCS. For this dataset already it's hosted in GCS as a TF Record.
\n\nAlso Kaggle notebooks are usually a docker instance to best of my knowledge
", "votes": null, "replies": [ { "id": 758328, "author_name": "ianwilkinson", "author_url": "", "post_date": "02/27/2020 16:35:16", "content": "Yes, i know that. I created my own TFRecord dataset and store in my own GCP Storage. After that im using Kaggle TPUs and works fine.\nThe problem is that I am suffering charges for the transmission of storage data to kaggle tpus for being in different areas.
", "votes": null, "replies": [] }, { "id": 758348, "author_name": "ghostskipper", "author_url": "", "post_date": "02/27/2020 16:54:46", "content": "Hi, please can you explain how you have used your own GCP storage with Kaggle TPUs, I am struggling to get it to work.
", "votes": null, "replies": [] }, { "id": 763606, "author_name": "ianwilkinson", "author_url": "", "post_date": "03/04/2020 16:55:02", "content": "Yes. I had to pass the kaggle dataset to TFRecord in my own gcs bucket. I use something similar to this code https://github.com/Ian-O-W/Computer-Vision/blob/master/Pass%20dataset%20to%20gcs%20in%20tfRecords%20format%20for%20TPU%20use.ipynb \nAfter that i can connect kaggle with my GCP project with Google credentials variable enviroment to give kaggle notebook access to my GCS bucket.
\n\nAt the end of the code you can see how to execute tpus model in kaggle with this format.
", "votes": null, "replies": [] } ] }, { "id": 758583, "author_name": "mgorner", "author_url": "", "post_date": "02/28/2020 00:16:36", "content": "Not a direct answer but the info might be useful:
\n\nKaggle notebook VMs can be anywhere in the world. When you request GCS access yo a Kaggle dataset using KaggleDatasets().get_gcs_path(), the dataset is copied to a GCS bucket in the same region as your VM and TPU for optimal data access speed.
Okey. Then is that copy wich generate me charges from transfer between regions.\nThanks!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "758176": "I'm working with my own dataset in a gcp storage. The problem is that i donth have the storage in the same zone as the notebook is. So they charge me a lot more of expected when i use tpus because the tpus call the dataset a lot of times.\n\nThis is the list of prices from GCS https://cloud.google.com/storage/pricing", "758300": "I think that's because current TPU's can only be used with GCS. For this dataset already it's hosted in GCS as a TF Record.\n\nAlso Kaggle notebooks are usually a docker instance to best of my knowledge", "758328": "Yes, i know that. I created my own TFRecord dataset and store in my own GCP Storage. After that im using Kaggle TPUs and works fine.\nThe problem is that I am suffering charges for the transmission of storage data to kaggle tpus for being in different areas.", "758348": "Hi, please can you explain how you have used your own GCP storage with Kaggle TPUs, I am struggling to get it to work.", "758583": "Not a direct answer but the info might be useful:\n\nKaggle notebook VMs can be anywhere in the world. When you request GCS access yo a Kaggle dataset using `KaggleDatasets().get_gcs_path()`, the dataset is copied to a GCS bucket in the same region as your VM and TPU for optimal data access speed.", "763606": "Yes. I had to pass the kaggle dataset to TFRecord in my own gcs bucket. I use something similar to this code https://github.com/Ian-O-W/Computer-Vision/blob/master/Pass%20dataset%20to%20gcs%20in%20tfRecords%20format%20for%20TPU%20use.ipynb \nAfter that i can connect kaggle with my GCP project with Google credentials variable enviroment to give kaggle notebook access to my GCS bucket.\n\nAt the end of the code you can see how to execute tpus model in kaggle with this format.", "763612": "Okey. Then is that copy wich generate me charges from transfer between regions.\nThanks!" }, "source": "meta" }