Hi! I'm trying to use in gcp jupyter notebooks (ai platform) the TPU api.\nI have my own data (actually is the NIH Chest X-ray dataset) with a lot of preprocess. I'm available to train a on gpus from the numpy arrays alocated in RAM.\nWhat transformations i should do to this arrays for be the inputs of my tpu model.
\n\nPS: My conexion with the tpu api is ok and i can create my tpu model.
\n\nThanks!
", "messages": [ { "id": "744210", "postDate": "02/12/2020 16:50:27", "content": "Hi! I'm trying to use in gcp jupyter notebooks (ai platform) the TPU api.\nI have my own data (actually is the NIH Chest X-ray dataset) with a lot of preprocess. I'm available to train a on gpus from the numpy arrays alocated in RAM.\nWhat transformations i should do to this arrays for be the inputs of my tpu model.
\n\nPS: My conexion with the tpu api is ok and i can create my tpu model.
\n\nThanks!
", "rawMarkdown": "Hi! I'm trying to use in gcp jupyter notebooks (ai platform) the TPU api.\nI have my own data (actually is the NIH Chest X-ray dataset) with a lot of preprocess. I'm available to train a on gpus from the numpy arrays alocated in RAM.\nWhat transformations i should do to this arrays for be the inputs of my tpu model.\n\nPS: My conexion with the tpu api is ok and i can create my tpu model.\n\nThanks!", "votes": null }, { "id": "744296", "postDate": "02/12/2020 17:47:43", "content": "Yes, you can use TPUs in AI Platform Notebooks. If you are on GCP already, your training data should go into a GCS (Google Cloud Storage) bucket located in the same region as your TPU (for optimal performance, avoid multi-region buckets).
\n\nTo spin up an AI Platform Jupyter VM + TPU pair, you gave to use gcloud on the command line since there isn't an option for that in the UI yet. I can share a script that does that: create-tpu-deep-learning-vm.sh.\ngcloud init # to select your project, account, zones, ...\nthen\n./create-tpu-deep-learning-vm.sh choose-a-name --tpu-type v3-8
For reading data in, numpy arrays in memory will work for TPUs but might not be the most performant option. If your data fits in memory, it might be OK. For out of memory datasets, us the tf.data.Dataset API. Tutorial here: TPU-speed data pipelines: tf.data.Dataset and TFRecords
Thank you Martin. I made the connexion with the tpu without gcloud command line, and its work. I just need to set the enviroment variable to the tpu node.
\n\nIn the other hand my dataset in arrays way fit in memory well, but when i tried pass it to the model the model didnt work.\nI tried to pass this arrays (i did a lot of preprocess in the data with the arrays) to tensorflow datasets with tensorflow.dataset.to_tensor, but still not working when i train the model (even passing this new dataset with the bacth option)
", "rawMarkdown": "Thank you Martin. I made the connexion with the tpu without gcloud command line, and its work. I just need to set the enviroment variable to the tpu node.\n\nIn the other hand my dataset in arrays way fit in memory well, but when i tried pass it to the model the model didnt work.\nI tried to pass this arrays (i did a lot of preprocess in the data with the arrays) to tensorflow datasets with tensorflow.dataset.to_tensor, but still not working when i train the model (even passing this new dataset with the bacth option)", "votes": null }, { "id": "745357", "postDate": "02/13/2020 18:31:52", "content": "If you just want to convert an in-memory array to a tf.data.Dataset, the correct function is Dataset.from_tensor_slices.
But passing a numpy array to model.fit() directly works, even on TPU.
", "rawMarkdown": "If you just want to convert an in-memory array to a tf.data.Dataset, the correct function is [`Dataset.from_tensor_slices`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#from_tensor_slices).\n\nBut passing a numpy array to model.fit() directly works, even on TPU.", "votes": null }, { "id": "745360", "postDate": "02/13/2020 18:34:10", "content": "In the script I sent earlier (create-tpu-deep-learning-vm.sh), you will find a way of setting the TPU_NAME environment variable sot that it persists across reboots. And yes of course you can do this by hand.
", "rawMarkdown": "In the script I sent earlier ([create-tpu-deep-learning-vm.sh](https://raw.githubusercontent.com/GoogleCloudPlatform/training-data-analyst/master/courses/fast-and-lean-data-science/create-tpu-deep-learning-vm.sh)), you will find a way of setting the TPU_NAME environment variable sot that it persists across reboots. And yes of course you can do this by hand.", "votes": null }, { "id": "745833", "postDate": "02/14/2020 09:02:48", "content": "Pass arrays didn't work on my case. Gave me error \"failed to serialize\".\nI tried the functions dataset.from_tensor_slices, but my data was larger than 2GB and y couldn't create the tensor.\nSo im trying work with tfrecords now. \nThanks!
", "rawMarkdown": "Pass arrays didn't work on my case. Gave me error \"failed to serialize\".\nI tried the functions dataset.from_tensor_slices, but my data was larger than 2GB and y couldn't create the tensor.\nSo im trying work with tfrecords now. \nThanks!", "votes": null }, { "id": "746178", "postDate": "02/14/2020 17:56:33", "content": "Not entirely surprised 2GB \"failed to serialize\" 😊
", "rawMarkdown": "Not entirely surprised 2GB \"failed to serialize\" 😊", "votes": null }, { "id": "801610", "postDate": "04/08/2020 16:18:40", "content": "Hi,\nI have a notebook I have only images. How can use them in TPU. Because in TPU I must use tf.data.Dataset like in this kernel
\n\nhttps://www.kaggle.com/hatemamine/flowertpuwin
\n\nMy code is like that
\n\nhttps://www.kaggle.com/furkankati/person-mask-u-net-model-tpu
\n\nI read data directly from images. I convert images to numpy arrays.
\n\nHow can I upload my files to google cloud storage and use them for tpu ?
", "rawMarkdown": "Hi,\nI have a notebook I have only images. How can use them in TPU. Because in TPU I must use tf.data.Dataset like in this kernel\n\n[https://www.kaggle.com/hatemamine/flowertpuwin](https://www.kaggle.com/hatemamine/flowertpuwin)\n\n\nMy code is like that \n\n[https://www.kaggle.com/furkankati/person-mask-u-net-model-tpu](https://www.kaggle.com/furkankati/person-mask-u-net-model-tpu)\n\nI read data directly from images. I convert images to numpy arrays.\n\nHow can I upload my files to google cloud storage and use them for tpu ?", "votes": null }, { "id": "920574", "postDate": "07/08/2020 17:20:07", "content": "Hey Martin,
\n\nIs there a way to use my own TPU on a Kaggle dataset directly? Would it be possible for me to provide my own personal TPU the GCS paths for the public dataset on Kaggle?
", "rawMarkdown": "Hey Martin, \n\nIs there a way to use my own TPU on a Kaggle dataset directly? Would it be possible for me to provide my own personal TPU the GCS paths for the public dataset on Kaggle?", "votes": null }, { "id": "932296", "postDate": "07/16/2020 23:04:48", "content": "Sure.
\nHere is the script to provision a TPU + Jupyter on GCP.
To access the data, run KaggleDatasets().get_gcs_path() on Kaggle once. It will return a GCS bucket name like 'gs://xyz12234567890'. That's where Kaggle stores the attached dataset. You can use this bucket on GCP as well. Just be careful: the bucket is a cache. It will be good for a few days and then be wiped out. If you want something more stable, copy to your own bucekt.
Yes, you can use TPUs in AI Platform Notebooks. If you are on GCP already, your training data should go into a GCS (Google Cloud Storage) bucket located in the same region as your TPU (for optimal performance, avoid multi-region buckets).
\n\nTo spin up an AI Platform Jupyter VM + TPU pair, you gave to use gcloud on the command line since there isn't an option for that in the UI yet. I can share a script that does that: create-tpu-deep-learning-vm.sh.\ngcloud init # to select your project, account, zones, ...\nthen\n./create-tpu-deep-learning-vm.sh choose-a-name --tpu-type v3-8
For reading data in, numpy arrays in memory will work for TPUs but might not be the most performant option. If your data fits in memory, it might be OK. For out of memory datasets, us the tf.data.Dataset API. Tutorial here: TPU-speed data pipelines: tf.data.Dataset and TFRecords
Thank you Martin. I made the connexion with the tpu without gcloud command line, and its work. I just need to set the enviroment variable to the tpu node.
\n\nIn the other hand my dataset in arrays way fit in memory well, but when i tried pass it to the model the model didnt work.\nI tried to pass this arrays (i did a lot of preprocess in the data with the arrays) to tensorflow datasets with tensorflow.dataset.to_tensor, but still not working when i train the model (even passing this new dataset with the bacth option)
", "votes": null, "replies": [] }, { "id": 745357, "author_name": "mgorner", "author_url": "", "post_date": "02/13/2020 18:31:52", "content": "If you just want to convert an in-memory array to a tf.data.Dataset, the correct function is Dataset.from_tensor_slices.
But passing a numpy array to model.fit() directly works, even on TPU.
", "votes": null, "replies": [] }, { "id": 745360, "author_name": "mgorner", "author_url": "", "post_date": "02/13/2020 18:34:10", "content": "In the script I sent earlier (create-tpu-deep-learning-vm.sh), you will find a way of setting the TPU_NAME environment variable sot that it persists across reboots. And yes of course you can do this by hand.
", "votes": null, "replies": [] }, { "id": 745833, "author_name": "ianwilkinson", "author_url": "", "post_date": "02/14/2020 09:02:48", "content": "Pass arrays didn't work on my case. Gave me error \"failed to serialize\".\nI tried the functions dataset.from_tensor_slices, but my data was larger than 2GB and y couldn't create the tensor.\nSo im trying work with tfrecords now. \nThanks!
", "votes": null, "replies": [] }, { "id": 746178, "author_name": "mgorner", "author_url": "", "post_date": "02/14/2020 17:56:33", "content": "Not entirely surprised 2GB \"failed to serialize\" 😊
", "votes": null, "replies": [] }, { "id": 920574, "author_name": "hooong", "author_url": "", "post_date": "07/08/2020 17:20:07", "content": "Hey Martin,
\n\nIs there a way to use my own TPU on a Kaggle dataset directly? Would it be possible for me to provide my own personal TPU the GCS paths for the public dataset on Kaggle?
", "votes": null, "replies": [] }, { "id": 932296, "author_name": "mgorner", "author_url": "", "post_date": "07/16/2020 23:04:48", "content": "Sure.
\nHere is the script to provision a TPU + Jupyter on GCP.
To access the data, run KaggleDatasets().get_gcs_path() on Kaggle once. It will return a GCS bucket name like 'gs://xyz12234567890'. That's where Kaggle stores the attached dataset. You can use this bucket on GCP as well. Just be careful: the bucket is a cache. It will be good for a few days and then be wiped out. If you want something more stable, copy to your own bucekt.
Hi,\nI have a notebook I have only images. How can use them in TPU. Because in TPU I must use tf.data.Dataset like in this kernel
\n\nhttps://www.kaggle.com/hatemamine/flowertpuwin
\n\nMy code is like that
\n\nhttps://www.kaggle.com/furkankati/person-mask-u-net-model-tpu
\n\nI read data directly from images. I convert images to numpy arrays.
\n\nHow can I upload my files to google cloud storage and use them for tpu ?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "744210": "Hi! I'm trying to use in gcp jupyter notebooks (ai platform) the TPU api.\nI have my own data (actually is the NIH Chest X-ray dataset) with a lot of preprocess. I'm available to train a on gpus from the numpy arrays alocated in RAM.\nWhat transformations i should do to this arrays for be the inputs of my tpu model.\n\nPS: My conexion with the tpu api is ok and i can create my tpu model.\n\nThanks!", "744296": "Yes, you can use TPUs in AI Platform Notebooks. If you are on GCP already, your training data should go into a GCS (Google Cloud Storage) bucket located in the same region as your TPU (for optimal performance, avoid multi-region buckets).\n\nTo spin up an AI Platform Jupyter VM + TPU pair, you gave to use `gcloud` on the command line since there isn't an option for that in the UI yet. I can share a script that does that: [create-tpu-deep-learning-vm.sh](https://raw.githubusercontent.com/GoogleCloudPlatform/training-data-analyst/master/courses/fast-and-lean-data-science/create-tpu-deep-learning-vm.sh).\n`gcloud init # to select your project, account, zones, ...`\nthen\n`./create-tpu-deep-learning-vm.sh choose-a-name --tpu-type v3-8`\n\nFor reading data in, numpy arrays in memory will work for TPUs but might not be the most performant option. If your data fits in memory, it might be OK. For out of memory datasets, us the `tf.data.Dataset` API. Tutorial here: [TPU-speed data pipelines: tf.data.Dataset and TFRecords](https://codelabs.developers.google.com/codelabs/keras-flowers-data/#0)", "744903": "Thank you Martin. I made the connexion with the tpu without gcloud command line, and its work. I just need to set the enviroment variable to the tpu node.\n\nIn the other hand my dataset in arrays way fit in memory well, but when i tried pass it to the model the model didnt work.\nI tried to pass this arrays (i did a lot of preprocess in the data with the arrays) to tensorflow datasets with tensorflow.dataset.to_tensor, but still not working when i train the model (even passing this new dataset with the bacth option)", "745357": "If you just want to convert an in-memory array to a tf.data.Dataset, the correct function is [`Dataset.from_tensor_slices`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#from_tensor_slices).\n\nBut passing a numpy array to model.fit() directly works, even on TPU.", "745360": "In the script I sent earlier ([create-tpu-deep-learning-vm.sh](https://raw.githubusercontent.com/GoogleCloudPlatform/training-data-analyst/master/courses/fast-and-lean-data-science/create-tpu-deep-learning-vm.sh)), you will find a way of setting the TPU_NAME environment variable sot that it persists across reboots. And yes of course you can do this by hand.", "745833": "Pass arrays didn't work on my case. Gave me error \"failed to serialize\".\nI tried the functions dataset.from_tensor_slices, but my data was larger than 2GB and y couldn't create the tensor.\nSo im trying work with tfrecords now. \nThanks!", "746178": "Not entirely surprised 2GB \"failed to serialize\" 😊", "801610": "Hi,\nI have a notebook I have only images. How can use them in TPU. Because in TPU I must use tf.data.Dataset like in this kernel\n\n[https://www.kaggle.com/hatemamine/flowertpuwin](https://www.kaggle.com/hatemamine/flowertpuwin)\n\n\nMy code is like that \n\n[https://www.kaggle.com/furkankati/person-mask-u-net-model-tpu](https://www.kaggle.com/furkankati/person-mask-u-net-model-tpu)\n\nI read data directly from images. I convert images to numpy arrays.\n\nHow can I upload my files to google cloud storage and use them for tpu ?", "920574": "Hey Martin, \n\nIs there a way to use my own TPU on a Kaggle dataset directly? Would it be possible for me to provide my own personal TPU the GCS paths for the public dataset on Kaggle?", "932296": "Sure.\nHere is the [script to provision a TPU + Jupyter on GCP](https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/fast-and-lean-data-science/README-TF2.1.md).\n\nTo access the data, run `KaggleDatasets().get_gcs_path()` **on Kaggle** once. It will return a GCS bucket name like 'gs://xyz12234567890'. That's where Kaggle stores the attached dataset. You can use this bucket on GCP as well. Just be careful: the bucket is a cache. It will be good for a few days and then be wiped out. If you want something more stable, copy to your own bucekt." }, "source": "meta" }