I'm trying to utilize colab TPU on @cdeotte 's custom TFRecord files. But it appears there are some issues with that. We may need to store files on the GCS. Right now, we've faced the following error:
\n\n\nUnimplementedError: 6 root error(s) found.\n (0) Unimplemented: {{function_node __inference_train_function_121761}} File system scheme '[local]' not implemented (file: 'ISIC2019-768x768/train04-858.tfrec')\n [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n [[RemoteCall]]\n [[IteratorGetNextAsOptional_5]]\n
Similar issue: on stackoverflow and github issue.
\n\nIs there anyone also facing this? Any way out to make it work without GCS? FYI, I've uploaded required files on the drive and it actually worked on GPU mode.
", "messages": [ { "id": "946020", "postDate": "07/26/2020 09:36:13", "content": "I'm trying to utilize colab TPU on @cdeotte 's custom TFRecord files. But it appears there are some issues with that. We may need to store files on the GCS. Right now, we've faced the following error:
\n\n\nUnimplementedError: 6 root error(s) found.\n (0) Unimplemented: {{function_node __inference_train_function_121761}} File system scheme '[local]' not implemented (file: 'ISIC2019-768x768/train04-858.tfrec')\n [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n [[RemoteCall]]\n [[IteratorGetNextAsOptional_5]]\n
Similar issue: on stackoverflow and github issue.
\n\nIs there anyone also facing this? Any way out to make it work without GCS? FYI, I've uploaded required files on the drive and it actually worked on GPU mode.
", "rawMarkdown": "I'm trying to utilize colab TPU on @cdeotte 's custom TFRecord files. But it appears there are some issues with that. We may need to store files on the GCS. Right now, we've faced the following error:\n\n```\nUnimplementedError: 6 root error(s) found.\n (0) Unimplemented: {{function_node __inference_train_function_121761}} File system scheme '[local]' not implemented (file: 'ISIC2019-768x768/train04-858.tfrec')\n\t [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n\t [[RemoteCall]]\n\t [[IteratorGetNextAsOptional_5]]\n```\n\nSimilar issue: on [stackoverflow](https://stackoverflow.com/questions/58009029/tf1-14tpucan-not-use-custom-tfrecord-dataset-on-colab-using-tpu) and github [issue](https://github.com/tensorflow/tensorflow/issues/32651).\n\nIs there anyone also facing this? Any way out to make it work without GCS? FYI, I've uploaded required files on the drive and it actually worked on GPU mode.", "votes": null }, { "id": "946023", "postDate": "07/26/2020 09:39:48", "content": "i get the gcs address of @cdeotte tfrecord data from kaggle and directly use that address in colab like this :
\n\nGCS_PATH[0] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'
\n\nGCS_PATH[1] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'
\n\nGCS_PATH[2] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'
\n\nGCS_PATH[3] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'
\n\nGCS_PATH[4] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'
\n\nGCS_PATH2[0] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'
\n\nGCS_PATH2[1] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'
\n\nGCS_PATH2[2] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'
\n\nGCS_PATH2[3] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'
\n\nGCS_PATH2[4] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'
", "rawMarkdown": "i get the gcs address of @cdeotte tfrecord data from kaggle and directly use that address in colab like this : \n\nGCS_PATH[0] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'\n\nGCS_PATH[1] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'\n\nGCS_PATH[2] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'\n\nGCS_PATH[3] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'\n\nGCS_PATH[4] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'\n\n\nGCS_PATH2[0] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'\n\nGCS_PATH2[1] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'\n\nGCS_PATH2[2] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'\n\nGCS_PATH2[3] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'\n\nGCS_PATH2[4] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'", "votes": null }, { "id": "946024", "postDate": "07/26/2020 09:40:12", "content": "Did you setup all the steps to access Kaggle data from Colab?
", "rawMarkdown": "Did you setup all the steps to access [Kaggle data from Colab](https://www.kaggle.com/general/74235)?", "votes": null }, { "id": "946059", "postDate": "07/26/2020 10:19:34", "content": "@sirishks yes, I'm aware of that.
", "rawMarkdown": "sirishks yes, I'm aware of that.", "votes": null }, { "id": "946061", "postDate": "07/26/2020 10:21:15", "content": "I don't have a problem using GCS but this process, It's quite inconsistent, isn't it?
", "rawMarkdown": "I don't have a problem using GCS but this process, It's quite inconsistent, isn't it?", "votes": null }, { "id": "946078", "postDate": "07/26/2020 10:38:51", "content": "yes it is inconsistent but nothing to do,,you will have to get gcs address from kaggle and use that in colab,, you can't do it from google drive when you use TPU,i searched a lot about this few days back and no solution!!\nthe only solution is using GCS and then colab will give awesome tpu v2 which takes a lot of time for training model (same for colab pro)
\n\nif you find a way of doing it from google drive then let me know,thanks
", "rawMarkdown": "yes it is inconsistent but nothing to do,,you will have to get gcs address from kaggle and use that in colab,, you can't do it from google drive when you use TPU,i searched a lot about this few days back and no solution!!\nthe only solution is using GCS and then colab will give awesome tpu v2 which takes a lot of time for training model (same for colab pro)\n\nif you find a way of doing it from google drive then let me know,thanks", "votes": null }, { "id": "946747", "postDate": "07/26/2020 19:34:02", "content": "Mobssir is correct -- you are getting this error because you are trying to read your tfrecords from the drive. The right way to do it is to read them from Google Cloud Storage buckets. To do that you will need to know the GCS addresses. Yes, they are not stable in a sense that they change every 5-10 days. To deal with this you can use the following public notebook to determine the most recent addresses:
\n\nSIIM Show GCS bucket addresses
\n\nThe notebook can be easily modified to show the GCS address for any other dataset.
", "rawMarkdown": "Mobssir is correct -- you are getting this error because you are trying to read your tfrecords from the drive. The right way to do it is to read them from Google Cloud Storage buckets. To do that you will need to know the GCS addresses. Yes, they are not stable in a sense that they change every 5-10 days. To deal with this you can use the following public notebook to determine the most recent addresses:\n\n[SIIM Show GCS bucket addresses](https://www.kaggle.com/graf10a/siim-show-gcs-bucket-addresses)\n\nThe notebook can be easily modified to show the GCS address for any other dataset.", "votes": null }, { "id": "946782", "postDate": "07/26/2020 20:25:14", "content": "@graf10a I've noticed your notebook right after this post. Thank you. -)
", "rawMarkdown": "graf10a I've noticed your notebook right after this post. Thank you. -)", "votes": null }, { "id": "946934", "postDate": "07/27/2020 00:52:46", "content": "Use GS buckets because all TPUs are in Google Cloud.
", "rawMarkdown": "Use GS buckets because all TPUs are in Google Cloud.", "votes": null } ], "comments": [ { "id": 946023, "author_name": "mobassir", "author_url": "", "post_date": "07/26/2020 09:39:48", "content": "i get the gcs address of @cdeotte tfrecord data from kaggle and directly use that address in colab like this :
\n\nGCS_PATH[0] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'
\n\nGCS_PATH[1] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'
\n\nGCS_PATH[2] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'
\n\nGCS_PATH[3] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'
\n\nGCS_PATH[4] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'
\n\nGCS_PATH2[0] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'
\n\nGCS_PATH2[1] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'
\n\nGCS_PATH2[2] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'
\n\nGCS_PATH2[3] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'
\n\nGCS_PATH2[4] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'
", "votes": null, "replies": [ { "id": 946061, "author_name": "ipythonx", "author_url": "", "post_date": "07/26/2020 10:21:15", "content": "I don't have a problem using GCS but this process, It's quite inconsistent, isn't it?
", "votes": null, "replies": [] }, { "id": 946078, "author_name": "mobassir", "author_url": "", "post_date": "07/26/2020 10:38:51", "content": "yes it is inconsistent but nothing to do,,you will have to get gcs address from kaggle and use that in colab,, you can't do it from google drive when you use TPU,i searched a lot about this few days back and no solution!!\nthe only solution is using GCS and then colab will give awesome tpu v2 which takes a lot of time for training model (same for colab pro)
\n\nif you find a way of doing it from google drive then let me know,thanks
", "votes": null, "replies": [] } ] }, { "id": 946024, "author_name": "sirishks", "author_url": "", "post_date": "07/26/2020 09:40:12", "content": "Did you setup all the steps to access Kaggle data from Colab?
", "votes": null, "replies": [ { "id": 946059, "author_name": "ipythonx", "author_url": "", "post_date": "07/26/2020 10:19:34", "content": "@sirishks yes, I'm aware of that.
", "votes": null, "replies": [] } ] }, { "id": 946747, "author_name": "graf10a", "author_url": "", "post_date": "07/26/2020 19:34:02", "content": "Mobssir is correct -- you are getting this error because you are trying to read your tfrecords from the drive. The right way to do it is to read them from Google Cloud Storage buckets. To do that you will need to know the GCS addresses. Yes, they are not stable in a sense that they change every 5-10 days. To deal with this you can use the following public notebook to determine the most recent addresses:
\n\nSIIM Show GCS bucket addresses
\n\nThe notebook can be easily modified to show the GCS address for any other dataset.
", "votes": null, "replies": [ { "id": 946782, "author_name": "ipythonx", "author_url": "", "post_date": "07/26/2020 20:25:14", "content": "@graf10a I've noticed your notebook right after this post. Thank you. -)
", "votes": null, "replies": [] } ] }, { "id": 946934, "author_name": "underwearfitting", "author_url": "", "post_date": "07/27/2020 00:52:46", "content": "Use GS buckets because all TPUs are in Google Cloud.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "946020": "I'm trying to utilize colab TPU on @cdeotte 's custom TFRecord files. But it appears there are some issues with that. We may need to store files on the GCS. Right now, we've faced the following error:\n\n```\nUnimplementedError: 6 root error(s) found.\n (0) Unimplemented: {{function_node __inference_train_function_121761}} File system scheme '[local]' not implemented (file: 'ISIC2019-768x768/train04-858.tfrec')\n\t [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n\t [[RemoteCall]]\n\t [[IteratorGetNextAsOptional_5]]\n```\n\nSimilar issue: on [stackoverflow](https://stackoverflow.com/questions/58009029/tf1-14tpucan-not-use-custom-tfrecord-dataset-on-colab-using-tpu) and github [issue](https://github.com/tensorflow/tensorflow/issues/32651).\n\nIs there anyone also facing this? Any way out to make it work without GCS? FYI, I've uploaded required files on the drive and it actually worked on GPU mode.", "946023": "i get the gcs address of @cdeotte tfrecord data from kaggle and directly use that address in colab like this : \n\nGCS_PATH[0] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'\n\nGCS_PATH[1] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'\n\nGCS_PATH[2] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'\n\nGCS_PATH[3] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'\n\nGCS_PATH[4] = 'gs://kds-abd9b34c7b6341bb0160d0614650d6a51734f29989b33aa5995ba9c7'\n\n\nGCS_PATH2[0] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'\n\nGCS_PATH2[1] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'\n\nGCS_PATH2[2] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'\n\nGCS_PATH2[3] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'\n\nGCS_PATH2[4] = 'gs://kds-28746e7cef66f1ba158e633f4dea61912d69a985fcc38b624bc8def6'", "946024": "Did you setup all the steps to access [Kaggle data from Colab](https://www.kaggle.com/general/74235)?", "946059": "sirishks yes, I'm aware of that.", "946061": "I don't have a problem using GCS but this process, It's quite inconsistent, isn't it?", "946078": "yes it is inconsistent but nothing to do,,you will have to get gcs address from kaggle and use that in colab,, you can't do it from google drive when you use TPU,i searched a lot about this few days back and no solution!!\nthe only solution is using GCS and then colab will give awesome tpu v2 which takes a lot of time for training model (same for colab pro)\n\nif you find a way of doing it from google drive then let me know,thanks", "946747": "Mobssir is correct -- you are getting this error because you are trying to read your tfrecords from the drive. The right way to do it is to read them from Google Cloud Storage buckets. To do that you will need to know the GCS addresses. Yes, they are not stable in a sense that they change every 5-10 days. To deal with this you can use the following public notebook to determine the most recent addresses:\n\n[SIIM Show GCS bucket addresses](https://www.kaggle.com/graf10a/siim-show-gcs-bucket-addresses)\n\nThe notebook can be easily modified to show the GCS address for any other dataset.", "946782": "graf10a I've noticed your notebook right after this post. Thank you. -)", "946934": "Use GS buckets because all TPUs are in Google Cloud." }, "source": "meta" }