Hey all,
\n\nFrom what I understood going through many discussions and through tensorflow's GitHub, we can't read local TFRecords, TFRecords from @cdeotte's Kaggle datasets or even local TFRecords from the competition's data because it raises an UnimplementedError: File system scheme '[local]' not implemented. I just tried to do it and it seems the issue still remains, but maybe there is something wrong in what I tried so please tell me if someone has already been able to read local TFRecords on TPUs.
Could we expect to have this implemented soon? Otherwise, is there an existing kernel which explains how to create our own GCS and link it to a Kaggle kernel in order to read custom TFRecords on TPUs? I didn't find one but I would be surprised if no one has tried before!
\n\nMany thanks for your help! 👍
", "messages": [ { "id": "882913", "postDate": "06/12/2020 08:24:50", "content": "Hey all,
\n\nFrom what I understood going through many discussions and through tensorflow's GitHub, we can't read local TFRecords, TFRecords from @cdeotte's Kaggle datasets or even local TFRecords from the competition's data because it raises an UnimplementedError: File system scheme '[local]' not implemented. I just tried to do it and it seems the issue still remains, but maybe there is something wrong in what I tried so please tell me if someone has already been able to read local TFRecords on TPUs.
Could we expect to have this implemented soon? Otherwise, is there an existing kernel which explains how to create our own GCS and link it to a Kaggle kernel in order to read custom TFRecords on TPUs? I didn't find one but I would be surprised if no one has tried before!
\n\nMany thanks for your help! 👍
", "rawMarkdown": "Hey all,\n\nFrom what I understood going through many discussions and through `tensorflow`'s GitHub, we can't read local TFRecords, TFRecords from @cdeotte's Kaggle datasets or even local TFRecords from the competition's data because it raises an `UnimplementedError: File system scheme '[local]' not implemented`. I just tried to do it and it seems the issue still remains, but maybe there is something wrong in what I tried so please tell me if someone has already been able to read local TFRecords on TPUs.\n\nCould we expect to have this implemented soon? Otherwise, is there an existing kernel which explains how to create our own GCS and link it to a Kaggle kernel in order to read custom TFRecords on TPUs? I didn't find one but I would be surprised if no one has tried before!\n\nMany thanks for your help! 👍", "votes": null }, { "id": "882928", "postDate": "06/12/2020 08:33:17", "content": "I tried on Colab failed. Even Kaggle private datasets don't have GCS path.
", "rawMarkdown": "I tried on Colab failed. Even Kaggle private datasets don't have GCS path.", "votes": null }, { "id": "883175", "postDate": "06/12/2020 12:23:21", "content": "You can only use public datasets for TPU's.\nI also learned it the hard way.
\n\nThey somehow decided to be like that.\nI think they intentionally decided it to be like that because of dataprotection or something like that.\nSadly can't remember where I read that.
\n\nWish I could give more information.
", "rawMarkdown": "You can only use public datasets for TPU's.\nI also learned it the hard way.\n\nThey somehow decided to be like that.\nI think they intentionally decided it to be like that because of dataprotection or something like that.\nSadly can't remember where I read that.\n\nWish I could give more information.", "votes": null }, { "id": "883263", "postDate": "06/12/2020 13:50:05", "content": "@crazyt not entirely correct i.e. data doesn't have to public. TPUs can also use private datasets.
\n\nThe only rule is that TPUs read data only from GCS buckets.\nSo, create Google Cloud Storage buckets (similar to S3 buckets) perhaps using $300 free credits given by GCP.
", "rawMarkdown": "crazyt not entirely correct i.e. data doesn't have to public. TPUs can also use private datasets.\n\nThe only rule is that TPUs read data only from **GCS buckets**.\nSo, create Google Cloud Storage buckets (similar to S3 buckets) perhaps using $300 [free credits](https://cloud.google.com/free/) given by GCP.", "votes": null }, { "id": "884694", "postDate": "06/13/2020 14:58:34", "content": "Is it correct that TPUs can only train from TFRecords??\nBecause when i use images directly TPUs MXU is at zero percentage and idle time is like --
I think you can do it without TFRecords, but the problem is that you will get errors similar to \"Compilation failure: Ran out of memory in memory space hbm. Used 39.26G of 16.00G hbm\" really fast.\nAtleast thats what I got, although I was using a \"Data Generator\" in keras.
\n\nBut now that I think about it, there is probably nothing that stops you from using the fit-command in a loop with small chunked datasets.
\n\nEdit:\nMaybe was wrong about the error, now I think \"Timeout error trying to communicate with service\" was the correct one.
", "rawMarkdown": "I think you can do it without TFRecords, but the problem is that you will get errors similar to \"Compilation failure: Ran out of memory in memory space hbm. Used 39.26G of 16.00G hbm\" really fast.\nAtleast thats what I got, although I was using a \"Data Generator\" in keras.\n\nBut now that I think about it, there is probably nothing that stops you from using the fit-command in a loop with small chunked datasets.\n\nEdit:\nMaybe was wrong about the error, now I think \"Timeout error trying to communicate with service\" was the correct one.", "votes": null } ], "comments": [ { "id": 882928, "author_name": "awsaf49", "author_url": "", "post_date": "06/12/2020 08:33:17", "content": "I tried on Colab failed. Even Kaggle private datasets don't have GCS path.
", "votes": null, "replies": [] }, { "id": 883175, "author_name": "crazyt", "author_url": "", "post_date": "06/12/2020 12:23:21", "content": "You can only use public datasets for TPU's.\nI also learned it the hard way.
\n\nThey somehow decided to be like that.\nI think they intentionally decided it to be like that because of dataprotection or something like that.\nSadly can't remember where I read that.
\n\nWish I could give more information.
", "votes": null, "replies": [ { "id": 883263, "author_name": "sirishks", "author_url": "", "post_date": "06/12/2020 13:50:05", "content": "@crazyt not entirely correct i.e. data doesn't have to public. TPUs can also use private datasets.
\n\nThe only rule is that TPUs read data only from GCS buckets.\nSo, create Google Cloud Storage buckets (similar to S3 buckets) perhaps using $300 free credits given by GCP.
", "votes": null, "replies": [] }, { "id": 884694, "author_name": "msharuk589", "author_url": "", "post_date": "06/13/2020 14:58:34", "content": "Is it correct that TPUs can only train from TFRecords??\nBecause when i use images directly TPUs MXU is at zero percentage and idle time is like --
I think you can do it without TFRecords, but the problem is that you will get errors similar to \"Compilation failure: Ran out of memory in memory space hbm. Used 39.26G of 16.00G hbm\" really fast.\nAtleast thats what I got, although I was using a \"Data Generator\" in keras.
\n\nBut now that I think about it, there is probably nothing that stops you from using the fit-command in a loop with small chunked datasets.
\n\nEdit:\nMaybe was wrong about the error, now I think \"Timeout error trying to communicate with service\" was the correct one.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "882913": "Hey all,\n\nFrom what I understood going through many discussions and through `tensorflow`'s GitHub, we can't read local TFRecords, TFRecords from @cdeotte's Kaggle datasets or even local TFRecords from the competition's data because it raises an `UnimplementedError: File system scheme '[local]' not implemented`. I just tried to do it and it seems the issue still remains, but maybe there is something wrong in what I tried so please tell me if someone has already been able to read local TFRecords on TPUs.\n\nCould we expect to have this implemented soon? Otherwise, is there an existing kernel which explains how to create our own GCS and link it to a Kaggle kernel in order to read custom TFRecords on TPUs? I didn't find one but I would be surprised if no one has tried before!\n\nMany thanks for your help! 👍", "882928": "I tried on Colab failed. Even Kaggle private datasets don't have GCS path.", "883175": "You can only use public datasets for TPU's.\nI also learned it the hard way.\n\nThey somehow decided to be like that.\nI think they intentionally decided it to be like that because of dataprotection or something like that.\nSadly can't remember where I read that.\n\nWish I could give more information.", "883263": "crazyt not entirely correct i.e. data doesn't have to public. TPUs can also use private datasets.\n\nThe only rule is that TPUs read data only from **GCS buckets**.\nSo, create Google Cloud Storage buckets (similar to S3 buckets) perhaps using $300 [free credits](https://cloud.google.com/free/) given by GCP.", "884694": "Is it correct that TPUs can only train from TFRecords??\nBecause when i use images directly TPUs MXU is at zero percentage and idle time is like `--`", "884716": "I think you can do it without TFRecords, but the problem is that you will get errors similar to \"Compilation failure: Ran out of memory in memory space hbm. Used 39.26G of 16.00G hbm\" really fast.\nAtleast thats what I got, although I was using a \"Data Generator\" in keras.\n\nBut now that I think about it, there is probably nothing that stops you from using the fit-command in a loop with small chunked datasets.\n\nEdit:\nMaybe was wrong about the error, now I think \"Timeout error trying to communicate with service\" was the correct one." }, "source": "meta" }