@osamir this looks like an intermittent error. I assume your Internet was ON and that was a TPU enabled notebook, right? If so, please retry. I did work at least in a couple of notebooks I tried.
", "rawMarkdown": "@osamir this looks like an intermittent error. I assume your Internet was ON and that was a TPU enabled notebook, right? If so, please retry. I did work at least in a couple of notebooks I tried.", "votes": 1 }, { "id": 969352, "postDate": "2020-08-13T16:18:50.900Z", "content": "I double checked that TPU is enabled and internet is on but still facing this error. I will try it again and see if it still exists or not.
", "rawMarkdown": "I double checked that TPU is enabled and internet is on but still facing this error. I will try it again and see if it still exists or not.", "votes": 1 }, { "id": 969381, "postDate": "2020-08-13T16:39:33.330Z", "content": "We are planning to downgrade back to TF 2.2 for the TPU machines. So, hopefully, you will not need the workaround soon.
", "rawMarkdown": "We are planning to downgrade back to TF 2.2 for the TPU machines. So, hopefully, you will not need the workaround soon.", "votes": 2, "replies": [ { "id": 969387, "postDate": "2020-08-13T16:42:12.087Z", "content": "Upgrade to TF brings a mess many times. IMHO, authority should double check before launch, one day waste at the end time; very lossy.
", "rawMarkdown": "Upgrade to TF brings a mess many times. IMHO, authority should double check before launch, one day waste at the end time; very lossy. ", "votes": 1 }, { "id": 969405, "postDate": "2020-08-13T16:53:04.030Z", "content": "actually the workaround doesn't work for me as it gives the same error as reported, and I've tried many times
", "rawMarkdown": "actually the workaround doesn't work for me as it gives the same error as reported, and I've tried many times", "votes": 1 } ] }, { "id": 969481, "postDate": "2020-08-13T17:56:41.110Z", "content": "Workaround does not work for me. Using workaround gives new weird errors.
", "rawMarkdown": "Workaround does not work for me. Using workaround gives new weird errors." }, { "id": 969696, "postDate": "2020-08-13T21:27:12.607Z", "content": "Same for me, in Kaggle and on Colab
", "rawMarkdown": "Same for me, in Kaggle and on Colab" } ] }, { "id": 968490, "postDate": "2020-08-13T04:12:23.387Z", "content": "I agree. I cannot use TPU tonight. Everything that worked yesterday does not work today
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2F1a0fcdf9beca61fe3fc18c03ec13cbc5%2Fe1.png?generation=1597291913411797&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2F25b4e74365ae124d7e6689411cdc9f7f%2Fe2.png?generation=1597291925498803&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2F7f6528504e281e6355766caf4ab6a086%2Fe3.png?generation=1597291935745289&alt=media\")
hope it will be fixed soon.
", "rawMarkdown": "hope it will be fixed soon.", "votes": 5 }, { "id": 969492, "postDate": "2020-08-13T18:07:41.563Z", "content": "The rollback to TF 2.2 is currently in progress. If you restart your session or create a new one, it is likely that you'll get yesterday's TF 2.2.
", "rawMarkdown": "The rollback to TF 2.2 is currently in progress. If you restart your session or create a new one, it is likely that you'll get yesterday's TF 2.2.", "votes": 1 }, { "id": 969844, "postDate": "2020-08-14T01:55:17.390Z", "content": "Thanks @ifigotin , after the rollback everything works like it used to.
", "rawMarkdown": "Thanks @ifigotin , after the rollback everything works like it used to.", "votes": 2 }, { "id": 969857, "postDate": "2020-08-14T02:29:53.523Z", "content": "Sounds great. As @mgornergoogle mentioned in this or the other discussion, we filed several issues with the TF team internally.
", "rawMarkdown": "Sounds great. As @mgornergoogle mentioned in this or the other discussion, we filed several issues with the TF team internally. ", "votes": 2 } ] }, { "id": 968548, "postDate": "2020-08-13T05:13:23.233Z", "content": "I was facing this Resource Exhausted Error for quite a while with big image and model size, I don't know if you are facing it due to the same reasons, but lowering the batch size worked for me.
Yes, that is the normal fix. But something changed at Kaggle. My popular notebook which ran yesterday does not run today. Something is different about the TPU
", "rawMarkdown": "Yes, that is the normal fix. But something changed at Kaggle. My popular notebook which ran yesterday does not run today. Something is different about the TPU", "votes": 2 }, { "id": 968597, "postDate": "2020-08-13T06:15:26.527Z", "content": "Yeah, it definitely has changed, it didn't happen before the last week or so! :/
", "rawMarkdown": "Yeah, it definitely has changed, it didn't happen before the last week or so! :/", "votes": 1 } ] }, { "id": 968510, "postDate": "2020-08-13T04:37:30.203Z", "content": "I have the same issue here
", "rawMarkdown": "I have the same issue here", "votes": 1, "replies": [ { "id": 968512, "postDate": "2020-08-13T04:41:59.347Z", "content": "It's frustrating. I wonder what happened. Did TF 2.3 cause this?
", "rawMarkdown": "It's frustrating. I wonder what happened. Did TF 2.3 cause this?", "votes": 1 }, { "id": 969125, "postDate": "2020-08-13T13:26:55.713Z", "content": "Yes it is very frustrating. I was trying a new idea yesterday and blocked by this problem. I think it is related to TF 2.3. I tried to install TF 2.2.0 on kaggle but i haven't succeeded yet.
", "rawMarkdown": "Yes it is very frustrating. I was trying a new idea yesterday and blocked by this problem. I think it is related to TF 2.3. I tried to install TF 2.2.0 on kaggle but i haven't succeeded yet." } ] }, { "id": 969503, "postDate": "2020-08-13T18:17:53.790Z", "content": "[Joking aside] After getting back TF 2.2, I think the TPU users can demand session limit [at least one hour extra, which means 3 hours + 1 ] - due to lossy one-day. 😏
", "rawMarkdown": "[Joking aside] After getting back TF 2.2, I think the TPU users can demand session limit [at least one hour extra, which means 3 hours + 1 ] - due to lossy one-day. 😏", "votes": 2 }, { "id": 968984, "postDate": "2020-08-13T11:39:29.660Z", "content": "It's the same problem as in colab 1 week ago, the TPU memory seems to be halved. Can kaggle please solve this, as training time is also very important in 3-hour kernels ???
", "rawMarkdown": "It's the same problem as in colab 1 week ago, the TPU memory seems to be halved. Can kaggle please solve this, as training time is also very important in 3-hour kernels ???", "votes": 2 }, { "id": 969841, "postDate": "2020-08-14T01:51:44.050Z", "content": "For info, if anyone is trying to access private datasets in Colab, the following authentication code needs to be added in the colab notebook.
\nfrom google.colab import auth\nauth.authenticate_user()\nThe rest remains same as @deepkim explained.
", "rawMarkdown": "For info, if anyone is trying to access private datasets in Colab, the following authentication code needs to be added in the colab notebook.\n```\nfrom google.colab import auth\nauth.authenticate_user()\n```\nThe rest remains same as @deepkim explained." }, { "id": 969449, "postDate": "2020-08-13T17:33:29.123Z", "content": "What a mess !! Doesn't work on colab or kaggle for hundreds of users. Doesn't Tensorflow team check basic stuff before upgrades ?
", "rawMarkdown": "What a mess !! Doesn't work on colab or kaggle for hundreds of users. Doesn't Tensorflow team check basic stuff before upgrades ?" }, { "id": 969243, "postDate": "2020-08-13T15:09:17.867Z", "content": "I encountered the same error.
\nSo I used google colab TPU and the auc dropped 0.01 :(
So, that's the issue, causing OOM -_-
\nThe competition is about to end, and most of us rely on TPU and now it's giving this error at this endpoint. Damn!
@tahsin vai, I think we're heading this issue.
", "rawMarkdown": "So, that's the issue, causing OOM -_-\nThe competition is about to end, and most of us rely on TPU and now it's giving this error at this endpoint. Damn! \n\n@tahsin vai, I think we're heading this issue. " }, { "id": 968784, "postDate": "2020-08-13T08:30:53.393Z", "content": "Can anyone select \"Pin to original environment (2020-07-29)\" on option Environment preferences? I tried it to solve the problem, but it's disabled on my end.
", "rawMarkdown": "Can anyone select \"Pin to original environment (2020-07-29)\" on option Environment preferences? I tried it to solve the problem, but it's disabled on my end." }, { "id": 968782, "postDate": "2020-08-13T08:28:13.177Z", "content": "Am currently using both Kaggle and Colab TPUs. Both are working fine as of writing this.
\nColab TPU needs the downgrade to 2.2 as mentioned in another thread though.
Hmm. not sure what causes it… But for your information in Colab downgrading to TF 2.2 is working fine.
", "rawMarkdown": "Hmm. not sure what causes it... But for your information in Colab downgrading to TF 2.2 is working fine.", "replies": [ { "id": 968575, "postDate": "2020-08-13T05:49:56.727Z", "content": "Yes, downgrading to TF 2.2 works in Colab but doesn't work in kaggle notebook.
", "rawMarkdown": "Yes, downgrading to TF 2.2 works in Colab but doesn't work in kaggle notebook. ", "replies": [ { "id": 968628, "postDate": "2020-08-13T06:33:16.470Z", "content": "Hi @gmhost ! could you tell me how you downgrade to TF 2.2 in Colab.
\n is it pip install tensorflow-gpu==2.2.0 ?
!pip install tensorflow~=2.2.0 tensorflow_gcs_config~=2.2.0
", "rawMarkdown": "!pip install tensorflow~=2.2.0 tensorflow\\_gcs\\_config~=2.2.0" } ] } ] }, { "id": 969189, "postDate": "2020-08-13T14:30:24.313Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 968664, "author_name": "kaggler", "author_url": "", "post_date": "2020-08-13T06:58:33.253000", "content": "USE COLAB! Colab is working well
\njust add these lines to colab before importing tensorflow module (downgrade to TF 2.2)
\n!pip install tensorflow~=2.2.0 tensorflowgcsconfig~=2.2.0
\nimport tensorflow as tf
\nimport requests
\nimport os
\nresp = requests.post(\"http://{}:8475/requestversion/{}\".format(os.environ[\"COLABTPUADDR\"].split(\":\")[0], tf.version))
\nif resp.status_code != 200:
\n print(\"Failed to switch the TPU to TF {}\".format(version))
To access data from kaggle to COLAB, you need to manually input GCS PATHs like this on colab
\ngcs_path = 'gs://kds-cfb16ed5f55adb5ab35a75a2fe74c3dc11a4b869dc8cfb3d9b1759e1';
\ngcs_path2 = 'gs://kds-79d9c56dcae7978f98a6bbe8318ac6866ba4778fc95c5871fcfd0f03';
\ngcs_path3 = 'gs://kds-032edf427ad6e338f5392f73759ff404a1a3f25c2abdc4995136de30';
\nGCSPATH = [None]*FOLDS; GCSPATH2 = [None]*FOLDS; GCS_PATH3 = [None]*FOLDS
\nfor i,k in enumerate(IMG_SIZES):
\nGCS_PATH[i] = gcs_path;\n\n\nGCS_PATH2[i] = gcs_path2;\n\n\nGCS_PATH3[i] = gcs_path3;\nfilestrain = np.sort(np.array(tf.io.gfile.glob(GCSPATH[0] + '/train*.tfrec')));
\nfilestest = np.sort(np.array(tf.io.gfile.glob(GCSPATH[0] + '/test*.tfrec')));
\nYou can get directly GCS PATHs FROM KAGGLE ENVIRONMENT(In kaggle to get GCS PATHs)
\nGCSPATH = [None]*FOLDS; GCSPATH2 = [None]*FOLDS; GCS_PATH3 = [None]*FOLDS
\nfor i,k in enumerate(IMG_SIZES[:FOLDS]):
\nGCS_PATH[i] = KaggleDatasets().get_gcs_path('melanoma-%ix%i'%(k,k))\n\n\nGCS_PATH2[i] = KaggleDatasets().get_gcs_path('isic2019-%ix%i'%(k,k))\n\n\nGCS_PATH3[i] = KaggleDatasets().get_gcs_path('malignant-v2-%ix%i'%(k,k))\nfilestrain = np.sort(np.array(tf.io.gfile.glob(GCSPATH[0] + '/train*.tfrec')))
\nfilestest = np.sort(np.array(tf.io.gfile.glob(GCSPATH[0] + '/test*.tfrec')))
\nprint(GCSPATH, GCSPATH2, GCS_PATH3)
", "votes": 7, "replies": [ { "id": 968910, "author_name": "Joeran", "author_url": "", "post_date": "2020-08-13T10:34:08.100000", "content": "I tried your method and it seems to be working, I can train on Colab now. However, I still get OOM when using the same batch size as yesterday, which worked well on Kaggle then.
\nDo you know if the initialisation of the TPU should be any different on Colab than on Kaggle?
\nI use the initialisation from Chris Deotte's popular notebook now.
halve the batch size!
\ncolab tpu memory is much smaller than kaggle's
\nif you set a batch size at 32 on kaggle machine, you should set it to 16 on colab( this is based on my trial and error)
@deepkim were you able to load/access private kaggle datasets in colab?
\nI saved all of my tpu resources for training at last and this happened.
@ankitsajwan i haven tried accessing private data yet. I have been just using public data.
", "votes": 1, "replies": [] } ] }, { "id": 969021, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T12:09:52.600000", "content": "@mgornergoogle Do you have any idea why TPU batch sizes that worked before August 12th don't work after August 12th? Before August 12th we could train 384x384 batch size 256 = 32 x 8 on EffNetB6 here. Now we cannot.
", "votes": 5, "replies": [ { "id": 969436, "author_name": "Martin Görner", "author_url": "", "post_date": "2020-08-13T17:18:22.773000", "content": "Yes, I saw this issue with TF 2.3 as well. Investigating. Did you encounter this with other models besides EfficientNet ?
\n(The next thing I am going to try is to load EfficientNet from TFHub directly - that works now in TF 2.3 with the /job:localhost LoadOption - instead of the pip-installed lib. Maybe there was a bug fix in there…)
I tried tf.keras.applications.EfficinetNetBx and it did not work any better under TF 2.3. I have not been able to load EfficientNet from TF Hub yes because the tensorflow-hub library has not yet been updated with the new hub.KerasLayer(load_options) parameter. The TF team is investigating…
", "votes": 0, "replies": [ { "id": 969742, "author_name": "Innat", "author_url": "", "post_date": "2020-08-13T22:36:10.527000", "content": "@mgornergoogle AFAIK, about tf.keras.applicaiton.EfficientNetBx - This API is new and only available in tf-nightly.
no tf.keras.applicaiton.EfficientNetBx is there in TF 2.3
", "votes": 0, "replies": [] } ] }, { "id": 969608, "author_name": "Martin Görner", "author_url": "", "post_date": "2020-08-13T19:59:35.660000", "content": "Everyone, have you seen regressions with other models than EfficientNetBx ?
", "votes": 0, "replies": [] }, { "id": 969668, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T20:51:39.003000", "content": "I have tried all models on TF 2.2 but I only tried EfficientNet on TF 2.3 so far.
", "votes": 0, "replies": [] } ] }, { "id": 969299, "author_name": "Ilya Figotin", "author_url": "", "post_date": "2020-08-13T15:47:36.867000", "content": "The code snippet to downgrade to TF 2.2 for now should be as follows:
\n!pip install tensorflow~=2.2.0 tensorflow_gcs_config~=2.2.0\nimport tensorflow as tf\nimport requests\nimport os\nresp = requests.post(\"http://{}:8475/requestversion/{}\".format(os.environ[\"TPU_NAME\"].split(\":\")[0], tf.__version__))\nif resp.status_code != 200:\n print(\"Failed to switch the TPU to TF {}\".format(version))\nLet us know if this unblocks you, and we will be looking at the issue in the meantime.
", "votes": 4, "replies": [ { "id": 969327, "author_name": "omarsamir", "author_url": "", "post_date": "2020-08-13T16:03:19.987000", "content": "Hi llya, Thanks for sharing this code snippets. I tried it but still couldn't successful to setup it and i found the following error:
\nConnectionError: HTTPConnectionPool(host='grpc', port=8475): Max retries exceeded with url: /requestversion/2.2.0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known'))
", "votes": 3, "replies": [] }, { "id": 969345, "author_name": "Ilya Figotin", "author_url": "", "post_date": "2020-08-13T16:10:20.657000", "content": "@osamir this looks like an intermittent error. I assume your Internet was ON and that was a TPU enabled notebook, right? If so, please retry. I did work at least in a couple of notebooks I tried.
", "votes": 1, "replies": [] }, { "id": 969352, "author_name": "omarsamir", "author_url": "", "post_date": "2020-08-13T16:18:50.900000", "content": "I double checked that TPU is enabled and internet is on but still facing this error. I will try it again and see if it still exists or not.
", "votes": 1, "replies": [] }, { "id": 969381, "author_name": "Ilya Figotin", "author_url": "", "post_date": "2020-08-13T16:39:33.330000", "content": "We are planning to downgrade back to TF 2.2 for the TPU machines. So, hopefully, you will not need the workaround soon.
", "votes": 2, "replies": [ { "id": 969387, "author_name": "Innat", "author_url": "", "post_date": "2020-08-13T16:42:12.087000", "content": "Upgrade to TF brings a mess many times. IMHO, authority should double check before launch, one day waste at the end time; very lossy.
", "votes": 1, "replies": [] }, { "id": 969405, "author_name": "Phi", "author_url": "", "post_date": "2020-08-13T16:53:04.030000", "content": "actually the workaround doesn't work for me as it gives the same error as reported, and I've tried many times
", "votes": 1, "replies": [] } ] }, { "id": 969481, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T17:56:41.110000", "content": "Workaround does not work for me. Using workaround gives new weird errors.
", "votes": 0, "replies": [] }, { "id": 969696, "author_name": "Ronaldo S.A. Batista", "author_url": "", "post_date": "2020-08-13T21:27:12.607000", "content": "Same for me, in Kaggle and on Colab
", "votes": 0, "replies": [] } ] }, { "id": 968490, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T04:12:23.387000", "content": "I agree. I cannot use TPU tonight. Everything that worked yesterday does not work today
\n
hope it will be fixed soon.
", "votes": 5, "replies": [] }, { "id": 969492, "author_name": "Ilya Figotin", "author_url": "", "post_date": "2020-08-13T18:07:41.563000", "content": "The rollback to TF 2.2 is currently in progress. If you restart your session or create a new one, it is likely that you'll get yesterday's TF 2.2.
", "votes": 1, "replies": [] }, { "id": 969844, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-14T01:55:17.390000", "content": "Thanks @ifigotin , after the rollback everything works like it used to.
", "votes": 2, "replies": [] }, { "id": 969857, "author_name": "Ilya Figotin", "author_url": "", "post_date": "2020-08-14T02:29:53.523000", "content": "Sounds great. As @mgornergoogle mentioned in this or the other discussion, we filed several issues with the TF team internally.
", "votes": 2, "replies": [] } ] }, { "id": 968548, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-08-13T05:13:23.233000", "content": "I was facing this Resource Exhausted Error for quite a while with big image and model size, I don't know if you are facing it due to the same reasons, but lowering the batch size worked for me.
Yes, that is the normal fix. But something changed at Kaggle. My popular notebook which ran yesterday does not run today. Something is different about the TPU
", "votes": 2, "replies": [] }, { "id": 968597, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-08-13T06:15:26.527000", "content": "Yeah, it definitely has changed, it didn't happen before the last week or so! :/
", "votes": 1, "replies": [] } ] }, { "id": 968510, "author_name": "omarsamir", "author_url": "", "post_date": "2020-08-13T04:37:30.203000", "content": "I have the same issue here
", "votes": 1, "replies": [ { "id": 968512, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T04:41:59.347000", "content": "It's frustrating. I wonder what happened. Did TF 2.3 cause this?
", "votes": 1, "replies": [] }, { "id": 969125, "author_name": "omarsamir", "author_url": "", "post_date": "2020-08-13T13:26:55.713000", "content": "Yes it is very frustrating. I was trying a new idea yesterday and blocked by this problem. I think it is related to TF 2.3. I tried to install TF 2.2.0 on kaggle but i haven't succeeded yet.
", "votes": 0, "replies": [] } ] }, { "id": 969503, "author_name": "Innat", "author_url": "", "post_date": "2020-08-13T18:17:53.790000", "content": "[Joking aside] After getting back TF 2.2, I think the TPU users can demand session limit [at least one hour extra, which means 3 hours + 1 ] - due to lossy one-day. 😏
", "votes": 2, "replies": [] }, { "id": 968984, "author_name": "Phi", "author_url": "", "post_date": "2020-08-13T11:39:29.660000", "content": "It's the same problem as in colab 1 week ago, the TPU memory seems to be halved. Can kaggle please solve this, as training time is also very important in 3-hour kernels ???
", "votes": 2, "replies": [] }, { "id": 969841, "author_name": "sajwankit", "author_url": "", "post_date": "2020-08-14T01:51:44.050000", "content": "For info, if anyone is trying to access private datasets in Colab, the following authentication code needs to be added in the colab notebook.
\nfrom google.colab import auth\nauth.authenticate_user()\nThe rest remains same as @deepkim explained.
", "votes": 0, "replies": [] }, { "id": 969449, "author_name": "VSR", "author_url": "", "post_date": "2020-08-13T17:33:29.123000", "content": "What a mess !! Doesn't work on colab or kaggle for hundreds of users. Doesn't Tensorflow team check basic stuff before upgrades ?
", "votes": 0, "replies": [] }, { "id": 969243, "author_name": "fantastic_hirarin", "author_url": "", "post_date": "2020-08-13T15:09:17.867000", "content": "I encountered the same error.
\nSo I used google colab TPU and the auc dropped 0.01 :(
So, that's the issue, causing OOM -_-
\nThe competition is about to end, and most of us rely on TPU and now it's giving this error at this endpoint. Damn!
@tahsin vai, I think we're heading this issue.
", "votes": 0, "replies": [] }, { "id": 968784, "author_name": "Sandy Khosasi", "author_url": "", "post_date": "2020-08-13T08:30:53.393000", "content": "Can anyone select \"Pin to original environment (2020-07-29)\" on option Environment preferences? I tried it to solve the problem, but it's disabled on my end.
", "votes": 0, "replies": [] }, { "id": 968782, "author_name": "Vee", "author_url": "", "post_date": "2020-08-13T08:28:13.177000", "content": "Am currently using both Kaggle and Colab TPUs. Both are working fine as of writing this.
\nColab TPU needs the downgrade to 2.2 as mentioned in another thread though.
Hmm. not sure what causes it… But for your information in Colab downgrading to TF 2.2 is working fine.
", "votes": 0, "replies": [ { "id": 968575, "author_name": "kwang", "author_url": "", "post_date": "2020-08-13T05:49:56.727000", "content": "Yes, downgrading to TF 2.2 works in Colab but doesn't work in kaggle notebook.
", "votes": 0, "replies": [ { "id": 968628, "author_name": "Mehul Sampat", "author_url": "", "post_date": "2020-08-13T06:33:16.470000", "content": "Hi @gmhost ! could you tell me how you downgrade to TF 2.2 in Colab.
\n is it pip install tensorflow-gpu==2.2.0 ?
!pip install tensorflow~=2.2.0 tensorflow_gcs_config~=2.2.0
", "votes": 0, "replies": [] } ] } ] }, { "id": 969189, "author_name": "", "author_url": "", "post_date": "2020-08-13T14:30:24.313000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "968488": "since kaggle has updated tensorflow version to 2.3, the notebook cann't run in the new version.\n\nit would cause ResourceExhaustedError , something like [this](https://github.com/googlecolab/colabtools/issues/1470).\n\nAnd downgrade tf to 2.2 didn't work. \n", "968664": "**USE COLAB! Colab is working well**\n\n**just add these lines to colab before importing tensorflow module (downgrade to TF 2.2)**\n\n!pip install tensorflow~=2.2.0 tensorflow_gcs_config~=2.2.0\nimport tensorflow as tf\nimport requests\nimport os\nresp = requests.post(\"http://{}:8475/requestversion/{}\".format(os.environ[\"COLAB_TPU_ADDR\"].split(\":\")[0], tf.__version__))\nif resp.status_code != 200:\n print(\"Failed to switch the TPU to TF {}\".format(version))\n\n\n**To access data from kaggle to COLAB, you need to manually input GCS PATHs like this on colab**\n\n\ngcs_path = 'gs://kds-cfb16ed5f55adb5ab35a75a2fe74c3dc11a4b869dc8cfb3d9b1759e1';\n\n\ngcs_path2 = 'gs://kds-79d9c56dcae7978f98a6bbe8318ac6866ba4778fc95c5871fcfd0f03';\n\n\ngcs_path3 = 'gs://kds-032edf427ad6e338f5392f73759ff404a1a3f25c2abdc4995136de30';\n\n\n\n\nGCS_PATH = [None]\\*FOLDS; GCS_PATH2 = [None]\\*FOLDS; GCS_PATH3 = [None]\\*FOLDS\n\n\nfor i,k in enumerate(IMG_SIZES):\n\n\n GCS_PATH[i] = gcs_path;\n\n\n GCS_PATH2[i] = gcs_path2;\n\n\n GCS_PATH3[i] = gcs_path3;\n\n\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/train\\*.tfrec')));\n\n \nfiles_test = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/test\\*.tfrec'))); \n\n**You can get directly GCS PATHs FROM KAGGLE ENVIRONMENT(In kaggle to get GCS PATHs)**\n\nGCS_PATH = [None]\\*FOLDS; GCS_PATH2 = [None]\\*FOLDS; GCS_PATH3 = [None]\\*FOLDS\n\n\nfor i,k in enumerate(IMG_SIZES[:FOLDS]):\n\n\n GCS_PATH[i] = KaggleDatasets().get_gcs_path('melanoma-%ix%i'%(k,k))\n\n\n GCS_PATH2[i] = KaggleDatasets().get_gcs_path('isic2019-%ix%i'%(k,k))\n\n\n GCS_PATH3[i] = KaggleDatasets().get_gcs_path('malignant-v2-%ix%i'%(k,k))\n\n\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/train\\*.tfrec')))\n\n\nfiles_test = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/test\\*.tfrec')))\n\nprint(GCS_PATH, GCS_PATH2, GCS_PATH3)", "969021": "@mgornergoogle Do you have any idea why TPU batch sizes that worked before August 12th don't work after August 12th? Before August 12th we could train 384x384 batch size 256 = 32 x 8 on EffNetB6 [here][1]. Now we cannot.\n\n[1]: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords", "969299": "The code snippet to downgrade to TF 2.2 for now should be as follows:\n```\n!pip install tensorflow~=2.2.0 tensorflow_gcs_config~=2.2.0\nimport tensorflow as tf\nimport requests\nimport os\nresp = requests.post(\"http://{}:8475/requestversion/{}\".format(os.environ[\"TPU_NAME\"].split(\":\")[0], tf.__version__))\nif resp.status_code != 200:\n print(\"Failed to switch the TPU to TF {}\".format(version))\n```\nLet us know if this unblocks you, and we will be looking at the issue in the meantime. ", "968490": "I agree. I cannot use TPU tonight. Everything that worked yesterday does not work today\n\n\n\n\n\n", "968548": "I was facing this `Resource Exhausted Error` for quite a while with big image and model size, I don't know if you are facing it due to the same reasons, but lowering the batch size worked for me.", "968510": "I have the same issue here", "969503": "[Joking aside] After getting back TF 2.2, I think the TPU users can demand session limit [at least one hour extra, which means 3 hours + 1 ] - due to lossy one-day. 😏", "968984": "It's the same problem as in colab 1 week ago, the TPU memory seems to be halved. Can kaggle please solve this, as training time is also very important in 3-hour kernels ???", "969841": "For info, if anyone is trying to access private datasets in Colab, the following authentication code needs to be added in the colab notebook.\n```\nfrom google.colab import auth\nauth.authenticate_user()\n```\nThe rest remains same as @deepkim explained.", "969449": "What a mess !! Doesn't work on colab or kaggle for hundreds of users. Doesn't Tensorflow team check basic stuff before upgrades ?", "969243": "I encountered the same error.\nSo I used google colab TPU and the auc dropped 0.01 :(", "969044": "So, that's the issue, causing OOM -_-\nThe competition is about to end, and most of us rely on TPU and now it's giving this error at this endpoint. Damn! \n\n@tahsin vai, I think we're heading this issue. ", "968784": "Can anyone select \"Pin to original environment (2020-07-29)\" on option Environment preferences? I tried it to solve the problem, but it's disabled on my end.", "968782": "Am currently using both Kaggle and Colab TPUs. Both are working fine as of writing this.\nColab TPU needs the downgrade to 2.2 as mentioned in another thread though.", "968553": "Hmm. not sure what causes it... But for your information in Colab downgrading to TF 2.2 is working fine.", "969189": "" } }