![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1182060%2F31ec47aac80e52cd67cf34b2744cdbc7%2FScreenshot%20from%202020-02-15%2020-10-58.png?generation=1581808283744088&alt=media\")
For some reason I keep getting errors related to TPU usage like the ones below, they all happen while I call model.fit().
\n\nI was successfully running the code some times, and it works perfectly in interactive mode.
\n\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1182060%2F8de19b0e81bfa003fd77ac9d0f2ffc7c%2FScreenshot%20from%202020-02-15%2020-10-49.png?generation=1581808287088816&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1182060%2Fbed489c2152337ad366ab602bbdf1289%2FScreenshot%20from%202020-02-15%2020-10-41.png?generation=1581808285414697&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1182060%2F13af6ea00d540e9f6ee37217149ed4c3%2FScreenshot%20from%202020-02-15%2020-10-32.png?generation=1581808287390303&alt=media\")
Anyone know the reason this happens or how to fix it?
\n\nlink for the kernel: https://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline
", "messages": [ { "id": "747071", "postDate": "02/15/2020 23:14:04", "content": "For some reason I keep getting errors related to TPU usage like the ones below, they all happen while I call model.fit().
\n\nI was successfully running the code some times, and it works perfectly in interactive mode.
\n\n
\n\n
Anyone know the reason this happens or how to fix it?
\n\nlink for the kernel: https://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline
", "rawMarkdown": "For some reason I keep getting errors related to TPU usage like the ones below, they all happen while I call model.fit().\n\nI was successfully running the code some times, and it works perfectly in interactive mode.\n\n\n\n\n\n\n\n\n\nAnyone know the reason this happens or how to fix it?\n\nlink for the kernel: https://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline", "votes": null }, { "id": "747240", "postDate": "02/16/2020 06:44:11", "content": "I see you fixed it and I assume the issue was drop_remainder=True in get_validation_dataset?
Did you know why it caused all these issues?
", "rawMarkdown": "I see you fixed it and I assume the issue was `drop_remainder=True` in `get_validation_dataset`? \n\nDid you know why it caused all these issues?", "votes": null }, { "id": "747414", "postDate": "02/16/2020 11:57:50", "content": "Hey @msheriey , I'm also assuming it was drop_remainder=True, I've used it both in train and validation sets, but I have no Idea why it happened, and if it was the real issue. I was just following this hint
I think we need to have @mgornergoogle's input on this.
", "rawMarkdown": "I think we need to have @mgornergoogle's input on this.", "votes": null }, { "id": "749614", "postDate": "02/18/2020 20:08:04", "content": "I kind of doubt the drop_remainder=True changed anything here. The training dataset is indefinitely repeated so there is no \"remainder\" when you batch it. Could it be that numerics go awry on the validation dataset which is finite ?
\n\nThe error above looks like the TPU disconnected. It might be a fluke. It might also be a TPU crash. And it might a an XLA crash. If the XLA compiler cannot compile your TF graph into TPU code, it can result in a TPU disconnection.
\n\nIf you manage to isolate the problem in a reproducible way, I'd be interested in having a look.
", "rawMarkdown": "I kind of doubt the drop_remainder=True changed anything here. The training dataset is indefinitely repeated so there is no \"remainder\" when you batch it. Could it be that numerics go awry on the validation dataset which is finite ?\n\nThe error above looks like the TPU disconnected. It might be a fluke. It might also be a TPU crash. And it might a an XLA crash. If the XLA compiler cannot compile your TF graph into TPU code, it can result in a TPU disconnection.\n\nIf you manage to isolate the problem in a reproducible way, I'd be interested in having a look.", "votes": null }, { "id": "751069", "postDate": "02/20/2020 01:21:15", "content": "Hi @mgornergoogle , I have managed to do some more experiments to isolate the problem.
\n\nOn my kernel I got the following experiments:\n- Version 32: I've not used drop_remainder=True on any set and got no error.\n- Version 33: I've used drop_remainder=True on train and validation sets and got error.\n- Version 34: I've used drop_remainder=True only on train and got no error.\n- Version 35: I've used drop_remainder=True only on validation and got error.
So it seems the errors are related to using drop_remainder=True on the validation set.
Oh, it's adding drop_remainder=True that is causing the issue ?\nThe easy workaround is to remove it. The validation dataset is exactly 29*128 elements. With a batch size of 128, there is no remainder anyway.
\n\nI agree with you that it should not crash though. Could you please file a bug against Tensorflow ? I'll surface it with the TF team. Thank you.
", "rawMarkdown": "Oh, it's adding drop_remainder=True that is causing the issue ?\nThe easy workaround is to remove it. The validation dataset is exactly 29*128 elements. With a batch size of 128, there is no remainder anyway.\n\nI agree with you that it should not crash though. Could you please file a bug against Tensorflow ? I'll surface it with the TF team. Thank you.", "votes": null }, { "id": "751837", "postDate": "02/20/2020 15:04:15", "content": "Hey @mgornergoogle , for now I'm removing it as you suggested.
\n\nI have created the issue, let me know if I can help with anything else.
", "rawMarkdown": "Hey @mgornergoogle , for now I'm removing it as you suggested.\n\nI have created the [issue](https://github.com/tensorflow/tensorflow/issues/36932), let me know if I can help with anything else.", "votes": null }, { "id": "755417", "postDate": "02/24/2020 18:59:01", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": null } ], "comments": [ { "id": 747240, "author_name": "msheriey", "author_url": "", "post_date": "02/16/2020 06:44:11", "content": "I see you fixed it and I assume the issue was drop_remainder=True in get_validation_dataset?
Did you know why it caused all these issues?
", "votes": null, "replies": [ { "id": 747414, "author_name": "dimitreoliveira", "author_url": "", "post_date": "02/16/2020 11:57:50", "content": "Hey @msheriey , I'm also assuming it was drop_remainder=True, I've used it both in train and validation sets, but I have no Idea why it happened, and if it was the real issue. I was just following this hint
I think we need to have @mgornergoogle's input on this.
", "votes": null, "replies": [] }, { "id": 749614, "author_name": "mgorner", "author_url": "", "post_date": "02/18/2020 20:08:04", "content": "I kind of doubt the drop_remainder=True changed anything here. The training dataset is indefinitely repeated so there is no \"remainder\" when you batch it. Could it be that numerics go awry on the validation dataset which is finite ?
\n\nThe error above looks like the TPU disconnected. It might be a fluke. It might also be a TPU crash. And it might a an XLA crash. If the XLA compiler cannot compile your TF graph into TPU code, it can result in a TPU disconnection.
\n\nIf you manage to isolate the problem in a reproducible way, I'd be interested in having a look.
", "votes": null, "replies": [] }, { "id": 751069, "author_name": "dimitreoliveira", "author_url": "", "post_date": "02/20/2020 01:21:15", "content": "Hi @mgornergoogle , I have managed to do some more experiments to isolate the problem.
\n\nOn my kernel I got the following experiments:\n- Version 32: I've not used drop_remainder=True on any set and got no error.\n- Version 33: I've used drop_remainder=True on train and validation sets and got error.\n- Version 34: I've used drop_remainder=True only on train and got no error.\n- Version 35: I've used drop_remainder=True only on validation and got error.
So it seems the errors are related to using drop_remainder=True on the validation set.
Oh, it's adding drop_remainder=True that is causing the issue ?\nThe easy workaround is to remove it. The validation dataset is exactly 29*128 elements. With a batch size of 128, there is no remainder anyway.
\n\nI agree with you that it should not crash though. Could you please file a bug against Tensorflow ? I'll surface it with the TF team. Thank you.
", "votes": null, "replies": [] }, { "id": 751837, "author_name": "dimitreoliveira", "author_url": "", "post_date": "02/20/2020 15:04:15", "content": "Hey @mgornergoogle , for now I'm removing it as you suggested.
\n\nI have created the issue, let me know if I can help with anything else.
", "votes": null, "replies": [] }, { "id": 755417, "author_name": "mgorner", "author_url": "", "post_date": "02/24/2020 18:59:01", "content": "Thanks!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "747071": "For some reason I keep getting errors related to TPU usage like the ones below, they all happen while I call model.fit().\n\nI was successfully running the code some times, and it works perfectly in interactive mode.\n\n\n\n\n\n\n\n\n\nAnyone know the reason this happens or how to fix it?\n\nlink for the kernel: https://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline", "747240": "I see you fixed it and I assume the issue was `drop_remainder=True` in `get_validation_dataset`? \n\nDid you know why it caused all these issues?", "747414": "Hey @msheriey , I'm also assuming it was `drop_remainder=True`, I've used it both in train and validation sets, but I have no Idea why it happened, and if it was the real issue. I was just following this [hint](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/130703)", "747734": "I think we need to have @mgornergoogle's input on this.", "749614": "I kind of doubt the drop_remainder=True changed anything here. The training dataset is indefinitely repeated so there is no \"remainder\" when you batch it. Could it be that numerics go awry on the validation dataset which is finite ?\n\nThe error above looks like the TPU disconnected. It might be a fluke. It might also be a TPU crash. And it might a an XLA crash. If the XLA compiler cannot compile your TF graph into TPU code, it can result in a TPU disconnection.\n\nIf you manage to isolate the problem in a reproducible way, I'd be interested in having a look.", "751069": "Hi @mgornergoogle , I have managed to do some more experiments to isolate the problem.\n\nOn my [kernel](https://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline) I got the following experiments:\n- [Version 32](https://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline?scriptVersionId=28912464): I've not used `drop_remainder=True` on any set and got no error.\n- [Version 33](https://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline?scriptVersionId=28930091): I've used `drop_remainder=True` on train and validation sets and got error.\n- [Version 34](https://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline?scriptVersionId=28931447): I've used `drop_remainder=True` only on train and got no error.\n- [Version 35](https://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline/notebook?scriptVersionId=28932681): I've used `drop_remainder=True` only on validation and got error.\n\nSo it seems the errors are related to using `drop_remainder=True` on the validation set.", "751112": "Oh, it's adding drop_remainder=True that is causing the issue ?\nThe easy workaround is to remove it. The validation dataset is exactly 29*128 elements. With a batch size of 128, there is no remainder anyway.\n\nI agree with you that it should not crash though. Could you please file a bug against Tensorflow ? I'll surface it with the TF team. Thank you.", "751837": "Hey @mgornergoogle , for now I'm removing it as you suggested.\n\nI have created the [issue](https://github.com/tensorflow/tensorflow/issues/36932), let me know if I can help with anything else.", "755417": "Thanks!" }, "source": "meta" }