I am trying to encode the data of mine where my input and output dimensions are \"( 512, 512, 3)\" and my loss is \"MSE\". when i am trying to train my model i am facing TPU compilation error.
\n\n```\nUnimplementedError: {{function_node __inference_train_function_50515}} Compilation failure: CustomCall is not supported to have a dynamic dimension\n TPU compilation failed\n [[{{node tpu_compile_succeeded_assert/_8679107639378603097/_5}}]]
\n\n```\nAny suggestions?
", "messages": [ { "id": "962727", "postDate": "08/08/2020 11:39:01", "content": "I am trying to encode the data of mine where my input and output dimensions are \"( 512, 512, 3)\" and my loss is \"MSE\". when i am trying to train my model i am facing TPU compilation error.
\n\n```\nUnimplementedError: {{function_node __inference_train_function_50515}} Compilation failure: CustomCall is not supported to have a dynamic dimension\n TPU compilation failed\n [[{{node tpu_compile_succeeded_assert/_8679107639378603097/_5}}]]
\n\n```\nAny suggestions?
", "rawMarkdown": "I am trying to encode the data of mine where my input and output dimensions are \"( 512, 512, 3)\" and my loss is \"MSE\". when i am trying to train my model i am facing TPU compilation error. \n\n```\nUnimplementedError: {{function_node __inference_train_function_50515}} Compilation failure: CustomCall is not supported to have a dynamic dimension\n\tTPU compilation failed\n\t [[{{node tpu_compile_succeeded_assert/_8679107639378603097/_5}}]]\n\n```\nAny suggestions?", "votes": null }, { "id": "962780", "postDate": "08/08/2020 12:40:47", "content": "Hi @ashoksrinivas can you check this [https://stackoverflow.com/questions/57658114/how-to-solve-propagation-of-dynamic-dimension-failed-error-in-tf-keras-with-tp]
", "rawMarkdown": "Hi @ashoksrinivas can you check this [https://stackoverflow.com/questions/57658114/how-to-solve-propagation-of-dynamic-dimension-failed-error-in-tf-keras-with-tp]", "votes": null }, { "id": "962799", "postDate": "08/08/2020 12:54:46", "content": "I have received this error before. If you are writing your own data augmentation functions for tf.data.Dataset, you must explicitly tell TPU what the output dimension is. For example in my notebook here which performs rotation augmentation, the last line of the transform function is reshape
def transform(image, DIM=256):\n # CODE HERE TO ROTATE\n return tf.reshape(d,[DIM, DIM,3])\nNote the last line returns tf.reshape which explicitly tells TPU was output dimension is. This is not needed for GPU but I found it is needed for TPU.
@cdeotte I am usingtf.reshape but still facing the same issue...
I will follow up on @cdeotte 's solution for everybody googling this issue in the future. If using tf.reshape() doesn't solve it for you, try adding drop_remainder=True to dataset.batch() function. Otherwise, batch_size is dynamic, and so is the shape of input.
", "rawMarkdown": "I will follow up on @cdeotte 's solution for everybody googling this issue in the future. If using tf.reshape() doesn't solve it for you, try adding ***drop_remainder=True*** to dataset.batch() function. Otherwise, batch_size is dynamic, and so is the shape of input.", "votes": null } ], "comments": [ { "id": 962799, "author_name": "cdeotte", "author_url": "", "post_date": "08/08/2020 12:54:46", "content": "I have received this error before. If you are writing your own data augmentation functions for tf.data.Dataset, you must explicitly tell TPU what the output dimension is. For example in my notebook here which performs rotation augmentation, the last line of the transform function is reshape
def transform(image, DIM=256):\n # CODE HERE TO ROTATE\n return tf.reshape(d,[DIM, DIM,3])\nNote the last line returns tf.reshape which explicitly tells TPU was output dimension is. This is not needed for GPU but I found it is needed for TPU.
@cdeotte I am usingtf.reshape but still facing the same issue...
I will follow up on @cdeotte 's solution for everybody googling this issue in the future. If using tf.reshape() doesn't solve it for you, try adding drop_remainder=True to dataset.batch() function. Otherwise, batch_size is dynamic, and so is the shape of input.
", "votes": null, "replies": [] }, { "id": 962780, "author_name": "kishor1210", "author_url": "", "post_date": "08/08/2020 12:40:47", "content": "Hi @ashoksrinivas can you check this [https://stackoverflow.com/questions/57658114/how-to-solve-propagation-of-dynamic-dimension-failed-error-in-tf-keras-with-tp]
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "962727": "I am trying to encode the data of mine where my input and output dimensions are \"( 512, 512, 3)\" and my loss is \"MSE\". when i am trying to train my model i am facing TPU compilation error. \n\n```\nUnimplementedError: {{function_node __inference_train_function_50515}} Compilation failure: CustomCall is not supported to have a dynamic dimension\n\tTPU compilation failed\n\t [[{{node tpu_compile_succeeded_assert/_8679107639378603097/_5}}]]\n\n```\nAny suggestions?", "962780": "Hi @ashoksrinivas can you check this [https://stackoverflow.com/questions/57658114/how-to-solve-propagation-of-dynamic-dimension-failed-error-in-tf-keras-with-tp]", "962799": "I have received this error before. If you are writing your own data augmentation functions for `tf.data.Dataset`, you must explicitly tell TPU what the output dimension is. For example in my notebook [here][1] which performs rotation augmentation, the last line of the transform function is reshape\n\n def transform(image, DIM=256):\n # CODE HERE TO ROTATE\n return tf.reshape(d,[DIM, DIM,3])\n\nNote the last line returns `tf.reshape` which explicitly tells TPU was output dimension is. This is not needed for GPU but I found it is needed for TPU.\n\n[1]: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords", "962911": "cdeotte I am using` tf.reshape` but still facing the same issue...", "1060656": "I will follow up on @cdeotte 's solution for everybody googling this issue in the future. If using tf.reshape() doesn't solve it for you, try adding ***drop_remainder=True*** to dataset.batch() function. Otherwise, batch_size is dynamic, and so is the shape of input." }, "source": "meta" }