While working with TPU I come across this error.
\nUnimplementedError: {{function_node __inference_train_function_53064}} Compilation failure: CustomCall is not supported to have a dynamic dimension TPU compilation failed [[{{node tpu_compile_succeeded_assert/_12959906650459098508/_6}}]]
After a bit of research, I found this discussion here. This problem occurs when we use UpSampling2D() while working with Images and masks and not tfrecords since Marcos Novaes used tfrecord and UpSampling2D() in his notebook here which works perfectly fine with TPU.
\nSo when I tried to work with segmentation_models by Pavel Yakubovskiy. This throws the above error again and the same occurred when I tried to use the model from this notebook. I guess the problem may not occur if we use tfrecords. I hope this helps.
", "messages": [ { "id": "1098492", "postDate": "12/01/2020 17:08:52", "content": "While working with TPU I come across this error.
\nUnimplementedError: {{function_node __inference_train_function_53064}} Compilation failure: CustomCall is not supported to have a dynamic dimension TPU compilation failed [[{{node tpu_compile_succeeded_assert/_12959906650459098508/_6}}]]
After a bit of research, I found this discussion here. This problem occurs when we use UpSampling2D() while working with Images and masks and not tfrecords since Marcos Novaes used tfrecord and UpSampling2D() in his notebook here which works perfectly fine with TPU.
\nSo when I tried to work with segmentation_models by Pavel Yakubovskiy. This throws the above error again and the same occurred when I tried to use the model from this notebook. I guess the problem may not occur if we use tfrecords. I hope this helps.
", "rawMarkdown": "While working with TPU I come across this error. \n\n`UnimplementedError: {{function_node __inference_train_function_53064}} Compilation failure: CustomCall is not supported to have a dynamic dimension TPU compilation failed [[{{node tpu_compile_succeeded_assert/_12959906650459098508/_6}}]]`\n\nAfter a bit of research, I found this [discussion](https://github.com/tensorflow/tensorflow/issues/35708) here. This problem occurs when we use UpSampling2D() while working with Images and masks and **not tfrecords** since Marcos Novaes used tfrecord and UpSampling2D() in his [notebook](https://www.kaggle.com/marcosnovaes/hubmap-unet-keras-model-fit-with-tpu/notebook) here which works perfectly fine with TPU.\n\nSo when I tried to work with [segmentation_models](https://github.com/qubvel/segmentation_models) by Pavel Yakubovskiy. This throws the above error again and the same occurred when I tried to use the model from this [notebook](https://www.kaggle.com/mistag/train-keras-u-net-with-tfrecords-input). I guess the problem may not occur if we use **tfrecords**. I hope this helps.", "votes": null }, { "id": "1098523", "postDate": "12/01/2020 17:27:43", "content": "Hello @kool777:
\nI did encounter this error before, as did other competitors. The reason is that TPU compilation does not support \"dynamic\" shapes for some layers. You will notice in my TFRecord code that I use LITERALS when I define the shape of the tiles I am using (ex. TILE_SIZE = 512). After reading the TFRecord, I explicitly call tf.reshape and pass the specific dimensions as LITERALS. That way the matrix shape is \"fixed\" as opposed to dynamic, and everything works. In my first notebook (read data and build TFRecords) I had this problem, as I was determining size dynamically, ex:
\nimg_array = tf.reshape( img_bytes, (img_height, img_width, num_channels))
\nIn my TPU notebook, you will noticed I changed it to use literals for this reason:
\nimg_array = tf.reshape( img_bytes, (512, 512, 3))
\nI am about to publish another TFRecord notebook has a little more elegant solution passing a literal as a parameter, but in the end the shape needs to resolve to a literal at TPU compile time. Hope this helps!!
\nCheers,
\n--Marcos
", "rawMarkdown": "Hello @kool777:\n\nI did encounter this error before, as did other competitors. The reason is that TPU compilation does not support \"dynamic\" shapes for some layers. You will notice in my TFRecord code that I use LITERALS when I define the shape of the tiles I am using (ex. TILE_SIZE = 512). After reading the TFRecord, I explicitly call tf.reshape and pass the specific dimensions as LITERALS. That way the matrix shape is \"fixed\" as opposed to dynamic, and everything works. In my first notebook (read data and build TFRecords) I had this problem, as I was determining size dynamically, ex:\n\nimg_array = tf.reshape( img_bytes, (img_height, img_width, num_channels))\n\nIn my TPU notebook, you will noticed I changed it to use literals for this reason:\n\nimg_array = tf.reshape( img_bytes, (512, 512, 3))\n\nI am about to publish another TFRecord notebook has a little more elegant solution passing a literal as a parameter, but in the end the shape needs to resolve to a literal at TPU compile time. Hope this helps!! \n\nCheers,\n\n--Marcos", "votes": null }, { "id": "1099630", "postDate": "12/02/2020 13:45:41", "content": "Thank you very much for the reply.
\nThe reason is that TPU compilation does not support \"dynamic\" shapes for some layers. I think I found the solution to use qubvel/segmentation_models with TPU. After checking the source code I come to know that we can pass decoder_block_type == 'transpose' argument inside the
Unet('seresnet18', input_shape=(256, 256, 3), encoder_weights=None, decoder_block_type='transpose') function which will use Conv2DTranspose
\n instead of UpSampling2D which solved the compilation problem.
I guess now we can run experiments with various segmentation models using TPU with fewer efforts since qubvel/segmentation_models is working fine with tf-Keras on TPU.
\nHi, I was facing the same issue and I was able to solve it by using the drop_remainder parameter in dataset.batch(BATCH_SIZE, drop_remainder=True) in the data pipeline
", "rawMarkdown": "Hi, I was facing the same issue and I was able to solve it by using the **drop_remainder** parameter in dataset.batch(BATCH_SIZE, drop_remainder=True) in the data pipeline", "votes": null } ], "comments": [ { "id": 1098523, "author_name": "marcosnovaes", "author_url": "", "post_date": "12/01/2020 17:27:43", "content": "Hello @kool777:
\nI did encounter this error before, as did other competitors. The reason is that TPU compilation does not support \"dynamic\" shapes for some layers. You will notice in my TFRecord code that I use LITERALS when I define the shape of the tiles I am using (ex. TILE_SIZE = 512). After reading the TFRecord, I explicitly call tf.reshape and pass the specific dimensions as LITERALS. That way the matrix shape is \"fixed\" as opposed to dynamic, and everything works. In my first notebook (read data and build TFRecords) I had this problem, as I was determining size dynamically, ex:
\nimg_array = tf.reshape( img_bytes, (img_height, img_width, num_channels))
\nIn my TPU notebook, you will noticed I changed it to use literals for this reason:
\nimg_array = tf.reshape( img_bytes, (512, 512, 3))
\nI am about to publish another TFRecord notebook has a little more elegant solution passing a literal as a parameter, but in the end the shape needs to resolve to a literal at TPU compile time. Hope this helps!!
\nCheers,
\n--Marcos
", "votes": null, "replies": [ { "id": 1099630, "author_name": "kool777", "author_url": "", "post_date": "12/02/2020 13:45:41", "content": "Thank you very much for the reply.
\nThe reason is that TPU compilation does not support \"dynamic\" shapes for some layers. I think I found the solution to use qubvel/segmentation_models with TPU. After checking the source code I come to know that we can pass decoder_block_type == 'transpose' argument inside the
Unet('seresnet18', input_shape=(256, 256, 3), encoder_weights=None, decoder_block_type='transpose') function which will use Conv2DTranspose
\n instead of UpSampling2D which solved the compilation problem.
I guess now we can run experiments with various segmentation models using TPU with fewer efforts since qubvel/segmentation_models is working fine with tf-Keras on TPU.
\nHi, I was facing the same issue and I was able to solve it by using the drop_remainder parameter in dataset.batch(BATCH_SIZE, drop_remainder=True) in the data pipeline
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1098492": "While working with TPU I come across this error. \n\n`UnimplementedError: {{function_node __inference_train_function_53064}} Compilation failure: CustomCall is not supported to have a dynamic dimension TPU compilation failed [[{{node tpu_compile_succeeded_assert/_12959906650459098508/_6}}]]`\n\nAfter a bit of research, I found this [discussion](https://github.com/tensorflow/tensorflow/issues/35708) here. This problem occurs when we use UpSampling2D() while working with Images and masks and **not tfrecords** since Marcos Novaes used tfrecord and UpSampling2D() in his [notebook](https://www.kaggle.com/marcosnovaes/hubmap-unet-keras-model-fit-with-tpu/notebook) here which works perfectly fine with TPU.\n\nSo when I tried to work with [segmentation_models](https://github.com/qubvel/segmentation_models) by Pavel Yakubovskiy. This throws the above error again and the same occurred when I tried to use the model from this [notebook](https://www.kaggle.com/mistag/train-keras-u-net-with-tfrecords-input). I guess the problem may not occur if we use **tfrecords**. I hope this helps.", "1098523": "Hello @kool777:\n\nI did encounter this error before, as did other competitors. The reason is that TPU compilation does not support \"dynamic\" shapes for some layers. You will notice in my TFRecord code that I use LITERALS when I define the shape of the tiles I am using (ex. TILE_SIZE = 512). After reading the TFRecord, I explicitly call tf.reshape and pass the specific dimensions as LITERALS. That way the matrix shape is \"fixed\" as opposed to dynamic, and everything works. In my first notebook (read data and build TFRecords) I had this problem, as I was determining size dynamically, ex:\n\nimg_array = tf.reshape( img_bytes, (img_height, img_width, num_channels))\n\nIn my TPU notebook, you will noticed I changed it to use literals for this reason:\n\nimg_array = tf.reshape( img_bytes, (512, 512, 3))\n\nI am about to publish another TFRecord notebook has a little more elegant solution passing a literal as a parameter, but in the end the shape needs to resolve to a literal at TPU compile time. Hope this helps!! \n\nCheers,\n\n--Marcos", "1099630": "Thank you very much for the reply. \n\n**The reason is that TPU compilation does not support \"dynamic\" shapes for some layers.** I think I found the solution to use [qubvel/segmentation_models](https://github.com/qubvel/segmentation_models) with TPU. After checking the source code I come to know that we can pass `decoder_block_type == 'transpose'` argument inside the \n\n`Unet('seresnet18', input_shape=(256, 256, 3), encoder_weights=None, decoder_block_type='transpose')` function which will use **Conv2DTranspose**\n instead of **UpSampling2D** which solved the compilation problem. \n\nI guess now we can run experiments with various segmentation models using TPU with fewer efforts since [qubvel/segmentation_models](https://github.com/qubvel/segmentation_models) is working fine with tf-Keras on TPU. \n\n###[NOTE] - This is not the problem with tfrecords.", "1103023": "Hi, I was facing the same issue and I was able to solve it by using the **drop_remainder** parameter in dataset.batch(BATCH_SIZE, drop_remainder=True) in the data pipeline" }, "source": "meta" }