{
  "id": 174856,
  "title": "how to train delf model with tpu?",
  "url": "/competitions/landmark-retrieval-2020/discussion/174856",
  "author_name": "",
  "post_date": "2020-08-15T19:43:40.468660900Z",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I tried to train the delf model following <a href=\"https://www.kaggle.com/c/landmark-retrieval-2020/discussion/163350\" target=\"_blank\">this</a> guide, with two change.</p>\n<ol>\n<li>replace MirroredStrategy to TPUStrategy</li>\n<li>remove all tf.summary lines.<br>\nbut encountered the following error, i am new to tpu and tensorflow, does anyone have the same problem?   </li>\n</ol>\n<pre><code>tensorflow.python.framework.errors_impl.InvalidArgumentError: 9 root error(s) found.\n  (0) Invalid argument: {{function_node __inference_distributed_train_step_43711}} Compilation failure: Input 1 to node `backbone/Mean` with op Mean must be a compile-time constant.\n\nXLA compilation requires that operator arguments that represent shapes or dimensions be evaluated to concrete values at compile time. This error means that a shape or dimension argument could not be evaluated at compile time, usually because the value of the argument depends on a parameter to the computation, on a variable, or on a stateful operation such as a random number generator.\n     [[{{node backbone/Mean}}]]\n    TPU compilation failed\n     [[tpu_compile_succeeded_assert/_12534948015575227521/_5]]\n     [[tpu_compile_succeeded_assert/_12534948015575227521/_5/_59]]\n</code></pre>",
  "messages": [
    {
      "id": "971674",
      "postDate": "08/15/2020 19:43:40",
      "content": "<p>I tried to train the delf model following <a href=\"https://www.kaggle.com/c/landmark-retrieval-2020/discussion/163350\" target=\"_blank\">this</a> guide, with two change.</p>\n<ol>\n<li>replace MirroredStrategy to TPUStrategy</li>\n<li>remove all tf.summary lines.<br>\nbut encountered the following error, i am new to tpu and tensorflow, does anyone have the same problem?   </li>\n</ol>\n<pre><code>tensorflow.python.framework.errors_impl.InvalidArgumentError: 9 root error(s) found.\n  (0) Invalid argument: {{function_node __inference_distributed_train_step_43711}} Compilation failure: Input 1 to node `backbone/Mean` with op Mean must be a compile-time constant.\n\nXLA compilation requires that operator arguments that represent shapes or dimensions be evaluated to concrete values at compile time. This error means that a shape or dimension argument could not be evaluated at compile time, usually because the value of the argument depends on a parameter to the computation, on a variable, or on a stateful operation such as a random number generator.\n     [[{{node backbone/Mean}}]]\n    TPU compilation failed\n     [[tpu_compile_succeeded_assert/_12534948015575227521/_5]]\n     [[tpu_compile_succeeded_assert/_12534948015575227521/_5/_59]]\n</code></pre>",
      "rawMarkdown": "I tried to train the delf model following [this](https://www.kaggle.com/c/landmark-retrieval-2020/discussion/163350) guide, with two change.\n1. replace MirroredStrategy to TPUStrategy\n2. remove all tf.summary lines.\nbut encountered the following error, i am new to tpu and tensorflow, does anyone have the same problem?   \n\n```\ntensorflow.python.framework.errors_impl.InvalidArgumentError: 9 root error(s) found.\n  (0) Invalid argument: {{function_node __inference_distributed_train_step_43711}} Compilation failure: Input 1 to node `backbone/Mean` with op Mean must be a compile-time constant.\n\nXLA compilation requires that operator arguments that represent shapes or dimensions be evaluated to concrete values at compile time. This error means that a shape or dimension argument could not be evaluated at compile time, usually because the value of the argument depends on a parameter to the computation, on a variable, or on a stateful operation such as a random number generator.\n\t [[{{node backbone/Mean}}]]\n\tTPU compilation failed\n\t [[tpu_compile_succeeded_assert/_12534948015575227521/_5]]\n\t [[tpu_compile_succeeded_assert/_12534948015575227521/_5/_59]]\n```",
      "votes": null
    },
    {
      "id": "972110",
      "postDate": "08/16/2020 08:42:52",
      "content": "<p>I believe you're getting this error because you're not setting the shape of image tensors in your parsing function.</p>\n<pre><code>...\nimage = tf.image.resize(image, [image_size, image_size])\nimage.set_shape([image_size, image_size, 3]) \n...\n</code></pre>",
      "rawMarkdown": "I believe you're getting this error because you're not setting the shape of image tensors in your parsing function.\n```\n...\nimage = tf.image.resize(image, [image_size, image_size])\nimage.set_shape([image_size, image_size, 3]) \n...\n```",
      "votes": null
    },
    {
      "id": "972482",
      "postDate": "08/16/2020 15:20:59",
      "content": "<p>Finally, I replaced tf op Mean with keras layer.</p>",
      "rawMarkdown": "Finally, I replaced tf op Mean with keras layer.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 972110,
      "author_name": "doncalculator",
      "author_url": "",
      "post_date": "08/16/2020 08:42:52",
      "content": "<p>I believe you're getting this error because you're not setting the shape of image tensors in your parsing function.</p>\n<pre><code>...\nimage = tf.image.resize(image, [image_size, image_size])\nimage.set_shape([image_size, image_size, 3]) \n...\n</code></pre>",
      "votes": null,
      "replies": [
        {
          "id": 972482,
          "author_name": "feiwofeifeixiaowo",
          "author_url": "",
          "post_date": "08/16/2020 15:20:59",
          "content": "<p>Finally, I replaced tf op Mean with keras layer.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "971674": "I tried to train the delf model following [this](https://www.kaggle.com/c/landmark-retrieval-2020/discussion/163350) guide, with two change.\n1. replace MirroredStrategy to TPUStrategy\n2. remove all tf.summary lines.\nbut encountered the following error, i am new to tpu and tensorflow, does anyone have the same problem?   \n\n```\ntensorflow.python.framework.errors_impl.InvalidArgumentError: 9 root error(s) found.\n  (0) Invalid argument: {{function_node __inference_distributed_train_step_43711}} Compilation failure: Input 1 to node `backbone/Mean` with op Mean must be a compile-time constant.\n\nXLA compilation requires that operator arguments that represent shapes or dimensions be evaluated to concrete values at compile time. This error means that a shape or dimension argument could not be evaluated at compile time, usually because the value of the argument depends on a parameter to the computation, on a variable, or on a stateful operation such as a random number generator.\n\t [[{{node backbone/Mean}}]]\n\tTPU compilation failed\n\t [[tpu_compile_succeeded_assert/_12534948015575227521/_5]]\n\t [[tpu_compile_succeeded_assert/_12534948015575227521/_5/_59]]\n```",
    "972110": "I believe you're getting this error because you're not setting the shape of image tensors in your parsing function.\n```\n...\nimage = tf.image.resize(image, [image_size, image_size])\nimage.set_shape([image_size, image_size, 3]) \n...\n```",
    "972482": "Finally, I replaced tf op Mean with keras layer."
  },
  "source": "meta"
}