{
  "id": 310243,
  "title": "Using TPU with Swin backbone issue",
  "url": "/competitions/happy-whale-and-dolphin/discussion/310243",
  "author_name": "",
  "post_date": "2022-02-28T10:35:25.189273300Z",
  "votes": 7,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi Kagglers,<br>\nI'm trying to change the backbone in this <a href=\"https://www.kaggle.com/lextoumbourou\" target=\"_blank\">@lextoumbourou</a> <a href=\"https://www.kaggle.com/lextoumbourou/happywhale-effnet-b6-fork-with-detic-crop\" target=\"_blank\">Effnet B6 fork with Detic crop</a> notebook from EffNet to Swin (<a href=\"https://github.com/rishigami/Swin-Transformer-TF)\" target=\"_blank\">https://github.com/rishigami/Swin-Transformer-TF)</a>. During the training, I have to add the parameter <code>drop_remainder=True</code> when batching in <code>get_dataset</code> functions; if not, the process would raise this kind of error:</p>\n<blockquote>\n  <p>/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)<br>\n       65     except Exception as e:  # pylint: disable=broad-except<br>\n       66       filtered_tb = _process_traceback_frames(e.<strong>traceback</strong>)<br>\n  ---&gt; 67       raise e.with_traceback(filtered_tb) from None<br>\n       68     finally:<br>\n       69       del filtered_tb</p>\n</blockquote>\n<p>/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py in _numpy(self)<br>\n   1189       return self._numpy_internal()<br>\n   1190     except core._NotOkStatusException as e:  # pylint: disable=protected-access<br>\n-&gt; 1191       raise core._status_to_exception(e) from None  # pylint: disable=protected-access<br>\n   1192 <br>\n   1193   <a href=\"https://www.kaggle.com/property\" target=\"_blank\">@property</a><br>\nInvalidArgumentError: 9 root error(s) found.<br>\n  (0) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. <br>\n     [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]<br>\n  (1) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. <br>\n     [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]<br>\n     [[tpu_compile_succeeded_assert/_15410396513388965835/_6/_243]]<br>\n  (2) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. <br>\n     [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]<br>\n     [[tpu_compile_succeeded_assert/_15410396513388965835/_6/_215]]<br>\n  (3) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. <br>\n     [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]<br>\n     [[tpu_compile_succeeded_assert/_15410396513388965835/_6/_187]]<br>\n  (4) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. <br>\n     [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]<br>\n     [[tpu_compile_succeeded_assert/_154103965133889 … [truncated]</p>\n<p>It's ok for the training/evaluation step. The problem is that, when doing inference, the number of test images will not be enough because of the dropping. I have tried removing <code>drop_remainder=True</code> or setting the batch size into 1 in the <code>get_test_dataset</code> function, but none of them works (raise the above error)</p>\n<p>Note: I'm using Colab TPU<br>\nCould anyone help me out, please? Thanks in advance</p>",
  "messages": [
    {
      "id": "1707328",
      "postDate": "02/28/2022 10:35:25",
      "content": "<p>Hi Kagglers,<br>\nI'm trying to change the backbone in this <a href=\"https://www.kaggle.com/lextoumbourou\" target=\"_blank\">@lextoumbourou</a> <a href=\"https://www.kaggle.com/lextoumbourou/happywhale-effnet-b6-fork-with-detic-crop\" target=\"_blank\">Effnet B6 fork with Detic crop</a> notebook from EffNet to Swin (<a href=\"https://github.com/rishigami/Swin-Transformer-TF)\" target=\"_blank\">https://github.com/rishigami/Swin-Transformer-TF)</a>. During the training, I have to add the parameter <code>drop_remainder=True</code> when batching in <code>get_dataset</code> functions; if not, the process would raise this kind of error:</p>\n<blockquote>\n  <p>/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)<br>\n       65     except Exception as e:  # pylint: disable=broad-except<br>\n       66       filtered_tb = _process_traceback_frames(e.<strong>traceback</strong>)<br>\n  ---&gt; 67       raise e.with_traceback(filtered_tb) from None<br>\n       68     finally:<br>\n       69       del filtered_tb</p>\n</blockquote>\n<p>/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py in _numpy(self)<br>\n   1189       return self._numpy_internal()<br>\n   1190     except core._NotOkStatusException as e:  # pylint: disable=protected-access<br>\n-&gt; 1191       raise core._status_to_exception(e) from None  # pylint: disable=protected-access<br>\n   1192 <br>\n   1193   <a href=\"https://www.kaggle.com/property\" target=\"_blank\">@property</a><br>\nInvalidArgumentError: 9 root error(s) found.<br>\n  (0) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. <br>\n     [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]<br>\n  (1) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. <br>\n     [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]<br>\n     [[tpu_compile_succeeded_assert/_15410396513388965835/_6/_243]]<br>\n  (2) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. <br>\n     [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]<br>\n     [[tpu_compile_succeeded_assert/_15410396513388965835/_6/_215]]<br>\n  (3) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. <br>\n     [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]<br>\n     [[tpu_compile_succeeded_assert/_15410396513388965835/_6/_187]]<br>\n  (4) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. <br>\n     [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]<br>\n     [[tpu_compile_succeeded_assert/_154103965133889 … [truncated]</p>\n<p>It's ok for the training/evaluation step. The problem is that, when doing inference, the number of test images will not be enough because of the dropping. I have tried removing <code>drop_remainder=True</code> or setting the batch size into 1 in the <code>get_test_dataset</code> function, but none of them works (raise the above error)</p>\n<p>Note: I'm using Colab TPU<br>\nCould anyone help me out, please? Thanks in advance</p>",
      "rawMarkdown": "Hi Kagglers,\nI'm trying to change the backbone in this @lextoumbourou [Effnet B6 fork with Detic crop](https://www.kaggle.com/lextoumbourou/happywhale-effnet-b6-fork-with-detic-crop) notebook from EffNet to Swin (https://github.com/rishigami/Swin-Transformer-TF). During the training, I have to add the parameter `drop_remainder=True` when batching in `get_dataset` functions; if not, the process would raise this kind of error:\n> /usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)\n     65     except Exception as e:  # pylint: disable=broad-except\n     66       filtered_tb = _process_traceback_frames(e.__traceback__)\n---> 67       raise e.with_traceback(filtered_tb) from None\n     68     finally:\n     69       del filtered_tb\n\n/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py in _numpy(self)\n   1189       return self._numpy_internal()\n   1190     except core._NotOkStatusException as e:  # pylint: disable=protected-access\n-> 1191       raise core._status_to_exception(e) from None  # pylint: disable=protected-access\n   1192 \n   1193   @property\nInvalidArgumentError: 9 root error(s) found.\n  (0) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. \n\t [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]\n  (1) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. \n\t [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]\n\t [[tpu_compile_succeeded_assert/_15410396513388965835/_6/_243]]\n  (2) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. \n\t [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]\n\t [[tpu_compile_succeeded_assert/_15410396513388965835/_6/_215]]\n  (3) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. \n\t [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]\n\t [[tpu_compile_succeeded_assert/_15410396513388965835/_6/_187]]\n  (4) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. \n\t [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]\n\t [[tpu_compile_succeeded_assert/_154103965133889 ... [truncated]\n\nIt's ok for the training/evaluation step. The problem is that, when doing inference, the number of test images will not be enough because of the dropping. I have tried removing `drop_remainder=True` or setting the batch size into 1 in the `get_test_dataset` function, but none of them works (raise the above error)\n\nNote: I'm using Colab TPU\nCould anyone help me out, please? Thanks in advance",
      "votes": null
    },
    {
      "id": "1710309",
      "postDate": "03/02/2022 23:28:18",
      "content": "<p>usually drop_remainder is only used for the training ds - so do it differently with val and test</p>",
      "rawMarkdown": "usually drop_remainder is only used for the training ds - so do it differently with val and test",
      "votes": null
    },
    {
      "id": "1735782",
      "postDate": "03/26/2022 15:51:47",
      "content": "<p><a href=\"https://www.kaggle.com/minhtu123\" target=\"_blank\">@minhtu123</a>. Thank you for posting this issue. I encountered the same problem. Have you fixed it?</p>",
      "rawMarkdown": "minhtu123. Thank you for posting this issue. I encountered the same problem. Have you fixed it?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1710309,
      "author_name": "romanweilguny",
      "author_url": "",
      "post_date": "03/02/2022 23:28:18",
      "content": "<p>usually drop_remainder is only used for the training ds - so do it differently with val and test</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1735782,
      "author_name": "niuniu3412",
      "author_url": "",
      "post_date": "03/26/2022 15:51:47",
      "content": "<p><a href=\"https://www.kaggle.com/minhtu123\" target=\"_blank\">@minhtu123</a>. Thank you for posting this issue. I encountered the same problem. Have you fixed it?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1707328": "Hi Kagglers,\nI'm trying to change the backbone in this @lextoumbourou [Effnet B6 fork with Detic crop](https://www.kaggle.com/lextoumbourou/happywhale-effnet-b6-fork-with-detic-crop) notebook from EffNet to Swin (https://github.com/rishigami/Swin-Transformer-TF). During the training, I have to add the parameter `drop_remainder=True` when batching in `get_dataset` functions; if not, the process would raise this kind of error:\n> /usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)\n     65     except Exception as e:  # pylint: disable=broad-except\n     66       filtered_tb = _process_traceback_frames(e.__traceback__)\n---> 67       raise e.with_traceback(filtered_tb) from None\n     68     finally:\n     69       del filtered_tb\n\n/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py in _numpy(self)\n   1189       return self._numpy_internal()\n   1190     except core._NotOkStatusException as e:  # pylint: disable=protected-access\n-> 1191       raise core._status_to_exception(e) from None  # pylint: disable=protected-access\n   1192 \n   1193   @property\nInvalidArgumentError: 9 root error(s) found.\n  (0) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. \n\t [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]\n  (1) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. \n\t [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]\n\t [[tpu_compile_succeeded_assert/_15410396513388965835/_6/_243]]\n  (2) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. \n\t [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]\n\t [[tpu_compile_succeeded_assert/_15410396513388965835/_6/_215]]\n  (3) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. \n\t [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]\n\t [[tpu_compile_succeeded_assert/_15410396513388965835/_6/_187]]\n  (4) INVALID_ARGUMENT: {{function_node __inference_train_function_60913}} Reshape's input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can't infer the output dimension %reshape.8171 = f32[1024,49,288]{2,1,0} reshape(f32[50176,288]{1,0} %transpose.8163), metadata={op_type=\"Reshape\" op_name=\"model/swin_tiny_224/sequential_4/basic_layer/sequential/swin_transformer_block_1/window_attention_1/layers0/blocks1/attn/qkv/Tensordot\"}. \n\t [[{{node TPUReplicate/_compile/_14828560742528468134/_5}}]]\n\t [[tpu_compile_succeeded_assert/_154103965133889 ... [truncated]\n\nIt's ok for the training/evaluation step. The problem is that, when doing inference, the number of test images will not be enough because of the dropping. I have tried removing `drop_remainder=True` or setting the batch size into 1 in the `get_test_dataset` function, but none of them works (raise the above error)\n\nNote: I'm using Colab TPU\nCould anyone help me out, please? Thanks in advance",
    "1710309": "usually drop_remainder is only used for the training ds - so do it differently with val and test",
    "1735782": "minhtu123. Thank you for posting this issue. I encountered the same problem. Have you fixed it?"
  },
  "source": "meta"
}