{
  "id": 134472,
  "title": "Multi-Output on TPUs",
  "url": "/competitions/bengaliai-cv19/discussion/134472",
  "author_name": "",
  "post_date": "2020-03-08T11:25:01.753034400Z",
  "votes": 8,
  "comment_count": 8,
  "views": 0,
  "content": "<p>I have started from the great work by @seesee (<a href=\"https://www.kaggle.com/c/bengaliai-cv19/discussion/134161\">post</a> and <a href=\"https://www.kaggle.com/seesee/2-train\">kernel</a>) and then extended the efficientnet model to have multiple outputs. \nThe kernel runs fine with GPU acceleration, but resets when ran on the TPU with no errors. If you commit it you get the log I have attached, which hasn't helped me debug this myself.</p>\n\n<p>@mgornergoogle I know you have been helping with the \"Flower Classification with TPUs\" competition, do you know if multi-output networks are supported on TPUs?</p>\n\n<p>More generally has anyone managed to get multi-output networks to work on the kaggle TPUs?</p>",
  "messages": [
    {
      "id": "766560",
      "postDate": "03/08/2020 11:25:01",
      "content": "<p>I have started from the great work by @seesee (<a href=\"https://www.kaggle.com/c/bengaliai-cv19/discussion/134161\">post</a> and <a href=\"https://www.kaggle.com/seesee/2-train\">kernel</a>) and then extended the efficientnet model to have multiple outputs. \nThe kernel runs fine with GPU acceleration, but resets when ran on the TPU with no errors. If you commit it you get the log I have attached, which hasn't helped me debug this myself.</p>\n\n<p>@mgornergoogle I know you have been helping with the \"Flower Classification with TPUs\" competition, do you know if multi-output networks are supported on TPUs?</p>\n\n<p>More generally has anyone managed to get multi-output networks to work on the kaggle TPUs?</p>",
      "rawMarkdown": "I have started from the great work by @seesee ([post](https://www.kaggle.com/c/bengaliai-cv19/discussion/134161) and [kernel](https://www.kaggle.com/seesee/2-train)) and then extended the efficientnet model to have multiple outputs. \nThe kernel runs fine with GPU acceleration, but resets when ran on the TPU with no errors. If you commit it you get the log I have attached, which hasn't helped me debug this myself.\n\n@mgornergoogle I know you have been helping with the \"Flower Classification with TPUs\" competition, do you know if multi-output networks are supported on TPUs?\n\nMore generally has anyone managed to get multi-output networks to work on the kaggle TPUs?",
      "votes": null
    },
    {
      "id": "766589",
      "postDate": "03/08/2020 12:23:33",
      "content": "<p>Mine is okay</p>\n\n<p>This is how I decode the tfrecord example\n```\ndef decode_example(self, example):\n        features = {\n            'img' : tf.io.FixedLenFeature([], tf.string),\n            'grapheme_root' : tf.io.FixedLenFeature([], tf.int64),\n            'vowel_diacritic' : tf.io.FixedLenFeature([], tf.int64),\n            'consonant_diacritic' : tf.io.FixedLenFeature([], tf.int64)\n        }</p>\n\n<pre><code>    example = tf.io.parse_single_example(example, features)\n    image = tf.image.decode_image(example['img'])\n    image = tf.reshape(image, self.image_shape)\n    image = tf.expand_dims(image, axis=-1)\n    image = tf.cast(image, tf.float32)\n    image = tf.repeat(image, 3, -1)\n    image -= tf.constant([0.485 * 255, 0.456 * 255, 0.406 * 255])  \n    image /= tf.constant([0.229 * 255, 0.224 * 255, 0.225 * 255])  \n    g_label = tf.cast(example['grapheme_root'], tf.int32)\n    v_label = tf.cast(example['vowel_diacritic'], tf.int32)\n    c_label = tf.cast(example['consonant_diacritic'], tf.int32)\n    g_label = tf.one_hot(g_label, 168)\n    v_label = tf.one_hot(v_label, 11)\n    c_label = tf.one_hot(c_label, 7)\n    return image, {'g_output' : g_label, 'v_output' : v_label, 'c_output' : c_label}\n</code></pre>\n\n<p>```</p>\n\n<p>And this is how I build the model:\n```\nwith strategy.scope():\n    base_model = efn.EfficientNetB0(weights='noisy-student', input_shape=(*IMG_DIM ,3), include_top=False)\n    Input = tf.keras.layers.Input(shape=(*IMG_DIM,3), dtype=tf.float32)\n    x = base_model(Input)\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)</p>\n\n<pre><code>g_output = tf.keras.layers.Dense(units=168, activation='softmax', name='g_output')(x)\nv_output = tf.keras.layers.Dense(units=11, activation='softmax', name='v_output')(x)\nc_output = tf.keras.layers.Dense(units=7, activation='softmax', name='c_output')(x)\n\nmodel = tf.keras.models.Model(inputs=[Input], outputs=[g_output, v_output, c_output])\nadam = tf.keras.optimizers.Adam(lr = LEARNING_RATE)\nmodel.compile(optimizer=adam,\n              loss=categorical_crossentropy,\n              metrics=[categorical_accuracy])\nmodel.summary()\n</code></pre>\n\n<p>```</p>",
      "rawMarkdown": "Mine is okay\n\nThis is how I decode the tfrecord example\n```\ndef decode_example(self, example):\n        features = {\n            'img' : tf.io.FixedLenFeature([], tf.string),\n            'grapheme_root' : tf.io.FixedLenFeature([], tf.int64),\n            'vowel_diacritic' : tf.io.FixedLenFeature([], tf.int64),\n            'consonant_diacritic' : tf.io.FixedLenFeature([], tf.int64)\n        }\n        \n        example = tf.io.parse_single_example(example, features)\n        image = tf.image.decode_image(example['img'])\n        image = tf.reshape(image, self.image_shape)\n        image = tf.expand_dims(image, axis=-1)\n        image = tf.cast(image, tf.float32)\n        image = tf.repeat(image, 3, -1)\n        image -= tf.constant([0.485 * 255, 0.456 * 255, 0.406 * 255])  \n        image /= tf.constant([0.229 * 255, 0.224 * 255, 0.225 * 255])  \n        g_label = tf.cast(example['grapheme_root'], tf.int32)\n        v_label = tf.cast(example['vowel_diacritic'], tf.int32)\n        c_label = tf.cast(example['consonant_diacritic'], tf.int32)\n        g_label = tf.one_hot(g_label, 168)\n        v_label = tf.one_hot(v_label, 11)\n        c_label = tf.one_hot(c_label, 7)\n        return image, {'g_output' : g_label, 'v_output' : v_label, 'c_output' : c_label}\n```\n\nAnd this is how I build the model:\n```\nwith strategy.scope():\n    base_model = efn.EfficientNetB0(weights='noisy-student', input_shape=(*IMG_DIM ,3), include_top=False)\n    Input = tf.keras.layers.Input(shape=(*IMG_DIM,3), dtype=tf.float32)\n    x = base_model(Input)\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)\n    \n    g_output = tf.keras.layers.Dense(units=168, activation='softmax', name='g_output')(x)\n    v_output = tf.keras.layers.Dense(units=11, activation='softmax', name='v_output')(x)\n    c_output = tf.keras.layers.Dense(units=7, activation='softmax', name='c_output')(x)\n    \n    model = tf.keras.models.Model(inputs=[Input], outputs=[g_output, v_output, c_output])\n    adam = tf.keras.optimizers.Adam(lr = LEARNING_RATE)\n    model.compile(optimizer=adam,\n                  loss=categorical_crossentropy,\n                  metrics=[categorical_accuracy])\n    model.summary()\n```",
      "votes": null
    },
    {
      "id": "766676",
      "postDate": "03/08/2020 14:49:08",
      "content": "<p>Thank you for sharing. \nMy code is quite similar to yours, especially the loading of the examples so it didn't take long to eliminate that part. The way I setup model was similar, but I was trying to use a different pooling and that turned out to be the issue. By switching to the GlobalAveragePooling2D you have used fixed it. 👍 </p>",
      "rawMarkdown": "Thank you for sharing. \nMy code is quite similar to yours, especially the loading of the examples so it didn't take long to eliminate that part. The way I setup model was similar, but I was trying to use a different pooling and that turned out to be the issue. By switching to the GlobalAveragePooling2D you have used fixed it. 👍",
      "votes": null
    },
    {
      "id": "766724",
      "postDate": "03/08/2020 16:01:26",
      "content": "<p>I experienced same thing, somehow gem didn't work with TPU:(</p>",
      "rawMarkdown": "I experienced same thing, somehow gem didn't work with TPU:(",
      "votes": null
    },
    {
      "id": "767169",
      "postDate": "03/09/2020 08:59:09",
      "content": "<p>I think TPU has a time limit of 3 hours? </p>\n\n<p>Not sure.</p>\n\n<p><a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/discussion/133391\">https://www.kaggle.com/c/flower-classification-with-tpus/discussion/133391</a></p>",
      "rawMarkdown": "I think TPU has a time limit of 3 hours? \n\nNot sure.\n\nhttps://www.kaggle.com/c/flower-classification-with-tpus/discussion/133391",
      "votes": null
    },
    {
      "id": "769163",
      "postDate": "03/11/2020 15:39:29",
      "content": "<p>Are you using gem in your model? i am also facing the same issue because of gem layer..did you find any solution?</p>",
      "rawMarkdown": "Are you using gem in your model? i am also facing the same issue because of gem layer..did you find any solution?",
      "votes": null
    },
    {
      "id": "769742",
      "postDate": "03/12/2020 08:00:33",
      "content": "<p>Hello. I try to finish in less than 3 hours because of the same situation.</p>",
      "rawMarkdown": "Hello. I try to finish in less than 3 hours because of the same situation.",
      "votes": null
    },
    {
      "id": "770428",
      "postDate": "03/12/2020 23:32:11",
      "content": "<p><a href=\"/hermione36\">@hermione36</a> no I removed gem to get it to work.</p>",
      "rawMarkdown": "hermione36 no I removed gem to get it to work.",
      "votes": null
    },
    {
      "id": "770869",
      "postDate": "03/13/2020 13:50:46",
      "content": "<p><a href=\"/bopengiowa\">@bopengiowa</a> Yes TPU time limit is 3 hours. </p>",
      "rawMarkdown": "bopengiowa Yes TPU time limit is 3 hours.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 766589,
      "author_name": "raytsai",
      "author_url": "",
      "post_date": "03/08/2020 12:23:33",
      "content": "<p>Mine is okay</p>\n\n<p>This is how I decode the tfrecord example\n```\ndef decode_example(self, example):\n        features = {\n            'img' : tf.io.FixedLenFeature([], tf.string),\n            'grapheme_root' : tf.io.FixedLenFeature([], tf.int64),\n            'vowel_diacritic' : tf.io.FixedLenFeature([], tf.int64),\n            'consonant_diacritic' : tf.io.FixedLenFeature([], tf.int64)\n        }</p>\n\n<pre><code>    example = tf.io.parse_single_example(example, features)\n    image = tf.image.decode_image(example['img'])\n    image = tf.reshape(image, self.image_shape)\n    image = tf.expand_dims(image, axis=-1)\n    image = tf.cast(image, tf.float32)\n    image = tf.repeat(image, 3, -1)\n    image -= tf.constant([0.485 * 255, 0.456 * 255, 0.406 * 255])  \n    image /= tf.constant([0.229 * 255, 0.224 * 255, 0.225 * 255])  \n    g_label = tf.cast(example['grapheme_root'], tf.int32)\n    v_label = tf.cast(example['vowel_diacritic'], tf.int32)\n    c_label = tf.cast(example['consonant_diacritic'], tf.int32)\n    g_label = tf.one_hot(g_label, 168)\n    v_label = tf.one_hot(v_label, 11)\n    c_label = tf.one_hot(c_label, 7)\n    return image, {'g_output' : g_label, 'v_output' : v_label, 'c_output' : c_label}\n</code></pre>\n\n<p>```</p>\n\n<p>And this is how I build the model:\n```\nwith strategy.scope():\n    base_model = efn.EfficientNetB0(weights='noisy-student', input_shape=(*IMG_DIM ,3), include_top=False)\n    Input = tf.keras.layers.Input(shape=(*IMG_DIM,3), dtype=tf.float32)\n    x = base_model(Input)\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)</p>\n\n<pre><code>g_output = tf.keras.layers.Dense(units=168, activation='softmax', name='g_output')(x)\nv_output = tf.keras.layers.Dense(units=11, activation='softmax', name='v_output')(x)\nc_output = tf.keras.layers.Dense(units=7, activation='softmax', name='c_output')(x)\n\nmodel = tf.keras.models.Model(inputs=[Input], outputs=[g_output, v_output, c_output])\nadam = tf.keras.optimizers.Adam(lr = LEARNING_RATE)\nmodel.compile(optimizer=adam,\n              loss=categorical_crossentropy,\n              metrics=[categorical_accuracy])\nmodel.summary()\n</code></pre>\n\n<p>```</p>",
      "votes": null,
      "replies": [
        {
          "id": 766676,
          "author_name": "ghostskipper",
          "author_url": "",
          "post_date": "03/08/2020 14:49:08",
          "content": "<p>Thank you for sharing. \nMy code is quite similar to yours, especially the loading of the examples so it didn't take long to eliminate that part. The way I setup model was similar, but I was trying to use a different pooling and that turned out to be the issue. By switching to the GlobalAveragePooling2D you have used fixed it. 👍 </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 766724,
          "author_name": "bamps53",
          "author_url": "",
          "post_date": "03/08/2020 16:01:26",
          "content": "<p>I experienced same thing, somehow gem didn't work with TPU:(</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 767169,
          "author_name": "bopengiowa",
          "author_url": "",
          "post_date": "03/09/2020 08:59:09",
          "content": "<p>I think TPU has a time limit of 3 hours? </p>\n\n<p>Not sure.</p>\n\n<p><a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/discussion/133391\">https://www.kaggle.com/c/flower-classification-with-tpus/discussion/133391</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 769163,
          "author_name": "hermione36",
          "author_url": "",
          "post_date": "03/11/2020 15:39:29",
          "content": "<p>Are you using gem in your model? i am also facing the same issue because of gem layer..did you find any solution?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 769742,
          "author_name": "koshirosato",
          "author_url": "",
          "post_date": "03/12/2020 08:00:33",
          "content": "<p>Hello. I try to finish in less than 3 hours because of the same situation.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 770428,
          "author_name": "ghostskipper",
          "author_url": "",
          "post_date": "03/12/2020 23:32:11",
          "content": "<p><a href=\"/hermione36\">@hermione36</a> no I removed gem to get it to work.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 770869,
          "author_name": "vishal1310",
          "author_url": "",
          "post_date": "03/13/2020 13:50:46",
          "content": "<p><a href=\"/bopengiowa\">@bopengiowa</a> Yes TPU time limit is 3 hours. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "766560": "I have started from the great work by @seesee ([post](https://www.kaggle.com/c/bengaliai-cv19/discussion/134161) and [kernel](https://www.kaggle.com/seesee/2-train)) and then extended the efficientnet model to have multiple outputs. \nThe kernel runs fine with GPU acceleration, but resets when ran on the TPU with no errors. If you commit it you get the log I have attached, which hasn't helped me debug this myself.\n\n@mgornergoogle I know you have been helping with the \"Flower Classification with TPUs\" competition, do you know if multi-output networks are supported on TPUs?\n\nMore generally has anyone managed to get multi-output networks to work on the kaggle TPUs?",
    "766589": "Mine is okay\n\nThis is how I decode the tfrecord example\n```\ndef decode_example(self, example):\n        features = {\n            'img' : tf.io.FixedLenFeature([], tf.string),\n            'grapheme_root' : tf.io.FixedLenFeature([], tf.int64),\n            'vowel_diacritic' : tf.io.FixedLenFeature([], tf.int64),\n            'consonant_diacritic' : tf.io.FixedLenFeature([], tf.int64)\n        }\n        \n        example = tf.io.parse_single_example(example, features)\n        image = tf.image.decode_image(example['img'])\n        image = tf.reshape(image, self.image_shape)\n        image = tf.expand_dims(image, axis=-1)\n        image = tf.cast(image, tf.float32)\n        image = tf.repeat(image, 3, -1)\n        image -= tf.constant([0.485 * 255, 0.456 * 255, 0.406 * 255])  \n        image /= tf.constant([0.229 * 255, 0.224 * 255, 0.225 * 255])  \n        g_label = tf.cast(example['grapheme_root'], tf.int32)\n        v_label = tf.cast(example['vowel_diacritic'], tf.int32)\n        c_label = tf.cast(example['consonant_diacritic'], tf.int32)\n        g_label = tf.one_hot(g_label, 168)\n        v_label = tf.one_hot(v_label, 11)\n        c_label = tf.one_hot(c_label, 7)\n        return image, {'g_output' : g_label, 'v_output' : v_label, 'c_output' : c_label}\n```\n\nAnd this is how I build the model:\n```\nwith strategy.scope():\n    base_model = efn.EfficientNetB0(weights='noisy-student', input_shape=(*IMG_DIM ,3), include_top=False)\n    Input = tf.keras.layers.Input(shape=(*IMG_DIM,3), dtype=tf.float32)\n    x = base_model(Input)\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)\n    \n    g_output = tf.keras.layers.Dense(units=168, activation='softmax', name='g_output')(x)\n    v_output = tf.keras.layers.Dense(units=11, activation='softmax', name='v_output')(x)\n    c_output = tf.keras.layers.Dense(units=7, activation='softmax', name='c_output')(x)\n    \n    model = tf.keras.models.Model(inputs=[Input], outputs=[g_output, v_output, c_output])\n    adam = tf.keras.optimizers.Adam(lr = LEARNING_RATE)\n    model.compile(optimizer=adam,\n                  loss=categorical_crossentropy,\n                  metrics=[categorical_accuracy])\n    model.summary()\n```",
    "766676": "Thank you for sharing. \nMy code is quite similar to yours, especially the loading of the examples so it didn't take long to eliminate that part. The way I setup model was similar, but I was trying to use a different pooling and that turned out to be the issue. By switching to the GlobalAveragePooling2D you have used fixed it. 👍",
    "766724": "I experienced same thing, somehow gem didn't work with TPU:(",
    "767169": "I think TPU has a time limit of 3 hours? \n\nNot sure.\n\nhttps://www.kaggle.com/c/flower-classification-with-tpus/discussion/133391",
    "769163": "Are you using gem in your model? i am also facing the same issue because of gem layer..did you find any solution?",
    "769742": "Hello. I try to finish in less than 3 hours because of the same situation.",
    "770428": "hermione36 no I removed gem to get it to work.",
    "770869": "bopengiowa Yes TPU time limit is 3 hours."
  },
  "source": "meta"
}