{
  "id": 174312,
  "title": "Trying to load previously trained model weights but getting a TPU Compilation Error",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/174312",
  "author_name": "",
  "post_date": "2020-08-13T05:52:36.651079200Z",
  "votes": 2,
  "comment_count": 4,
  "views": 0,
  "content": "<p>PFA the screenshot of the issue <br>\nIs anyone facing a similar issue?<br>\nOr does anyone know how to resolve this?</p>\n<p>Thanks in advance</p>",
  "messages": [
    {
      "id": "968577",
      "postDate": "08/13/2020 05:52:36",
      "content": "<p>PFA the screenshot of the issue <br>\nIs anyone facing a similar issue?<br>\nOr does anyone know how to resolve this?</p>\n<p>Thanks in advance</p>",
      "rawMarkdown": "PFA the screenshot of the issue \nIs anyone facing a similar issue?\nOr does anyone know how to resolve this?\n\nThanks in advance",
      "votes": null
    },
    {
      "id": "968766",
      "postDate": "08/13/2020 08:22:20",
      "content": "<p>One of the possible reason can be not compiling the model before loading the weights. Have you tried that?</p>",
      "rawMarkdown": "One of the possible reason can be not compiling the model before loading the weights. Have you tried that?",
      "votes": null
    },
    {
      "id": "968832",
      "postDate": "08/13/2020 09:10:59",
      "content": "<p>Are you doing this to continue training? I did that with a coding block:</p>\n<pre><code>with strategy.scope():\n    model = ...\n    loss = ...\n    model.compile...\n    model.build(input_shape=(None, *image_size, 3)\n    model.load_weights(...)\n</code></pre>\n<p>with image_size = (dim, dim)</p>",
      "rawMarkdown": "Are you doing this to continue training? I did that with a coding block:\n\n    with strategy.scope():\n        model = ...\n        loss = ...\n        model.compile...\n        model.build(input_shape=(None, *image_size, 3)\n        model.load_weights(...)\n\nwith image_size = (dim, dim)",
      "votes": null
    },
    {
      "id": "968880",
      "postDate": "08/13/2020 10:06:18",
      "content": "<p>Not working on this competition but faced similar issue. Everything was fine before today and I did not change anything. I have just switched to GPU and it works as usual.</p>",
      "rawMarkdown": "Not working on this competition but faced similar issue. Everything was fine before today and I did not change anything. I have just switched to GPU and it works as usual.",
      "votes": null
    },
    {
      "id": "969576",
      "postDate": "08/13/2020 19:21:19",
      "content": "<p>Could you please share your code and checkpoints for investigation ?<br>\nAlso, can you confirm that you have saved the checkpoints using TF 2.2 and tried to reload them using TF 2.3 ? Is that the case that is failing ?</p>",
      "rawMarkdown": "Could you please share your code and checkpoints for investigation ?\nAlso, can you confirm that you have saved the checkpoints using TF 2.2 and tried to reload them using TF 2.3 ? Is that the case that is failing ?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 968766,
      "author_name": "azaemon",
      "author_url": "",
      "post_date": "08/13/2020 08:22:20",
      "content": "<p>One of the possible reason can be not compiling the model before loading the weights. Have you tried that?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 968832,
      "author_name": "helgith",
      "author_url": "",
      "post_date": "08/13/2020 09:10:59",
      "content": "<p>Are you doing this to continue training? I did that with a coding block:</p>\n<pre><code>with strategy.scope():\n    model = ...\n    loss = ...\n    model.compile...\n    model.build(input_shape=(None, *image_size, 3)\n    model.load_weights(...)\n</code></pre>\n<p>with image_size = (dim, dim)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 968880,
      "author_name": "bunnyxx",
      "author_url": "",
      "post_date": "08/13/2020 10:06:18",
      "content": "<p>Not working on this competition but faced similar issue. Everything was fine before today and I did not change anything. I have just switched to GPU and it works as usual.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 969576,
      "author_name": "mgorner",
      "author_url": "",
      "post_date": "08/13/2020 19:21:19",
      "content": "<p>Could you please share your code and checkpoints for investigation ?<br>\nAlso, can you confirm that you have saved the checkpoints using TF 2.2 and tried to reload them using TF 2.3 ? Is that the case that is failing ?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "968577": "PFA the screenshot of the issue \nIs anyone facing a similar issue?\nOr does anyone know how to resolve this?\n\nThanks in advance",
    "968766": "One of the possible reason can be not compiling the model before loading the weights. Have you tried that?",
    "968832": "Are you doing this to continue training? I did that with a coding block:\n\n    with strategy.scope():\n        model = ...\n        loss = ...\n        model.compile...\n        model.build(input_shape=(None, *image_size, 3)\n        model.load_weights(...)\n\nwith image_size = (dim, dim)",
    "968880": "Not working on this competition but faced similar issue. Everything was fine before today and I did not change anything. I have just switched to GPU and it works as usual.",
    "969576": "Could you please share your code and checkpoints for investigation ?\nAlso, can you confirm that you have saved the checkpoints using TF 2.2 and tried to reload them using TF 2.3 ? Is that the case that is failing ?"
  },
  "source": "meta"
}