{
  "id": 140970,
  "title": "Saving Model from TPU",
  "url": "/competitions/jigsaw-multilingual-toxic-comment-classification/discussion/140970",
  "author_name": "",
  "post_date": "2020-04-04T02:56:54.293145800Z",
  "votes": 3,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Hi! How do you save a model that is running on TPU? </p>\n\n<p>Take a look at this kernel that I've made: <a href=\"https://www.kaggle.com/ilhamfp31/jigsaw-tpu-xlm-roberta\">https://www.kaggle.com/ilhamfp31/jigsaw-tpu-xlm-roberta</a></p>\n\n<p>I used simple <code>model.save('./my_model')</code>, and it output:  </p>\n\n<p><code>\nUnimplementedError: File system scheme '[local]' not implemented (file: './my_model/variables/variables_temp_3bc690ff0b614af59d0e6e93af7fd362')\n    Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. [Op:Identity]\n</code></p>\n\n<p>What should I do?</p>",
  "messages": [
    {
      "id": "796872",
      "postDate": "04/04/2020 02:56:54",
      "content": "<p>Hi! How do you save a model that is running on TPU? </p>\n\n<p>Take a look at this kernel that I've made: <a href=\"https://www.kaggle.com/ilhamfp31/jigsaw-tpu-xlm-roberta\">https://www.kaggle.com/ilhamfp31/jigsaw-tpu-xlm-roberta</a></p>\n\n<p>I used simple <code>model.save('./my_model')</code>, and it output:  </p>\n\n<p><code>\nUnimplementedError: File system scheme '[local]' not implemented (file: './my_model/variables/variables_temp_3bc690ff0b614af59d0e6e93af7fd362')\n    Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. [Op:Identity]\n</code></p>\n\n<p>What should I do?</p>",
      "rawMarkdown": "Hi! How do you save a model that is running on TPU? \n\nTake a look at this kernel that I've made: https://www.kaggle.com/ilhamfp31/jigsaw-tpu-xlm-roberta\n\nI used simple `model.save('./my_model')`, and it output:  \n  \n`\nUnimplementedError: File system scheme '[local]' not implemented (file: './my_model/variables/variables_temp_3bc690ff0b614af59d0e6e93af7fd362')\n\tEncountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. [Op:Identity]\n`\n\n  \nWhat should I do?",
      "votes": null
    },
    {
      "id": "797193",
      "postDate": "04/04/2020 10:01:21",
      "content": "<p><code>save_weights</code></p>",
      "rawMarkdown": "`save_weights`",
      "votes": null
    },
    {
      "id": "797443",
      "postDate": "04/04/2020 15:02:07",
      "content": "<p>Hey, it works! Thank you. </p>",
      "rawMarkdown": "Hey, it works! Thank you.",
      "votes": null
    },
    {
      "id": "799799",
      "postDate": "04/06/2020 19:03:17",
      "content": "<p>saving in HD5 format works. See the sample in the TPU documentation: <a href=\"http://kaggle.com/docs/tpu\">http://kaggle.com/docs/tpu</a></p>",
      "rawMarkdown": "saving in HD5 format works. See the sample in the TPU documentation: http://kaggle.com/docs/tpu",
      "votes": null
    },
    {
      "id": "800585",
      "postDate": "04/07/2020 14:51:08",
      "content": "<p>If you'd like to use ModelCheckpoint, it's also possible. These lines worked for me:\n<code>\nmodel_path = \"model_best.h5\"\ncheckpoint = ModelCheckpoint(model_path, monitor='val_auc', mode='max', save_best_only=True, save_weights_only=True)\n</code>\nNote: setting <code>save_weights_only=True</code> is important, otherwise getting a <code>NotImplementedError</code></p>",
      "rawMarkdown": "If you'd like to use ModelCheckpoint, it's also possible. These lines worked for me:\n```\nmodel_path = \"model_best.h5\"\ncheckpoint = ModelCheckpoint(model_path, monitor='val_auc', mode='max', save_best_only=True, save_weights_only=True)\n```\nNote: setting `save_weights_only=True` is important, otherwise getting a `NotImplementedError`",
      "votes": null
    },
    {
      "id": "888748",
      "postDate": "06/16/2020 14:40:37",
      "content": "<p>But when i am loading the weights then the training starts from initial.Can you please help me out? </p>",
      "rawMarkdown": "But when i am loading the weights then the training starts from initial.Can you please help me out?",
      "votes": null
    },
    {
      "id": "888764",
      "postDate": "06/16/2020 14:51:48",
      "content": "<p>Hmm, I'm not sure how that happens. If you load your weight into the model, then it should start training from that pretrained weight.</p>",
      "rawMarkdown": "Hmm, I'm not sure how that happens. If you load your weight into the model, then it should start training from that pretrained weight.",
      "votes": null
    },
    {
      "id": "1581556",
      "postDate": "11/13/2021 23:59:16",
      "content": "<p>saving weights does not work for me. I get the following error when i try to load them </p>\n<p>2021-11-13 23:40:50.607077: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open ../input/roberta-model-a/roberta_model_1: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?</p>\n<p>Anyone know  a solution for this?</p>",
      "rawMarkdown": "saving weights does not work for me. I get the following error when i try to load them \n\n2021-11-13 23:40:50.607077: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open ../input/roberta-model-a/roberta_model_1: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?\n\nAnyone know  a solution for this?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 797193,
      "author_name": "philippsinger",
      "author_url": "",
      "post_date": "04/04/2020 10:01:21",
      "content": "<p><code>save_weights</code></p>",
      "votes": null,
      "replies": [
        {
          "id": 797443,
          "author_name": "ilhamfp31",
          "author_url": "",
          "post_date": "04/04/2020 15:02:07",
          "content": "<p>Hey, it works! Thank you. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 888748,
          "author_name": "akhilsingla",
          "author_url": "",
          "post_date": "06/16/2020 14:40:37",
          "content": "<p>But when i am loading the weights then the training starts from initial.Can you please help me out? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 888764,
          "author_name": "ilhamfp31",
          "author_url": "",
          "post_date": "06/16/2020 14:51:48",
          "content": "<p>Hmm, I'm not sure how that happens. If you load your weight into the model, then it should start training from that pretrained weight.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1581556,
          "author_name": "linusj79",
          "author_url": "",
          "post_date": "11/13/2021 23:59:16",
          "content": "<p>saving weights does not work for me. I get the following error when i try to load them </p>\n<p>2021-11-13 23:40:50.607077: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open ../input/roberta-model-a/roberta_model_1: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?</p>\n<p>Anyone know  a solution for this?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 799799,
      "author_name": "mgorner",
      "author_url": "",
      "post_date": "04/06/2020 19:03:17",
      "content": "<p>saving in HD5 format works. See the sample in the TPU documentation: <a href=\"http://kaggle.com/docs/tpu\">http://kaggle.com/docs/tpu</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 800585,
      "author_name": "rafiko1",
      "author_url": "",
      "post_date": "04/07/2020 14:51:08",
      "content": "<p>If you'd like to use ModelCheckpoint, it's also possible. These lines worked for me:\n<code>\nmodel_path = \"model_best.h5\"\ncheckpoint = ModelCheckpoint(model_path, monitor='val_auc', mode='max', save_best_only=True, save_weights_only=True)\n</code>\nNote: setting <code>save_weights_only=True</code> is important, otherwise getting a <code>NotImplementedError</code></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "796872": "Hi! How do you save a model that is running on TPU? \n\nTake a look at this kernel that I've made: https://www.kaggle.com/ilhamfp31/jigsaw-tpu-xlm-roberta\n\nI used simple `model.save('./my_model')`, and it output:  \n  \n`\nUnimplementedError: File system scheme '[local]' not implemented (file: './my_model/variables/variables_temp_3bc690ff0b614af59d0e6e93af7fd362')\n\tEncountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. [Op:Identity]\n`\n\n  \nWhat should I do?",
    "797193": "`save_weights`",
    "797443": "Hey, it works! Thank you.",
    "799799": "saving in HD5 format works. See the sample in the TPU documentation: http://kaggle.com/docs/tpu",
    "800585": "If you'd like to use ModelCheckpoint, it's also possible. These lines worked for me:\n```\nmodel_path = \"model_best.h5\"\ncheckpoint = ModelCheckpoint(model_path, monitor='val_auc', mode='max', save_best_only=True, save_weights_only=True)\n```\nNote: setting `save_weights_only=True` is important, otherwise getting a `NotImplementedError`",
    "888748": "But when i am loading the weights then the training starts from initial.Can you please help me out?",
    "888764": "Hmm, I'm not sure how that happens. If you load your weight into the model, then it should start training from that pretrained weight.",
    "1581556": "saving weights does not work for me. I get the following error when i try to load them \n\n2021-11-13 23:40:50.607077: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open ../input/roberta-model-a/roberta_model_1: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?\n\nAnyone know  a solution for this?"
  },
  "source": "meta"
}