I get an error when using keras callback model checkpoint. The error is \"unimplemented error\". I searched about it and it seems that means it does not work with TPU yet. Is there any solutions for this or another way for model checkpointing.
", "messages": [ { "id": "755578", "postDate": "02/24/2020 22:48:38", "content": "I get an error when using keras callback model checkpoint. The error is \"unimplemented error\". I searched about it and it seems that means it does not work with TPU yet. Is there any solutions for this or another way for model checkpointing.
", "rawMarkdown": "I get an error when using keras callback model checkpoint. The error is \"unimplemented error\". I searched about it and it seems that means it does not work with TPU yet. Is there any solutions for this or another way for model checkpointing.", "votes": null }, { "id": "755618", "postDate": "02/25/2020 00:29:26", "content": "The TPU can only write to GCS and Kaggle does not offer a writable GCS bucket for your TPUs.\nThere is model saving code in the TPU starter notebook in the TPU docs. Not exactly the same thing as a checkpoint but close. Look for cells \"Save the mode;\" and \"Reload the model\"
", "rawMarkdown": "The TPU can only write to GCS and Kaggle does not offer a writable GCS bucket for your TPUs.\nThere is model saving code in the [TPU starter notebook](https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu) in the [TPU docs](https://www.kaggle.com/docs/tpu). Not exactly the same thing as a checkpoint but close. Look for cells \"Save the mode;\" and \"Reload the model\"", "votes": null }, { "id": "755659", "postDate": "02/25/2020 01:45:26", "content": "I checked the kernel you mentioned. This means that if I want to use the model checkpoint callback, I would need to re implement it using the technique in your notebook.
", "rawMarkdown": "I checked the kernel you mentioned. This means that if I want to use the model checkpoint callback, I would need to re implement it using the technique in your notebook.", "votes": null }, { "id": "755721", "postDate": "02/25/2020 03:57:56", "content": "sir i need lung cancer CT dataset. How can i get it?
", "rawMarkdown": "sir i need lung cancer CT dataset. How can i get it?", "votes": null }, { "id": "756628", "postDate": "02/25/2020 22:46:10", "content": "yes, or not use a callback at all and just save the model at the end of the training
", "rawMarkdown": "yes, or not use a callback at all and just save the model at the end of the training", "votes": null }, { "id": "757455", "postDate": "02/26/2020 19:27:49", "content": "Hey @ibrahimsherify and @mgornergoogle I'm using Keras ModelCheckpoint on my code here and it seems to be working fine.
Thank for the notebook. So it looks like ModelCheckpoint will work with a TPU if the model_dir is local and the checkpoint format is .h5. Thank you for the find. There is definitely room for improvement on our end here.
", "rawMarkdown": "Thank for the notebook. So it looks like ModelCheckpoint will work with a TPU if the model_dir is local and the checkpoint format is .h5. Thank you for the find. There is definitely room for improvement on our end here.", "votes": null }, { "id": "757498", "postDate": "02/26/2020 20:59:28", "content": "You're welcome @mgornergoogle , that's the beauty of open-sourced code, the community integration!
", "rawMarkdown": "You're welcome @mgornergoogle , that's the beauty of open-sourced code, the community integration!", "votes": null }, { "id": "757681", "postDate": "02/27/2020 02:37:23", "content": "I tried it and it worked. Thanks for your help.
", "rawMarkdown": "I tried it and it worked. Thanks for your help.", "votes": null }, { "id": "758038", "postDate": "02/27/2020 11:45:37", "content": "You're welcome @ibrahimsherify
", "rawMarkdown": "You're welcome @ibrahimsherify", "votes": null }, { "id": "759408", "postDate": "02/29/2020 02:07:26", "content": "Thanks! I hope ModelCheckpoint can also be used in a custom training loop also. I will try it.
It should work but in a custom training loop, you have access to the model at every step so you can just call model.save()
", "rawMarkdown": "It should work but in a custom training loop, you have access to the model at every step so you can just call model.save()", "votes": null } ], "comments": [ { "id": 755618, "author_name": "mgorner", "author_url": "", "post_date": "02/25/2020 00:29:26", "content": "The TPU can only write to GCS and Kaggle does not offer a writable GCS bucket for your TPUs.\nThere is model saving code in the TPU starter notebook in the TPU docs. Not exactly the same thing as a checkpoint but close. Look for cells \"Save the mode;\" and \"Reload the model\"
", "votes": null, "replies": [ { "id": 755659, "author_name": "ibrahimsherify", "author_url": "", "post_date": "02/25/2020 01:45:26", "content": "I checked the kernel you mentioned. This means that if I want to use the model checkpoint callback, I would need to re implement it using the technique in your notebook.
", "votes": null, "replies": [] }, { "id": 756628, "author_name": "mgorner", "author_url": "", "post_date": "02/25/2020 22:46:10", "content": "yes, or not use a callback at all and just save the model at the end of the training
", "votes": null, "replies": [] } ] }, { "id": 755721, "author_name": "halakajalms", "author_url": "", "post_date": "02/25/2020 03:57:56", "content": "sir i need lung cancer CT dataset. How can i get it?
", "votes": null, "replies": [] }, { "id": 757455, "author_name": "dimitreoliveira", "author_url": "", "post_date": "02/26/2020 19:27:49", "content": "Hey @ibrahimsherify and @mgornergoogle I'm using Keras ModelCheckpoint on my code here and it seems to be working fine.
Thank for the notebook. So it looks like ModelCheckpoint will work with a TPU if the model_dir is local and the checkpoint format is .h5. Thank you for the find. There is definitely room for improvement on our end here.
", "votes": null, "replies": [] }, { "id": 757498, "author_name": "dimitreoliveira", "author_url": "", "post_date": "02/26/2020 20:59:28", "content": "You're welcome @mgornergoogle , that's the beauty of open-sourced code, the community integration!
", "votes": null, "replies": [] }, { "id": 757681, "author_name": "ibrahimsherify", "author_url": "", "post_date": "02/27/2020 02:37:23", "content": "I tried it and it worked. Thanks for your help.
", "votes": null, "replies": [] }, { "id": 758038, "author_name": "dimitreoliveira", "author_url": "", "post_date": "02/27/2020 11:45:37", "content": "You're welcome @ibrahimsherify
", "votes": null, "replies": [] }, { "id": 759408, "author_name": "yihdarshieh", "author_url": "", "post_date": "02/29/2020 02:07:26", "content": "Thanks! I hope ModelCheckpoint can also be used in a custom training loop also. I will try it.
It should work but in a custom training loop, you have access to the model at every step so you can just call model.save()
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "755578": "I get an error when using keras callback model checkpoint. The error is \"unimplemented error\". I searched about it and it seems that means it does not work with TPU yet. Is there any solutions for this or another way for model checkpointing.", "755618": "The TPU can only write to GCS and Kaggle does not offer a writable GCS bucket for your TPUs.\nThere is model saving code in the [TPU starter notebook](https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu) in the [TPU docs](https://www.kaggle.com/docs/tpu). Not exactly the same thing as a checkpoint but close. Look for cells \"Save the mode;\" and \"Reload the model\"", "755659": "I checked the kernel you mentioned. This means that if I want to use the model checkpoint callback, I would need to re implement it using the technique in your notebook.", "755721": "sir i need lung cancer CT dataset. How can i get it?", "756628": "yes, or not use a callback at all and just save the model at the end of the training", "757455": "Hey @ibrahimsherify and @mgornergoogle I'm using Keras `ModelCheckpoint` on [my code here](https://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline/notebook) and it seems to be working fine.", "757474": "Thank for the notebook. So it looks like ModelCheckpoint will work with a TPU if the model_dir is local and the checkpoint format is .h5. Thank you for the find. There is definitely room for improvement on our end here.", "757498": "You're welcome @mgornergoogle , that's the beauty of open-sourced code, the community integration!", "757681": "I tried it and it worked. Thanks for your help.", "758038": "You're welcome @ibrahimsherify", "759408": "Thanks! I hope `ModelCheckpoint` can also be used in a custom training loop also. I will try it.", "759410": "It should work but in a custom training loop, you have access to the model at every step so you can just call model.save()" }, "source": "meta" }