Apologies if this is a beginner problem, but I can't get my Tensorflow models to save using the same callbacks that I've encountered in several different notebooks when using TPU. I've gone so far as copying line-by-line the exact callbacks that were used in:
\nhttps://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline
\nand https://www.kaggle.com/xhlulu/ranzcr-efficientnet-tpu-training
Still, nothing has worked. The kaggle instructions for saving checkpoints with TPU models explain how to save them using this code snippet:
\nsave_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')\ncheckpoints_cb = tf.keras.callbacks.ModelCheckpoint('./checkpoints', options=save_locally)\nmodel.fit(…, callbacks=[checkpoints_cb])\nBut it doesn't explain how to retrieve it. I can save the model to the working directory after training has completed, but this will only save the model in its final form and this is not necessarily the best performing one (https://github.com/keras-team/keras/issues/2768). I've also made sure that the models I've tried saving with callbacks all have generic names that end in '.h5'.
\nI've been tearing my hair out over this since I can't identify anything in my code that would cause the callbacks not to save their output unlike the other notebooks I linked. If anyone has experienced a similar issue I would appreciate your input, as I've already lost 7 hours of TPU time.
", "messages": [ { "id": "1218356", "postDate": "02/25/2021 18:46:44", "content": "Apologies if this is a beginner problem, but I can't get my Tensorflow models to save using the same callbacks that I've encountered in several different notebooks when using TPU. I've gone so far as copying line-by-line the exact callbacks that were used in:
\nhttps://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline
\nand https://www.kaggle.com/xhlulu/ranzcr-efficientnet-tpu-training
Still, nothing has worked. The kaggle instructions for saving checkpoints with TPU models explain how to save them using this code snippet:
\nsave_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')\ncheckpoints_cb = tf.keras.callbacks.ModelCheckpoint('./checkpoints', options=save_locally)\nmodel.fit(…, callbacks=[checkpoints_cb])\nBut it doesn't explain how to retrieve it. I can save the model to the working directory after training has completed, but this will only save the model in its final form and this is not necessarily the best performing one (https://github.com/keras-team/keras/issues/2768). I've also made sure that the models I've tried saving with callbacks all have generic names that end in '.h5'.
\nI've been tearing my hair out over this since I can't identify anything in my code that would cause the callbacks not to save their output unlike the other notebooks I linked. If anyone has experienced a similar issue I would appreciate your input, as I've already lost 7 hours of TPU time.
", "rawMarkdown": "Apologies if this is a beginner problem, but I can't get my Tensorflow models to save using the same callbacks that I've encountered in several different notebooks when using TPU. I've gone so far as copying line-by-line the exact callbacks that were used in:\n\nhttps://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline\nand https://www.kaggle.com/xhlulu/ranzcr-efficientnet-tpu-training\n\nStill, nothing has worked. The kaggle instructions for saving checkpoints with TPU models explain how to save them using this code snippet:\n\n```\nsave_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')\ncheckpoints_cb = tf.keras.callbacks.ModelCheckpoint('./checkpoints', options=save_locally)\nmodel.fit(…, callbacks=[checkpoints_cb])\n```\n\nBut it doesn't explain how to retrieve it. I can save the model to the working directory after training has completed, but this will only save the model in its final form and this is not necessarily the best performing one (https://github.com/keras-team/keras/issues/2768). I've also made sure that the models I've tried saving with callbacks all have generic names that end in '.h5'.\n\nI've been tearing my hair out over this since I can't identify anything in my code that would cause the callbacks not to save their output unlike the other notebooks I linked. If anyone has experienced a similar issue I would appreciate your input, as I've already lost 7 hours of TPU time.", "votes": null }, { "id": "1218445", "postDate": "02/25/2021 22:04:21", "content": "Shoudn't you use save_best_only=True parameter in ModelCheckpoint?
Yes, I have been using that parameter. The main code I've used is:
\ncheckpoint = tf.keras.callbacks.ModelCheckpoint(model_name, save_best_only=True, save_weights_only=True, monitor='val_auc', mode='max')\nlr_reducer = tf.keras.callbacks.ReduceLROnPlateau(\n monitor=\"val_auc\", patience=1, min_lr=1e-6, mode='max') \n\nhistory = model.fit(\n train_dataset, \n epochs=25,\n verbose=1,\n callbacks=[checkpoint, lr_reducer],\n steps_per_epoch=steps_per_epoch,\n validation_data=valid_dataset)\nI've tried removing save_weights_only=True, results are the same - nothing is saved to the working directory.
Can you show how your model.compile(...) looks?
model.compile( \n optimizer=tf.keras.optimizers.Adam(lr=1e-4),\n loss='binary_crossentropy',\n metrics=[tf.keras.metrics.AUC(multi_label=True)])\nLooks OK. Can you try to save model based on val_loss, not on val_auc(and don't forget to change mode to min)?
I faced the same issue, looks like some issue with the recent upgrade of tensorflow version and h5py. The sequential api is working but issue with the functional one. I tried running exactly same code which is using functional api and was working before the tf upgrade but now it does not seem to work.
", "rawMarkdown": "I faced the same issue, looks like some issue with the recent upgrade of tensorflow version and h5py. The sequential api is working but issue with the functional one. I tried running exactly same code which is using functional api and was working before the tf upgrade but now it does not seem to work.", "votes": null }, { "id": "1220206", "postDate": "02/27/2021 17:58:58", "content": "I tried that last night, again no positive results.
\nI've been very careful with my code and can't see anything that would obstruct the callback saving its output. I'm considering contacting the Kaggle technical team over this since I can't share my code privately in this competition.
", "rawMarkdown": "I tried that last night, again no positive results.\n\nI've been very careful with my code and can't see anything that would obstruct the callback saving its output. I'm considering contacting the Kaggle technical team over this since I can't share my code privately in this competition.", "votes": null }, { "id": "1220233", "postDate": "02/27/2021 18:47:47", "content": "You should try previous TF versions with !pip install tensorflow==<version>
model_save = ModelCheckpoint('./model.h5',
\n save_best_only = True,
\n save_weights_only = True,
\n monitor = 'val_loss',
\n mode = 'min', verbose = 1)
See here :
\nhttps://www.kaggle.com/maksymshkliarevskyi/ranzcr-xception-tpu-baseline
Shoudn't you use save_best_only=True parameter in ModelCheckpoint?
Yes, I have been using that parameter. The main code I've used is:
\ncheckpoint = tf.keras.callbacks.ModelCheckpoint(model_name, save_best_only=True, save_weights_only=True, monitor='val_auc', mode='max')\nlr_reducer = tf.keras.callbacks.ReduceLROnPlateau(\n monitor=\"val_auc\", patience=1, min_lr=1e-6, mode='max') \n\nhistory = model.fit(\n train_dataset, \n epochs=25,\n verbose=1,\n callbacks=[checkpoint, lr_reducer],\n steps_per_epoch=steps_per_epoch,\n validation_data=valid_dataset)\nI've tried removing save_weights_only=True, results are the same - nothing is saved to the working directory.
Can you show how your model.compile(...) looks?
model.compile( \n optimizer=tf.keras.optimizers.Adam(lr=1e-4),\n loss='binary_crossentropy',\n metrics=[tf.keras.metrics.AUC(multi_label=True)])\nLooks OK. Can you try to save model based on val_loss, not on val_auc(and don't forget to change mode to min)?
I tried that last night, again no positive results.
\nI've been very careful with my code and can't see anything that would obstruct the callback saving its output. I'm considering contacting the Kaggle technical team over this since I can't share my code privately in this competition.
", "votes": null, "replies": [] }, { "id": 1220233, "author_name": "atamazian", "author_url": "", "post_date": "02/27/2021 18:47:47", "content": "You should try previous TF versions with !pip install tensorflow==<version>
I faced the same issue, looks like some issue with the recent upgrade of tensorflow version and h5py. The sequential api is working but issue with the functional one. I tried running exactly same code which is using functional api and was working before the tf upgrade but now it does not seem to work.
", "votes": null, "replies": [] }, { "id": 1221100, "author_name": "faisalalsrheed", "author_url": "", "post_date": "02/28/2021 17:08:15", "content": "model_save = ModelCheckpoint('./model.h5',
\n save_best_only = True,
\n save_weights_only = True,
\n monitor = 'val_loss',
\n mode = 'min', verbose = 1)
See here :
\nhttps://www.kaggle.com/maksymshkliarevskyi/ranzcr-xception-tpu-baseline