Hi everyone, I would like to implement ReduceLRonPlateau in my custom training loop in TF. I know how to implement it in model.fit(), but it's a bit tricky when implementing it in a custom tf.data pipeline.
\nI'm not sure how to create a feedback loop that reads my validation accuracy and then adjusts the learning rate after, given that the learning rate is predefined in the optimizer and if I try to pass a schedule with the variable that defines my validation accuracy, it doesn't work since the validation accuracy has not yet been called. So I'm stuck in sort of a catch-22.
\nAny help would be greatly appreciated, thanks!
", "messages": [ { "id": "1177129", "postDate": "01/30/2021 04:47:31", "content": "Hi everyone, I would like to implement ReduceLRonPlateau in my custom training loop in TF. I know how to implement it in model.fit(), but it's a bit tricky when implementing it in a custom tf.data pipeline.
\nI'm not sure how to create a feedback loop that reads my validation accuracy and then adjusts the learning rate after, given that the learning rate is predefined in the optimizer and if I try to pass a schedule with the variable that defines my validation accuracy, it doesn't work since the validation accuracy has not yet been called. So I'm stuck in sort of a catch-22.
\nAny help would be greatly appreciated, thanks!
", "rawMarkdown": "Hi everyone, I would like to implement ReduceLRonPlateau in my custom training loop in TF. I know how to implement it in model.fit(), but it's a bit tricky when implementing it in a custom tf.data pipeline. \n\nI'm not sure how to create a feedback loop that reads my validation accuracy and then adjusts the learning rate after, given that the learning rate is predefined in the optimizer and if I try to pass a schedule with the variable that defines my validation accuracy, it doesn't work since the validation accuracy has not yet been called. So I'm stuck in sort of a catch-22. \n\nAny help would be greatly appreciated, thanks!", "votes": null }, { "id": "1177466", "postDate": "01/30/2021 09:54:31", "content": "Hello!
\nI'm not pretty sure about the exact implementation, but after a peer at the Keras callbacks.py I'd suggest the following:
\ntrain_steps(train_dataset)
\nvalidation_steps(validation_dataset)
\n# evaluating model -> score -> patience_count
\nif patience >= max_patience: tf.keras.backend.set_value(model.optimizer.lr, new_lr)
\nAlso consider looking through this notebook with a custom TensorFlow loop and the EarlyStopping Callback hard-coded in pretty the same way.
Thanks! I'll give it a try.
", "rawMarkdown": "Thanks! I'll give it a try.", "votes": null } ], "comments": [ { "id": 1177466, "author_name": "nickuzmenkov", "author_url": "", "post_date": "01/30/2021 09:54:31", "content": "Hello!
\nI'm not pretty sure about the exact implementation, but after a peer at the Keras callbacks.py I'd suggest the following:
\ntrain_steps(train_dataset)
\nvalidation_steps(validation_dataset)
\n# evaluating model -> score -> patience_count
\nif patience >= max_patience: tf.keras.backend.set_value(model.optimizer.lr, new_lr)
\nAlso consider looking through this notebook with a custom TensorFlow loop and the EarlyStopping Callback hard-coded in pretty the same way.
Thanks! I'll give it a try.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1177129": "Hi everyone, I would like to implement ReduceLRonPlateau in my custom training loop in TF. I know how to implement it in model.fit(), but it's a bit tricky when implementing it in a custom tf.data pipeline. \n\nI'm not sure how to create a feedback loop that reads my validation accuracy and then adjusts the learning rate after, given that the learning rate is predefined in the optimizer and if I try to pass a schedule with the variable that defines my validation accuracy, it doesn't work since the validation accuracy has not yet been called. So I'm stuck in sort of a catch-22. \n\nAny help would be greatly appreciated, thanks!", "1177466": "Hello!\n\nI'm not pretty sure about the exact implementation, but after a peer at the Keras **[callbacks.py](https://github.com/keras-team/keras/blob/master/keras/callbacks.py)** I'd suggest the following:\n`train_steps(train_dataset)`\n`validation_steps(validation_dataset)`\n`# evaluating model -> score -> patience_count`\n`if patience >= max_patience: tf.keras.backend.set_value(model.optimizer.lr, new_lr)`\nAlso consider looking through this **[notebook](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-training-with-tpu-v2-pods)** with a custom TensorFlow loop and the EarlyStopping Callback hard-coded in pretty the same way.", "1177580": "Thanks! I'll give it a try." }, "source": "meta" }