Hi, I just published my kernel for demonstrating how to do custom training with TPU -- Custom training with TPU.
\n\nHope you find it helpful!
", "messages": [ { "id": "745630", "postDate": "02/14/2020 02:45:41", "content": "Hi, I just published my kernel for demonstrating how to do custom training with TPU -- Custom training with TPU.
\n\nHope you find it helpful!
", "rawMarkdown": "Hi, I just published my kernel for demonstrating how to do custom training with TPU -- [Custom training with TPU](https://www.kaggle.com/yihdarshieh/custom-training-with-tpu?scriptVersionId=28624481).\n\nHope you find it helpful!", "votes": null }, { "id": "745633", "postDate": "02/14/2020 02:51:14", "content": "I personally have question for one part in my kernel
\n\nfor epoch in range(last_epoch, EPOCHS):\n\n epoch_start_time = datetime.datetime.now()\n\n # We need to shuffle the training dataset in every epoch.\n # I don't know if there is a better way than the following way.\n\n training_dataset = get_training_dataset()\n training_dist_dataset = strategy.experimental_distribute_dataset(training_dataset)\nIf we don't call get_training_dataset(), and we just iterate again over the same object training_dataset, do the elements appear in the same order as in the previous iteratings?
If yes, we should avoid this situation. Other than calling get_training_dataset(), is there a better way??
Congrats for being the first to experiment with distributed custom training loops !
\n\nThe training dataset returned by get_training_dataset() is iterable. The training loop should iterate on it. I have a custom training loop sample here:\nhttps://colab.research.google.com/github/GoogleCloudPlatform/training-data-analyst/blob/master/courses/fast-and-lean-data-science/keras_flowers_customtrainloop_tf2.1.ipynb
@mgornergoogle , I observed that, in your notebook above, you use tf.reduce_mean in the loss function and the loss function is called inside train_step.
However, in [https://www.tensorflow.org/tutorials/distribute/custom_training#define_the_loss_function](Custom training with tf.distribute.Strategy), it specifically says the following:
\n\nIf you're writing a custom training loop, as in this tutorial, you should sum the per example losses and divide the sum by the GLOBAL_BATCH_SIZE: scale_loss = tf.reduce_sum(loss) * (1. / GLOBAL_BATCH_SIZE) or you can use tf.nn.compute_average_loss which takes the per example loss, optional sample weights, and GLOBAL_BATCH_SIZE as arguments and returns the scaled loss.\nand
\n\nUsing tf.reduce_mean is not recommended. Doing so divides the loss by actual per replica batch size which may vary step to step.\nand
\n\nWhy do this?\n\nThis needs to be done because after the gradients are calculated on each replica, they are synced across the replicas by summing them.\nI think in your notebook about custom training, the loss is larger than the correct one. (If divide your loss function by the number of replica, which is 8 there, it should be correct).
\n\nI didn't run several times, but I tried once to change your loss function and run it. The following is the training results of the last epochs, which has training/val acc ~0.80, which a bit better than the result in the original notebook (0.77/0.75). Be careful, the loss value in the reported is not the correct value to be reported. We should sum the loss values from all the replica, but I didn't spent my time to do this.
\n\nEPOCH: 51\nloss: 0.078226216 , accuracy_: 0.8203125 , val_loss: 0.08729416 , val_acc_: 0.7246094 , lr: 0.012249263566363598\n=======================\nEPOCH: 52\nloss: 0.063389845 , accuracy_: 0.81827444 , val_loss: 0.08148105 , val_acc_: 0.7871094 , lr: 0.01168680038804542\n=======================\nEPOCH: 53\nloss: 0.057514176 , accuracy_: 0.8226902 , val_loss: 0.07985326 , val_acc_: 0.7949219 , lr: 0.011152460368643147\n=======================\nEPOCH: 54\nloss: 0.07713041 , accuracy_: 0.8141984 , val_loss: 0.08316609 , val_acc_: 0.7890625 , lr: 0.01064483735021099\n=======================\nEPOCH: 55\nloss: 0.0724027 , accuracy_: 0.8345788 , val_loss: 0.05968307 , val_acc_: 0.8203125 , lr: 0.010162595482700439\n=======================\nEPOCH: 56\nloss: 0.066552304 , accuracy_: 0.8213315 , val_loss: 0.07661423 , val_acc_: 0.80078125 , lr: 0.009704465708565417\n=======================\nEPOCH: 57\nloss: 0.07287082 , accuracy_: 0.8199728 , val_loss: 0.07448967 , val_acc_: 0.8222656 , lr: 0.009269242423137147\n=======================\nEPOCH: 58\nloss: 0.053799704 , accuracy_: 0.83186144 , val_loss: 0.11246389 , val_acc_: 0.73828125 , lr: 0.008855780301980289\n=======================\nEPOCH: 59\nloss: 0.051523175 , accuracy_: 0.8413723 , val_loss: 0.07519884 , val_acc_: 0.7910156 , lr: 0.008462991286881274\n=======================\nEPOCH: 60\nloss: 0.052226786 , accuracy_: 0.84714675 , val_loss: 0.080301926 , val_acc_: 0.8046875 , lr: 0.008089841722537211\nI have to admit I did not pay much attention to replicating exactly the same loss a with model.fit() since a constant factor in the loss can be compensated for in the learning rate.
\n\nYou are correct, my loss computation would be wrong if batches were of varying sizes. With a repeated training dataset, batch size does not change throughout training though.
\n\nThe easiest way to fix it is probably to remove reduce_mean from the loss_fn and let tf.distribute.ReduceOp.MEAN do the work.
Yes, tf.distribute.ReduceOp should work (I am not familiar with it, currently, I do the ReduceOP in my own way, which is not a good practice ....)
Your own custom reduce is absolutely fine. Get the per-replica loss with strategy.experiemental_local_results(loss) and then you can do anything you want.
Data returned from strategy.experimental_run_v2 is in a custom format that has more info for tf.distribute.ReduceOpto work. strategy.experiemental_local_results() turns it into a regular per-replica list.
I personally have question for one part in my kernel
\n\nfor epoch in range(last_epoch, EPOCHS):\n\n epoch_start_time = datetime.datetime.now()\n\n # We need to shuffle the training dataset in every epoch.\n # I don't know if there is a better way than the following way.\n\n training_dataset = get_training_dataset()\n training_dist_dataset = strategy.experimental_distribute_dataset(training_dataset)\nIf we don't call get_training_dataset(), and we just iterate again over the same object training_dataset, do the elements appear in the same order as in the previous iteratings?
If yes, we should avoid this situation. Other than calling get_training_dataset(), is there a better way??
Congrats for being the first to experiment with distributed custom training loops !
\n\nThe training dataset returned by get_training_dataset() is iterable. The training loop should iterate on it. I have a custom training loop sample here:\nhttps://colab.research.google.com/github/GoogleCloudPlatform/training-data-analyst/blob/master/courses/fast-and-lean-data-science/keras_flowers_customtrainloop_tf2.1.ipynb
@mgornergoogle , I observed that, in your notebook above, you use tf.reduce_mean in the loss function and the loss function is called inside train_step.
However, in [https://www.tensorflow.org/tutorials/distribute/custom_training#define_the_loss_function](Custom training with tf.distribute.Strategy), it specifically says the following:
\n\nIf you're writing a custom training loop, as in this tutorial, you should sum the per example losses and divide the sum by the GLOBAL_BATCH_SIZE: scale_loss = tf.reduce_sum(loss) * (1. / GLOBAL_BATCH_SIZE) or you can use tf.nn.compute_average_loss which takes the per example loss, optional sample weights, and GLOBAL_BATCH_SIZE as arguments and returns the scaled loss.\nand
\n\nUsing tf.reduce_mean is not recommended. Doing so divides the loss by actual per replica batch size which may vary step to step.\nand
\n\nWhy do this?\n\nThis needs to be done because after the gradients are calculated on each replica, they are synced across the replicas by summing them.\nI think in your notebook about custom training, the loss is larger than the correct one. (If divide your loss function by the number of replica, which is 8 there, it should be correct).
\n\nI didn't run several times, but I tried once to change your loss function and run it. The following is the training results of the last epochs, which has training/val acc ~0.80, which a bit better than the result in the original notebook (0.77/0.75). Be careful, the loss value in the reported is not the correct value to be reported. We should sum the loss values from all the replica, but I didn't spent my time to do this.
\n\nEPOCH: 51\nloss: 0.078226216 , accuracy_: 0.8203125 , val_loss: 0.08729416 , val_acc_: 0.7246094 , lr: 0.012249263566363598\n=======================\nEPOCH: 52\nloss: 0.063389845 , accuracy_: 0.81827444 , val_loss: 0.08148105 , val_acc_: 0.7871094 , lr: 0.01168680038804542\n=======================\nEPOCH: 53\nloss: 0.057514176 , accuracy_: 0.8226902 , val_loss: 0.07985326 , val_acc_: 0.7949219 , lr: 0.011152460368643147\n=======================\nEPOCH: 54\nloss: 0.07713041 , accuracy_: 0.8141984 , val_loss: 0.08316609 , val_acc_: 0.7890625 , lr: 0.01064483735021099\n=======================\nEPOCH: 55\nloss: 0.0724027 , accuracy_: 0.8345788 , val_loss: 0.05968307 , val_acc_: 0.8203125 , lr: 0.010162595482700439\n=======================\nEPOCH: 56\nloss: 0.066552304 , accuracy_: 0.8213315 , val_loss: 0.07661423 , val_acc_: 0.80078125 , lr: 0.009704465708565417\n=======================\nEPOCH: 57\nloss: 0.07287082 , accuracy_: 0.8199728 , val_loss: 0.07448967 , val_acc_: 0.8222656 , lr: 0.009269242423137147\n=======================\nEPOCH: 58\nloss: 0.053799704 , accuracy_: 0.83186144 , val_loss: 0.11246389 , val_acc_: 0.73828125 , lr: 0.008855780301980289\n=======================\nEPOCH: 59\nloss: 0.051523175 , accuracy_: 0.8413723 , val_loss: 0.07519884 , val_acc_: 0.7910156 , lr: 0.008462991286881274\n=======================\nEPOCH: 60\nloss: 0.052226786 , accuracy_: 0.84714675 , val_loss: 0.080301926 , val_acc_: 0.8046875 , lr: 0.008089841722537211\nI have to admit I did not pay much attention to replicating exactly the same loss a with model.fit() since a constant factor in the loss can be compensated for in the learning rate.
\n\nYou are correct, my loss computation would be wrong if batches were of varying sizes. With a repeated training dataset, batch size does not change throughout training though.
\n\nThe easiest way to fix it is probably to remove reduce_mean from the loss_fn and let tf.distribute.ReduceOp.MEAN do the work.
Yes, tf.distribute.ReduceOp should work (I am not familiar with it, currently, I do the ReduceOP in my own way, which is not a good practice ....)
Your own custom reduce is absolutely fine. Get the per-replica loss with strategy.experiemental_local_results(loss) and then you can do anything you want.
Data returned from strategy.experimental_run_v2 is in a custom format that has more info for tf.distribute.ReduceOpto work. strategy.experiemental_local_results() turns it into a regular per-replica list.