We were trying to train with multi-gpu by the following code:\n\nstrategy=tf.distribute.MirroredStrategy()\nwith strategy.scope():\n ....#define model/compile\nmodel.fit(X,y)\nmodel.save_weights('model.h5')\n\nbut after loading the saved weights, the results are different from the original model. We are aware of this is because of the weights are saved on different gpus(or something similar like that). Can anyone please help us? Thank you in advance.
We were trying to train with multi-gpu by the following code:\n\nstrategy=tf.distribute.MirroredStrategy()\nwith strategy.scope():\n ....#define model/compile\nmodel.fit(X,y)\nmodel.save_weights('model.h5')\n\nbut after loading the saved weights, the results are different from the original model. We are aware of this is because of the weights are saved on different gpus(or something similar like that). Can anyone please help us? Thank you in advance.
While using MirroredStrategy(), use callbacks as below.
\n\nmodel_checkpoint = ModelCheckpoint(filepath, monitor = 'val_loss', save_best_only = False)\ncallback_list = [model_checkpoint]\nmodel.fit(......., callbacks = callback_list,....)
\n\nThis method works fine for me.
\n\nHowever, you can't load this model back and start training with multi gpu again. That is currently broken in tf 2.1
", "rawMarkdown": "While using MirroredStrategy(), use callbacks as below.\n\nmodel_checkpoint = ModelCheckpoint(filepath, monitor = 'val_loss', save_best_only = False)\ncallback_list = [model_checkpoint]\nmodel.fit(......., callbacks = callback_list,....)\n\nThis method works fine for me.\n\nHowever, you can't load this model back and start training with multi gpu again. That is currently broken in tf 2.1", "votes": null }, { "id": "766454", "postDate": "03/08/2020 06:48:39", "content": "We are able to save our models, but in any case, the weights get changed when the same model is loaded on single gpu and makes the model preforms worse and that is no everyday worse performance, makes ~0.05 lower on leaderboard.
", "rawMarkdown": "We are able to save our models, but in any case, the weights get changed when the same model is loaded on single gpu and makes the model preforms worse and that is no everyday worse performance, makes ~0.05 lower on leaderboard.", "votes": null }, { "id": "766743", "postDate": "03/08/2020 16:21:14", "content": "So what @harshitsheoran means is, the prediction on multi-gpu is different than prediction of weights loaded on single-gpu.
", "rawMarkdown": "So what @harshitsheoran means is, the prediction on multi-gpu is different than prediction of weights loaded on single-gpu.", "votes": null }, { "id": "766767", "postDate": "03/08/2020 17:03:59", "content": "I just verified this on my work system, though on a different work related project. The outputs for the model remains the same when loaded into a single gpu, irrespective of whether it was trained in multi gpu or single gpu. Verified with 4x2080Ti.
", "rawMarkdown": "I just verified this on my work system, though on a different work related project. The outputs for the model remains the same when loaded into a single gpu, irrespective of whether it was trained in multi gpu or single gpu. Verified with 4x2080Ti.", "votes": null }, { "id": "766778", "postDate": "03/08/2020 17:25:51", "content": "Did you use any tricks like tf.device? Simple keras.models.load_model doesn't work and model.load_weights have different results.
I don't save only the weights. I save the entire model as a whole using ModelCheckpoint in callbacks. After training is done, I simply use tf.keras.models.load_model()
", "rawMarkdown": "I don't save only the weights. I save the entire model as a whole using ModelCheckpoint in callbacks. After training is done, I simply use tf.keras.models.load_model()", "votes": null }, { "id": "767027", "postDate": "03/09/2020 03:58:54", "content": "I used model.save('model.h5') then use tf.keras.models.load_model('model.h5), but it gave a weird error saying init doesn't have *. I'll try ModelCheckpoint. Thank you for your solution for us!
While using MirroredStrategy(), use callbacks as below.
\n\nmodel_checkpoint = ModelCheckpoint(filepath, monitor = 'val_loss', save_best_only = False)\ncallback_list = [model_checkpoint]\nmodel.fit(......., callbacks = callback_list,....)
\n\nThis method works fine for me.
\n\nHowever, you can't load this model back and start training with multi gpu again. That is currently broken in tf 2.1
", "votes": null, "replies": [ { "id": 766454, "author_name": "harshitsheoran", "author_url": "", "post_date": "03/08/2020 06:48:39", "content": "We are able to save our models, but in any case, the weights get changed when the same model is loaded on single gpu and makes the model preforms worse and that is no everyday worse performance, makes ~0.05 lower on leaderboard.
", "votes": null, "replies": [] }, { "id": 766743, "author_name": "unkownhihi", "author_url": "", "post_date": "03/08/2020 16:21:14", "content": "So what @harshitsheoran means is, the prediction on multi-gpu is different than prediction of weights loaded on single-gpu.
", "votes": null, "replies": [] }, { "id": 766767, "author_name": "akashnandi", "author_url": "", "post_date": "03/08/2020 17:03:59", "content": "I just verified this on my work system, though on a different work related project. The outputs for the model remains the same when loaded into a single gpu, irrespective of whether it was trained in multi gpu or single gpu. Verified with 4x2080Ti.
", "votes": null, "replies": [] }, { "id": 766778, "author_name": "unkownhihi", "author_url": "", "post_date": "03/08/2020 17:25:51", "content": "Did you use any tricks like tf.device? Simple keras.models.load_model doesn't work and model.load_weights have different results.
I don't save only the weights. I save the entire model as a whole using ModelCheckpoint in callbacks. After training is done, I simply use tf.keras.models.load_model()
", "votes": null, "replies": [] }, { "id": 767027, "author_name": "unkownhihi", "author_url": "", "post_date": "03/09/2020 03:58:54", "content": "I used model.save('model.h5') then use tf.keras.models.load_model('model.h5), but it gave a weird error saying init doesn't have *. I'll try ModelCheckpoint. Thank you for your solution for us!