Some new gpu's (RTX and Volta) have tensor cores for mixed precision training (fp16 instead of fp32).
\nAutomatic Mixed Precision for Deep Learning
It's really easy to implement (tensorflow, other frameworks in the article above) just wrap your optimizer:
\nopt = tf.keras.optimizers.Adam(learning_rate=0.001)
\nopt = tf.train.experimental.enable_mixed_precision_graph_rewrite(opt)
EffNet training time faster for ~40%
", "messages": [ { "id": "967212", "postDate": "08/12/2020 05:09:18", "content": "Some new gpu's (RTX and Volta) have tensor cores for mixed precision training (fp16 instead of fp32).
\nAutomatic Mixed Precision for Deep Learning
It's really easy to implement (tensorflow, other frameworks in the article above) just wrap your optimizer:
\nopt = tf.keras.optimizers.Adam(learning_rate=0.001)
\nopt = tf.train.experimental.enable_mixed_precision_graph_rewrite(opt)
EffNet training time faster for ~40%
", "rawMarkdown": "Some new gpu's (RTX and Volta) have tensor cores for mixed precision training (fp16 instead of fp32). \n[Automatic Mixed Precision for Deep Learning](https://developer.nvidia.com/automatic-mixed-precision)\n\nIt's really easy to implement (tensorflow, other frameworks in the article above) just wrap your optimizer:\n`opt = tf.keras.optimizers.Adam(learning_rate=0.001)`\n`opt = tf.train.experimental.enable_mixed_precision_graph_rewrite(opt)`\n\nEffNet training time faster for ~40%", "votes": null }, { "id": "967509", "postDate": "08/12/2020 09:55:18", "content": "thanks for mentioning it, since 6 days left have to do it!
", "rawMarkdown": "thanks for mentioning it, since 6 days left have to do it!", "votes": null }, { "id": "968258", "postDate": "08/12/2020 20:01:49", "content": "@uspangaliev thanks for mentioning it. -)
\nHere is a short summary of usages for all frameworks. (Direct for the above link).
\nopt = tf.keras.optimizers.Adam(learning_rate=0.001)\nopt = tf.train.experimental.enable_mixed_precision_graph_rewrite(opt)\nmodel, optimizer = amp.initialize(model, optimizer, opt_level=\"O1\")\nwith amp.scale_loss(loss, optimizer) as scaled_loss:\n scaled_loss.backward()\namp.init()\namp.init_trainer(trainer)\nwith amp.scale_loss(loss, trainer) as scaled_loss:\n autograd.backward(scaled_loss)\nsgd = SGDOptimizer()\nmp_sgd = fluid.contrib.mixed_precision.decorator.decorate(sgd)\nmp_sgd.minimize(loss)\nAlso, see this discussion, it might help. mixed precision with TPU tf.distrubute
\n+
\nCheck this interesting discussion, might help. - Mixed precision training with tf.keras 2.0
Efficientnet don't use tensorcores on gpu. It is why they run faster on tpu.
", "rawMarkdown": "Efficientnet don't use tensorcores on gpu. It is why they run faster on tpu.", "votes": null }, { "id": "968357", "postDate": "08/12/2020 23:29:38", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "968981", "postDate": "08/13/2020 11:36:30", "content": "With PyTorch 1.6, now you can use FP16 with Pascal architecture GPUs as well. You can have a look at my kernel where I use FP16 with GPU (PyTorch 1.6) instead of TPU. It's really easy and stable as well. Works with GTX 10 series GPU like a charm.\nhttps://www.kaggle.com/sovitrath/caltech-256-amp-pytorch/comments
", "rawMarkdown": "With PyTorch 1.6, now you can use FP16 with Pascal architecture GPUs as well. You can have a look at my [kernel](https://www.kaggle.com/sovitrath/caltech-256-amp-pytorch/comments) where I use FP16 with GPU (PyTorch 1.6) instead of TPU. It's really easy and stable as well. Works with GTX 10 series GPU like a charm.\nhttps://www.kaggle.com/sovitrath/caltech-256-amp-pytorch/comments", "votes": null }, { "id": "969086", "postDate": "08/13/2020 12:56:09", "content": "@cpmpml is there a principle why that happens?
", "rawMarkdown": "cpmpml is there a principle why that happens?", "votes": null }, { "id": "969200", "postDate": "08/13/2020 14:41:41", "content": "Some fp16 kernels are missing in cuda. We are looking into it. The P100 available in Kaggle kernels have no tensorcores anyway, hence the issue is not relevant. But is is if you use V100 for instance.
\nGood news is that using mixed precision helps GPU still. I am using amp in pytorch, and TF users should also use mixed precision. It helps, even if tensorcores aren't used.
", "rawMarkdown": "Some fp16 kernels are missing in cuda. We are looking into it. The P100 available in Kaggle kernels have no tensorcores anyway, hence the issue is not relevant. But is is if you use V100 for instance.\n\nGood news is that using mixed precision helps GPU still. I am using amp in pytorch, and TF users should also use mixed precision. It helps, even if tensorcores aren't used.", "votes": null }, { "id": "969286", "postDate": "08/13/2020 15:35:28", "content": "Exactly. Using AMP with even P100 GPUs we can literally use double the batch size of what we can without AMP. It helps a lot and the training is faster as well. It has become even easier with PyTorch 1.6.
", "rawMarkdown": "Exactly. Using AMP with even P100 GPUs we can literally use double the batch size of what we can without AMP. It helps a lot and the training is faster as well. It has become even easier with PyTorch 1.6.", "votes": null }, { "id": "970458", "postDate": "08/14/2020 13:25:27", "content": "Thanx for the information :)
", "rawMarkdown": "Thanx for the information :)", "votes": null } ], "comments": [ { "id": 967509, "author_name": "yash612", "author_url": "", "post_date": "08/12/2020 09:55:18", "content": "thanks for mentioning it, since 6 days left have to do it!
", "votes": null, "replies": [] }, { "id": 968258, "author_name": "ipythonx", "author_url": "", "post_date": "08/12/2020 20:01:49", "content": "@uspangaliev thanks for mentioning it. -)
\nHere is a short summary of usages for all frameworks. (Direct for the above link).
\nopt = tf.keras.optimizers.Adam(learning_rate=0.001)\nopt = tf.train.experimental.enable_mixed_precision_graph_rewrite(opt)\nmodel, optimizer = amp.initialize(model, optimizer, opt_level=\"O1\")\nwith amp.scale_loss(loss, optimizer) as scaled_loss:\n scaled_loss.backward()\namp.init()\namp.init_trainer(trainer)\nwith amp.scale_loss(loss, trainer) as scaled_loss:\n autograd.backward(scaled_loss)\nsgd = SGDOptimizer()\nmp_sgd = fluid.contrib.mixed_precision.decorator.decorate(sgd)\nmp_sgd.minimize(loss)\nAlso, see this discussion, it might help. mixed precision with TPU tf.distrubute
\n+
\nCheck this interesting discussion, might help. - Mixed precision training with tf.keras 2.0
Efficientnet don't use tensorcores on gpu. It is why they run faster on tpu.
", "votes": null, "replies": [ { "id": 969086, "author_name": "hiramcho", "author_url": "", "post_date": "08/13/2020 12:56:09", "content": "@cpmpml is there a principle why that happens?
", "votes": null, "replies": [] }, { "id": 969200, "author_name": "cpmpml", "author_url": "", "post_date": "08/13/2020 14:41:41", "content": "Some fp16 kernels are missing in cuda. We are looking into it. The P100 available in Kaggle kernels have no tensorcores anyway, hence the issue is not relevant. But is is if you use V100 for instance.
\nGood news is that using mixed precision helps GPU still. I am using amp in pytorch, and TF users should also use mixed precision. It helps, even if tensorcores aren't used.
", "votes": null, "replies": [] }, { "id": 969286, "author_name": "sovitrath", "author_url": "", "post_date": "08/13/2020 15:35:28", "content": "Exactly. Using AMP with even P100 GPUs we can literally use double the batch size of what we can without AMP. It helps a lot and the training is faster as well. It has become even easier with PyTorch 1.6.
", "votes": null, "replies": [] } ] }, { "id": 968357, "author_name": "darkcore", "author_url": "", "post_date": "08/12/2020 23:29:38", "content": "Thanks for sharing!
", "votes": null, "replies": [] }, { "id": 970458, "author_name": "luisr55", "author_url": "", "post_date": "08/14/2020 13:25:27", "content": "Thanx for the information :)
", "votes": null, "replies": [] }, { "id": 968981, "author_name": "sovitrath", "author_url": "", "post_date": "08/13/2020 11:36:30", "content": "With PyTorch 1.6, now you can use FP16 with Pascal architecture GPUs as well. You can have a look at my kernel where I use FP16 with GPU (PyTorch 1.6) instead of TPU. It's really easy and stable as well. Works with GTX 10 series GPU like a charm.\nhttps://www.kaggle.com/sovitrath/caltech-256-amp-pytorch/comments
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "967212": "Some new gpu's (RTX and Volta) have tensor cores for mixed precision training (fp16 instead of fp32). \n[Automatic Mixed Precision for Deep Learning](https://developer.nvidia.com/automatic-mixed-precision)\n\nIt's really easy to implement (tensorflow, other frameworks in the article above) just wrap your optimizer:\n`opt = tf.keras.optimizers.Adam(learning_rate=0.001)`\n`opt = tf.train.experimental.enable_mixed_precision_graph_rewrite(opt)`\n\nEffNet training time faster for ~40%", "967509": "thanks for mentioning it, since 6 days left have to do it!", "968258": "uspangaliev thanks for mentioning it. -)\n\nHere is a short summary of usages for all frameworks. (Direct for the above link).\n\n## TensorFlow\n```\nopt = tf.keras.optimizers.Adam(learning_rate=0.001)\nopt = tf.train.experimental.enable_mixed_precision_graph_rewrite(opt)\n```\n\n## PyTorch\n```\nmodel, optimizer = amp.initialize(model, optimizer, opt_level=\"O1\")\nwith amp.scale_loss(loss, optimizer) as scaled_loss:\n scaled_loss.backward()\n```\n## MXNet\n```\namp.init()\namp.init_trainer(trainer)\nwith amp.scale_loss(loss, trainer) as scaled_loss:\n autograd.backward(scaled_loss)\n```\n\n## PaddlePaddle\n```\nsgd = SGDOptimizer()\nmp_sgd = fluid.contrib.mixed_precision.decorator.decorate(sgd)\nmp_sgd.minimize(loss)\n```\n\n---\n\nAlso, see this discussion, it might help. [mixed precision with TPU tf.distrubute](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/169897)\n+\nCheck this interesting discussion, might help. - [Mixed precision training with tf.keras 2.0](https://github.com/tensorflow/tensorflow/issues/34406)", "968342": "Efficientnet don't use tensorcores on gpu. It is why they run faster on tpu.", "968357": "Thanks for sharing!", "968981": "With PyTorch 1.6, now you can use FP16 with Pascal architecture GPUs as well. You can have a look at my [kernel](https://www.kaggle.com/sovitrath/caltech-256-amp-pytorch/comments) where I use FP16 with GPU (PyTorch 1.6) instead of TPU. It's really easy and stable as well. Works with GTX 10 series GPU like a charm.\nhttps://www.kaggle.com/sovitrath/caltech-256-amp-pytorch/comments", "969086": "cpmpml is there a principle why that happens?", "969200": "Some fp16 kernels are missing in cuda. We are looking into it. The P100 available in Kaggle kernels have no tensorcores anyway, hence the issue is not relevant. But is is if you use V100 for instance.\n\nGood news is that using mixed precision helps GPU still. I am using amp in pytorch, and TF users should also use mixed precision. It helps, even if tensorcores aren't used.", "969286": "Exactly. Using AMP with even P100 GPUs we can literally use double the batch size of what we can without AMP. It helps a lot and the training is faster as well. It has become even easier with PyTorch 1.6.", "970458": "Thanx for the information :)" }, "source": "meta" }