Every kaggle competition solves a different problem and i learn a different thing. In this Carvana Image Masking Challenge, able to hange large input and output (e.g. prediction of mask at 1024x1024) is may be an advantage.
\n\nThis thread is for using fp16 (16-bit float) and multi-gpu training and inference. I hope experienced kagglers who had worked with fp16 or multi-gpu can answer our question here.
\n\nFor a start, I have probelm with BN layer.
\n\nwill BN layer be affected. (e.g. the moving vars needs high precision) by fp16?
for multi gpu, how to compute BN statistics (moving mean and vars) over 4 gpu? The results doesn't seems to be stable? I read that if one solution is to freeze BN statistics (assuming that you have a pretrain model to begin with). Is that what you really do?
There is a new paper on batch renormalisation. https://arxiv.org/abs/1702.03275.\nHas any try this and with this solve the issues above?
\n\nI will post some code later, if the results are stable.
", "messages": [ { "id": "209357", "postDate": "08/02/2017 00:50:09", "content": "Every kaggle competition solves a different problem and i learn a different thing. In this Carvana Image Masking Challenge, able to hange large input and output (e.g. prediction of mask at 1024x1024) is may be an advantage.
\n\nThis thread is for using fp16 (16-bit float) and multi-gpu training and inference. I hope experienced kagglers who had worked with fp16 or multi-gpu can answer our question here.
\n\nFor a start, I have probelm with BN layer.
\n\nwill BN layer be affected. (e.g. the moving vars needs high precision) by fp16?
for multi gpu, how to compute BN statistics (moving mean and vars) over 4 gpu? The results doesn't seems to be stable? I read that if one solution is to freeze BN statistics (assuming that you have a pretrain model to begin with). Is that what you really do?
There is a new paper on batch renormalisation. https://arxiv.org/abs/1702.03275.\nHas any try this and with this solve the issues above?
\n\nI will post some code later, if the results are stable.
", "rawMarkdown": "Every kaggle competition solves a different problem and i learn a different thing. In this Carvana Image Masking Challenge, able to hange large input and output (e.g. prediction of mask at 1024x1024) is may be an advantage. \n\nThis thread is for using fp16 (16-bit float) and multi-gpu training and inference. I hope experienced kagglers who had worked with fp16 or multi-gpu can answer our question here.\n\nFor a start, I have probelm with BN layer.\n\n 1. will BN layer be affected. (e.g. the moving vars needs high precision) by fp16?\n \n 2. for multi gpu, how to compute BN statistics (moving mean and vars) over 4 gpu? The results doesn't seems to be stable? I read that if one solution is to freeze BN statistics (assuming that you have a pretrain model to begin with). Is that what you really do?\n \nThere is a new paper on batch renormalisation. https://arxiv.org/abs/1702.03275.\nHas any try this and with this solve the issues above?\n\nI will post some code later, if the results are stable.", "votes": null }, { "id": "209404", "postDate": "08/02/2017 05:44:02", "content": "You don't want to use fp16 on usual GPUs, it is much slower on them (basically on everything except Teslas). Also, it is usually fine for inference but while training we still need fp32 precision.
\n\nBut I like how you are open to experiments, keep going!
", "rawMarkdown": "You don't want to use fp16 on usual GPUs, it is much slower on them (basically on everything except Teslas). Also, it is usually fine for inference but while training we still need fp32 precision. \n\nBut I like how you are open to experiments, keep going!", "votes": null }, { "id": "209436", "postDate": "08/02/2017 07:58:46", "content": "I agree with Sergey, for training I think fp32 is needed.
\n\nHowever on inference you could try using int8. NVIDIA has a tool called TensorRT that allows to optimize a network for using int8 with a small loss on precision.
\nhttp://on-demand.gputechconf.com/gtc/2017/presentation/s7310-8-bit-inference-with-tensorrt.pdf
However it's a quite complicated process on this days. I believe it will become much easier in the future.
", "rawMarkdown": "I agree with Sergey, for training I think fp32 is needed. \n\nHowever on inference you could try using int8. NVIDIA has a tool called TensorRT that allows to optimize a network for using int8 with a small loss on precision. \nhttp://on-demand.gputechconf.com/gtc/2017/presentation/s7310-8-bit-inference-with-tensorrt.pdf\n\nHowever it's a quite complicated process on this days. I believe it will become much easier in the future.", "votes": null }, { "id": "211087", "postDate": "08/08/2017 01:39:51", "content": "sorry for this noob question. But what do you all mean by inference in this context? I see the word mentioned a lot. Do you mean predicting the test set only or something else?
", "rawMarkdown": "sorry for this noob question. But what do you all mean by inference in this context? I see the word mentioned a lot. Do you mean predicting the test set only or something else?", "votes": null }, { "id": "211097", "postDate": "08/08/2017 02:29:05", "content": "Yeah, in this particular context, it means predicting the bitmap of the crop mask for data we don't have the ground truth to. This competition has the additional requirement of packaging predictions as run-length encodings in order to limit submission sizes
", "rawMarkdown": "Yeah, in this particular context, it means predicting the bitmap of the crop mask for data we don't have the ground truth to. This competition has the additional requirement of packaging predictions as run-length encodings in order to limit submission sizes", "votes": null }, { "id": "211108", "postDate": "08/08/2017 03:24:22", "content": "it means test mode. back propagation is not required and intermediate values are not cached.
", "rawMarkdown": "it means test mode. back propagation is not required and intermediate values are not cached.", "votes": null }, { "id": "218177", "postDate": "09/02/2017 19:36:33", "content": "BN can be trained in fp16 without overflow and loss of accuracy here. Example training loop is shown below. it reduces GPU memory by 40% for the feature maps so that the network can be wider (i.e. using more channels). The conversion between fp16 and fp32 however does take some time.
\n\n images = Variable(images).cuda()\n labels = Variable(labels).cuda()\n\n\n net.half() ## \n images = images.half() ##\n\n #forward\n logits = net(images)\n logits = logits.float() ##\n\n loss = criterion(logits, labels) \n\n loss.backward() \n net.float() ##\n optimizer.step()\nIsn't fp16 much slower than fp32 on Geforce GPUs?
", "rawMarkdown": "Isn't fp16 much slower than fp32 on Geforce GPUs?", "votes": null }, { "id": "218236", "postDate": "09/03/2017 03:18:14", "content": "I tried using np.float16 in planet very briefly since my gpu wasn't able to compete so much on large batch sizes and resolutions. I think I faced an error that was taking too much time or maybe it was saying fp16 wasn't supported. Looks like the HalfTensor is a relatively new addition:\nhttps://github.com/pytorch/pytorch/commit/67f94557ff26428ac911d8c08c7f9b619a41950e
\n\nDon't remember what the exact problem was but I think this would definitely be worth a little more time, just depends on how much more time. Luckily, you can likely see if it's worth it or not after a handful of epochs. The 1080 seems to handle half floats well for me. I think nvidia increased the fp16 tflops for the new series as a move to be more supportive of building larger networks, but that's just a guess.
", "rawMarkdown": "I tried using np.float16 in planet very briefly since my gpu wasn't able to compete so much on large batch sizes and resolutions. I think I faced an error that was taking too much time or maybe it was saying fp16 wasn't supported. Looks like the HalfTensor is a relatively new addition:\nhttps://github.com/pytorch/pytorch/commit/67f94557ff26428ac911d8c08c7f9b619a41950e\n\nDon't remember what the exact problem was but I think this would definitely be worth a little more time, just depends on how much more time. Luckily, you can likely see if it's worth it or not after a handful of epochs. The 1080 seems to handle half floats well for me. I think nvidia increased the fp16 tflops for the new series as a move to be more supportive of building larger networks, but that's just a guess.", "votes": null }, { "id": "218239", "postDate": "09/03/2017 03:41:29", "content": "it is slower. but you can have a bigger model
", "rawMarkdown": "it is slower. but you can have a bigger model", "votes": null }, { "id": "432818", "postDate": "12/04/2018 11:13:10", "content": "Hi, I was experimenting with this for a while and made a POC, so I've yet to test on Volta or Tesla GPU, but on Pascal architecture I got about 15s less per epoch, that is almost 50mins less than the training time in FP32. Best per till now is that the storage is exactly halved.\nYou can checkout my code here: suvojit-0x55aa/mixed-precision-pytorch \nAny suggestion and opinions are most welcome.
", "rawMarkdown": "Hi, I was experimenting with this for a while and made a POC, so I've yet to test on Volta or Tesla GPU, but on Pascal architecture I got about 15s less per epoch, that is almost 50mins less than the training time in FP32. Best per till now is that the storage is exactly halved.\nYou can checkout my code here: [suvojit-0x55aa/mixed-precision-pytorch](https://github.com/suvojit-0x55aa/mixed-precision-pytorch) \nAny suggestion and opinions are most welcome.", "votes": null }, { "id": "755090", "postDate": "02/24/2020 12:20:05", "content": "I'm only commenting because this is a top result from Google, even 3 years later. \nFP16 (half float) is considerably faster on any up-to-date GPU (Pascal and later) and you can\neasily see this for your self by training using cuda().half() vs cuda().float().\nHowever, half often leads to numerical instability, resulting in nan or other issues.\nThere are workarounds, but it depends on the network parameters and the optimizer.
You don't want to use fp16 on usual GPUs, it is much slower on them (basically on everything except Teslas). Also, it is usually fine for inference but while training we still need fp32 precision.
\n\nBut I like how you are open to experiments, keep going!
", "votes": null, "replies": [] }, { "id": 209436, "author_name": "ironbar", "author_url": "", "post_date": "08/02/2017 07:58:46", "content": "I agree with Sergey, for training I think fp32 is needed.
\n\nHowever on inference you could try using int8. NVIDIA has a tool called TensorRT that allows to optimize a network for using int8 with a small loss on precision.
\nhttp://on-demand.gputechconf.com/gtc/2017/presentation/s7310-8-bit-inference-with-tensorrt.pdf
However it's a quite complicated process on this days. I believe it will become much easier in the future.
", "votes": null, "replies": [] }, { "id": 211087, "author_name": "cjansen", "author_url": "", "post_date": "08/08/2017 01:39:51", "content": "sorry for this noob question. But what do you all mean by inference in this context? I see the word mentioned a lot. Do you mean predicting the test set only or something else?
", "votes": null, "replies": [ { "id": 211097, "author_name": "cpruce", "author_url": "", "post_date": "08/08/2017 02:29:05", "content": "Yeah, in this particular context, it means predicting the bitmap of the crop mask for data we don't have the ground truth to. This competition has the additional requirement of packaging predictions as run-length encodings in order to limit submission sizes
", "votes": null, "replies": [] }, { "id": 211108, "author_name": "hengck23", "author_url": "", "post_date": "08/08/2017 03:24:22", "content": "it means test mode. back propagation is not required and intermediate values are not cached.
", "votes": null, "replies": [] } ] }, { "id": 218177, "author_name": "hengck23", "author_url": "", "post_date": "09/02/2017 19:36:33", "content": "BN can be trained in fp16 without overflow and loss of accuracy here. Example training loop is shown below. it reduces GPU memory by 40% for the feature maps so that the network can be wider (i.e. using more channels). The conversion between fp16 and fp32 however does take some time.
\n\n images = Variable(images).cuda()\n labels = Variable(labels).cuda()\n\n\n net.half() ## \n images = images.half() ##\n\n #forward\n logits = net(images)\n logits = logits.float() ##\n\n loss = criterion(logits, labels) \n\n loss.backward() \n net.float() ##\n optimizer.step()\nIsn't fp16 much slower than fp32 on Geforce GPUs?
", "votes": null, "replies": [] }, { "id": 218236, "author_name": "cpruce", "author_url": "", "post_date": "09/03/2017 03:18:14", "content": "I tried using np.float16 in planet very briefly since my gpu wasn't able to compete so much on large batch sizes and resolutions. I think I faced an error that was taking too much time or maybe it was saying fp16 wasn't supported. Looks like the HalfTensor is a relatively new addition:\nhttps://github.com/pytorch/pytorch/commit/67f94557ff26428ac911d8c08c7f9b619a41950e
\n\nDon't remember what the exact problem was but I think this would definitely be worth a little more time, just depends on how much more time. Luckily, you can likely see if it's worth it or not after a handful of epochs. The 1080 seems to handle half floats well for me. I think nvidia increased the fp16 tflops for the new series as a move to be more supportive of building larger networks, but that's just a guess.
", "votes": null, "replies": [] }, { "id": 218239, "author_name": "hengck23", "author_url": "", "post_date": "09/03/2017 03:41:29", "content": "it is slower. but you can have a bigger model
", "votes": null, "replies": [] } ] }, { "id": 432818, "author_name": "shinmigami", "author_url": "", "post_date": "12/04/2018 11:13:10", "content": "Hi, I was experimenting with this for a while and made a POC, so I've yet to test on Volta or Tesla GPU, but on Pascal architecture I got about 15s less per epoch, that is almost 50mins less than the training time in FP32. Best per till now is that the storage is exactly halved.\nYou can checkout my code here: suvojit-0x55aa/mixed-precision-pytorch \nAny suggestion and opinions are most welcome.
", "votes": null, "replies": [] }, { "id": 755090, "author_name": "alexgiokas", "author_url": "", "post_date": "02/24/2020 12:20:05", "content": "I'm only commenting because this is a top result from Google, even 3 years later. \nFP16 (half float) is considerably faster on any up-to-date GPU (Pascal and later) and you can\neasily see this for your self by training using cuda().half() vs cuda().float().\nHowever, half often leads to numerical instability, resulting in nan or other issues.\nThere are workarounds, but it depends on the network parameters and the optimizer.