![\"\"](\"https://i.imgur.com/jXvP3fW.png\")
I frequently encounter NAN in loss when training transformer in fp16.
\n(the problem disappears in fp32).
Does anyone have a suggestion to avoid this problem?
\ne.g.
It is difficult for me to catch the overflow and right now i do not know where the overflow has occurred. Does anyone have a good suggestion?
\nbut i thought each layer is already normalized by layerNorm?
", "messages": [ { "id": "1304266", "postDate": "05/12/2021 14:14:37", "content": "I frequently encounter NAN in loss when training transformer in fp16.
\n(the problem disappears in fp32).
Does anyone have a suggestion to avoid this problem?
\ne.g.
It is difficult for me to catch the overflow and right now i do not know where the overflow has occurred. Does anyone have a good suggestion?
\nbut i thought each layer is already normalized by layerNorm?
", "rawMarkdown": "I frequently encounter NAN in loss when training transformer in fp16.\n(the problem disappears in fp32).\n\nDoes anyone have a suggestion to avoid this problem?\ne.g. \n- adding clipping function to feed-forward layer?\n- or should i modify the softmax weight computation (minus off the max value)\n\nIt is difficult for me to catch the overflow and right now i do not know where the overflow has occurred. Does anyone have a good suggestion?\n\nbut i thought each layer is already normalized by layerNorm?", "votes": null }, { "id": "1304289", "postDate": "05/12/2021 14:35:27", "content": "Do you mean transformer decoder or the transformer image encoder you are (apparently) using?
\nI have found that it helps to add LayerNorm before input into transformer decoder. Although it worsens performance.
Have you tried clipping the gradient norm at optimizer level? That also helps to avoid NaNs (without using mentioned LayerNorm) in general.
\nOnce gain it worsens my performance instead of improving it, but I have the suspicion that the cause lies elsewhere.
In my case there is also no problem with fp32 training
\nYou can determine cause of NaNs by checking intermediate tensors for NaNs. For PyTorch e.g. not tensor.isnan().all()
It depends on the encoder, some visual transformer will not show this behaviour :) The decoder will not have this problem. You can test that: freeze encoder and train with the half preicision.
", "rawMarkdown": "It depends on the encoder, some visual transformer will not show this behaviour :) The decoder will not have this problem. You can test that: freeze encoder and train with the half preicision.", "votes": null }, { "id": "1304302", "postDate": "05/12/2021 14:43:36", "content": "yes=)
\nfor some transformers decoders I had to clip to 0.5 for others to 1.0 plus very low lr and batch_size not lower than 32.
I fight that, too.
\nWhen it happens, I try to just start from the last model that had no nan losses yet. Sometimes it runs without any issues after that, but there are times when it runs for only a few epochs and I start getting nans again, and in this case if I just try to continue as the last time, it will fail again very soon.
\nIn these cases, increasing the batch size can help, even with gradient accumulation. Somewhat equivalent to increasing the batch size is lowering the learning rate.
\nIf these doesn't help, I load back older model, like 5-10 epochs before the first nan was encountered and continue from there, usually with increased batch size.
\nNote that I don't save and load the optimizer state and don't use warmup after restarting - I act like nothing has happened.
I'm not sure of it, as I didn't go after what causes these nans, but I think that it happens when the model encounters very rare cases of inchis on which the model performs poorly. I only tried training on 40k training images in which I put some images with inchis that were unique in some ways compared to all the other (40k-1) images. The model was training well, there was no sign of sickness in the training loss, but the validation was a rollercoaster. If I removed those unique images, the validation was normal.
\nIf these unique inchis are really the causes, then eliminating those can help.
\nOr just monitoring if there are too high losses in the current batch, and if so, you can skip the update. (I didn't try this, as I had success continuing training so far with what I wrote before.)
I found out recently that there is more elegant way of detecting what causing underflow or overflow … here is a whole page from huggingface its pretty easy to implement https://huggingface.co/transformers/master/debugging.html#underflow-and-overflow-detection
huggingface solution is good.
\ni need to study it.
\nThanks a lot.
they are using pytorch hook function for callback
", "rawMarkdown": "huggingface solution is good.\n\ni need to study it.\nThanks a lot.\n\nthey are using pytorch hook function for callback", "votes": null }, { "id": "1305698", "postDate": "05/13/2021 12:35:04", "content": "if you are using fp16, you can finetune in fp32. my experiments show that fp32 produces better results
", "rawMarkdown": "if you are using fp16, you can finetune in fp32. my experiments show that fp32 produces better results", "votes": null }, { "id": "1306154", "postDate": "05/13/2021 16:28:59", "content": "I'm running into this issue at the moment. I tried using fp32 for the optimizer only to rule out that being the problem - it still ended up with nans. Now trying with a combination of lower lr and lower value for clip_grad_norm. If that fails I'll switch to fp32 for the whole thing. But then it ends up being S___L___O___W…
I met the same problem. my forked TPU training notebook work fine on KAGGLE. I manage to run it on Colab and continue training. Due to OOM problem, I have to reduce batchsize, however, then after several epochs, NAN pops up and training break.
", "rawMarkdown": "I met the same problem. my forked TPU training notebook work fine on KAGGLE. I manage to run it on Colab and continue training. Due to OOM problem, I have to reduce batchsize, however, then after several epochs, NAN pops up and training break.", "votes": null }, { "id": "1306509", "postDate": "05/13/2021 20:33:01", "content": "Beware if you use fp32 on GPU on Pytorch with models like EffNets
\nThe implementation of depthwise convolution is not optimal compared to tensorflow https://github.com/pytorch/pytorch/issues/18631
", "rawMarkdown": "Beware if you use fp32 on GPU on Pytorch with models like EffNets \n\nThe implementation of depthwise convolution is not optimal compared to tensorflow [https://github.com/pytorch/pytorch/issues/18631](https://github.com/pytorch/pytorch/issues/18631)", "votes": null }, { "id": "1306517", "postDate": "05/13/2021 20:42:42", "content": "I just wanted to point out an overview of my observations (some again) at one place. It might help others to determine their cause. Others could post alike below?
\nFramework: Pytorch, AMP Mixed precision training
=> Best solution so far: Keep dropout lower, no gradient norm, no NaNs, not best theoretical performance by a good margin :/
\n=> Second best: LayerNorm after CNN.
I haven't played too much with visual transformers (as @tugstugi suggests, I agree the encoder is the likely place where you are seeing the issues), but it seems they are generally trained with lower learning rates than we are used to with CNNs.
\nAlso, seems people devote quite a few epochs to warm up.
\nI have been consistently getting NaNs when training with a flat LR without warmup, but then I switched to this schedule:
from fastai.callback.schedule import *\n\np = torch.linspace(0.,1,100)\nf = combine_scheds([0.05, 0.95], [SchedLin(LR/1000, LR), SchedCos(LR, LR/100)])\nplt.plot(p, [f(o) for o in p]);\n
I train with roughly 1/3 max LR how the arch was trained for the paper, and scale the LR linearly with batch size. Seems transformers really like low LRs so this might still be too high.
fastai is very modular, so I am grabbing the scheduler and have been using it with pytorch and ignite as well.
Doing the above, I still see a weird jump in train loss some time into the training and subsequently the model trains very poorly. If I add gradient clipping, the problem goes away.
", "rawMarkdown": "I haven't played too much with visual transformers (as @tugstugi suggests, I agree the encoder is the likely place where you are seeing the issues), but it seems they are generally trained with lower learning rates than we are used to with CNNs.\n\nAlso, seems people devote quite a few epochs to warm up.\n\nI have been consistently getting `NaNs` when training with a flat `LR` without warmup, but then I switched to this schedule:\n\n```\nfrom fastai.callback.schedule import *\n\np = torch.linspace(0.,1,100)\nf = combine_scheds([0.05, 0.95], [SchedLin(LR/1000, LR), SchedCos(LR, LR/100)])\nplt.plot(p, [f(o) for o in p]);\n```\n\n\n\nI train with roughly 1/3 max `LR` how the arch was trained for the paper, and scale the `LR` linearly with batch size. Seems transformers really like low `LRs` so this might still be too high.\n\n`fastai` is very modular, so I am grabbing the scheduler and have been using it with pytorch and ignite as well.\n\nDoing the above, I still see a weird jump in train loss some time into the training and subsequently the model trains very poorly. If I add gradient clipping, the problem goes away.", "votes": null }, { "id": "1308757", "postDate": "05/15/2021 12:44:14", "content": "I had this issue with the LSTM approach and I found my NaN's were coming from overflow in the attention calculation for one random sample in one random batch. The sample didn't seem particularly special in any way. Therefore grad clipping seemed to have nothing to do with it.
", "rawMarkdown": "I had this issue with the LSTM approach and I found my NaN's were coming from overflow in the attention calculation for one random sample in one random batch. The sample didn't seem particularly special in any way. Therefore grad clipping seemed to have nothing to do with it.", "votes": null }, { "id": "1309498", "postDate": "05/16/2021 03:31:00", "content": "Certain Transformer variants play poorly with FP16 - I've encountered this with the T5 architecture for example in an NLP task at the GeGLU layer.
\nTwo common solutions :-
\nit is important to cross check your results.
\nNAN can happens in both training and inference
Do you mean transformer decoder or the transformer image encoder you are (apparently) using?
\nI have found that it helps to add LayerNorm before input into transformer decoder. Although it worsens performance.
Have you tried clipping the gradient norm at optimizer level? That also helps to avoid NaNs (without using mentioned LayerNorm) in general.
\nOnce gain it worsens my performance instead of improving it, but I have the suspicion that the cause lies elsewhere.
In my case there is also no problem with fp32 training
\nYou can determine cause of NaNs by checking intermediate tensors for NaNs. For PyTorch e.g. not tensor.isnan().all()
It depends on the encoder, some visual transformer will not show this behaviour :) The decoder will not have this problem. You can test that: freeze encoder and train with the half preicision.
", "votes": null, "replies": [ { "id": 1304302, "author_name": "drhabib", "author_url": "", "post_date": "05/12/2021 14:43:36", "content": "yes=)
\nfor some transformers decoders I had to clip to 0.5 for others to 1.0 plus very low lr and batch_size not lower than 32.
I fight that, too.
\nWhen it happens, I try to just start from the last model that had no nan losses yet. Sometimes it runs without any issues after that, but there are times when it runs for only a few epochs and I start getting nans again, and in this case if I just try to continue as the last time, it will fail again very soon.
\nIn these cases, increasing the batch size can help, even with gradient accumulation. Somewhat equivalent to increasing the batch size is lowering the learning rate.
\nIf these doesn't help, I load back older model, like 5-10 epochs before the first nan was encountered and continue from there, usually with increased batch size.
\nNote that I don't save and load the optimizer state and don't use warmup after restarting - I act like nothing has happened.
I'm not sure of it, as I didn't go after what causes these nans, but I think that it happens when the model encounters very rare cases of inchis on which the model performs poorly. I only tried training on 40k training images in which I put some images with inchis that were unique in some ways compared to all the other (40k-1) images. The model was training well, there was no sign of sickness in the training loss, but the validation was a rollercoaster. If I removed those unique images, the validation was normal.
\nIf these unique inchis are really the causes, then eliminating those can help.
\nOr just monitoring if there are too high losses in the current batch, and if so, you can skip the update. (I didn't try this, as I had success continuing training so far with what I wrote before.)
I found out recently that there is more elegant way of detecting what causing underflow or overflow … here is a whole page from huggingface its pretty easy to implement https://huggingface.co/transformers/master/debugging.html#underflow-and-overflow-detection
huggingface solution is good.
\ni need to study it.
\nThanks a lot.
they are using pytorch hook function for callback
", "votes": null, "replies": [] } ] }, { "id": 1305698, "author_name": "hengck23", "author_url": "", "post_date": "05/13/2021 12:35:04", "content": "if you are using fp16, you can finetune in fp32. my experiments show that fp32 produces better results
", "votes": null, "replies": [] }, { "id": 1306154, "author_name": "fergusoci", "author_url": "", "post_date": "05/13/2021 16:28:59", "content": "I'm running into this issue at the moment. I tried using fp32 for the optimizer only to rule out that being the problem - it still ended up with nans. Now trying with a combination of lower lr and lower value for clip_grad_norm. If that fails I'll switch to fp32 for the whole thing. But then it ends up being S___L___O___W…
I met the same problem. my forked TPU training notebook work fine on KAGGLE. I manage to run it on Colab and continue training. Due to OOM problem, I have to reduce batchsize, however, then after several epochs, NAN pops up and training break.
", "votes": null, "replies": [] }, { "id": 1306509, "author_name": "serigne", "author_url": "", "post_date": "05/13/2021 20:33:01", "content": "Beware if you use fp32 on GPU on Pytorch with models like EffNets
\nThe implementation of depthwise convolution is not optimal compared to tensorflow https://github.com/pytorch/pytorch/issues/18631
", "votes": null, "replies": [] }, { "id": 1306517, "author_name": "cepheidq", "author_url": "", "post_date": "05/13/2021 20:42:42", "content": "I just wanted to point out an overview of my observations (some again) at one place. It might help others to determine their cause. Others could post alike below?
\nFramework: Pytorch, AMP Mixed precision training
=> Best solution so far: Keep dropout lower, no gradient norm, no NaNs, not best theoretical performance by a good margin :/
\n=> Second best: LayerNorm after CNN.
I haven't played too much with visual transformers (as @tugstugi suggests, I agree the encoder is the likely place where you are seeing the issues), but it seems they are generally trained with lower learning rates than we are used to with CNNs.
\nAlso, seems people devote quite a few epochs to warm up.
\nI have been consistently getting NaNs when training with a flat LR without warmup, but then I switched to this schedule:
from fastai.callback.schedule import *\n\np = torch.linspace(0.,1,100)\nf = combine_scheds([0.05, 0.95], [SchedLin(LR/1000, LR), SchedCos(LR, LR/100)])\nplt.plot(p, [f(o) for o in p]);\n
I train with roughly 1/3 max LR how the arch was trained for the paper, and scale the LR linearly with batch size. Seems transformers really like low LRs so this might still be too high.
fastai is very modular, so I am grabbing the scheduler and have been using it with pytorch and ignite as well.
Doing the above, I still see a weird jump in train loss some time into the training and subsequently the model trains very poorly. If I add gradient clipping, the problem goes away.
", "votes": null, "replies": [] }, { "id": 1308757, "author_name": "alexandersoare", "author_url": "", "post_date": "05/15/2021 12:44:14", "content": "I had this issue with the LSTM approach and I found my NaN's were coming from overflow in the attention calculation for one random sample in one random batch. The sample didn't seem particularly special in any way. Therefore grad clipping seemed to have nothing to do with it.
", "votes": null, "replies": [] }, { "id": 1309498, "author_name": "leecming", "author_url": "", "post_date": "05/16/2021 03:31:00", "content": "Certain Transformer variants play poorly with FP16 - I've encountered this with the T5 architecture for example in an NLP task at the GeGLU layer.
\nTwo common solutions :-
\nit is important to cross check your results.
\nNAN can happens in both training and inference