Hi All, I am trying to implement label smoothing in pytorch as below:
\n\n```\nclass LabelSmoothing(nn.Module):\n def init(self, smoothing = 0.05):\n super(LabelSmoothing, self).init()\n self.confidence = 1.0 - smoothing\n self.smoothing = smoothing
\n\ndef forward(self, x, target):\n if self.training:\n x = x.float()\n target = target.float()\n logprobs = torch.nn.functional.log_softmax(x, dim = -1)\n\n nll_loss = -logprobs * target\n nll_loss = nll_loss.sum(-1)\n\n smooth_loss = -logprobs.mean(dim=-1)\n\n loss = self.confidence * nll_loss + self.smoothing * smooth_loss\n\n return loss.mean()\n else:\n return torch.nn.functional.cross_entropy(x, target)\n```
\n\nHowever, my loss remains at 0.0 throughout. My gut feeling is that it has to do with one-hot encoding the vectors (or lack of it in my implementation). Looking for thoughts on how I can make it work. My original loss fn is BCE withlogits.
", "messages": [ { "id": "928773", "postDate": "07/14/2020 07:55:03", "content": "Hi All, I am trying to implement label smoothing in pytorch as below:
\n\n```\nclass LabelSmoothing(nn.Module):\n def init(self, smoothing = 0.05):\n super(LabelSmoothing, self).init()\n self.confidence = 1.0 - smoothing\n self.smoothing = smoothing
\n\ndef forward(self, x, target):\n if self.training:\n x = x.float()\n target = target.float()\n logprobs = torch.nn.functional.log_softmax(x, dim = -1)\n\n nll_loss = -logprobs * target\n nll_loss = nll_loss.sum(-1)\n\n smooth_loss = -logprobs.mean(dim=-1)\n\n loss = self.confidence * nll_loss + self.smoothing * smooth_loss\n\n return loss.mean()\n else:\n return torch.nn.functional.cross_entropy(x, target)\n```
\n\nHowever, my loss remains at 0.0 throughout. My gut feeling is that it has to do with one-hot encoding the vectors (or lack of it in my implementation). Looking for thoughts on how I can make it work. My original loss fn is BCE withlogits.
", "rawMarkdown": "Hi All, I am trying to implement label smoothing in pytorch as below:\n\n```\nclass LabelSmoothing(nn.Module):\n def __init__(self, smoothing = 0.05):\n super(LabelSmoothing, self).__init__()\n self.confidence = 1.0 - smoothing\n self.smoothing = smoothing\n\n def forward(self, x, target):\n if self.training:\n x = x.float()\n target = target.float()\n logprobs = torch.nn.functional.log_softmax(x, dim = -1)\n\n nll_loss = -logprobs * target\n nll_loss = nll_loss.sum(-1)\n \n smooth_loss = -logprobs.mean(dim=-1)\n\n loss = self.confidence * nll_loss + self.smoothing * smooth_loss\n\n return loss.mean()\n else:\n return torch.nn.functional.cross_entropy(x, target)\n```\n\nHowever, my loss remains at 0.0 throughout. My gut feeling is that it has to do with one-hot encoding the vectors (or lack of it in my implementation). Looking for thoughts on how I can make it work. My original loss fn is BCE withlogits.", "votes": null }, { "id": "928930", "postDate": "07/14/2020 10:31:02", "content": "Why do you write smooth_loss this way? I would expect something like
\nsmooth_loss = -logprobs * (1-target)\nsmooth_loss = smooth_loss .sum(-1)\n
@zaharch @pheadrus Please correct me if I'm wrong, but I'd think that you can simply smooth the targets directly
\n\nUnless it's ignored, it seems to work fine here: https://www.kaggle.com/hmendonca/melanoma-neat-pytorch-lightning-native-amp#Model\n\n y_smo = y.float() * (1 - label_smoothing) + 0.5 * label_smoothing\n loss = F.binary_cross_entropy_with_logits(y_hat, y_smo.type_as(y_hat),\n pos_weight=torch.tensor(pos_weight))\n
I've just checked, and the code above outputs the same as tf.keras.losses.binary_crossentropy with the same label_smoothing
Interestingly, higher label_smoothing actually increases the loss, which seems a bit counter-intuitive to me...
Agree, and it is better than what I proposed (yours is shorter).
\n\nThe loss increases because now there is less certainty in the targets. Consider the case that you don't have any information besides that there are 98% target 0 and 2% target 1. Then if you give 0.02 score to all samples you get small loss:\n\n-0.02*np.log(0.02) -0.98*np.log(0.98) = 0.098\n\nbut if now your targets are 90% to 10% then only with the prior one can get at best:\n\n-0.1*np.log(0.1) -0.9*np.log(0.9) = 0.325\n
Yep, I think this should do it. Thanks @hmendonca
", "rawMarkdown": "Yep, I think this should do it. Thanks @hmendonca", "votes": null }, { "id": "929587", "postDate": "07/14/2020 19:07:12", "content": "@hmendonca , I am a bit confused by this implementation. Because it makes smoothing only by half of label_smoothing value. Maybe something like torch.abs(y.float() - label_smoothing) will be more correct for binary case?
Yes, it does @vladimirsydor \nlabel_smoothing = 0.02 makes your binary targets 0.01 and 0.99
as in https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy :
\n\n\n\n\nlabel_smoothing\n Float in [0, 1]. When 0, no smoothing occurs. When > 0, we compute the loss between the predicted labels and a smoothed version of the true labels, where the smoothing squeezes the labels towards 0.5. Larger values of
\nlabel_smoothingcorrespond to heavier smoothing.
label_smoothing is a ratio of smoothing, i.e. label_smoothing = 1.0 push all targets all the way to 0.5 (which is not particularly useful ;)
@hmendonca I think your implementation works great, just did a test and single model BCEloss gives 0.933 CV and 0.915 LB. With your label smoothing i get 0.938 CV and 0.925 LB
\n\nDo you think your label smoothing can work with focal loss?
", "rawMarkdown": "hmendonca I think your implementation works great, just did a test and single model BCEloss gives 0.933 CV and 0.915 LB. With your label smoothing i get 0.938 CV and 0.925 LB\n\nDo you think your label smoothing can work with focal loss?", "votes": null }, { "id": "930044", "postDate": "07/15/2020 06:47:48", "content": "@yannmajewski please be careful about taking conclusions from any isolated experiment, as DL has a very high intrinsic variance/stochasticity\nI.e. the same pipeline may give you very different results with just slight changes in hyper parameters and/or even just the order of training data points, as well as, the random weights initialisation, augmentation, etc...
\n\nHaving said that, label_smoothing normally helps generalisation (test accuracy and AUC) in the presence of label noise, which is generally the case.\nAnd it should also work fine together with focal loss.\nHappy Kaggling :)
", "rawMarkdown": "yannmajewski please be careful about taking conclusions from any isolated experiment, as DL has a very high intrinsic variance/stochasticity\nI.e. the same pipeline may give you very different results with just slight changes in hyper parameters and/or even just the order of training data points, as well as, the random weights initialisation, augmentation, etc...\n\nHaving said that, label_smoothing normally helps generalisation (test accuracy and AUC) in the presence of label noise, which is generally the case.\nAnd it should also work fine together with focal loss.\nHappy Kaggling :)", "votes": null }, { "id": "930136", "postDate": "07/15/2020 08:15:00", "content": "This is what I'm using as a direct drop-in for torch.nn.CrossEntropyLoss.\n```\nclass LabelSmoothingLoss(torch.nn.Module):\n def init(self, smoothing: float = 0.1, reduction=\"mean\", weight=None):\n super(LabelSmoothingLoss, self).init()\n self.epsilon = smoothing\n self.reduction = reduction\n self.weight = weight
def reduce_loss(self, loss):\n if self.reduction == \"mean\":\n return loss.mean()\n elif self.reduction == \"sum\":\n return loss.sum()\n else:\n return loss\n\ndef linear_combination(self, x, y):\n return self.epsilon * x + (1 - self.epsilon) * y\n\ndef forward(self, preds, target):\n if self.weight is not None:\n self.weight = self.weight.to(preds.device)\n\n if self.training:\n n = preds.size(-1)\n log_preds = F.log_softmax(preds, dim=-1)\n loss = self.reduce_loss(-log_preds.sum(dim=-1))\n nll = F.nll_loss(\n log_preds, target, reduction=self.reduction, weight=self.weight\n )\n return self.linear_combination(loss / n, nll)\n else:\n return torch.nn.functional.cross_entropy(preds, target, weight=self.weight)\n```\nEdit: Updated with a version that accepts weights
", "rawMarkdown": "This is what I'm using as a direct drop-in for `torch.nn.CrossEntropyLoss`.\n```\nclass LabelSmoothingLoss(torch.nn.Module):\n def __init__(self, smoothing: float = 0.1, reduction=\"mean\", weight=None):\n super(LabelSmoothingLoss, self).__init__()\n self.epsilon = smoothing\n self.reduction = reduction\n self.weight = weight\n\n def reduce_loss(self, loss):\n if self.reduction == \"mean\":\n return loss.mean()\n elif self.reduction == \"sum\":\n return loss.sum()\n else:\n return loss\n\n def linear_combination(self, x, y):\n return self.epsilon * x + (1 - self.epsilon) * y\n\n def forward(self, preds, target):\n if self.weight is not None:\n self.weight = self.weight.to(preds.device)\n\n if self.training:\n n = preds.size(-1)\n log_preds = F.log_softmax(preds, dim=-1)\n loss = self.reduce_loss(-log_preds.sum(dim=-1))\n nll = F.nll_loss(\n log_preds, target, reduction=self.reduction, weight=self.weight\n )\n return self.linear_combination(loss / n, nll)\n else:\n return torch.nn.functional.cross_entropy(preds, target, weight=self.weight)\n```\nEdit: Updated with a version that accepts weights", "votes": null }, { "id": "963323", "postDate": "08/08/2020 22:38:23", "content": "@hmendonca any particular reason you use the F.binary_cross_entropy_with_logits()instead of the NN version?
No @brianfeeny it's the same function.
\nBCEWithLogitsLoss is just a Module wrapper for F.binarycrossentropywithlogits, which can be nice for reusability in some cases, e.g. if you keep your hyper params somewhere else
check out the code:
\nhttps://github.com/albanie/pytorch/blob/master/torch/nn/modules/loss.py#L492-L496
Why do you write smooth_loss this way? I would expect something like
\nsmooth_loss = -logprobs * (1-target)\nsmooth_loss = smooth_loss .sum(-1)\n
@zaharch @pheadrus Please correct me if I'm wrong, but I'd think that you can simply smooth the targets directly
\n\nUnless it's ignored, it seems to work fine here: https://www.kaggle.com/hmendonca/melanoma-neat-pytorch-lightning-native-amp#Model\n\n y_smo = y.float() * (1 - label_smoothing) + 0.5 * label_smoothing\n loss = F.binary_cross_entropy_with_logits(y_hat, y_smo.type_as(y_hat),\n pos_weight=torch.tensor(pos_weight))\n
I've just checked, and the code above outputs the same as tf.keras.losses.binary_crossentropy with the same label_smoothing
Interestingly, higher label_smoothing actually increases the loss, which seems a bit counter-intuitive to me...
Agree, and it is better than what I proposed (yours is shorter).
\n\nThe loss increases because now there is less certainty in the targets. Consider the case that you don't have any information besides that there are 98% target 0 and 2% target 1. Then if you give 0.02 score to all samples you get small loss:\n\n-0.02*np.log(0.02) -0.98*np.log(0.98) = 0.098\n\nbut if now your targets are 90% to 10% then only with the prior one can get at best:\n\n-0.1*np.log(0.1) -0.9*np.log(0.9) = 0.325\n
Yep, I think this should do it. Thanks @hmendonca
", "votes": null, "replies": [] }, { "id": 929788, "author_name": "yannmajewski", "author_url": "", "post_date": "07/15/2020 00:30:49", "content": "@hmendonca I think your implementation works great, just did a test and single model BCEloss gives 0.933 CV and 0.915 LB. With your label smoothing i get 0.938 CV and 0.925 LB
\n\nDo you think your label smoothing can work with focal loss?
", "votes": null, "replies": [] }, { "id": 930044, "author_name": "hmendonca", "author_url": "", "post_date": "07/15/2020 06:47:48", "content": "@yannmajewski please be careful about taking conclusions from any isolated experiment, as DL has a very high intrinsic variance/stochasticity\nI.e. the same pipeline may give you very different results with just slight changes in hyper parameters and/or even just the order of training data points, as well as, the random weights initialisation, augmentation, etc...
\n\nHaving said that, label_smoothing normally helps generalisation (test accuracy and AUC) in the presence of label noise, which is generally the case.\nAnd it should also work fine together with focal loss.\nHappy Kaggling :)
", "votes": null, "replies": [] }, { "id": 963323, "author_name": "brianfeeny", "author_url": "", "post_date": "08/08/2020 22:38:23", "content": "@hmendonca any particular reason you use the F.binary_cross_entropy_with_logits()instead of the NN version?
No @brianfeeny it's the same function.
\nBCEWithLogitsLoss is just a Module wrapper for F.binarycrossentropywithlogits, which can be nice for reusability in some cases, e.g. if you keep your hyper params somewhere else
check out the code:
\nhttps://github.com/albanie/pytorch/blob/master/torch/nn/modules/loss.py#L492-L496
@hmendonca , I am a bit confused by this implementation. Because it makes smoothing only by half of label_smoothing value. Maybe something like torch.abs(y.float() - label_smoothing) will be more correct for binary case?
Yes, it does @vladimirsydor \nlabel_smoothing = 0.02 makes your binary targets 0.01 and 0.99
as in https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy :
\n\n\n\n\nlabel_smoothing\n Float in [0, 1]. When 0, no smoothing occurs. When > 0, we compute the loss between the predicted labels and a smoothed version of the true labels, where the smoothing squeezes the labels towards 0.5. Larger values of
\nlabel_smoothingcorrespond to heavier smoothing.
label_smoothing is a ratio of smoothing, i.e. label_smoothing = 1.0 push all targets all the way to 0.5 (which is not particularly useful ;)
This is what I'm using as a direct drop-in for torch.nn.CrossEntropyLoss.\n```\nclass LabelSmoothingLoss(torch.nn.Module):\n def init(self, smoothing: float = 0.1, reduction=\"mean\", weight=None):\n super(LabelSmoothingLoss, self).init()\n self.epsilon = smoothing\n self.reduction = reduction\n self.weight = weight
def reduce_loss(self, loss):\n if self.reduction == \"mean\":\n return loss.mean()\n elif self.reduction == \"sum\":\n return loss.sum()\n else:\n return loss\n\ndef linear_combination(self, x, y):\n return self.epsilon * x + (1 - self.epsilon) * y\n\ndef forward(self, preds, target):\n if self.weight is not None:\n self.weight = self.weight.to(preds.device)\n\n if self.training:\n n = preds.size(-1)\n log_preds = F.log_softmax(preds, dim=-1)\n loss = self.reduce_loss(-log_preds.sum(dim=-1))\n nll = F.nll_loss(\n log_preds, target, reduction=self.reduction, weight=self.weight\n )\n return self.linear_combination(loss / n, nll)\n else:\n return torch.nn.functional.cross_entropy(preds, target, weight=self.weight)\n```\nEdit: Updated with a version that accepts weights
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "928773": "Hi All, I am trying to implement label smoothing in pytorch as below:\n\n```\nclass LabelSmoothing(nn.Module):\n def __init__(self, smoothing = 0.05):\n super(LabelSmoothing, self).__init__()\n self.confidence = 1.0 - smoothing\n self.smoothing = smoothing\n\n def forward(self, x, target):\n if self.training:\n x = x.float()\n target = target.float()\n logprobs = torch.nn.functional.log_softmax(x, dim = -1)\n\n nll_loss = -logprobs * target\n nll_loss = nll_loss.sum(-1)\n \n smooth_loss = -logprobs.mean(dim=-1)\n\n loss = self.confidence * nll_loss + self.smoothing * smooth_loss\n\n return loss.mean()\n else:\n return torch.nn.functional.cross_entropy(x, target)\n```\n\nHowever, my loss remains at 0.0 throughout. My gut feeling is that it has to do with one-hot encoding the vectors (or lack of it in my implementation). Looking for thoughts on how I can make it work. My original loss fn is BCE withlogits.", "928930": "Why do you write `smooth_loss` this way? I would expect something like\n\n```\nsmooth_loss = -logprobs * (1-target)\nsmooth_loss = smooth_loss .sum(-1)\n```", "929222": "zaharch @pheadrus Please correct me if I'm wrong, but I'd think that you can simply smooth the targets directly\n\nUnless it's ignored, it seems to work fine here: https://www.kaggle.com/hmendonca/melanoma-neat-pytorch-lightning-native-amp#Model\n```\n y_smo = y.float() * (1 - label_smoothing) + 0.5 * label_smoothing\n loss = F.binary_cross_entropy_with_logits(y_hat, y_smo.type_as(y_hat),\n pos_weight=torch.tensor(pos_weight))\n```", "929250": "I've just checked, and the code above outputs the same as `tf.keras.losses.binary_crossentropy` with the same `label_smoothing`\n\nInterestingly, higher `label_smoothing` actually increases the loss, which seems a bit counter-intuitive to me...", "929373": "Agree, and it is better than what I proposed (yours is shorter). \n\nThe loss increases because now there is less certainty in the targets. Consider the case that you don't have any information besides that there are 98% target 0 and 2% target 1. Then if you give 0.02 score to all samples you get small loss:\n```\n-0.02*np.log(0.02) -0.98*np.log(0.98) = 0.098\n```\nbut if now your targets are 90% to 10% then only with the prior one can get at best:\n```\n-0.1*np.log(0.1) -0.9*np.log(0.9) = 0.325\n```", "929405": "Yep, I think this should do it. Thanks @hmendonca", "929587": "hmendonca , I am a bit confused by this implementation. Because it makes smoothing only by half of `label_smoothing` value. Maybe something like `torch.abs(y.float() - label_smoothing)` will be more correct for binary case?", "929677": "Yes, it does @vladimirsydor \n`label_smoothing = 0.02` makes your binary targets 0.01 and 0.99\n\nas in https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy :\n> **label\\_smoothing**\n> \tFloat in [0, 1]. When 0, no smoothing occurs. When > 0, we compute the loss between the predicted labels and a smoothed version of the true labels, where the smoothing squeezes the labels towards 0.5. Larger values of `label_smoothing` correspond to heavier smoothing. \n\n`label_smoothing` is a ratio of smoothing, i.e. `label_smoothing = 1.0` push all targets all the way to 0.5 (which is not particularly useful ;)", "929788": "hmendonca I think your implementation works great, just did a test and single model BCEloss gives 0.933 CV and 0.915 LB. With your label smoothing i get 0.938 CV and 0.925 LB\n\nDo you think your label smoothing can work with focal loss?", "930044": "yannmajewski please be careful about taking conclusions from any isolated experiment, as DL has a very high intrinsic variance/stochasticity\nI.e. the same pipeline may give you very different results with just slight changes in hyper parameters and/or even just the order of training data points, as well as, the random weights initialisation, augmentation, etc...\n\nHaving said that, label_smoothing normally helps generalisation (test accuracy and AUC) in the presence of label noise, which is generally the case.\nAnd it should also work fine together with focal loss.\nHappy Kaggling :)", "930136": "This is what I'm using as a direct drop-in for `torch.nn.CrossEntropyLoss`.\n```\nclass LabelSmoothingLoss(torch.nn.Module):\n def __init__(self, smoothing: float = 0.1, reduction=\"mean\", weight=None):\n super(LabelSmoothingLoss, self).__init__()\n self.epsilon = smoothing\n self.reduction = reduction\n self.weight = weight\n\n def reduce_loss(self, loss):\n if self.reduction == \"mean\":\n return loss.mean()\n elif self.reduction == \"sum\":\n return loss.sum()\n else:\n return loss\n\n def linear_combination(self, x, y):\n return self.epsilon * x + (1 - self.epsilon) * y\n\n def forward(self, preds, target):\n if self.weight is not None:\n self.weight = self.weight.to(preds.device)\n\n if self.training:\n n = preds.size(-1)\n log_preds = F.log_softmax(preds, dim=-1)\n loss = self.reduce_loss(-log_preds.sum(dim=-1))\n nll = F.nll_loss(\n log_preds, target, reduction=self.reduction, weight=self.weight\n )\n return self.linear_combination(loss / n, nll)\n else:\n return torch.nn.functional.cross_entropy(preds, target, weight=self.weight)\n```\nEdit: Updated with a version that accepts weights", "963323": "hmendonca any particular reason you use the `F.binary_cross_entropy_with_logits() `instead of the NN version?", "964873": "No @brianfeeny it's the same function.\nBCEWithLogitsLoss is just a Module wrapper for F.binary_cross_entropy_with_logits, which can be nice for reusability in some cases, e.g. if you keep your hyper params somewhere else\n\ncheck out the code:\nhttps://github.com/albanie/pytorch/blob/master/torch/nn/modules/loss.py#L492-L496" }, "source": "meta" }