Which implementation of focal loss for pytorch do you use? I found there are multiple ideas how ot implement in in pytorch and I don't see any in public kernels for this competition.
", "messages": [ { "id": "903747", "postDate": "06/27/2020 04:44:28", "content": "Which implementation of focal loss for pytorch do you use? I found there are multiple ideas how ot implement in in pytorch and I don't see any in public kernels for this competition.
", "rawMarkdown": "Which implementation of focal loss for pytorch do you use? I found there are multiple ideas how ot implement in in pytorch and I don't see any in public kernels for this competition.", "votes": null }, { "id": "903986", "postDate": "06/27/2020 09:01:52", "content": "```\nclass WeightedFocalLoss(nn.Module):\n \"Weighted version of Focal Loss\"\n def init(self, alpha=.25, gamma=2):\n super(WeightedFocalLoss, self).init()\n self.alpha = torch.tensor([alpha, 1-alpha]).cuda()\n self.gamma = gamma
\n\ndef forward(self, inputs, targets):\n BCE_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction='none')\n targets = targets.type(torch.long)\n at = self.alpha.gather(0, targets.data.view(-1))\n pt = torch.exp(-BCE_loss)\n F_loss = at*(1-pt)**self.gamma * BCE_loss\n return F_loss.mean()\n```
\n\nYour inputs and targets are of shape [64,1].
\n\nHope this helps! :)
", "rawMarkdown": "```\nclass WeightedFocalLoss(nn.Module):\n \"Weighted version of Focal Loss\"\n def __init__(self, alpha=.25, gamma=2):\n super(WeightedFocalLoss, self).__init__()\n self.alpha = torch.tensor([alpha, 1-alpha]).cuda()\n self.gamma = gamma\n \n def forward(self, inputs, targets):\n BCE_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction='none')\n targets = targets.type(torch.long)\n at = self.alpha.gather(0, targets.data.view(-1))\n pt = torch.exp(-BCE_loss)\n F_loss = at*(1-pt)**self.gamma * BCE_loss\n return F_loss.mean()\n```\n\nYour inputs and targets are of shape [64,1].\n\nHope this helps! :)", "votes": null }, { "id": "903993", "postDate": "06/27/2020 09:08:17", "content": "Thank you :)\nSo this is similar to this one https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/65938\nI am asking because I have also seen code like that https://github.com/kornia/kornia/blob/master/kornia/losses/focal.py or https://github.com/kuangliu/pytorch-retinanet/blob/2d7c663350f330a34771a8fa6a4f37a2baa52a1d/loss.py or https://github.com/DingKe/pytorch_workplace/blob/4bc60a2c3640de522d0b72262667ba70391ba16e/focalloss/loss.py
", "rawMarkdown": "Thank you :)\nSo this is similar to this one https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/65938\nI am asking because I have also seen code like that https://github.com/kornia/kornia/blob/master/kornia/losses/focal.py or https://github.com/kuangliu/pytorch-retinanet/blob/2d7c663350f330a34771a8fa6a4f37a2baa52a1d/loss.py or https://github.com/DingKe/pytorch_workplace/blob/4bc60a2c3640de522d0b72262667ba70391ba16e/focalloss/loss.py", "votes": null }, { "id": "904086", "postDate": "06/27/2020 10:52:38", "content": "Yes, similar but this uses weighted focal loss. I basically corrected the at values after reading the research paper from the same implementation you have shared.
If you see in this implementation, alpha is actually pretty useless and just scales focal loss down instead of really down any weighting.
Please correct me if I'm wrong about it, My understanding of focal loss is that we assign high weights to the minority class so when our model predicts this class wrong, the penalty is very high in comparison to the majority class, so by doing it, we make our model focus on predicting the minority class correctly right?
", "rawMarkdown": "Please correct me if I'm wrong about it, My understanding of focal loss is that we assign high weights to the minority class so when our model predicts this class wrong, the penalty is very high in comparison to the majority class, so by doing it, we make our model focus on predicting the minority class correctly right?", "votes": null }, { "id": "905556", "postDate": "06/28/2020 16:31:38", "content": "Uhmm that would be any weighted loss. Focal loss does something more, it reduces loss for well classified examples (prob>.5) and higher loss otherwise.
\n\nHave a skim through the paper, it’s really nicely written. :)
", "rawMarkdown": "Uhmm that would be any weighted loss. Focal loss does something more, it reduces loss for well classified examples (prob>.5) and higher loss otherwise. \n\nHave a skim through the paper, it’s really nicely written. :)", "votes": null }, { "id": "905563", "postDate": "06/28/2020 16:36:57", "content": "Sure, thanks, will see it!
\n\nAlso, how much improvement are you getting after using it?
", "rawMarkdown": "Sure, thanks, will see it!\n\nAlso, how much improvement are you getting after using it?", "votes": null }, { "id": "920735", "postDate": "07/08/2020 19:14:40", "content": "```\ngamma = float(gamma)\nalpha = float(alpha)
\n\ndef focal_loss_fixed(y_true, y_pred):\n \"\"\"Focal loss for multi-classification\n FL(p_t)=-alpha(1-p_t)^{gamma}ln(p_t)\n Notice: y_pred is probability after softmax\n gradient is d(Fl)/d(p_t) not d(Fl)/d(x) as described in paper\n d(Fl)/d(p_t) * [p_t(1-p_t)] = d(Fl)/d(x)\n Focal Loss for Dense Object Detection\n https://arxiv.org/abs/1708.02002\n Arguments:\n y_true {tensor} -- ground truth labels, shape of [batch_size, num_cls]\n y_pred {tensor} -- model's output, shape of [batch_size, num_cls]\n Keyword Arguments:\n gamma {float} -- (default: {2.0})\n alpha {float} -- (default: {4.0})\n Returns:\n [tensor] -- loss.\n \"\"\"\n epsilon = 1.e-9\n y_true = tf.convert_to_tensor(y_true, tf.float32)\n y_pred = tf.convert_to_tensor(y_pred, tf.float32)
\n\nmodel_out = tf.add(y_pred, epsilon)\nce = tf.multiply(y_true, -tf.log(model_out))\nweight = tf.multiply(y_true, tf.pow(tf.subtract(1., model_out), gamma))\nfl = tf.multiply(alpha, tf.multiply(weight, ce))\nreduced_fl = tf.reduce_max(fl, axis=1)\nreturn tf.reduce_mean(reduced_fl)\nreturn focal_loss_fixed\n```
", "rawMarkdown": "```\ngamma = float(gamma)\nalpha = float(alpha)\n\ndef focal_loss_fixed(y_true, y_pred):\n \"\"\"Focal loss for multi-classification\n FL(p_t)=-alpha(1-p_t)^{gamma}ln(p_t)\n Notice: y_pred is probability after softmax\n gradient is d(Fl)/d(p_t) not d(Fl)/d(x) as described in paper\n d(Fl)/d(p_t) * [p_t(1-p_t)] = d(Fl)/d(x)\n Focal Loss for Dense Object Detection\n https://arxiv.org/abs/1708.02002\n Arguments:\n y_true {tensor} -- ground truth labels, shape of [batch_size, num_cls]\n y_pred {tensor} -- model's output, shape of [batch_size, num_cls]\n Keyword Arguments:\n gamma {float} -- (default: {2.0})\n alpha {float} -- (default: {4.0})\n Returns:\n [tensor] -- loss.\n \"\"\"\n epsilon = 1.e-9\n y_true = tf.convert_to_tensor(y_true, tf.float32)\n y_pred = tf.convert_to_tensor(y_pred, tf.float32)\n\n model_out = tf.add(y_pred, epsilon)\n ce = tf.multiply(y_true, -tf.log(model_out))\n weight = tf.multiply(y_true, tf.pow(tf.subtract(1., model_out), gamma))\n fl = tf.multiply(alpha, tf.multiply(weight, ce))\n reduced_fl = tf.reduce_max(fl, axis=1)\n return tf.reduce_mean(reduced_fl)\nreturn focal_loss_fixed\n```", "votes": null }, { "id": "940541", "postDate": "07/23/2020 02:31:40", "content": "@aroraaman have you noticed big difference in scale of loss between BCE and your focal loss?\nBCE values are like 0.1-0.5\nyour focal loss values are 100 times smaller\nprobably we should scale gamma 100 times
", "rawMarkdown": "aroraaman have you noticed big difference in scale of loss between BCE and your focal loss?\nBCE values are like 0.1-0.5\nyour focal loss values are 100 times smaller\nprobably we should scale gamma 100 times", "votes": null } ], "comments": [ { "id": 903986, "author_name": "aroraaman", "author_url": "", "post_date": "06/27/2020 09:01:52", "content": "```\nclass WeightedFocalLoss(nn.Module):\n \"Weighted version of Focal Loss\"\n def init(self, alpha=.25, gamma=2):\n super(WeightedFocalLoss, self).init()\n self.alpha = torch.tensor([alpha, 1-alpha]).cuda()\n self.gamma = gamma
\n\ndef forward(self, inputs, targets):\n BCE_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction='none')\n targets = targets.type(torch.long)\n at = self.alpha.gather(0, targets.data.view(-1))\n pt = torch.exp(-BCE_loss)\n F_loss = at*(1-pt)**self.gamma * BCE_loss\n return F_loss.mean()\n```
\n\nYour inputs and targets are of shape [64,1].
\n\nHope this helps! :)
", "votes": null, "replies": [ { "id": 903993, "author_name": "jacekpoplawski", "author_url": "", "post_date": "06/27/2020 09:08:17", "content": "Thank you :)\nSo this is similar to this one https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/65938\nI am asking because I have also seen code like that https://github.com/kornia/kornia/blob/master/kornia/losses/focal.py or https://github.com/kuangliu/pytorch-retinanet/blob/2d7c663350f330a34771a8fa6a4f37a2baa52a1d/loss.py or https://github.com/DingKe/pytorch_workplace/blob/4bc60a2c3640de522d0b72262667ba70391ba16e/focalloss/loss.py
", "votes": null, "replies": [] }, { "id": 904086, "author_name": "aroraaman", "author_url": "", "post_date": "06/27/2020 10:52:38", "content": "Yes, similar but this uses weighted focal loss. I basically corrected the at values after reading the research paper from the same implementation you have shared.
If you see in this implementation, alpha is actually pretty useless and just scales focal loss down instead of really down any weighting.
Please correct me if I'm wrong about it, My understanding of focal loss is that we assign high weights to the minority class so when our model predicts this class wrong, the penalty is very high in comparison to the majority class, so by doing it, we make our model focus on predicting the minority class correctly right?
", "votes": null, "replies": [] }, { "id": 905556, "author_name": "aroraaman", "author_url": "", "post_date": "06/28/2020 16:31:38", "content": "Uhmm that would be any weighted loss. Focal loss does something more, it reduces loss for well classified examples (prob>.5) and higher loss otherwise.
\n\nHave a skim through the paper, it’s really nicely written. :)
", "votes": null, "replies": [] }, { "id": 905563, "author_name": "sarques", "author_url": "", "post_date": "06/28/2020 16:36:57", "content": "Sure, thanks, will see it!
\n\nAlso, how much improvement are you getting after using it?
", "votes": null, "replies": [] }, { "id": 940541, "author_name": "jacekpoplawski", "author_url": "", "post_date": "07/23/2020 02:31:40", "content": "@aroraaman have you noticed big difference in scale of loss between BCE and your focal loss?\nBCE values are like 0.1-0.5\nyour focal loss values are 100 times smaller\nprobably we should scale gamma 100 times
", "votes": null, "replies": [] } ] }, { "id": 920735, "author_name": "doanquanvietnamca", "author_url": "", "post_date": "07/08/2020 19:14:40", "content": "```\ngamma = float(gamma)\nalpha = float(alpha)
\n\ndef focal_loss_fixed(y_true, y_pred):\n \"\"\"Focal loss for multi-classification\n FL(p_t)=-alpha(1-p_t)^{gamma}ln(p_t)\n Notice: y_pred is probability after softmax\n gradient is d(Fl)/d(p_t) not d(Fl)/d(x) as described in paper\n d(Fl)/d(p_t) * [p_t(1-p_t)] = d(Fl)/d(x)\n Focal Loss for Dense Object Detection\n https://arxiv.org/abs/1708.02002\n Arguments:\n y_true {tensor} -- ground truth labels, shape of [batch_size, num_cls]\n y_pred {tensor} -- model's output, shape of [batch_size, num_cls]\n Keyword Arguments:\n gamma {float} -- (default: {2.0})\n alpha {float} -- (default: {4.0})\n Returns:\n [tensor] -- loss.\n \"\"\"\n epsilon = 1.e-9\n y_true = tf.convert_to_tensor(y_true, tf.float32)\n y_pred = tf.convert_to_tensor(y_pred, tf.float32)
\n\nmodel_out = tf.add(y_pred, epsilon)\nce = tf.multiply(y_true, -tf.log(model_out))\nweight = tf.multiply(y_true, tf.pow(tf.subtract(1., model_out), gamma))\nfl = tf.multiply(alpha, tf.multiply(weight, ce))\nreduced_fl = tf.reduce_max(fl, axis=1)\nreturn tf.reduce_mean(reduced_fl)\nreturn focal_loss_fixed\n```
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "903747": "Which implementation of focal loss for pytorch do you use? I found there are multiple ideas how ot implement in in pytorch and I don't see any in public kernels for this competition.", "903986": "```\nclass WeightedFocalLoss(nn.Module):\n \"Weighted version of Focal Loss\"\n def __init__(self, alpha=.25, gamma=2):\n super(WeightedFocalLoss, self).__init__()\n self.alpha = torch.tensor([alpha, 1-alpha]).cuda()\n self.gamma = gamma\n \n def forward(self, inputs, targets):\n BCE_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction='none')\n targets = targets.type(torch.long)\n at = self.alpha.gather(0, targets.data.view(-1))\n pt = torch.exp(-BCE_loss)\n F_loss = at*(1-pt)**self.gamma * BCE_loss\n return F_loss.mean()\n```\n\nYour inputs and targets are of shape [64,1].\n\nHope this helps! :)", "903993": "Thank you :)\nSo this is similar to this one https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/65938\nI am asking because I have also seen code like that https://github.com/kornia/kornia/blob/master/kornia/losses/focal.py or https://github.com/kuangliu/pytorch-retinanet/blob/2d7c663350f330a34771a8fa6a4f37a2baa52a1d/loss.py or https://github.com/DingKe/pytorch_workplace/blob/4bc60a2c3640de522d0b72262667ba70391ba16e/focalloss/loss.py", "904086": "Yes, similar but this uses weighted focal loss. I basically corrected the `at` values after reading the research paper from the same implementation you have shared. \n\nIf you see in [this]( https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/65938) implementation, `alpha` is actually pretty useless and just scales focal loss down instead of really down any weighting.", "905116": "Please correct me if I'm wrong about it, My understanding of focal loss is that we assign high weights to the minority class so when our model predicts this class wrong, the penalty is very high in comparison to the majority class, so by doing it, we make our model focus on predicting the minority class correctly right?", "905556": "Uhmm that would be any weighted loss. Focal loss does something more, it reduces loss for well classified examples (prob>.5) and higher loss otherwise. \n\nHave a skim through the paper, it’s really nicely written. :)", "905563": "Sure, thanks, will see it!\n\nAlso, how much improvement are you getting after using it?", "920735": "```\ngamma = float(gamma)\nalpha = float(alpha)\n\ndef focal_loss_fixed(y_true, y_pred):\n \"\"\"Focal loss for multi-classification\n FL(p_t)=-alpha(1-p_t)^{gamma}ln(p_t)\n Notice: y_pred is probability after softmax\n gradient is d(Fl)/d(p_t) not d(Fl)/d(x) as described in paper\n d(Fl)/d(p_t) * [p_t(1-p_t)] = d(Fl)/d(x)\n Focal Loss for Dense Object Detection\n https://arxiv.org/abs/1708.02002\n Arguments:\n y_true {tensor} -- ground truth labels, shape of [batch_size, num_cls]\n y_pred {tensor} -- model's output, shape of [batch_size, num_cls]\n Keyword Arguments:\n gamma {float} -- (default: {2.0})\n alpha {float} -- (default: {4.0})\n Returns:\n [tensor] -- loss.\n \"\"\"\n epsilon = 1.e-9\n y_true = tf.convert_to_tensor(y_true, tf.float32)\n y_pred = tf.convert_to_tensor(y_pred, tf.float32)\n\n model_out = tf.add(y_pred, epsilon)\n ce = tf.multiply(y_true, -tf.log(model_out))\n weight = tf.multiply(y_true, tf.pow(tf.subtract(1., model_out), gamma))\n fl = tf.multiply(alpha, tf.multiply(weight, ce))\n reduced_fl = tf.reduce_max(fl, axis=1)\n return tf.reduce_mean(reduced_fl)\nreturn focal_loss_fixed\n```", "940541": "aroraaman have you noticed big difference in scale of loss between BCE and your focal loss?\nBCE values are like 0.1-0.5\nyour focal loss values are 100 times smaller\nprobably we should scale gamma 100 times" }, "source": "meta" }