I've implemented a custom loss function for training. But I use Focal Cosine Loss for validation. And the results are pretty much consistent with LB. Of course you can also use it for training and it will likely give better results than regular Cross Entropy
\nThe loss fuction was introcuded in this paper : Data-Efficient Deep Learning Method for Image Classification Using Data Augmentation, Focal Cosine Loss, and Ensemble.
\nHere you go :
\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\n\n\nclass FocalCosineLoss(nn.Module):\n def __init__(self, alpha=1, gamma=2, xent=.1):\n super(FocalCosineLoss, self).__init__()\n self.alpha = alpha\n self.gamma = gamma\n\n self.xent = xent\n\n self.y = torch.Tensor([1]).cuda()\n\n def forward(self, input, target, reduction=\"mean\"):\n cosine_loss = F.cosine_embedding_loss(input, F.one_hot(target, num_classes=input.size(-1)), self.y, reduction=reduction)\n\n cent_loss = F.cross_entropy(F.normalize(input), target, reduce=False)\n pt = torch.exp(-cent_loss)\n focal_loss = self.alpha * (1-pt)**self.gamma * cent_loss\n\n if reduction == \"mean\":\n focal_loss = torch.mean(focal_loss)\n\n return cosine_loss + self.xent * focal_loss\nThe implementation can also be found here
", "messages": [ { "id": 1112358, "postDate": "2020-12-14T14:09:15.387Z", "content": "I've implemented a custom loss function for training. But I use Focal Cosine Loss for validation. And the results are pretty much consistent with LB. Of course you can also use it for training and it will likely give better results than regular Cross Entropy
\nThe loss fuction was introcuded in this paper : Data-Efficient Deep Learning Method for Image Classification Using Data Augmentation, Focal Cosine Loss, and Ensemble.
\nHere you go :
\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\n\n\nclass FocalCosineLoss(nn.Module):\n def __init__(self, alpha=1, gamma=2, xent=.1):\n super(FocalCosineLoss, self).__init__()\n self.alpha = alpha\n self.gamma = gamma\n\n self.xent = xent\n\n self.y = torch.Tensor([1]).cuda()\n\n def forward(self, input, target, reduction=\"mean\"):\n cosine_loss = F.cosine_embedding_loss(input, F.one_hot(target, num_classes=input.size(-1)), self.y, reduction=reduction)\n\n cent_loss = F.cross_entropy(F.normalize(input), target, reduce=False)\n pt = torch.exp(-cent_loss)\n focal_loss = self.alpha * (1-pt)**self.gamma * cent_loss\n\n if reduction == \"mean\":\n focal_loss = torch.mean(focal_loss)\n\n return cosine_loss + self.xent * focal_loss\nThe implementation can also be found here
", "rawMarkdown": "I've implemented a custom loss function for training. But I use Focal Cosine Loss for validation. And the results are pretty much consistent with LB. Of course you can also use it for training and it will likely give better results than regular Cross Entropy\n\nThe loss fuction was introcuded [in this paper](https://arxiv.org/abs/2007.07805) : Data-Efficient Deep Learning Method for Image Classification Using Data Augmentation, Focal Cosine Loss, and Ensemble. \n\n\nHere you go :\n\n\n```\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\n\n\nclass FocalCosineLoss(nn.Module):\n def __init__(self, alpha=1, gamma=2, xent=.1):\n super(FocalCosineLoss, self).__init__()\n self.alpha = alpha\n self.gamma = gamma\n\n self.xent = xent\n \n self.y = torch.Tensor([1]).cuda()\n \n def forward(self, input, target, reduction=\"mean\"):\n cosine_loss = F.cosine_embedding_loss(input, F.one_hot(target, num_classes=input.size(-1)), self.y, reduction=reduction)\n\n cent_loss = F.cross_entropy(F.normalize(input), target, reduce=False)\n pt = torch.exp(-cent_loss)\n focal_loss = self.alpha * (1-pt)**self.gamma * cent_loss\n\n if reduction == \"mean\":\n focal_loss = torch.mean(focal_loss)\n \n return cosine_loss + self.xent * focal_loss\n```\n\n\n\n\nThe implementation can also be found [here](https://github.com/byeongjokim/VIPriors-Image-Classification-Challenge/blob/332e04fd3e82b20d312128bad302a9081f5c37ce/timm/loss/cosine.py) ", "votes": 103 }, { "id": 1113911, "postDate": "2020-12-15T19:48:17.700Z", "content": "Thanks a lot for sharing! I have couple of questions; 1) you said you are using it in validation does it mean local focal cosine loss is correlated with LB accuracy? 2) I thought focal loss was meant to focus on hard samples but in our case if we have noisy labels wouldn't it be even worse to use it since it will focus on hard noisy samples and try to learn them. Maybe, a cleaning needed on data?
", "rawMarkdown": "Thanks a lot for sharing! I have couple of questions; 1) you said you are using it in validation does it mean local focal cosine loss is correlated with LB accuracy? 2) I thought focal loss was meant to focus on hard samples but in our case if we have noisy labels wouldn't it be even worse to use it since it will focus on hard noisy samples and try to learn them. Maybe, a cleaning needed on data?", "votes": 3 }, { "id": 1148296, "postDate": "2021-01-11T04:14:53.697Z", "content": "Thanks for sharing @serigne
\nI tried FocalCosineLoss, and locla CV and public LB is similar more than softmax CE.
\nHowever, FocalCosineLoss score is lower than my best.
\nI don't know cause of this ( maybe learning rate or epoch?) :(
@serigne thanks for this! Would you happen to know of any publicly available paper that introduces or explains cosine focal loss?
", "rawMarkdown": "@serigne thanks for this! Would you happen to know of any publicly available paper that introduces or explains cosine focal loss?" }, { "id": 1120615, "postDate": "2020-12-21T01:33:54.643Z", "content": "After using your loss instead of smoothing CE, my CV increased but LB decreased.
", "rawMarkdown": "After using your loss instead of smoothing CE, my CV increased but LB decreased." }, { "id": 1116861, "postDate": "2020-12-17T14:21:15.930Z", "content": "Hi,I use the tf_efficientnet_b4_ns to train the model,the data is 2019 and 2020,I use you share the FocalCosineLoss to train the model,I use 5 fold and 512 size,but I get the cv is 0.8925.So I dont know what happend?Do you know there is why?Thanks!
", "rawMarkdown": "Hi,I use the tf_efficientnet_b4_ns to train the model,the data is 2019 and 2020,I use you share the FocalCosineLoss to train the model,I use 5 fold and 512 size,but I get the cv is 0.8925.So I dont know what happend?Do you know there is why?Thanks!" }, { "id": 1113961, "postDate": "2020-12-15T20:52:57.617Z", "content": "Since we are talking about losses, I thought I'd ask if anyone has tried OHEM loss and saw improvements with it.
", "rawMarkdown": "Since we are talking about losses, I thought I'd ask if anyone has tried OHEM loss and saw improvements with it." }, { "id": 1113680, "postDate": "2020-12-15T16:26:02.300Z", "content": "Hi @serigne thanks for sharing your work and insight.
", "rawMarkdown": "Hi @serigne thanks for sharing your work and insight." }, { "id": 1113055, "postDate": "2020-12-15T06:35:23.220Z", "content": "Hi, @serigne, here's a noob question.
\nI am not sure what is meant by using loss for validation. I thought loss is calculated for only training samples for backprop.
@mightyrains Normally validation loss is used to judge how well your model performs. even though there is no backpropagation of the loss during validation. If you see your train loss decreasing but valid loss not improving similarly, it would mean that the model is not learning properly. Basically, validation loss and validation metric helps you see how the model performs overall on unseen data.
", "rawMarkdown": "@mightyrains Normally validation loss is used to judge how well your model performs. even though there is no backpropagation of the loss during validation. If you see your train loss decreasing but valid loss not improving similarly, it would mean that the model is not learning properly. Basically, validation loss and validation metric helps you see how the model performs overall on unseen data.", "votes": 2, "replies": [ { "id": 1113161, "postDate": "2020-12-15T08:46:41.263Z", "content": "@yovinyahathugoda thanks for the reply.
\nGot it, we do that to see if the model is training properly or overfitting.
\nI hadn't come across anybody doing that with different losses for training and validation. Cool thing!!
It is very common, and actually the correct way, when you use label smoothing param to your train loss e.g. BCE. Then you define as val loss a \"clean\" BCE w/o smoothing to assess model performance
", "rawMarkdown": "It is very common, and actually the correct way, when you use label smoothing param to your train loss e.g. BCE. Then you define as val loss a \"clean\" BCE w/o smoothing to assess model performance ", "votes": 5 } ] }, { "id": 1112959, "postDate": "2020-12-15T04:04:15.400Z", "content": "Thanks for sharing I guess you are using focal cosine loss + some regularization method for training ? Am I correct ?
", "rawMarkdown": "Thanks for sharing I guess you are using focal cosine loss + some regularization method for training ? Am I correct ? " }, { "id": 1112543, "postDate": "2020-12-14T17:36:22.270Z", "content": "Can I ask that is there a reason it is only used for validation?
\n", "rawMarkdown": "Can I ask that is there a reason it is only used for validation?\n\n@serigne ", "replies": [ { "id": 1112677, "postDate": "2020-12-14T19:57:03.277Z", "content": "I used a custom loss function for training which is not relevant for validation.
\nBut I was using Focal Cosine for training earlier in the competition.
", "rawMarkdown": "I used a custom loss function for training which is not relevant for validation. \n\nBut I was using Focal Cosine for training earlier in the competition. ", "votes": 1 }, { "id": 1112687, "postDate": "2020-12-14T20:06:53.883Z", "content": "Interesting. Your custom loss is pretty good on this dataset.
\n", "rawMarkdown": "Interesting. Your custom loss is pretty good on this dataset.\n\n@serigne " } ] }, { "id": 1153842, "postDate": "2021-01-15T07:32:55.017Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1113461, "postDate": "2020-12-15T13:00:11.140Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1127101, "postDate": "2020-12-26T08:36:47.043Z", "content": "Thanks for sharing @Serigne!!!
", "rawMarkdown": "Thanks for sharing @Serigne!!!", "votes": 1 } ], "comments": [ { "id": 1113911, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-12-15T19:48:17.700000", "content": "Thanks a lot for sharing! I have couple of questions; 1) you said you are using it in validation does it mean local focal cosine loss is correlated with LB accuracy? 2) I thought focal loss was meant to focus on hard samples but in our case if we have noisy labels wouldn't it be even worse to use it since it will focus on hard noisy samples and try to learn them. Maybe, a cleaning needed on data?
", "votes": 3, "replies": [] }, { "id": 1148296, "author_name": "imori", "author_url": "", "post_date": "2021-01-11T04:14:53.697000", "content": "Thanks for sharing @serigne
\nI tried FocalCosineLoss, and locla CV and public LB is similar more than softmax CE.
\nHowever, FocalCosineLoss score is lower than my best.
\nI don't know cause of this ( maybe learning rate or epoch?) :(
@serigne thanks for this! Would you happen to know of any publicly available paper that introduces or explains cosine focal loss?
", "votes": 0, "replies": [] }, { "id": 1120615, "author_name": "sky_fly", "author_url": "", "post_date": "2020-12-21T01:33:54.643000", "content": "After using your loss instead of smoothing CE, my CV increased but LB decreased.
", "votes": 0, "replies": [] }, { "id": 1116861, "author_name": "Bcw93", "author_url": "", "post_date": "2020-12-17T14:21:15.930000", "content": "Hi,I use the tf_efficientnet_b4_ns to train the model,the data is 2019 and 2020,I use you share the FocalCosineLoss to train the model,I use 5 fold and 512 size,but I get the cv is 0.8925.So I dont know what happend?Do you know there is why?Thanks!
", "votes": 0, "replies": [] }, { "id": 1113961, "author_name": "Phaedrus", "author_url": "", "post_date": "2020-12-15T20:52:57.617000", "content": "Since we are talking about losses, I thought I'd ask if anyone has tried OHEM loss and saw improvements with it.
", "votes": 0, "replies": [] }, { "id": 1113680, "author_name": "Patrick Uzuwe", "author_url": "", "post_date": "2020-12-15T16:26:02.300000", "content": "Hi @serigne thanks for sharing your work and insight.
", "votes": 0, "replies": [] }, { "id": 1113055, "author_name": "Mighty Rains", "author_url": "", "post_date": "2020-12-15T06:35:23.220000", "content": "Hi, @serigne, here's a noob question.
\nI am not sure what is meant by using loss for validation. I thought loss is calculated for only training samples for backprop.
@mightyrains Normally validation loss is used to judge how well your model performs. even though there is no backpropagation of the loss during validation. If you see your train loss decreasing but valid loss not improving similarly, it would mean that the model is not learning properly. Basically, validation loss and validation metric helps you see how the model performs overall on unseen data.
", "votes": 2, "replies": [ { "id": 1113161, "author_name": "Mighty Rains", "author_url": "", "post_date": "2020-12-15T08:46:41.263000", "content": "@yovinyahathugoda thanks for the reply.
\nGot it, we do that to see if the model is training properly or overfitting.
\nI hadn't come across anybody doing that with different losses for training and validation. Cool thing!!
It is very common, and actually the correct way, when you use label smoothing param to your train loss e.g. BCE. Then you define as val loss a \"clean\" BCE w/o smoothing to assess model performance
", "votes": 5, "replies": [] } ] }, { "id": 1112959, "author_name": "Atharva Phatak", "author_url": "", "post_date": "2020-12-15T04:04:15.400000", "content": "Thanks for sharing I guess you are using focal cosine loss + some regularization method for training ? Am I correct ?
", "votes": 0, "replies": [] }, { "id": 1112543, "author_name": "Heroseo", "author_url": "", "post_date": "2020-12-14T17:36:22.270000", "content": "Can I ask that is there a reason it is only used for validation?
\n", "votes": 0, "replies": [ { "id": 1112677, "author_name": "Serigne ", "author_url": "", "post_date": "2020-12-14T19:57:03.277000", "content": "I used a custom loss function for training which is not relevant for validation.
\nBut I was using Focal Cosine for training earlier in the competition.
", "votes": 1, "replies": [] }, { "id": 1112687, "author_name": "Heroseo", "author_url": "", "post_date": "2020-12-14T20:06:53.883000", "content": "Interesting. Your custom loss is pretty good on this dataset.
\n", "votes": 0, "replies": [] } ] }, { "id": 1153842, "author_name": "", "author_url": "", "post_date": "2021-01-15T07:32:55.017000", "content": "", "votes": 1, "replies": [] }, { "id": 1113461, "author_name": "", "author_url": "", "post_date": "2020-12-15T13:00:11.140000", "content": "", "votes": 0, "replies": [] }, { "id": 1127101, "author_name": "Saurabh Shahane", "author_url": "", "post_date": "2020-12-26T08:36:47.043000", "content": "Thanks for sharing @Serigne!!!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1112358": "I've implemented a custom loss function for training. But I use Focal Cosine Loss for validation. And the results are pretty much consistent with LB. Of course you can also use it for training and it will likely give better results than regular Cross Entropy\n\nThe loss fuction was introcuded [in this paper](https://arxiv.org/abs/2007.07805) : Data-Efficient Deep Learning Method for Image Classification Using Data Augmentation, Focal Cosine Loss, and Ensemble. \n\n\nHere you go :\n\n\n```\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\n\n\nclass FocalCosineLoss(nn.Module):\n def __init__(self, alpha=1, gamma=2, xent=.1):\n super(FocalCosineLoss, self).__init__()\n self.alpha = alpha\n self.gamma = gamma\n\n self.xent = xent\n \n self.y = torch.Tensor([1]).cuda()\n \n def forward(self, input, target, reduction=\"mean\"):\n cosine_loss = F.cosine_embedding_loss(input, F.one_hot(target, num_classes=input.size(-1)), self.y, reduction=reduction)\n\n cent_loss = F.cross_entropy(F.normalize(input), target, reduce=False)\n pt = torch.exp(-cent_loss)\n focal_loss = self.alpha * (1-pt)**self.gamma * cent_loss\n\n if reduction == \"mean\":\n focal_loss = torch.mean(focal_loss)\n \n return cosine_loss + self.xent * focal_loss\n```\n\n\n\n\nThe implementation can also be found [here](https://github.com/byeongjokim/VIPriors-Image-Classification-Challenge/blob/332e04fd3e82b20d312128bad302a9081f5c37ce/timm/loss/cosine.py) ", "1113911": "Thanks a lot for sharing! I have couple of questions; 1) you said you are using it in validation does it mean local focal cosine loss is correlated with LB accuracy? 2) I thought focal loss was meant to focus on hard samples but in our case if we have noisy labels wouldn't it be even worse to use it since it will focus on hard noisy samples and try to learn them. Maybe, a cleaning needed on data?", "1148296": "Thanks for sharing @serigne \nI tried FocalCosineLoss, and locla CV and public LB is similar more than softmax CE.\nHowever, FocalCosineLoss score is lower than my best.\nI don't know cause of this ( maybe learning rate or epoch?) :(", "1135548": "@serigne thanks for this! Would you happen to know of any publicly available paper that introduces or explains cosine focal loss?", "1120615": "After using your loss instead of smoothing CE, my CV increased but LB decreased.", "1116861": "Hi,I use the tf_efficientnet_b4_ns to train the model,the data is 2019 and 2020,I use you share the FocalCosineLoss to train the model,I use 5 fold and 512 size,but I get the cv is 0.8925.So I dont know what happend?Do you know there is why?Thanks!", "1113961": "Since we are talking about losses, I thought I'd ask if anyone has tried OHEM loss and saw improvements with it.", "1113680": "Hi @serigne thanks for sharing your work and insight.", "1113055": "Hi, @serigne, here's a noob question.\nI am not sure what is meant by using loss for validation. I thought loss is calculated for only training samples for backprop. ", "1112959": "Thanks for sharing I guess you are using focal cosine loss + some regularization method for training ? Am I correct ? ", "1112543": "Can I ask that is there a reason it is only used for validation?\n\n@serigne ", "1153842": "", "1113461": "", "1127101": "Thanks for sharing @Serigne!!!" } }