{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Loss Function Reference for Keras & PyTorch\n---\nThis kernel provides a reference library for some popular custom loss functions that you can easily import into your code.\n\nLoss functions define how neural network models calculate the overall error from their residuals for each training batch. This in turn affects how they adjust their internal weights when performing backpropagation, so the choice of loss function has a direct influence on model performance.\n\nThe default choice of loss function for segmentation and other classification tasks is Binary Cross-Entropy (BCE). In situations where a particular metric, like the Dice Coefficient or Intersection over Union (IoU), is being used to judge model performance, competitors will sometimes experiment with loss functions that derive from these metrics - typically in the form `1 - f(x)` where `f(x)` is the metric in question. \n\nThese functions cannot simply be written in NumPy, as they must operate on tensors that also have gradient parameters which need to be calculated throughout the model during backpropagation. Accordingly, loss functions must be written using backend functions from the respective model library. This is less complicated than it sounds! For example in Keras, you would simply use the same familiar mathematical functions, albeit using the Keras backend imported as `K`, i.e `K.sum()`. Gradient calculation is handled automatically by the model libraries, although (at least in PyTorch) you can define this manually if you are confident in your mathematical ability and wish to make alterations of your own.\n\nWith multi-class classification or segmentation, we sometimes use loss functions that calculate the average loss for each class, rather than calculating loss from the prediction tensor as a whole. This kernel is meant as a template reference for the basic code, so all examples calculate loss on the entire tensor, but it should be trivial for you to modify it for multi-class averaging. \n\nI hope this kernel will be of use to you, and any corrections or suggestions are welcome.","metadata":{}},{"cell_type":"code","source":"import numpy\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport keras\nimport keras.backend as K","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-06T20:40:05.227725Z","iopub.execute_input":"2022-07-06T20:40:05.228214Z","iopub.status.idle":"2022-07-06T20:40:08.328158Z","shell.execute_reply.started":"2022-07-06T20:40:05.228154Z","shell.execute_reply":"2022-07-06T20:40:08.327365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dice Loss\n---\nThe Dice coefficient, or Dice-Sørensen coefficient, is a common metric for pixel segmentation that can also be modified to act as a loss function:\n\n![](https://wikimedia.org/api/rest_v1/media/math/render/svg/a80a97215e1afc0b222e604af1b2099dc9363d3b)","metadata":{}},{"cell_type":"code","source":"#PyTorch\nclass DiceLoss(nn.Module):\n    def __init__(self, weight=None, size_average=True):\n        super(DiceLoss, self).__init__()\n\n    def forward(self, inputs, targets, smooth=1):\n        \n        #comment out if your model contains a sigmoid or equivalent activation layer\n        inputs = F.sigmoid(inputs)       \n        \n        #flatten label and prediction tensors\n        inputs = inputs.view(-1)\n        targets = targets.view(-1)\n        \n        intersection = (inputs * targets).sum()                            \n        dice = (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth)  \n        \n        return 1 - dice","metadata":{"execution":{"iopub.status.busy":"2022-07-06T20:40:08.330025Z","iopub.execute_input":"2022-07-06T20:40:08.330258Z","iopub.status.idle":"2022-07-06T20:40:08.337881Z","shell.execute_reply.started":"2022-07-06T20:40:08.330224Z","shell.execute_reply":"2022-07-06T20:40:08.337058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Keras\ndef DiceLoss(targets, inputs, smooth=1e-6):\n    \n    #flatten label and prediction tensors\n    inputs = K.flatten(inputs)\n    targets = K.flatten(targets)\n    \n    intersection = K.sum(K.dot(targets, inputs))\n    dice = (2*intersection + smooth) / (K.sum(targets) + K.sum(inputs) + smooth)\n    return 1 - dice","metadata":{"execution":{"iopub.status.busy":"2022-07-06T20:40:08.339281Z","iopub.execute_input":"2022-07-06T20:40:08.339539Z","iopub.status.idle":"2022-07-06T20:40:08.349929Z","shell.execute_reply.started":"2022-07-06T20:40:08.339499Z","shell.execute_reply":"2022-07-06T20:40:08.349076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# BCE-Dice Loss\n---\nThis loss combines Dice loss with the standard binary cross-entropy (BCE) loss that is generally the default for segmentation models. Combining the two methods allows for some diversity in the loss, while benefitting from the stability of BCE. The equation for multi-class BCE by itself will be familiar to anyone who has studied logistic regression:\n\n![](https://wikimedia.org/api/rest_v1/media/math/render/svg/80f87a71d3a616a0939f5360cec24d702d2593a2)","metadata":{}},{"cell_type":"code","source":"#PyTorch\nclass DiceBCELoss(nn.Module):\n    def __init__(self, weight=None, size_average=True):\n        super(DiceBCELoss, self).__init__()\n\n    def forward(self, inputs, targets, smooth=1):\n        \n        #comment out if your model contains a sigmoid or equivalent activation layer\n        inputs = F.sigmoid(inputs)       \n        \n        #flatten label and prediction tensors\n        inputs = inputs.view(-1)\n        targets = targets.view(-1)\n        \n        intersection = (inputs * targets).sum()                            \n        dice_loss = 1 - (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth)  \n        BCE = F.binary_cross_entropy(inputs, targets, reduction='mean')\n        Dice_BCE = BCE + dice_loss\n        \n        return Dice_BCE","metadata":{"execution":{"iopub.status.busy":"2022-07-06T20:40:08.351444Z","iopub.execute_input":"2022-07-06T20:40:08.351822Z","iopub.status.idle":"2022-07-06T20:40:08.360449Z","shell.execute_reply.started":"2022-07-06T20:40:08.351754Z","shell.execute_reply":"2022-07-06T20:40:08.359744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Keras\ndef DiceBCELoss(targets, inputs, smooth=1e-6):    \n       \n    #flatten label and prediction tensors\n    inputs = K.flatten(inputs)\n    targets = K.flatten(targets)\n    \n    BCE =  binary_crossentropy(targets, inputs)\n    intersection = K.sum(K.dot(targets, inputs))    \n    dice_loss = 1 - (2*intersection + smooth) / (K.sum(targets) + K.sum(inputs) + smooth)\n    Dice_BCE = BCE + dice_loss\n    \n    return Dice_BCE","metadata":{"execution":{"iopub.status.busy":"2022-07-06T20:40:08.363439Z","iopub.execute_input":"2022-07-06T20:40:08.363854Z","iopub.status.idle":"2022-07-06T20:40:08.372902Z","shell.execute_reply.started":"2022-07-06T20:40:08.363673Z","shell.execute_reply":"2022-07-06T20:40:08.372240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Jaccard/Intersection over Union (IoU) Loss\n---\nThe IoU metric, or Jaccard Index, is similar to the Dice metric and is calculated as the ratio between the overlap of the positive instances between two sets, and their mutual combined values:\n\n![](https://wikimedia.org/api/rest_v1/media/math/render/svg/eaef5aa86949f49e7dc6b9c8c3dd8b233332c9e7)\n\nLike the Dice metric, it is a common means of evaluating the performance of pixel segmentation models.","metadata":{}},{"cell_type":"code","source":"#PyTorch\nclass IoULoss(nn.Module):\n    def __init__(self, weight=None, size_average=True):\n        super(IoULoss, self).__init__()\n\n    def forward(self, inputs, targets, smooth=1):\n        \n        #comment out if your model contains a sigmoid or equivalent activation layer\n        inputs = F.sigmoid(inputs)       \n        \n        #flatten label and prediction tensors\n        inputs = inputs.view(-1)\n        targets = targets.view(-1)\n        \n        #intersection is equivalent to True Positive count\n        #union is the mutually inclusive area of all labels & predictions \n        intersection = (inputs * targets).sum()\n        total = (inputs + targets).sum()\n        union = total - intersection \n        \n        IoU = (intersection + smooth)/(union + smooth)\n                \n        return 1 - IoU","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-06T20:40:08.374265Z","iopub.execute_input":"2022-07-06T20:40:08.374705Z","iopub.status.idle":"2022-07-06T20:40:08.387112Z","shell.execute_reply.started":"2022-07-06T20:40:08.374650Z","shell.execute_reply":"2022-07-06T20:40:08.386434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Keras\ndef IoULoss(targets, inputs, smooth=1e-6):\n    \n    #flatten label and prediction tensors\n    inputs = K.flatten(inputs)\n    targets = K.flatten(targets)\n    \n    intersection = K.sum(K.dot(targets, inputs))\n    total = K.sum(targets) + K.sum(inputs)\n    union = total - intersection\n    \n    IoU = (intersection + smooth) / (union + smooth)\n    return 1 - IoU","metadata":{"execution":{"iopub.status.busy":"2022-07-06T20:40:08.388304Z","iopub.execute_input":"2022-07-06T20:40:08.388757Z","iopub.status.idle":"2022-07-06T20:40:08.397952Z","shell.execute_reply.started":"2022-07-06T20:40:08.388643Z","shell.execute_reply":"2022-07-06T20:40:08.397301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Focal Loss\n---\nFocal Loss was introduced by *Lin et al* of Facebook AI Research in 2017 as a means of combatting extremely imbalanced datasets where positive cases were relatively rare. Their paper \"Focal Loss for Dense Object Detection\" is retrievable here: https://arxiv.org/abs/1708.02002. In practice, the researchers used an alpha-modified version of the function so I have included it in this implementation.","metadata":{}},{"cell_type":"code","source":"#PyTorch\nALPHA = 0.8\nGAMMA = 2\n\nclass FocalLoss(nn.Module):\n    def __init__(self, weight=None, size_average=True):\n        super(FocalLoss, self).__init__()\n\n    def forward(self, inputs, targets, alpha=ALPHA, gamma=GAMMA, smooth=1):\n        \n        #comment out if your model contains a sigmoid or equivalent activation layer\n        inputs = F.sigmoid(inputs)       \n        \n        #flatten label and prediction tensors\n        inputs = inputs.view(-1)\n        targets = targets.view(-1)\n        \n        #first compute binary cross-entropy \n        BCE = F.binary_cross_entropy(inputs, targets, reduction='mean')\n        BCE_EXP = torch.exp(-BCE)\n        focal_loss = alpha * (1-BCE_EXP)**gamma * BCE\n                       \n        return focal_loss","metadata":{"execution":{"iopub.status.busy":"2022-07-06T20:40:08.399358Z","iopub.execute_input":"2022-07-06T20:40:08.399747Z","iopub.status.idle":"2022-07-06T20:40:08.410823Z","shell.execute_reply.started":"2022-07-06T20:40:08.399679Z","shell.execute_reply":"2022-07-06T20:40:08.409866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Keras\nALPHA = 0.8\nGAMMA = 2\n\ndef FocalLoss(targets, inputs, alpha=ALPHA, gamma=GAMMA):    \n    \n    inputs = K.flatten(inputs)\n    targets = K.flatten(targets)\n    \n    BCE = K.binary_crossentropy(targets, inputs)\n    BCE_EXP = K.exp(-BCE)\n    focal_loss = K.mean(alpha * K.pow((1-BCE_EXP), gamma) * BCE)\n    \n    return focal_loss","metadata":{"execution":{"iopub.status.busy":"2022-07-06T20:40:08.412093Z","iopub.execute_input":"2022-07-06T20:40:08.412349Z","iopub.status.idle":"2022-07-06T20:40:08.426106Z","shell.execute_reply.started":"2022-07-06T20:40:08.412307Z","shell.execute_reply":"2022-07-06T20:40:08.425505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Tversky Loss\n---\nThis loss was introduced in \"Tversky loss function for image segmentationusing 3D fully convolutional deep networks\", retrievable here: https://arxiv.org/abs/1706.05721. It was designed to optimise segmentation on imbalanced medical datasets by utilising constants that can adjust how harshly different types of error are penalised in the loss function. From the paper:\n\n>... in the case of α=β=0.5 the Tversky index simplifies to be the same as the Dice coefficient, which is also equal to the F1 score.  With α=β=1, Equation 2 produces Tanimoto coefficient, and setting α+β=1 produces the set of Fβ scores. Larger βs weigh recall higher than precision (by placing more emphasis on false negatives).\n\nTo summarise, this loss function is weighted by the constants 'alpha' and 'beta' that penalise false positives and false negatives respectively to a higher degree in the loss function as their value is increased. The beta constant in particular has applications in situations where models can obtain misleadingly positive performance via highly conservative prediction. You may want to experiment with different values to find the optimum. With alpha==beta==0.5, this loss becomes equivalent to Dice Loss.","metadata":{}},{"cell_type":"code","source":"#PyTorch\nALPHA = 0.5\nBETA = 0.5\n\nclass TverskyLoss(nn.Module):\n    def __init__(self, weight=None, size_average=True):\n        super(TverskyLoss, self).__init__()\n\n    def forward(self, inputs, targets, smooth=1, alpha=ALPHA, beta=BETA):\n        \n        #comment out if your model contains a sigmoid or equivalent activation layer\n        inputs = F.sigmoid(inputs)       \n        \n        #flatten label and prediction tensors\n        inputs = inputs.view(-1)\n        targets = targets.view(-1)\n        \n        #True Positives, False Positives & False Negatives\n        TP = (inputs * targets).sum()    \n        FP = ((1-targets) * inputs).sum()\n        FN = (targets * (1-inputs)).sum()\n       \n        Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)  \n        \n        return 1 - Tversky","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2022-07-06T20:40:08.427051Z","iopub.execute_input":"2022-07-06T20:40:08.427368Z","iopub.status.idle":"2022-07-06T20:40:08.437861Z","shell.execute_reply.started":"2022-07-06T20:40:08.427333Z","shell.execute_reply":"2022-07-06T20:40:08.437140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Keras\nALPHA = 0.5\nBETA = 0.5\n\ndef TverskyLoss(targets, inputs, alpha=ALPHA, beta=BETA, smooth=1e-6):\n        \n        #flatten label and prediction tensors\n        inputs = K.flatten(inputs)\n        targets = K.flatten(targets)\n        \n        #True Positives, False Positives & False Negatives\n        TP = K.sum((inputs * targets))\n        FP = K.sum(((1-targets) * inputs))\n        FN = K.sum((targets * (1-inputs)))\n       \n        Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)  \n        \n        return 1 - Tversky","metadata":{"execution":{"iopub.status.busy":"2022-07-06T20:40:08.438781Z","iopub.execute_input":"2022-07-06T20:40:08.439110Z","iopub.status.idle":"2022-07-06T20:40:08.450910Z","shell.execute_reply.started":"2022-07-06T20:40:08.439074Z","shell.execute_reply":"2022-07-06T20:40:08.450185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Focal Tversky Loss\n---\n\nA variant on the Tversky loss that also includes the gamma modifier from Focal Loss.","metadata":{}},{"cell_type":"code","source":"#PyTorch\nALPHA = 0.5\nBETA = 0.5\nGAMMA = 1\n\nclass FocalTverskyLoss(nn.Module):\n    def __init__(self, weight=None, size_average=True):\n        super(FocalTverskyLoss, self).__init__()\n\n    def forward(self, inputs, targets, smooth=1, alpha=ALPHA, beta=BETA, gamma=GAMMA):\n        \n        #comment out if your model contains a sigmoid or equivalent activation layer\n        inputs = F.sigmoid(inputs)       \n        \n        #flatten label and prediction tensors\n        inputs = inputs.view(-1)\n        targets = targets.view(-1)\n        \n        #True Positives, False Positives & False Negatives\n        TP = (inputs * targets).sum()    \n        FP = ((1-targets) * inputs).sum()\n        FN = (targets * (1-inputs)).sum()\n        \n        Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)  \n        FocalTversky = (1 - Tversky)**gamma\n                       \n        return FocalTversky","metadata":{"execution":{"iopub.status.busy":"2022-07-06T20:40:08.452199Z","iopub.execute_input":"2022-07-06T20:40:08.452459Z","iopub.status.idle":"2022-07-06T20:40:08.465252Z","shell.execute_reply.started":"2022-07-06T20:40:08.452411Z","shell.execute_reply":"2022-07-06T20:40:08.464390Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Keras\nALPHA = 0.5\nBETA = 0.5\nGAMMA = 1\n\ndef FocalTverskyLoss(targets, inputs, alpha=ALPHA, beta=BETA, gamma=GAMMA, smooth=1e-6):\n    \n        #flatten label and prediction tensors\n        inputs = K.flatten(inputs)\n        targets = K.flatten(targets)\n        \n        #True Positives, False Positives & False Negatives\n        TP = K.sum((inputs * targets))\n        FP = K.sum(((1-targets) * inputs))\n        FN = K.sum((targets * (1-inputs)))\n               \n        Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)  \n        FocalTversky = K.pow((1 - Tversky), gamma)\n        \n        return FocalTversky","metadata":{"execution":{"iopub.status.busy":"2022-07-06T20:40:08.466546Z","iopub.execute_input":"2022-07-06T20:40:08.467024Z","iopub.status.idle":"2022-07-06T20:40:08.476333Z","shell.execute_reply.started":"2022-07-06T20:40:08.466981Z","shell.execute_reply":"2022-07-06T20:40:08.475757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Lovasz Hinge Loss\n---\nThis complex loss function was introduced by Berman, Triki and Blaschko in their paper \"The Lovasz-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks\", retrievable here: https://arxiv.org/abs/1705.08790. It is designed to optimise the Intersection over Union score for semantic segmentation, particularly for multi-class instances. Specifically, it sorts predictions by their error before calculating cumulatively how each error affects the IoU score. This gradient vector is then multiplied with the initial error vector to penalise most strongly the predictions that decreased the IoU score the most. This procedure is detailed by [jeandebleu](https://www.kaggle.com/jeandebleau) in his excellent summary [here](https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/67791).\n\nThis code is taken directly from the author's github repo here: https://github.com/bermanmaxim/LovaszSoftmax and all credit is to them.\n\nIn this kernel I have implemented the flat variant that uses reshaped rank-1 tensors as inputs for PyTorch. You can modify it accordingly with the dimensions and class number of your data as needed. This code takes raw logits so ensure your model does not contain an activation layer prior to the loss calculation.\n\nI have hidden the researchers' own code below for brevity; simply load it into your kernel for the losses to function. In the case of their tensorflow implementation, I am still working to make it compatible with Keras. There are differences between the Tensorflow and Keras function libraries that complicate this.","metadata":{}},{"cell_type":"code","source":"#PyTorch\n\ndef flatten_binary_scores(scores, labels, ignore=None):\n    \"\"\"\n    Flattens predictions in the batch (binary case)\n    Remove labels equal to 'ignore'\n    \"\"\"\n    scores = scores.view(-1)\n    labels = labels.view(-1)\n    if ignore is None:\n        return scores, labels\n    valid = (labels != ignore)\n    vscores = scores[valid]\n    vlabels = labels[valid]\n    return vscores, vlabels\n\ndef lovasz_grad(gt_sorted):\n    \"\"\"\n    Computes gradient of the Lovasz extension w.r.t sorted errors\n    See Alg. 1 in paper\n    \"\"\"\n    p = len(gt_sorted)\n    gts = gt_sorted.sum()\n    intersection = gts - gt_sorted.float().cumsum(0)\n    union = gts + (1 - gt_sorted).float().cumsum(0)\n    jaccard = 1. - intersection / union\n    if p > 1: # cover 1-pixel case\n        jaccard[1:p] = jaccard[1:p] - jaccard[0:-1]\n    return jaccard\n\ndef lovasz_hinge(logits, labels, per_image=True, ignore=None):\n    \"\"\"\n    Binary Lovasz hinge loss\n      logits: [B, H, W] Variable, logits at each pixel (between -\\infty and +\\infty)\n      labels: [B, H, W] Tensor, binary ground truth masks (0 or 1)\n      per_image: compute the loss per image instead of per batch\n      ignore: void class id\n    \"\"\"\n    if per_image:\n        loss = mean(lovasz_hinge_flat(*flatten_binary_scores(log.unsqueeze(0), lab.unsqueeze(0), ignore))\n                          for log, lab in zip(logits, labels))\n    else:\n        loss = lovasz_hinge_flat(*flatten_binary_scores(logits, labels, ignore))\n    return loss\n\ndef lovasz_hinge_flat(logits, labels):\n    \"\"\"\n    Binary Lovasz hinge loss\n      logits: [P] Variable, logits at each prediction (between -\\infty and +\\infty)\n      labels: [P] Tensor, binary ground truth labels (0 or 1)\n      ignore: label to ignore\n    \"\"\"\n    if len(labels) == 0:\n        # only void pixels, the gradients should be 0\n        return logits.sum() * 0.\n    signs = 2. * labels.float() - 1.\n    errors = (1. - logits * Variable(signs))\n    errors_sorted, perm = torch.sort(errors, dim=0, descending=True)\n    perm = perm.data\n    gt_sorted = labels[perm]\n    grad = lovasz_grad(gt_sorted)\n    loss = torch.dot(F.relu(errors_sorted), Variable(grad))\n    return loss\n\n#=====\n#Multi-class Lovasz loss\n#=====\n\ndef lovasz_softmax(probas, labels, classes='present', per_image=False, ignore=None):\n    \"\"\"\n    Multi-class Lovasz-Softmax loss\n      probas: [B, C, H, W] Variable, class probabilities at each prediction (between 0 and 1).\n              Interpreted as binary (sigmoid) output with outputs of size [B, H, W].\n      labels: [B, H, W] Tensor, ground truth labels (between 0 and C - 1)\n      classes: 'all' for all, 'present' for classes present in labels, or a list of classes to average.\n      per_image: compute the loss per image instead of per batch\n      ignore: void class labels\n    \"\"\"\n    if per_image:\n        loss = mean(lovasz_softmax_flat(*flatten_probas(prob.unsqueeze(0), lab.unsqueeze(0), ignore), classes=classes)\n                          for prob, lab in zip(probas, labels))\n    else:\n        loss = lovasz_softmax_flat(*flatten_probas(probas, labels, ignore), classes=classes)\n    return loss\n\n\ndef lovasz_softmax_flat(probas, labels, classes='present'):\n    \"\"\"\n    Multi-class Lovasz-Softmax loss\n      probas: [P, C] Variable, class probabilities at each prediction (between 0 and 1)\n      labels: [P] Tensor, ground truth labels (between 0 and C - 1)\n      classes: 'all' for all, 'present' for classes present in labels, or a list of classes to average.\n    \"\"\"\n    if probas.numel() == 0:\n        # only void pixels, the gradients should be 0\n        return probas * 0.\n    C = probas.size(1)\n    losses = []\n    class_to_sum = list(range(C)) if classes in ['all', 'present'] else classes\n    for c in class_to_sum:\n        fg = (labels == c).float() # foreground for class c\n        if (classes is 'present' and fg.sum() == 0):\n            continue\n        if C == 1:\n            if len(classes) > 1:\n                raise ValueError('Sigmoid output possible only with 1 class')\n            class_pred = probas[:, 0]\n        else:\n            class_pred = probas[:, c]\n        errors = (Variable(fg) - class_pred).abs()\n        errors_sorted, perm = torch.sort(errors, 0, descending=True)\n        perm = perm.data\n        fg_sorted = fg[perm]\n        losses.append(torch.dot(errors_sorted, Variable(lovasz_grad(fg_sorted))))\n    return mean(losses)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-06T20:40:08.477341Z","iopub.execute_input":"2022-07-06T20:40:08.477696Z","iopub.status.idle":"2022-07-06T20:40:08.505583Z","shell.execute_reply.started":"2022-07-06T20:40:08.477646Z","shell.execute_reply":"2022-07-06T20:40:08.504765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#PyTorch\nclass LovaszHingeLoss(nn.Module):\n    def __init__(self, weight=None, size_average=True):\n        super(LovaszHingeLoss, self).__init__()\n\n    def forward(self, inputs, targets):\n        inputs = F.sigmoid(inputs)    \n        Lovasz = lovasz_hinge(inputs, targets, per_image=False)                       \n        return Lovasz","metadata":{"execution":{"iopub.status.busy":"2022-07-06T20:40:08.506980Z","iopub.execute_input":"2022-07-06T20:40:08.507401Z","iopub.status.idle":"2022-07-06T20:40:08.519797Z","shell.execute_reply.started":"2022-07-06T20:40:08.507349Z","shell.execute_reply":"2022-07-06T20:40:08.519001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\nimport numpy as np\n\n\ndef lovasz_grad(gt_sorted):\n    \"\"\"\n    Computes gradient of the Lovasz extension w.r.t sorted errors\n    See Alg. 1 in paper\n    \"\"\"\n    gts = tf.reduce_sum(gt_sorted)\n    intersection = gts - tf.cumsum(gt_sorted)\n    union = gts + tf.cumsum(1. - gt_sorted)\n    jaccard = 1. - intersection / union\n    jaccard = tf.concat((jaccard[0:1], jaccard[1:] - jaccard[:-1]), 0)\n    return jaccard\n\n\n# --------------------------- BINARY LOSSES ---------------------------\n\n\ndef lovasz_hinge(logits, labels, per_image=True, ignore=None):\n    \"\"\"\n    Binary Lovasz hinge loss\n      logits: [B, H, W] Variable, logits at each pixel (between -\\infty and +\\infty)\n      labels: [B, H, W] Tensor, binary ground truth masks (0 or 1)\n      per_image: compute the loss per image instead of per batch\n      ignore: void class id\n    \"\"\"\n    if per_image:\n        def treat_image(log_lab):\n            log, lab = log_lab\n            log, lab = tf.expand_dims(log, 0), tf.expand_dims(lab, 0)\n            log, lab = flatten_binary_scores(log, lab, ignore)\n            return lovasz_hinge_flat(log, lab)\n        losses = tf.map_fn(treat_image, (logits, labels), dtype=tf.float32)\n        loss = tf.reduce_mean(losses)\n    else:\n        loss = lovasz_hinge_flat(*flatten_binary_scores(logits, labels, ignore))\n    return loss\n\n\ndef lovasz_hinge_flat(logits, labels):\n    \"\"\"\n    Binary Lovasz hinge loss\n      logits: [P] Variable, logits at each prediction (between -\\infty and +\\infty)\n      labels: [P] Tensor, binary ground truth labels (0 or 1)\n      ignore: label to ignore\n    \"\"\"\n\n    def compute_loss():\n        labelsf = tf.cast(labels, logits.dtype)\n        signs = 2. * labelsf - 1.\n        errors = 1. - logits * tf.stop_gradient(signs)\n        errors_sorted, perm = tf.nn.top_k(errors, k=tf.shape(errors)[0], name=\"descending_sort\")\n        gt_sorted = tf.gather(labelsf, perm)\n        grad = lovasz_grad(gt_sorted)\n        loss = tf.tensordot(tf.nn.relu(errors_sorted), grad, 1, name=\"loss_non_void\")\n        return loss\n\n    # deal with the void prediction case (only void pixels)\n    loss = tf.cond(tf.equal(tf.shape(logits)[0], 0),\n                   lambda: tf.reduce_sum(logits) * 0.,\n                   compute_loss,\n                   strict=True,\n                   name=\"loss\"\n                   )\n    return loss\n\n\ndef flatten_binary_scores(scores, labels, ignore=None):\n    \"\"\"\n    Flattens predictions in the batch (binary case)\n    Remove labels equal to 'ignore'\n    \"\"\"\n    scores = tf.reshape(scores, (-1,))\n    labels = tf.reshape(labels, (-1,))\n    if ignore is None:\n        return scores, labels\n    valid = tf.not_equal(labels, ignore)\n    vscores = tf.boolean_mask(scores, valid, name='valid_scores')\n    vlabels = tf.boolean_mask(labels, valid, name='valid_labels')\n    return vscores, vlabels","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-06T20:40:08.521483Z","iopub.execute_input":"2022-07-06T20:40:08.521791Z","iopub.status.idle":"2022-07-06T20:40:08.542280Z","shell.execute_reply.started":"2022-07-06T20:40:08.521737Z","shell.execute_reply":"2022-07-06T20:40:08.541485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Keras\n# not working yet\n# def LovaszHingeLoss(inputs, targets):\n#     return lovasz_hinge_loss(inputs, targets)","metadata":{"execution":{"iopub.status.busy":"2022-07-06T20:40:08.543658Z","iopub.execute_input":"2022-07-06T20:40:08.544200Z","iopub.status.idle":"2022-07-06T20:40:08.556173Z","shell.execute_reply.started":"2022-07-06T20:40:08.543862Z","shell.execute_reply":"2022-07-06T20:40:08.555219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Combo Loss\n---\nThis loss was introduced by Taghanaki et al in their paper \"Combo loss: Handling input and output imbalance in multi-organ segmentation\", retrievable here: https://arxiv.org/abs/1805.02798. Combo loss is a combination of Dice Loss and a modified Cross-Entropy function that, like Tversky loss, has additional constants which penalise either false positives or false negatives more respectively.\n\nSince my GPU quota has run out this week, as of V16 these functions have not been tested, so please leave any debugging notes in the comments section below.","metadata":{}},{"cell_type":"code","source":"#PyTorch\nALPHA = 0.5 # < 0.5 penalises FP more, > 0.5 penalises FN more\nCE_RATIO = 0.5 #weighted contribution of modified CE loss compared to Dice loss\n\nclass ComboLoss(nn.Module):\n    def __init__(self, weight=None, size_average=True):\n        super(ComboLoss, self).__init__()\n\n    def forward(self, inputs, targets, smooth=1, alpha=ALPHA, beta=BETA, eps=1e-9):\n        \n        #flatten label and prediction tensors\n        inputs = inputs.view(-1)\n        targets = targets.view(-1)\n        \n        #True Positives, False Positives & False Negatives\n        intersection = (inputs * targets).sum()    \n        dice = (2. * intersection + smooth) / (inputs.sum() + targets.sum() + smooth)\n        \n        inputs = torch.clamp(inputs, eps, 1.0 - eps)       \n        out = - (ALPHA * ((targets * torch.log(inputs)) + ((1 - ALPHA) * (1.0 - targets) * torch.log(1.0 - inputs))))\n        weighted_ce = out.mean(-1)\n        combo = (CE_RATIO * weighted_ce) - ((1 - CE_RATIO) * dice)\n        \n        return combo","metadata":{"execution":{"iopub.status.busy":"2022-07-06T20:40:08.557715Z","iopub.execute_input":"2022-07-06T20:40:08.558218Z","iopub.status.idle":"2022-07-06T20:40:08.571012Z","shell.execute_reply.started":"2022-07-06T20:40:08.558139Z","shell.execute_reply":"2022-07-06T20:40:08.570293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Keras\nALPHA = 0.5 # < 0.5 penalises FP more, > 0.5 penalises FN more\nCE_RATIO = 0.5 #weighted contribution of modified CE loss compared to Dice loss\n\ndef Combo_loss(targets, inputs, eps=1e-9):\n    targets = K.flatten(targets)\n    inputs = K.flatten(inputs)\n    \n    intersection = K.sum(targets * inputs)\n    dice = (2. * intersection + smooth) / (K.sum(targets) + K.sum(inputs) + smooth)\n    inputs = K.clip(inputs, eps, 1.0 - eps)\n    out = - (ALPHA * ((targets * K.log(inputs)) + ((1 - ALPHA) * (1.0 - targets) * K.log(1.0 - inputs))))\n    weighted_ce = K.mean(out, axis=-1)\n    combo = (CE_RATIO * weighted_ce) - ((1 - CE_RATIO) * dice)\n    \n    return combo","metadata":{"execution":{"iopub.status.busy":"2022-07-06T20:40:08.572246Z","iopub.execute_input":"2022-07-06T20:40:08.572664Z","iopub.status.idle":"2022-07-06T20:40:08.582300Z","shell.execute_reply.started":"2022-07-06T20:40:08.572602Z","shell.execute_reply":"2022-07-06T20:40:08.581750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Usage Tips\n---\n\nIn my experience testing and debugging these losses, I have some observations that may be useful to beginners experimenting with different loss functions. These are not rules that are set in stone; they are simply my findings and your results may vary.\n\n* Tversky and Focal-Tversky loss benefit from very low learning rates, of the order 5e-5 to 1e-4. They would not see much improvement in my kernels until around 7-10 epochs, upon which performance would improve significantly.\n\n* In general, if a loss function does not appear to be working well (or at all), experiment with modifying the learning rate before moving on to other options.\n\n* You can easily create your own loss functions by combining any of the above with Binary Cross-Entropy or any combination of other losses. Bear in mind that loss is calculated for every batch, so more complex losses will increase runtime.\n\n* Care must be taken when writing loss functions for PyTorch. If you call a function to modify the inputs that doesn't entirely use PyTorch's numerical methods, the tensor will 'detach' from the the graph that maps it back through the neural network for the purposes of backpropagation, making the loss function unusable. Discussion of this is available [here.](https://discuss.pytorch.org/t/some-problems-in-custom-loss-functions-and-so-on/36618)\n\nI hope this kernel is of use to you. Good luck with your work!","metadata":{}}]}