{
  "id": 189302,
  "title": "Surrogate loss for modified Laplace log likelihood (79th solution)",
  "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/189302",
  "author_name": "datasaurus",
  "post_date": "2020-10-07T08:19:21.107000",
  "votes": 3,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Congrats to all the winners and all those who survived the shakeup!</p>\n<p>I wasn't really present in the last month of this competition since I was moving house and my computer is in a shipping container somewhere on the sea, but I thought I'd share the loss function I made that served as a pretty good surrogate for the competition metric (in PyTorch). The predictions should be a tensor of shape (N, 2) where the first column is the FVC prediction, and the second is the confidence.</p>\n<pre><code>class LaplaceLogLikelihood(nn.Module):\n    def __init__(self):\n        super(LaplaceLogLikelihood, self).__init__()\n        self.l1_loss = nn.SmoothL1Loss(reduction=\"none\")\n        self.lrelu = nn.RReLU()\n        self.root2 = torch.sqrt(torch.tensor(2, dtype=torch.float, requires_grad=False))\n\n    def forward(self, predictions, target, clamp=False):\n        delta = self.l1_loss(predictions[:, 0], target)\n\n        if clamp:\n            delta = torch.clamp(delta, max=1000)\n            sigma = torch.clamp(predictions[:, 1], min=70)\n        # clip sigma without destroying gradient\n        else:\n            sigma = self.lrelu(predictions[:, 1] - 70) + 70\n\n        laplace_ll = -(self.root2 * delta) / sigma - torch.log(self.root2 * sigma)\n        laplace_ll_mean = torch.mean(laplace_ll)\n        return -laplace_ll_mean\n</code></pre>\n<p>The delta part of the metric was easy - we can just use <code>L1Loss</code> or <code>SmoothedL1Loss</code> as I have.</p>\n<p>The tricky part was the <code>max(sigma, 70)</code> for the clipping of confidence. For this, I used a leaky ReLU so that we could do something similar without killing the gradient.</p>\n<p>My best private LB score was <strong>-6.8383</strong> (gold) using tabular only but of course, I chose the wrong version ☹️. I'm not going to make excuses, and there were people who had worked harder than me who deserved it more. I would say though a) trust your CV and b) annotate/track your notebooks clearly to make picking submissions easy for yourself</p>\n<p>Here's the complete code: <a href=\"https://www.kaggle.com/anjum48/fully-connected-custom-loss-private-6-8383\" target=\"_blank\">https://www.kaggle.com/anjum48/fully-connected-custom-loss-private-6-8383</a></p>",
  "messages": [
    {
      "id": 1040564,
      "postDate": "2020-10-07T08:19:21.107Z",
      "content": "<p>Congrats to all the winners and all those who survived the shakeup!</p>\n<p>I wasn't really present in the last month of this competition since I was moving house and my computer is in a shipping container somewhere on the sea, but I thought I'd share the loss function I made that served as a pretty good surrogate for the competition metric (in PyTorch). The predictions should be a tensor of shape (N, 2) where the first column is the FVC prediction, and the second is the confidence.</p>\n<pre><code>class LaplaceLogLikelihood(nn.Module):\n    def __init__(self):\n        super(LaplaceLogLikelihood, self).__init__()\n        self.l1_loss = nn.SmoothL1Loss(reduction=\"none\")\n        self.lrelu = nn.RReLU()\n        self.root2 = torch.sqrt(torch.tensor(2, dtype=torch.float, requires_grad=False))\n\n    def forward(self, predictions, target, clamp=False):\n        delta = self.l1_loss(predictions[:, 0], target)\n\n        if clamp:\n            delta = torch.clamp(delta, max=1000)\n            sigma = torch.clamp(predictions[:, 1], min=70)\n        # clip sigma without destroying gradient\n        else:\n            sigma = self.lrelu(predictions[:, 1] - 70) + 70\n\n        laplace_ll = -(self.root2 * delta) / sigma - torch.log(self.root2 * sigma)\n        laplace_ll_mean = torch.mean(laplace_ll)\n        return -laplace_ll_mean\n</code></pre>\n<p>The delta part of the metric was easy - we can just use <code>L1Loss</code> or <code>SmoothedL1Loss</code> as I have.</p>\n<p>The tricky part was the <code>max(sigma, 70)</code> for the clipping of confidence. For this, I used a leaky ReLU so that we could do something similar without killing the gradient.</p>\n<p>My best private LB score was <strong>-6.8383</strong> (gold) using tabular only but of course, I chose the wrong version ☹️. I'm not going to make excuses, and there were people who had worked harder than me who deserved it more. I would say though a) trust your CV and b) annotate/track your notebooks clearly to make picking submissions easy for yourself</p>\n<p>Here's the complete code: <a href=\"https://www.kaggle.com/anjum48/fully-connected-custom-loss-private-6-8383\" target=\"_blank\">https://www.kaggle.com/anjum48/fully-connected-custom-loss-private-6-8383</a></p>",
      "rawMarkdown": "Congrats to all the winners and all those who survived the shakeup!\n\nI wasn't really present in the last month of this competition since I was moving house and my computer is in a shipping container somewhere on the sea, but I thought I'd share the loss function I made that served as a pretty good surrogate for the competition metric (in PyTorch). The predictions should be a tensor of shape (N, 2) where the first column is the FVC prediction, and the second is the confidence.\n\n```\nclass LaplaceLogLikelihood(nn.Module):\n    def __init__(self):\n        super(LaplaceLogLikelihood, self).__init__()\n        self.l1_loss = nn.SmoothL1Loss(reduction=\"none\")\n        self.lrelu = nn.RReLU()\n        self.root2 = torch.sqrt(torch.tensor(2, dtype=torch.float, requires_grad=False))\n\n    def forward(self, predictions, target, clamp=False):\n        delta = self.l1_loss(predictions[:, 0], target)\n\n        if clamp:\n            delta = torch.clamp(delta, max=1000)\n            sigma = torch.clamp(predictions[:, 1], min=70)\n        # clip sigma without destroying gradient\n        else:\n            sigma = self.lrelu(predictions[:, 1] - 70) + 70\n\n        laplace_ll = -(self.root2 * delta) / sigma - torch.log(self.root2 * sigma)\n        laplace_ll_mean = torch.mean(laplace_ll)\n        return -laplace_ll_mean\n```\nThe delta part of the metric was easy - we can just use `L1Loss` or `SmoothedL1Loss` as I have.\n\nThe tricky part was the `max(sigma, 70)` for the clipping of confidence. For this, I used a leaky ReLU so that we could do something similar without killing the gradient.\n\nMy best private LB score was **-6.8383** (gold) using tabular only but of course, I chose the wrong version ☹️. I'm not going to make excuses, and there were people who had worked harder than me who deserved it more. I would say though a) trust your CV and b) annotate/track your notebooks clearly to make picking submissions easy for yourself\n\nHere's the complete code: https://www.kaggle.com/anjum48/fully-connected-custom-loss-private-6-8383",
      "votes": 4
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1040564": "Congrats to all the winners and all those who survived the shakeup!\n\nI wasn't really present in the last month of this competition since I was moving house and my computer is in a shipping container somewhere on the sea, but I thought I'd share the loss function I made that served as a pretty good surrogate for the competition metric (in PyTorch). The predictions should be a tensor of shape (N, 2) where the first column is the FVC prediction, and the second is the confidence.\n\n```\nclass LaplaceLogLikelihood(nn.Module):\n    def __init__(self):\n        super(LaplaceLogLikelihood, self).__init__()\n        self.l1_loss = nn.SmoothL1Loss(reduction=\"none\")\n        self.lrelu = nn.RReLU()\n        self.root2 = torch.sqrt(torch.tensor(2, dtype=torch.float, requires_grad=False))\n\n    def forward(self, predictions, target, clamp=False):\n        delta = self.l1_loss(predictions[:, 0], target)\n\n        if clamp:\n            delta = torch.clamp(delta, max=1000)\n            sigma = torch.clamp(predictions[:, 1], min=70)\n        # clip sigma without destroying gradient\n        else:\n            sigma = self.lrelu(predictions[:, 1] - 70) + 70\n\n        laplace_ll = -(self.root2 * delta) / sigma - torch.log(self.root2 * sigma)\n        laplace_ll_mean = torch.mean(laplace_ll)\n        return -laplace_ll_mean\n```\nThe delta part of the metric was easy - we can just use `L1Loss` or `SmoothedL1Loss` as I have.\n\nThe tricky part was the `max(sigma, 70)` for the clipping of confidence. For this, I used a leaky ReLU so that we could do something similar without killing the gradient.\n\nMy best private LB score was **-6.8383** (gold) using tabular only but of course, I chose the wrong version ☹️. I'm not going to make excuses, and there were people who had worked harder than me who deserved it more. I would say though a) trust your CV and b) annotate/track your notebooks clearly to make picking submissions easy for yourself\n\nHere's the complete code: https://www.kaggle.com/anjum48/fully-connected-custom-loss-private-6-8383"
  }
}