{
  "id": 70225,
  "title": "direct F1 optmisation",
  "url": "/competitions/human-protein-atlas-image-classification/discussion/70225",
  "author_name": "",
  "post_date": "2018-11-01T04:02:23.654051800Z",
  "votes": 13,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Scalable Learning of Non-Decomposable Objectives\n- Elad Eban (elade@google.com)</p>\n\n<p><a href=\"http://proceedings.mlr.press/v54/eban17a/eban17a.pdf\">http://proceedings.mlr.press/v54/eban17a/eban17a.pdf</a></p>\n\n<p>Modern retrieval systems are often driven by an underlying machine learning model.\nThe goal of such systems is to identify and possibly rank the few most relevant items for\na given query or context. Thus, such systems are typically evaluated using a ranking-based\nperformance metric such as the area under the precision-recall curve, the Fβ score, precision\nat fixed recall, etc. Obviously, it is desirable to train such systems to optimize the metric of interest</p>\n\n<p><img src=\"https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/5c8a5f6f2b29f8dce2a07fcb3790f9160b36f87e/8-Figure2-1.png\" alt=\"enter image description here\"></p>",
  "messages": [
    {
      "id": "413512",
      "postDate": "11/01/2018 04:02:23",
      "content": "<p>Scalable Learning of Non-Decomposable Objectives\n- Elad Eban (elade@google.com)</p>\n\n<p><a href=\"http://proceedings.mlr.press/v54/eban17a/eban17a.pdf\">http://proceedings.mlr.press/v54/eban17a/eban17a.pdf</a></p>\n\n<p>Modern retrieval systems are often driven by an underlying machine learning model.\nThe goal of such systems is to identify and possibly rank the few most relevant items for\na given query or context. Thus, such systems are typically evaluated using a ranking-based\nperformance metric such as the area under the precision-recall curve, the Fβ score, precision\nat fixed recall, etc. Obviously, it is desirable to train such systems to optimize the metric of interest</p>\n\n<p><img src=\"https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/5c8a5f6f2b29f8dce2a07fcb3790f9160b36f87e/8-Figure2-1.png\" alt=\"enter image description here\"></p>",
      "rawMarkdown": "Scalable Learning of Non-Decomposable Objectives\n- Elad Eban (elade@google.com)\n\nhttp://proceedings.mlr.press/v54/eban17a/eban17a.pdf\n\nModern retrieval systems are often driven by an underlying machine learning model.\nThe goal of such systems is to identify and possibly rank the few most relevant items for\na given query or context. Thus, such systems are typically evaluated using a ranking-based\nperformance metric such as the area under the precision-recall curve, the Fβ score, precision\nat fixed recall, etc. Obviously, it is desirable to train such systems to optimize the metric of interest\n\n\n  ![enter image description here][1]\n\n\n  [1]: https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/5c8a5f6f2b29f8dce2a07fcb3790f9160b36f87e/8-Figure2-1.png",
      "votes": null
    },
    {
      "id": "413517",
      "postDate": "11/01/2018 04:08:01",
      "content": "<p>related papers:</p>\n\n<p>Optimizing Non-decomposable Measures with Deep Networks</p>\n\n<p>Metric-Optimized Example Weights</p>",
      "rawMarkdown": "related papers:\n\nOptimizing Non-decomposable Measures with Deep Networks\n\nMetric-Optimized Example Weights",
      "votes": null
    },
    {
      "id": "413853",
      "postDate": "11/01/2018 16:36:36",
      "content": "<p>Differentiable f1_loss, adapted from bestfitting's implementation:</p>\n\n<pre><code>def f1_loss(logits, labels):\n__small_value=1e-6\nbeta = 1\nbatch_size = logits.size()[0]\np = F.sigmoid(logits)\nl = labels\nnum_pos = torch.sum(p, 1) + __small_value\nnum_pos_hat = torch.sum(l, 1) + __small_value\ntp = torch.sum(l * p, 1)\nprecise = tp / num_pos\nrecall = tp / num_pos_hat\nfs = (1 + beta * beta) * precise * recall / (beta * beta * precise + recall + __small_value)\nloss = fs.sum() / batch_size\nreturn (1 - loss)\n</code></pre>",
      "rawMarkdown": "Differentiable f1_loss, adapted from bestfitting's implementation:\n\n    def f1_loss(logits, labels):\n    __small_value=1e-6\n    beta = 1\n    batch_size = logits.size()[0]\n    p = F.sigmoid(logits)\n    l = labels\n    num_pos = torch.sum(p, 1) + __small_value\n    num_pos_hat = torch.sum(l, 1) + __small_value\n    tp = torch.sum(l * p, 1)\n    precise = tp / num_pos\n    recall = tp / num_pos_hat\n    fs = (1 + beta * beta) * precise * recall / (beta * beta * precise + recall + __small_value)\n    loss = fs.sum() / batch_size\n    return (1 - loss)",
      "votes": null
    },
    {
      "id": "413862",
      "postDate": "11/01/2018 16:59:48",
      "content": "<p>do you know an implementation of this? I tried to do something like this, but the problem is pretty complex.</p>",
      "rawMarkdown": "do you know an implementation of this? I tried to do something like this, but the problem is pretty complex.",
      "votes": null
    },
    {
      "id": "430203",
      "postDate": "11/30/2018 02:33:00",
      "content": "<p>thx</p>",
      "rawMarkdown": "thx",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 413517,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "11/01/2018 04:08:01",
      "content": "<p>related papers:</p>\n\n<p>Optimizing Non-decomposable Measures with Deep Networks</p>\n\n<p>Metric-Optimized Example Weights</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 413853,
      "author_name": "alexanderliao",
      "author_url": "",
      "post_date": "11/01/2018 16:36:36",
      "content": "<p>Differentiable f1_loss, adapted from bestfitting's implementation:</p>\n\n<pre><code>def f1_loss(logits, labels):\n__small_value=1e-6\nbeta = 1\nbatch_size = logits.size()[0]\np = F.sigmoid(logits)\nl = labels\nnum_pos = torch.sum(p, 1) + __small_value\nnum_pos_hat = torch.sum(l, 1) + __small_value\ntp = torch.sum(l * p, 1)\nprecise = tp / num_pos\nrecall = tp / num_pos_hat\nfs = (1 + beta * beta) * precise * recall / (beta * beta * precise + recall + __small_value)\nloss = fs.sum() / batch_size\nreturn (1 - loss)\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 413862,
      "author_name": "tcapelle",
      "author_url": "",
      "post_date": "11/01/2018 16:59:48",
      "content": "<p>do you know an implementation of this? I tried to do something like this, but the problem is pretty complex.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 430203,
      "author_name": "zjucor",
      "author_url": "",
      "post_date": "11/30/2018 02:33:00",
      "content": "<p>thx</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "413512": "Scalable Learning of Non-Decomposable Objectives\n- Elad Eban (elade@google.com)\n\nhttp://proceedings.mlr.press/v54/eban17a/eban17a.pdf\n\nModern retrieval systems are often driven by an underlying machine learning model.\nThe goal of such systems is to identify and possibly rank the few most relevant items for\na given query or context. Thus, such systems are typically evaluated using a ranking-based\nperformance metric such as the area under the precision-recall curve, the Fβ score, precision\nat fixed recall, etc. Obviously, it is desirable to train such systems to optimize the metric of interest\n\n\n  ![enter image description here][1]\n\n\n  [1]: https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/5c8a5f6f2b29f8dce2a07fcb3790f9160b36f87e/8-Figure2-1.png",
    "413517": "related papers:\n\nOptimizing Non-decomposable Measures with Deep Networks\n\nMetric-Optimized Example Weights",
    "413853": "Differentiable f1_loss, adapted from bestfitting's implementation:\n\n    def f1_loss(logits, labels):\n    __small_value=1e-6\n    beta = 1\n    batch_size = logits.size()[0]\n    p = F.sigmoid(logits)\n    l = labels\n    num_pos = torch.sum(p, 1) + __small_value\n    num_pos_hat = torch.sum(l, 1) + __small_value\n    tp = torch.sum(l * p, 1)\n    precise = tp / num_pos\n    recall = tp / num_pos_hat\n    fs = (1 + beta * beta) * precise * recall / (beta * beta * precise + recall + __small_value)\n    loss = fs.sum() / batch_size\n    return (1 - loss)",
    "413862": "do you know an implementation of this? I tried to do something like this, but the problem is pretty complex.",
    "430203": "thx"
  },
  "source": "meta"
}