{
  "id": 73246,
  "title": "compute macro f1 score in pytorch",
  "url": "/competitions/human-protein-atlas-image-classification/discussion/73246",
  "author_name": "",
  "post_date": "2018-12-01T02:11:42.434514300Z",
  "votes": 2,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Compute f1 each batch ?Or each image?</p>",
  "messages": [
    {
      "id": "430817",
      "postDate": "12/01/2018 02:11:42",
      "content": "<p>Compute f1 each batch ?Or each image?</p>",
      "rawMarkdown": "Compute f1 each batch ?Or each image?",
      "votes": null
    },
    {
      "id": "430820",
      "postDate": "12/01/2018 02:15:14",
      "content": "<p>I tried to use f1 score in sklearn,like this:\n```python\nfor i,(images,target) in enumerate(train_loader):\n        images = images.cuda(non_blocking=True)\n        target = torch.from_numpy(np.array(target)).float().cuda(non_blocking=True)\n        # compute output\n        output = model(images)\n        loss = criterion(output,target)\n        losses.update(loss.item(),images.size(0))</p>\n\n<pre><code>    f1_batch = f1_score(target.cpu(),output.sigmoid().cpu() &gt; 0.15,average='macro')\n</code></pre>\n\n<p>```\nand here is my results:\ntrain f1: 0.56\nval f1: 0.417\ntest f1:0461</p>",
      "rawMarkdown": "I tried to use f1 score in sklearn,like this:\n```python\nfor i,(images,target) in enumerate(train_loader):\n        images = images.cuda(non_blocking=True)\n        target = torch.from_numpy(np.array(target)).float().cuda(non_blocking=True)\n        # compute output\n        output = model(images)\n        loss = criterion(output,target)\n        losses.update(loss.item(),images.size(0))\n        \n        f1_batch = f1_score(target.cpu(),output.sigmoid().cpu() &gt; 0.15,average='macro')\n```\nand here is my results:\ntrain f1: 0.56\nval f1: 0.417\ntest f1:0461",
      "votes": null
    },
    {
      "id": "430821",
      "postDate": "12/01/2018 02:16:51",
      "content": "<p>this compute method will get better score on test set,but haven't get improvement when val f1 get 0.43 </p>",
      "rawMarkdown": "this compute method will get better score on test set,but haven't get improvement when val f1 get 0.43",
      "votes": null
    },
    {
      "id": "430823",
      "postDate": "12/01/2018 02:20:41",
      "content": "<p>anthor compute method is :</p>\n\n<p>```python\nimport torch\ndef F_score(logit, label, threshold=0.5, beta=2):\n    prob = torch.sigmoid(logit)\n    prob = prob &gt; threshold\n    label = label &gt; threshold</p>\n\n<pre><code>TP = (prob &amp; label).sum(1).float()\nTN = ((~prob) &amp; (~label)).sum(1).float()\nFP = (prob &amp; (~label)).sum(1).float()\nFN = ((~prob) &amp; label).sum(1).float()\n\nprecision = torch.mean(TP / (TP + FP + 1e-12))\nrecall = torch.mean(TP / (TP + FN + 1e-12))\nF2 = (1 + beta**2) * precision * recall / (beta**2 * precision + recall + 1e-12)\nreturn F2.mean(0)\n</code></pre>\n\n<p>```</p>\n\n<p>here is my results :\ntrain:0.675\nval:0.6968\ntest:0.396</p>",
      "rawMarkdown": "anthor compute method is :\n\n```python\nimport torch\ndef F_score(logit, label, threshold=0.5, beta=2):\n    prob = torch.sigmoid(logit)\n    prob = prob &gt; threshold\n    label = label &gt; threshold\n\n    TP = (prob &amp; label).sum(1).float()\n    TN = ((~prob) &amp; (~label)).sum(1).float()\n    FP = (prob &amp; (~label)).sum(1).float()\n    FN = ((~prob) &amp; label).sum(1).float()\n\n    precision = torch.mean(TP / (TP + FP + 1e-12))\n    recall = torch.mean(TP / (TP + FN + 1e-12))\n    F2 = (1 + beta**2) * precision * recall / (beta**2 * precision + recall + 1e-12)\n    return F2.mean(0)\n\n```\n\nhere is my results :\ntrain:0.675\nval:0.6968\ntest:0.396",
      "votes": null
    },
    {
      "id": "430834",
      "postDate": "12/01/2018 03:02:38",
      "content": "<p>have you solved it ?</p>",
      "rawMarkdown": "have you solved it ?",
      "votes": null
    },
    {
      "id": "430996",
      "postDate": "12/01/2018 11:34:12",
      "content": "<p>You have to compute the macro f1 score on the whole validation set in one go - not on a per batch basis.</p>",
      "rawMarkdown": "You have to compute the macro f1 score on the whole validation set in one go - not on a per batch basis.",
      "votes": null
    },
    {
      "id": "431271",
      "postDate": "12/02/2018 01:00:12",
      "content": "<p>@Mark Worrall\nThanks a lot, this problem confued me so much</p>",
      "rawMarkdown": "Mark Worrall\nThanks a lot, this problem confued me so much",
      "votes": null
    },
    {
      "id": "805069",
      "postDate": "04/12/2020 10:01:40",
      "content": "<p><a href=\"/spytensor\">@spytensor</a> Hi i am going through the same problem right now. So after calculating \"f1_score\"  for epoch batch, did you directly add it like, \nf1+= batch_f1\nBecause I dont think thats the correct way to do it. We should calculate TP, FP,FN added them over each batch and then calculate precision, recision or f1_score.</p>",
      "rawMarkdown": "spytensor Hi i am going through the same problem right now. So after calculating \"f1_score\"  for epoch batch, did you directly add it like, \nf1+= batch_f1\nBecause I dont think thats the correct way to do it. We should calculate TP, FP,FN added them over each batch and then calculate precision, recision or f1_score.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 430820,
      "author_name": "spytensor",
      "author_url": "",
      "post_date": "12/01/2018 02:15:14",
      "content": "<p>I tried to use f1 score in sklearn,like this:\n```python\nfor i,(images,target) in enumerate(train_loader):\n        images = images.cuda(non_blocking=True)\n        target = torch.from_numpy(np.array(target)).float().cuda(non_blocking=True)\n        # compute output\n        output = model(images)\n        loss = criterion(output,target)\n        losses.update(loss.item(),images.size(0))</p>\n\n<pre><code>    f1_batch = f1_score(target.cpu(),output.sigmoid().cpu() &gt; 0.15,average='macro')\n</code></pre>\n\n<p>```\nand here is my results:\ntrain f1: 0.56\nval f1: 0.417\ntest f1:0461</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 430821,
      "author_name": "spytensor",
      "author_url": "",
      "post_date": "12/01/2018 02:16:51",
      "content": "<p>this compute method will get better score on test set,but haven't get improvement when val f1 get 0.43 </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 430823,
      "author_name": "spytensor",
      "author_url": "",
      "post_date": "12/01/2018 02:20:41",
      "content": "<p>anthor compute method is :</p>\n\n<p>```python\nimport torch\ndef F_score(logit, label, threshold=0.5, beta=2):\n    prob = torch.sigmoid(logit)\n    prob = prob &gt; threshold\n    label = label &gt; threshold</p>\n\n<pre><code>TP = (prob &amp; label).sum(1).float()\nTN = ((~prob) &amp; (~label)).sum(1).float()\nFP = (prob &amp; (~label)).sum(1).float()\nFN = ((~prob) &amp; label).sum(1).float()\n\nprecision = torch.mean(TP / (TP + FP + 1e-12))\nrecall = torch.mean(TP / (TP + FN + 1e-12))\nF2 = (1 + beta**2) * precision * recall / (beta**2 * precision + recall + 1e-12)\nreturn F2.mean(0)\n</code></pre>\n\n<p>```</p>\n\n<p>here is my results :\ntrain:0.675\nval:0.6968\ntest:0.396</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 430834,
      "author_name": "chaojiebuaa",
      "author_url": "",
      "post_date": "12/01/2018 03:02:38",
      "content": "<p>have you solved it ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 430996,
      "author_name": "maw501",
      "author_url": "",
      "post_date": "12/01/2018 11:34:12",
      "content": "<p>You have to compute the macro f1 score on the whole validation set in one go - not on a per batch basis.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 431271,
      "author_name": "spytensor",
      "author_url": "",
      "post_date": "12/02/2018 01:00:12",
      "content": "<p>@Mark Worrall\nThanks a lot, this problem confued me so much</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 805069,
      "author_name": "prateekgupta891",
      "author_url": "",
      "post_date": "04/12/2020 10:01:40",
      "content": "<p><a href=\"/spytensor\">@spytensor</a> Hi i am going through the same problem right now. So after calculating \"f1_score\"  for epoch batch, did you directly add it like, \nf1+= batch_f1\nBecause I dont think thats the correct way to do it. We should calculate TP, FP,FN added them over each batch and then calculate precision, recision or f1_score.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "430817": "Compute f1 each batch ?Or each image?",
    "430820": "I tried to use f1 score in sklearn,like this:\n```python\nfor i,(images,target) in enumerate(train_loader):\n        images = images.cuda(non_blocking=True)\n        target = torch.from_numpy(np.array(target)).float().cuda(non_blocking=True)\n        # compute output\n        output = model(images)\n        loss = criterion(output,target)\n        losses.update(loss.item(),images.size(0))\n        \n        f1_batch = f1_score(target.cpu(),output.sigmoid().cpu() &gt; 0.15,average='macro')\n```\nand here is my results:\ntrain f1: 0.56\nval f1: 0.417\ntest f1:0461",
    "430821": "this compute method will get better score on test set,but haven't get improvement when val f1 get 0.43",
    "430823": "anthor compute method is :\n\n```python\nimport torch\ndef F_score(logit, label, threshold=0.5, beta=2):\n    prob = torch.sigmoid(logit)\n    prob = prob &gt; threshold\n    label = label &gt; threshold\n\n    TP = (prob &amp; label).sum(1).float()\n    TN = ((~prob) &amp; (~label)).sum(1).float()\n    FP = (prob &amp; (~label)).sum(1).float()\n    FN = ((~prob) &amp; label).sum(1).float()\n\n    precision = torch.mean(TP / (TP + FP + 1e-12))\n    recall = torch.mean(TP / (TP + FN + 1e-12))\n    F2 = (1 + beta**2) * precision * recall / (beta**2 * precision + recall + 1e-12)\n    return F2.mean(0)\n\n```\n\nhere is my results :\ntrain:0.675\nval:0.6968\ntest:0.396",
    "430834": "have you solved it ?",
    "430996": "You have to compute the macro f1 score on the whole validation set in one go - not on a per batch basis.",
    "431271": "Mark Worrall\nThanks a lot, this problem confued me so much",
    "805069": "spytensor Hi i am going through the same problem right now. So after calculating \"f1_score\"  for epoch batch, did you directly add it like, \nf1+= batch_f1\nBecause I dont think thats the correct way to do it. We should calculate TP, FP,FN added them over each batch and then calculate precision, recision or f1_score."
  },
  "source": "meta"
}