{
  "id": 239747,
  "title": "Fast metric computation",
  "url": "/competitions/birdclef-2021/discussion/239747",
  "author_name": "",
  "post_date": "2021-05-17T14:05:55.544323Z",
  "votes": 24,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Here is a code I am using.  It runs with both numpy arrays or pytorch tensors (but not if you mix numpy arrays with pytorch tensors).  I don't know for tensorflow but adapting it should be straightforward if it does not run as is.</p>\n<p>It assumes that predictions and target are of size (n samples, n_species+1).  Each column codes the presence of one of the bird species or a no call.  I include the code for recall and precision as a bonus.</p>\n<pre><code>def fast_f1_score(predictions, target):\n    tp = (predictions * target).sum(1)\n    fp = (predictions * (1 - target)).sum(1)\n    fn = ((1 - predictions) * target).sum(1)\n    f1 = tp / (tp + (fp + fn) / 2)\n    precision = tp / (tp + fp)\n    recall = tp / (tp + fn)\n    return f1.mean(), precision.mean(), recall.mean()\n</code></pre>\n<p>On the machine I am using with numpy it runs in 8 milliseconds, to be compared with 2.8 seconds when using a code based on the strings used for submission.</p>\n<p>scikit-learn f1_score with average='sample' runs in 85ms, i.e. 10 times slower.  Note that the order of predictions and target arguments is different in scikit-learn.  I used the pytorch order.</p>\n<p>Note also that the presence of nocall as a target ensures that there is no division by 0.  You can set it from predicting only bird species: its value is 1 if no bird species are predicted.</p>",
  "messages": [
    {
      "id": "1311607",
      "postDate": "05/17/2021 14:05:55",
      "content": "<p>Here is a code I am using.  It runs with both numpy arrays or pytorch tensors (but not if you mix numpy arrays with pytorch tensors).  I don't know for tensorflow but adapting it should be straightforward if it does not run as is.</p>\n<p>It assumes that predictions and target are of size (n samples, n_species+1).  Each column codes the presence of one of the bird species or a no call.  I include the code for recall and precision as a bonus.</p>\n<pre><code>def fast_f1_score(predictions, target):\n    tp = (predictions * target).sum(1)\n    fp = (predictions * (1 - target)).sum(1)\n    fn = ((1 - predictions) * target).sum(1)\n    f1 = tp / (tp + (fp + fn) / 2)\n    precision = tp / (tp + fp)\n    recall = tp / (tp + fn)\n    return f1.mean(), precision.mean(), recall.mean()\n</code></pre>\n<p>On the machine I am using with numpy it runs in 8 milliseconds, to be compared with 2.8 seconds when using a code based on the strings used for submission.</p>\n<p>scikit-learn f1_score with average='sample' runs in 85ms, i.e. 10 times slower.  Note that the order of predictions and target arguments is different in scikit-learn.  I used the pytorch order.</p>\n<p>Note also that the presence of nocall as a target ensures that there is no division by 0.  You can set it from predicting only bird species: its value is 1 if no bird species are predicted.</p>",
      "rawMarkdown": "Here is a code I am using.  It runs with both numpy arrays or pytorch tensors (but not if you mix numpy arrays with pytorch tensors).  I don't know for tensorflow but adapting it should be straightforward if it does not run as is.\n\nIt assumes that predictions and target are of size (n samples, n_species+1).  Each column codes the presence of one of the bird species or a no call.  I include the code for recall and precision as a bonus.\n\n```\ndef fast_f1_score(predictions, target):\n    tp = (predictions * target).sum(1)\n    fp = (predictions * (1 - target)).sum(1)\n    fn = ((1 - predictions) * target).sum(1)\n    f1 = tp / (tp + (fp + fn) / 2)\n    precision = tp / (tp + fp)\n    recall = tp / (tp + fn)\n    return f1.mean(), precision.mean(), recall.mean()\n```\n\nOn the machine I am using with numpy it runs in 8 milliseconds, to be compared with 2.8 seconds when using a code based on the strings used for submission.\n\nscikit-learn f1_score with average='sample' runs in 85ms, i.e. 10 times slower.  Note that the order of predictions and target arguments is different in scikit-learn.  I used the pytorch order.\n\nNote also that the presence of nocall as a target ensures that there is no division by 0.  You can set it from predicting only bird species: its value is 1 if no bird species are predicted.",
      "votes": null
    },
    {
      "id": "1312182",
      "postDate": "05/17/2021 22:11:34",
      "content": "<p>Thanks for sharing. Looks training 'nocall' class with extra data is still working for you, I tried many ways but not work well. 😓</p>",
      "rawMarkdown": "Thanks for sharing. Looks training 'nocall' class with extra data is still working for you, I tried many ways but not work well. 😓",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1312182,
      "author_name": "superchenhao",
      "author_url": "",
      "post_date": "05/17/2021 22:11:34",
      "content": "<p>Thanks for sharing. Looks training 'nocall' class with extra data is still working for you, I tried many ways but not work well. 😓</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1311607": "Here is a code I am using.  It runs with both numpy arrays or pytorch tensors (but not if you mix numpy arrays with pytorch tensors).  I don't know for tensorflow but adapting it should be straightforward if it does not run as is.\n\nIt assumes that predictions and target are of size (n samples, n_species+1).  Each column codes the presence of one of the bird species or a no call.  I include the code for recall and precision as a bonus.\n\n```\ndef fast_f1_score(predictions, target):\n    tp = (predictions * target).sum(1)\n    fp = (predictions * (1 - target)).sum(1)\n    fn = ((1 - predictions) * target).sum(1)\n    f1 = tp / (tp + (fp + fn) / 2)\n    precision = tp / (tp + fp)\n    recall = tp / (tp + fn)\n    return f1.mean(), precision.mean(), recall.mean()\n```\n\nOn the machine I am using with numpy it runs in 8 milliseconds, to be compared with 2.8 seconds when using a code based on the strings used for submission.\n\nscikit-learn f1_score with average='sample' runs in 85ms, i.e. 10 times slower.  Note that the order of predictions and target arguments is different in scikit-learn.  I used the pytorch order.\n\nNote also that the presence of nocall as a target ensures that there is no division by 0.  You can set it from predicting only bird species: its value is 1 if no bird species are predicted.",
    "1312182": "Thanks for sharing. Looks training 'nocall' class with extra data is still working for you, I tried many ways but not work well. 😓"
  },
  "source": "meta"
}