{
  "id": 177690,
  "title": "Evaluation metric is F1 with average=\"samples\"",
  "url": "/competitions/birdsong-recognition/discussion/177690",
  "author_name": "",
  "post_date": "2020-08-26T23:18:16.639700900Z",
  "votes": 15,
  "comment_count": 12,
  "views": 0,
  "content": "<p>It looks like the metric is what sklearn calls F1 with average=\"samples\":</p>\n<blockquote>\n  <p>'samples':</p>\n<pre><code>Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from accuracy_score).\n</code></pre>\n</blockquote>\n<p>Am I right?  Feedback welcome.</p>\n<p>Edit: this is not news, was discussed here already: <a href=\"https://www.kaggle.com/c/birdsong-recognition/discussion/173144\" target=\"_blank\">https://www.kaggle.com/c/birdsong-recognition/discussion/173144</a></p>",
  "messages": [
    {
      "id": "987027",
      "postDate": "08/26/2020 23:18:16",
      "content": "<p>It looks like the metric is what sklearn calls F1 with average=\"samples\":</p>\n<blockquote>\n  <p>'samples':</p>\n<pre><code>Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from accuracy_score).\n</code></pre>\n</blockquote>\n<p>Am I right?  Feedback welcome.</p>\n<p>Edit: this is not news, was discussed here already: <a href=\"https://www.kaggle.com/c/birdsong-recognition/discussion/173144\" target=\"_blank\">https://www.kaggle.com/c/birdsong-recognition/discussion/173144</a></p>",
      "rawMarkdown": "It looks like the metric is what sklearn calls F1 with average=\"samples\":\n\n> 'samples':\n> \n>     Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from accuracy_score).\n\nAm I right?  Feedback welcome.\n\nEdit: this is not news, was discussed here already: https://www.kaggle.com/c/birdsong-recognition/discussion/173144",
      "votes": null
    },
    {
      "id": "987086",
      "postDate": "08/27/2020 01:59:04",
      "content": "<p>Yes =)</p>\n<p>here is excellent kernel: <a href=\"https://www.kaggle.com/shonenkov/sample-submission-using-custom-check\" target=\"_blank\">https://www.kaggle.com/shonenkov/sample-submission-using-custom-check</a> =) <br>\nand discussion here: <a href=\"https://www.kaggle.com/c/birdsong-recognition/discussion/173144\" target=\"_blank\">https://www.kaggle.com/c/birdsong-recognition/discussion/173144</a> </p>",
      "rawMarkdown": "Yes =)\n\nhere is excellent kernel: https://www.kaggle.com/shonenkov/sample-submission-using-custom-check =) \nand discussion here: https://www.kaggle.com/c/birdsong-recognition/discussion/173144",
      "votes": null
    },
    {
      "id": "987458",
      "postDate": "08/27/2020 09:24:26",
      "content": "<p>Thanks.  I updated my post.</p>",
      "rawMarkdown": "Thanks.  I updated my post.",
      "votes": null
    },
    {
      "id": "988434",
      "postDate": "08/28/2020 03:48:41",
      "content": "<p>For anyone using tensorflow I found </p>\n<pre><code>import tensorflow_addons as tfa\nnum_birds = 264\nmodel.compile(optimizer=optimizer, metrics=[\"accuracy\", tfa.metrics.F1Score(num_birds, average=\"micro\")])\n</code></pre>\n<p>helpful</p>",
      "rawMarkdown": "For anyone using tensorflow I found \n\n\n```\nimport tensorflow_addons as tfa\nnum_birds = 264\nmodel.compile(optimizer=optimizer, metrics=[\"accuracy\", tfa.metrics.F1Score(num_birds, average=\"micro\")])\n```\n\nhelpful",
      "votes": null
    },
    {
      "id": "988782",
      "postDate": "08/28/2020 09:15:49",
      "content": "<p>f1 micro is not the competition metric. </p>",
      "rawMarkdown": "f1 micro is not the competition metric.",
      "votes": null
    },
    {
      "id": "988854",
      "postDate": "08/28/2020 10:31:49",
      "content": "<p><a href=\"https://www.kaggle.com/drhabib\" target=\"_blank\">@drhabib</a> you linked to the submission kernel, the metrics kernel is <a href=\"https://www.kaggle.com/shonenkov/competition-metrics\" target=\"_blank\">https://www.kaggle.com/shonenkov/competition-metrics</a></p>",
      "rawMarkdown": "drhabib you linked to the submission kernel, the metrics kernel is https://www.kaggle.com/shonenkov/competition-metrics",
      "votes": null
    },
    {
      "id": "988998",
      "postDate": "08/28/2020 13:40:07",
      "content": "<p>Tx, I was wondering why he linked the submission kernel.  </p>",
      "rawMarkdown": "Tx, I was wondering why he linked the submission kernel.",
      "votes": null
    },
    {
      "id": "989868",
      "postDate": "08/29/2020 06:55:25",
      "content": "<p>Literally it says \"micro averaged\" so I've used f1_score with average=\"micro\" so far. Wish to know the right answer.</p>",
      "rawMarkdown": "Literally it says \"micro averaged\" so I've used f1_score with average=\"micro\" so far. Wish to know the right answer.",
      "votes": null
    },
    {
      "id": "990199",
      "postDate": "08/29/2020 12:31:13",
      "content": "<p>That's why I posted this.  The description reads:</p>\n<blockquote>\n  <p>Submissions will be evaluated based on their row-wise micro averaged F1 score.</p>\n</blockquote>\n<p>Row wise is <code>average='samples'</code> in sklearn.  Apparently the person who wrote the description did not know this.</p>\n<p>My CV is way higher wen I use f1 micro than when I use f1 samples.</p>",
      "rawMarkdown": "That's why I posted this.  The description reads:\n\n> Submissions will be evaluated based on their row-wise micro averaged F1 score.\n\nRow wise is `average='samples'` in sklearn.  Apparently the person who wrote the description did not know this.\n\nMy CV is way higher wen I use f1 micro than when I use f1 samples.",
      "votes": null
    },
    {
      "id": "990367",
      "postDate": "08/29/2020 15:02:26",
      "content": "<p>Makes me wonder what else have I been doing wrong this entire time, haha. </p>\n<p>Thanks for pointing this out <a href=\"https://www.kaggle.com/cpmpml\" target=\"_blank\">@cpmpml</a> :)</p>",
      "rawMarkdown": "Makes me wonder what else have I been doing wrong this entire time, haha. \n\nThanks for pointing this out @cpmpml :)",
      "votes": null
    },
    {
      "id": "991217",
      "postDate": "08/30/2020 08:31:12",
      "content": "<p><a href=\"https://www.kaggle.com/cpmpml\" target=\"_blank\">@cpmpml</a> yes, finally I can prove we deal with sklearn f1_score average=\"samples\" <br>\nthank you for pointing this out</p>",
      "rawMarkdown": "cpmpml yes, finally I can prove we deal with sklearn f1_score average=\"samples\" \nthank you for pointing this out",
      "votes": null
    },
    {
      "id": "991271",
      "postDate": "08/30/2020 09:25:38",
      "content": "<p>How do you prove it?</p>",
      "rawMarkdown": "How do you prove it?",
      "votes": null
    },
    {
      "id": "991272",
      "postDate": "08/30/2020 09:26:13",
      "content": "<p>You seem to do few things right given your LB score ;)</p>",
      "rawMarkdown": "You seem to do few things right given your LB score ;)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 987086,
      "author_name": "drhabib",
      "author_url": "",
      "post_date": "08/27/2020 01:59:04",
      "content": "<p>Yes =)</p>\n<p>here is excellent kernel: <a href=\"https://www.kaggle.com/shonenkov/sample-submission-using-custom-check\" target=\"_blank\">https://www.kaggle.com/shonenkov/sample-submission-using-custom-check</a> =) <br>\nand discussion here: <a href=\"https://www.kaggle.com/c/birdsong-recognition/discussion/173144\" target=\"_blank\">https://www.kaggle.com/c/birdsong-recognition/discussion/173144</a> </p>",
      "votes": null,
      "replies": [
        {
          "id": 987458,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "08/27/2020 09:24:26",
          "content": "<p>Thanks.  I updated my post.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 988854,
          "author_name": "sheriytm",
          "author_url": "",
          "post_date": "08/28/2020 10:31:49",
          "content": "<p><a href=\"https://www.kaggle.com/drhabib\" target=\"_blank\">@drhabib</a> you linked to the submission kernel, the metrics kernel is <a href=\"https://www.kaggle.com/shonenkov/competition-metrics\" target=\"_blank\">https://www.kaggle.com/shonenkov/competition-metrics</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 988998,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "08/28/2020 13:40:07",
          "content": "<p>Tx, I was wondering why he linked the submission kernel.  </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 988434,
      "author_name": "lewington",
      "author_url": "",
      "post_date": "08/28/2020 03:48:41",
      "content": "<p>For anyone using tensorflow I found </p>\n<pre><code>import tensorflow_addons as tfa\nnum_birds = 264\nmodel.compile(optimizer=optimizer, metrics=[\"accuracy\", tfa.metrics.F1Score(num_birds, average=\"micro\")])\n</code></pre>\n<p>helpful</p>",
      "votes": null,
      "replies": [
        {
          "id": 988782,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "08/28/2020 09:15:49",
          "content": "<p>f1 micro is not the competition metric. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 989868,
      "author_name": "yaroshevskiy",
      "author_url": "",
      "post_date": "08/29/2020 06:55:25",
      "content": "<p>Literally it says \"micro averaged\" so I've used f1_score with average=\"micro\" so far. Wish to know the right answer.</p>",
      "votes": null,
      "replies": [
        {
          "id": 990199,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "08/29/2020 12:31:13",
          "content": "<p>That's why I posted this.  The description reads:</p>\n<blockquote>\n  <p>Submissions will be evaluated based on their row-wise micro averaged F1 score.</p>\n</blockquote>\n<p>Row wise is <code>average='samples'</code> in sklearn.  Apparently the person who wrote the description did not know this.</p>\n<p>My CV is way higher wen I use f1 micro than when I use f1 samples.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 991217,
          "author_name": "yaroshevskiy",
          "author_url": "",
          "post_date": "08/30/2020 08:31:12",
          "content": "<p><a href=\"https://www.kaggle.com/cpmpml\" target=\"_blank\">@cpmpml</a> yes, finally I can prove we deal with sklearn f1_score average=\"samples\" <br>\nthank you for pointing this out</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 991271,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "08/30/2020 09:25:38",
          "content": "<p>How do you prove it?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 990367,
      "author_name": "ramarlina",
      "author_url": "",
      "post_date": "08/29/2020 15:02:26",
      "content": "<p>Makes me wonder what else have I been doing wrong this entire time, haha. </p>\n<p>Thanks for pointing this out <a href=\"https://www.kaggle.com/cpmpml\" target=\"_blank\">@cpmpml</a> :)</p>",
      "votes": null,
      "replies": [
        {
          "id": 991272,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "08/30/2020 09:26:13",
          "content": "<p>You seem to do few things right given your LB score ;)</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "987027": "It looks like the metric is what sklearn calls F1 with average=\"samples\":\n\n> 'samples':\n> \n>     Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from accuracy_score).\n\nAm I right?  Feedback welcome.\n\nEdit: this is not news, was discussed here already: https://www.kaggle.com/c/birdsong-recognition/discussion/173144",
    "987086": "Yes =)\n\nhere is excellent kernel: https://www.kaggle.com/shonenkov/sample-submission-using-custom-check =) \nand discussion here: https://www.kaggle.com/c/birdsong-recognition/discussion/173144",
    "987458": "Thanks.  I updated my post.",
    "988434": "For anyone using tensorflow I found \n\n\n```\nimport tensorflow_addons as tfa\nnum_birds = 264\nmodel.compile(optimizer=optimizer, metrics=[\"accuracy\", tfa.metrics.F1Score(num_birds, average=\"micro\")])\n```\n\nhelpful",
    "988782": "f1 micro is not the competition metric.",
    "988854": "drhabib you linked to the submission kernel, the metrics kernel is https://www.kaggle.com/shonenkov/competition-metrics",
    "988998": "Tx, I was wondering why he linked the submission kernel.",
    "989868": "Literally it says \"micro averaged\" so I've used f1_score with average=\"micro\" so far. Wish to know the right answer.",
    "990199": "That's why I posted this.  The description reads:\n\n> Submissions will be evaluated based on their row-wise micro averaged F1 score.\n\nRow wise is `average='samples'` in sklearn.  Apparently the person who wrote the description did not know this.\n\nMy CV is way higher wen I use f1 micro than when I use f1 samples.",
    "990367": "Makes me wonder what else have I been doing wrong this entire time, haha. \n\nThanks for pointing this out @cpmpml :)",
    "991217": "cpmpml yes, finally I can prove we deal with sklearn f1_score average=\"samples\" \nthank you for pointing this out",
    "991271": "How do you prove it?",
    "991272": "You seem to do few things right given your LB score ;)"
  },
  "source": "meta"
}