tf_efficientnet_b7 (single model)
\nn_mels: 256
\nCV (F1@0.30): 0.64
\nLB : 0.58
tf_efficientnet_b7 (single model)
\nn_mels: 256
\nCV (F1@0.30): 0.64
\nLB : 0.58
tf_efficientnet_b0_ns (single model, 5 folds but only one trained and scored yet)
\nCV: 0.63
\nLB: 0.54
Need to fix my CV strategy……
", "rawMarkdown": "tf_efficientnet_b0_ns (single model, 5 folds but only one trained and scored yet)\nCV: 0.63\nLB: 0.54\n\nNeed to fix my CV strategy......", "votes": null }, { "id": "1720183", "postDate": "03/12/2022 14:37:10", "content": "Hi @kfk42kfk what is F1@0.30 ? If i know then i can share my CV
", "rawMarkdown": "Hi @kfk42kfk what is F1@0.30 ? If i know then i can share my CV", "votes": null }, { "id": "1720856", "postDate": "03/13/2022 07:52:01", "content": "I calculated mine using this code snippet from this notebook: https://www.kaggle.com/kaerunantoka/birdclef2022-n001-training
\nclass MetricMeter(object):\n def __init__(self):\n self.reset()\n\n def reset(self):\n self.y_true = []\n self.y_pred = []\n\n def update(self, y_true, y_pred):\n self.y_true.extend(y_true.cpu().detach().numpy().tolist())\n # self.y_pred.extend(torch.sigmoid(y_pred).cpu().detach().numpy().tolist())\n # self.y_pred.extend(y_pred[\"clipwise_output\"].max(axis=1)[0].cpu().detach().numpy().tolist())\n self.y_pred.extend(y_pred[\"clipwise_output\"].cpu().detach().numpy().tolist())\n\n @property\n def avg(self):\n self.f1_03 = metrics.f1_score(np.array(self.y_true), np.array(self.y_pred) > 0.3, average=\"micro\")\n self.f1_05 = metrics.f1_score(np.array(self.y_true), np.array(self.y_pred) > 0.5, average=\"micro\")\n\n return {\n \"f1_at_03\" : self.f1_03,\n \"f1_at_05\" : self.f1_05,\n }\nWhy would you use \"micro\" ? Isn't the metric \"macro\"?
\n\n", "rawMarkdown": "Why would you use \"micro\" ? Isn't the metric \"macro\"?\n\n> Submissions are evaluated on a metric that is most similar to the **macro F1 score**. Given the amount of audio data used in this competition it wasn't feasible to label every single species found in every soundscape. Instead only a subset of species are actually scored for any given audio file. After dropping all of the un-scored rows we technically run a weighted classification accuracy with the weights set such that all of the species are assigned the same total weight and the true negatives and true positives for each species have the same weight. The extra complexity exists purely to allow us to have a great deal of control over which birds are scored for a given soundscape. For offline cross validation purposes, the macro F1 is the closest analogue to the actual metric.", "votes": null }, { "id": "1782219", "postDate": "05/09/2022 10:57:03", "content": "Submissions are evaluated on a metric that is most similar to the macro F1 score. Given the amount of audio data used in this competition it wasn't feasible to label every single species found in every soundscape. Instead only a subset of species are actually scored for any given audio file. After dropping all of the un-scored rows we technically run a weighted classification accuracy with the weights set such that all of the species are assigned the same total weight and the true negatives and true positives for each species have the same weight. The extra complexity exists purely to allow us to have a great deal of control over which birds are scored for a given soundscape. For offline cross validation purposes, the macro F1 is the closest analogue to the actual metric.
\n
Thanks for sharing you CV and LB scores.
\nThere seems to be a big gap between the CV and LB. Do you solve the gap?
Yes, my CV and LB also have a big gap. In fact, I'm not sure if I should trust CV or LB when submitting the final 2 results.
\nAs I said in the discussion, there are some reasons for each view, so I'm not sure how making a decision.
", "rawMarkdown": "Yes, my CV and LB also have a big gap. In fact, I'm not sure if I should trust CV or LB when submitting the final 2 results.\n\nAs I said in the [discussion](https://www.kaggle.com/competitions/birdclef-2022/discussion/324304#1784539), there are some reasons for each view, so I'm not sure how making a decision.", "votes": null } ], "comments": [ { "id": 1717991, "author_name": "hinepo", "author_url": "", "post_date": "03/10/2022 12:12:57", "content": "tf_efficientnet_b0_ns (single model, 5 folds but only one trained and scored yet)
\nCV: 0.63
\nLB: 0.54
Need to fix my CV strategy……
", "votes": null, "replies": [ { "id": 1782219, "author_name": "zhoumichael", "author_url": "", "post_date": "05/09/2022 10:57:03", "content": "Thanks for sharing you CV and LB scores.
\nThere seems to be a big gap between the CV and LB. Do you solve the gap?
Hi @kfk42kfk what is F1@0.30 ? If i know then i can share my CV
", "votes": null, "replies": [ { "id": 1720856, "author_name": "kfk42kfk", "author_url": "", "post_date": "03/13/2022 07:52:01", "content": "I calculated mine using this code snippet from this notebook: https://www.kaggle.com/kaerunantoka/birdclef2022-n001-training
\nclass MetricMeter(object):\n def __init__(self):\n self.reset()\n\n def reset(self):\n self.y_true = []\n self.y_pred = []\n\n def update(self, y_true, y_pred):\n self.y_true.extend(y_true.cpu().detach().numpy().tolist())\n # self.y_pred.extend(torch.sigmoid(y_pred).cpu().detach().numpy().tolist())\n # self.y_pred.extend(y_pred[\"clipwise_output\"].max(axis=1)[0].cpu().detach().numpy().tolist())\n self.y_pred.extend(y_pred[\"clipwise_output\"].cpu().detach().numpy().tolist())\n\n @property\n def avg(self):\n self.f1_03 = metrics.f1_score(np.array(self.y_true), np.array(self.y_pred) > 0.3, average=\"micro\")\n self.f1_05 = metrics.f1_score(np.array(self.y_true), np.array(self.y_pred) > 0.5, average=\"micro\")\n\n return {\n \"f1_at_03\" : self.f1_03,\n \"f1_at_05\" : self.f1_05,\n }\nWhy would you use \"micro\" ? Isn't the metric \"macro\"?
\n\n", "votes": null, "replies": [] } ] }, { "id": 1786552, "author_name": "jiedengsc", "author_url": "", "post_date": "05/13/2022 02:22:39", "content": "Submissions are evaluated on a metric that is most similar to the macro F1 score. Given the amount of audio data used in this competition it wasn't feasible to label every single species found in every soundscape. Instead only a subset of species are actually scored for any given audio file. After dropping all of the un-scored rows we technically run a weighted classification accuracy with the weights set such that all of the species are assigned the same total weight and the true negatives and true positives for each species have the same weight. The extra complexity exists purely to allow us to have a great deal of control over which birds are scored for a given soundscape. For offline cross validation purposes, the macro F1 is the closest analogue to the actual metric.
\n
Yes, my CV and LB also have a big gap. In fact, I'm not sure if I should trust CV or LB when submitting the final 2 results.
\nAs I said in the discussion, there are some reasons for each view, so I'm not sure how making a decision.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1717774": "tf_efficientnet_b7 (single model)\nn_mels: 256\nCV (F1@0.30): 0.64\nLB : 0.58", "1717991": "tf_efficientnet_b0_ns (single model, 5 folds but only one trained and scored yet)\nCV: 0.63\nLB: 0.54\n\nNeed to fix my CV strategy......", "1720183": "Hi @kfk42kfk what is F1@0.30 ? If i know then i can share my CV", "1720856": "I calculated mine using this code snippet from this notebook: https://www.kaggle.com/kaerunantoka/birdclef2022-n001-training\n```\nclass MetricMeter(object):\n def __init__(self):\n self.reset()\n \n def reset(self):\n self.y_true = []\n self.y_pred = []\n \n def update(self, y_true, y_pred):\n self.y_true.extend(y_true.cpu().detach().numpy().tolist())\n # self.y_pred.extend(torch.sigmoid(y_pred).cpu().detach().numpy().tolist())\n # self.y_pred.extend(y_pred[\"clipwise_output\"].max(axis=1)[0].cpu().detach().numpy().tolist())\n self.y_pred.extend(y_pred[\"clipwise_output\"].cpu().detach().numpy().tolist())\n\n @property\n def avg(self):\n self.f1_03 = metrics.f1_score(np.array(self.y_true), np.array(self.y_pred) > 0.3, average=\"micro\")\n self.f1_05 = metrics.f1_score(np.array(self.y_true), np.array(self.y_pred) > 0.5, average=\"micro\")\n \n return {\n \"f1_at_03\" : self.f1_03,\n \"f1_at_05\" : self.f1_05,\n }\n```", "1723765": "Why would you use \"micro\" ? Isn't the metric \"macro\"?\n\n> Submissions are evaluated on a metric that is most similar to the **macro F1 score**. Given the amount of audio data used in this competition it wasn't feasible to label every single species found in every soundscape. Instead only a subset of species are actually scored for any given audio file. After dropping all of the un-scored rows we technically run a weighted classification accuracy with the weights set such that all of the species are assigned the same total weight and the true negatives and true positives for each species have the same weight. The extra complexity exists purely to allow us to have a great deal of control over which birds are scored for a given soundscape. For offline cross validation purposes, the macro F1 is the closest analogue to the actual metric.", "1782219": "Thanks for sharing you CV and LB scores.\nThere seems to be a big gap between the CV and LB. Do you solve the gap?", "1786552": "Yes, my CV and LB also have a big gap. In fact, I'm not sure if I should trust CV or LB when submitting the final 2 results.\n\nAs I said in the [discussion](https://www.kaggle.com/competitions/birdclef-2022/discussion/324304#1784539), there are some reasons for each view, so I'm not sure how making a decision." }, "source": "meta" }