{
  "id": 505669,
  "title": "Suggestions to improve future BirdCLEFs",
  "url": "/competitions/birdclef-2024/discussion/505669",
  "author_name": "",
  "post_date": "2024-05-18T14:09:16.273752Z",
  "votes": 1,
  "comment_count": 8,
  "views": 0,
  "content": "<p>The goal of BirdCLEF is to produce better solutions that can be adapted and used by bird researchers. Here are a couple of suggestions which I think would produce better solutions:</p>\n<ol>\n<li><p>Publish 5% of the test data, including the ground truth labels. There's clearly something unusual about this year's data, since some solutions that perform better on the unlabelled soundscapes perform worse in the LB. We could produce better solutions if we understood that better.</p></li>\n<li><p>Show scores to three decimal places. For instance, it would be helpful to know if an experiment changed the score from .650 to .654.</p></li>\n</ol>\n<p>What do other competitors think?</p>",
  "messages": [
    {
      "id": "2822260",
      "postDate": "05/18/2024 14:09:16",
      "content": "<p>The goal of BirdCLEF is to produce better solutions that can be adapted and used by bird researchers. Here are a couple of suggestions which I think would produce better solutions:</p>\n<ol>\n<li><p>Publish 5% of the test data, including the ground truth labels. There's clearly something unusual about this year's data, since some solutions that perform better on the unlabelled soundscapes perform worse in the LB. We could produce better solutions if we understood that better.</p></li>\n<li><p>Show scores to three decimal places. For instance, it would be helpful to know if an experiment changed the score from .650 to .654.</p></li>\n</ol>\n<p>What do other competitors think?</p>",
      "rawMarkdown": "The goal of BirdCLEF is to produce better solutions that can be adapted and used by bird researchers. Here are a couple of suggestions which I think would produce better solutions:\n\n1. Publish 5% of the test data, including the ground truth labels. There's clearly something unusual about this year's data, since some solutions that perform better on the unlabelled soundscapes perform worse in the LB. We could produce better solutions if we understood that better.\n\n2. Show scores to three decimal places. For instance, it would be helpful to know if an experiment changed the score from .650 to .654.\n\nWhat do other competitors think?",
      "votes": null
    },
    {
      "id": "2822311",
      "postDate": "05/18/2024 14:27:56",
      "content": "<p>This is a great idea and should have been in this competition! However, the second suggestion could be a bit bad since in this competition LB only shows 2 decimal points to prevent LB probing.</p>",
      "rawMarkdown": "This is a great idea and should have been in this competition! However, the second suggestion could be a bit bad since in this competition LB only shows 2 decimal points to prevent LB probing.",
      "votes": null
    },
    {
      "id": "2822406",
      "postDate": "05/18/2024 15:19:08",
      "content": "<p>Yes, I guess that's a valid reason to restrict it to 2 decimal places. </p>",
      "rawMarkdown": "Yes, I guess that's a valid reason to restrict it to 2 decimal places.",
      "votes": null
    },
    {
      "id": "2822575",
      "postDate": "05/18/2024 17:10:21",
      "content": "<blockquote>\n  <p>since some solutions that perform better on the unlabelled soundscapes</p>\n</blockquote>\n<p>How do you know how a solution performs on unlabelled soundscapes?</p>",
      "rawMarkdown": "> since some solutions that perform better on the unlabelled soundscapes\n\nHow do you know how a solution performs on unlabelled soundscapes?",
      "votes": null
    },
    {
      "id": "2822598",
      "postDate": "05/18/2024 17:16:35",
      "content": "<p>I've tried to use the unlabeled soundscapes and it's only lowered my LB so I'm curious about this as well.</p>",
      "rawMarkdown": "I've tried to use the unlabeled soundscapes and it's only lowered my LB so I'm curious about this as well.",
      "votes": null
    },
    {
      "id": "2822622",
      "postDate": "05/18/2024 17:22:03",
      "content": "<p>I picked a bunch of unlabelled soundscapes and labelled them, using embeddings and visual checks to confirm the species matches.</p>",
      "rawMarkdown": "I picked a bunch of unlabelled soundscapes and labelled them, using embeddings and visual checks to confirm the species matches.",
      "votes": null
    },
    {
      "id": "2822646",
      "postDate": "05/18/2024 17:31:50",
      "content": "<p>I think it's more about my labeling model not being very good. I'm using my highest ranking one, but I imagine that higher ranking models would improve the labeling.</p>",
      "rawMarkdown": "I think it's more about my labeling model not being very good. I'm using my highest ranking one, but I imagine that higher ranking models would improve the labeling.",
      "votes": null
    },
    {
      "id": "2822773",
      "postDate": "05/18/2024 18:41:48",
      "content": "<blockquote>\n  <p>I picked a bunch of unlabelled soundscapes and labelled them</p>\n</blockquote>\n<p>I would not know how to label correctly any of these recordings. Congrats for being able to do so.</p>",
      "rawMarkdown": "> I picked a bunch of unlabelled soundscapes and labelled them\n\nI would not know how to label correctly any of these recordings. Congrats for being able to do so.",
      "votes": null
    },
    {
      "id": "2822790",
      "postDate": "05/18/2024 18:59:42",
      "content": "<p>I put all the training data in a database, added embeddings and then searched it, which works well if you’re not doing too many. You can see the tools/search.py script at <a href=\"https://github.com/jhuus/HawkEars\" target=\"_blank\">https://github.com/jhuus/HawkEars</a></p>",
      "rawMarkdown": "I put all the training data in a database, added embeddings and then searched it, which works well if you’re not doing too many. You can see the tools/search.py script at https://github.com/jhuus/HawkEars",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2822311,
      "author_name": "max1mum",
      "author_url": "",
      "post_date": "05/18/2024 14:27:56",
      "content": "<p>This is a great idea and should have been in this competition! However, the second suggestion could be a bit bad since in this competition LB only shows 2 decimal points to prevent LB probing.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2822406,
          "author_name": "janhuus",
          "author_url": "",
          "post_date": "05/18/2024 15:19:08",
          "content": "<p>Yes, I guess that's a valid reason to restrict it to 2 decimal places. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2822575,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "05/18/2024 17:10:21",
      "content": "<blockquote>\n  <p>since some solutions that perform better on the unlabelled soundscapes</p>\n</blockquote>\n<p>How do you know how a solution performs on unlabelled soundscapes?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2822598,
          "author_name": "willrice",
          "author_url": "",
          "post_date": "05/18/2024 17:16:35",
          "content": "<p>I've tried to use the unlabeled soundscapes and it's only lowered my LB so I'm curious about this as well.</p>",
          "votes": null,
          "replies": [
            {
              "id": 2822622,
              "author_name": "janhuus",
              "author_url": "",
              "post_date": "05/18/2024 17:22:03",
              "content": "<p>I picked a bunch of unlabelled soundscapes and labelled them, using embeddings and visual checks to confirm the species matches.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2822646,
                  "author_name": "willrice",
                  "author_url": "",
                  "post_date": "05/18/2024 17:31:50",
                  "content": "<p>I think it's more about my labeling model not being very good. I'm using my highest ranking one, but I imagine that higher ranking models would improve the labeling.</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 2822773,
                      "author_name": "cpmpml",
                      "author_url": "",
                      "post_date": "05/18/2024 18:41:48",
                      "content": "<blockquote>\n  <p>I picked a bunch of unlabelled soundscapes and labelled them</p>\n</blockquote>\n<p>I would not know how to label correctly any of these recordings. Congrats for being able to do so.</p>",
                      "votes": null,
                      "replies": [
                        {
                          "id": 2822790,
                          "author_name": "janhuus",
                          "author_url": "",
                          "post_date": "05/18/2024 18:59:42",
                          "content": "<p>I put all the training data in a database, added embeddings and then searched it, which works well if you’re not doing too many. You can see the tools/search.py script at <a href=\"https://github.com/jhuus/HawkEars\" target=\"_blank\">https://github.com/jhuus/HawkEars</a></p>",
                          "votes": null,
                          "replies": []
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2822260": "The goal of BirdCLEF is to produce better solutions that can be adapted and used by bird researchers. Here are a couple of suggestions which I think would produce better solutions:\n\n1. Publish 5% of the test data, including the ground truth labels. There's clearly something unusual about this year's data, since some solutions that perform better on the unlabelled soundscapes perform worse in the LB. We could produce better solutions if we understood that better.\n\n2. Show scores to three decimal places. For instance, it would be helpful to know if an experiment changed the score from .650 to .654.\n\nWhat do other competitors think?",
    "2822311": "This is a great idea and should have been in this competition! However, the second suggestion could be a bit bad since in this competition LB only shows 2 decimal points to prevent LB probing.",
    "2822406": "Yes, I guess that's a valid reason to restrict it to 2 decimal places.",
    "2822575": "> since some solutions that perform better on the unlabelled soundscapes\n\nHow do you know how a solution performs on unlabelled soundscapes?",
    "2822598": "I've tried to use the unlabeled soundscapes and it's only lowered my LB so I'm curious about this as well.",
    "2822622": "I picked a bunch of unlabelled soundscapes and labelled them, using embeddings and visual checks to confirm the species matches.",
    "2822646": "I think it's more about my labeling model not being very good. I'm using my highest ranking one, but I imagine that higher ranking models would improve the labeling.",
    "2822773": "> I picked a bunch of unlabelled soundscapes and labelled them\n\nI would not know how to label correctly any of these recordings. Congrats for being able to do so.",
    "2822790": "I put all the training data in a database, added embeddings and then searched it, which works well if you’re not doing too many. You can see the tools/search.py script at https://github.com/jhuus/HawkEars"
  },
  "source": "meta"
}