{ "id": 232645, "title": "Annotations for the training soundscapes: questions (and quibbles)", "url": "/competitions/birdclef-2021/discussion/232645", "author_name": "", "post_date": "2021-04-14T17:01:59.244404600Z", "votes": 6, "comment_count": 2, "views": 0, "content": "

I’ve been looking in more detail at the training soundscapes, and I have some questions about the “answer” annotations in the train_soundscape_labels.csv file. There’s inherently a some gray area in making annotations, whether a human or computer is doing it, and I hope to understand a bit more about the existing annotations so the two approaches have the best chance at meshing.

\n

Threshold of detectability?

\n

Is there an objective threshold for determining when a bird sound is detectable vs. “nocall”? There are a variety of places that have a “nocall” label; however, my ear can detect what is, I’m pretty sure, some kind of bird. Some of these might not be definitively identifiable.

\n

One specific example: Between the 1:50 and 2:00 marks (end times 115 and 120) of 10534_SSW_20170429, there’s a faint but pretty definite Black-capped Chickadee, plus some downslurs of what could be a Northern Cardinal, and possibly even a couple more species that I can’t quite distinguish.

\n

On the other hand, there are some barely discernible sounds with a species listing.
\nFor an example this, see 54955_SSW_20170617 at the 6:35 mark (segment end 395); there’s supposed to be a Gray Catbird, and I think it is right at the end of the segment and spilling into the next. I think I can convince myself that I can just hear it, but it is barely a smudge in my spectrogram view.

\n

Sound that spans two 5-second windows

\n
\n

For each row_id/time window, you need to provide a space delimited list of the set of unique birds that made a call beginning or ending in that time window.

\n
\n

This implies that if a sound spans the boundary between two five second windows, then it counts in both. Is that true even if it is a very brief bit of a call that spills over? In the above example (54955_SSW_20170617), the hard-to-see “mew” call of the catbird (grycat) appears to span the 5-second mark, but it is only listed in the first time period. (I thought I had found a better example, but I can’t find my notes on it right now.)

\n

More examples
\nHere are a few more specific examples for which I would appreciate some explanation. I think some of these might be errors or omissions.

\n

File: 14473_SSW_20170701
\nEnd time: 590
\nLabel: norwat
\nComment: The last 30 seconds of this soundscape includes consistent “mewing” of a catbird (grycat), but if there's actually a Northern Waterthrush, I can't find it. Should this be grycat?

\n

File: 11254_COR_20190904
\nEnd time: 10
\nLabel: nocall
\nComment: There’s bird sound, and it sure sounds like the same rubwre1 that is singing in the segments before and after. (Same file, time ending at 30 sec also seems to be a bird, presumably the same rubwre1, that is listed as nocall, and it seems to keep going to about the 35 second mark.)

\n

File: 11254_COR_20190904
\nEnd time: 195
\nLabel: nocall
\nComment: There’s a sharp chip that I’m pretty sure is a bird (if I had to guess, I’d say flycatcher), but it is listed as nocall here. It repeats within segments ending at 215, 220, 225, 230, 235, and 245. Through this section, there’s also something intermittently doing a really awesome song with a clear whistle, and it is not at all jay-like. (The Brown Jay is the only other species listed in this broader window of time).

\n

This isn’t a comprehensive list, just some things that came up from manually poking around at the data; I certainly haven’t gone over everything in great detail. But the more I look at the spectrograms, the more questions seem to come up.

\n

Thanks in advance.

", "messages": [ { "id": "1273805", "postDate": "04/14/2021 17:01:59", "content": "

I’ve been looking in more detail at the training soundscapes, and I have some questions about the “answer” annotations in the train_soundscape_labels.csv file. There’s inherently a some gray area in making annotations, whether a human or computer is doing it, and I hope to understand a bit more about the existing annotations so the two approaches have the best chance at meshing.

\n

Threshold of detectability?

\n

Is there an objective threshold for determining when a bird sound is detectable vs. “nocall”? There are a variety of places that have a “nocall” label; however, my ear can detect what is, I’m pretty sure, some kind of bird. Some of these might not be definitively identifiable.

\n

One specific example: Between the 1:50 and 2:00 marks (end times 115 and 120) of 10534_SSW_20170429, there’s a faint but pretty definite Black-capped Chickadee, plus some downslurs of what could be a Northern Cardinal, and possibly even a couple more species that I can’t quite distinguish.

\n

On the other hand, there are some barely discernible sounds with a species listing.
\nFor an example this, see 54955_SSW_20170617 at the 6:35 mark (segment end 395); there’s supposed to be a Gray Catbird, and I think it is right at the end of the segment and spilling into the next. I think I can convince myself that I can just hear it, but it is barely a smudge in my spectrogram view.

\n

Sound that spans two 5-second windows

\n
\n

For each row_id/time window, you need to provide a space delimited list of the set of unique birds that made a call beginning or ending in that time window.

\n
\n

This implies that if a sound spans the boundary between two five second windows, then it counts in both. Is that true even if it is a very brief bit of a call that spills over? In the above example (54955_SSW_20170617), the hard-to-see “mew” call of the catbird (grycat) appears to span the 5-second mark, but it is only listed in the first time period. (I thought I had found a better example, but I can’t find my notes on it right now.)

\n

More examples
\nHere are a few more specific examples for which I would appreciate some explanation. I think some of these might be errors or omissions.

\n

File: 14473_SSW_20170701
\nEnd time: 590
\nLabel: norwat
\nComment: The last 30 seconds of this soundscape includes consistent “mewing” of a catbird (grycat), but if there's actually a Northern Waterthrush, I can't find it. Should this be grycat?

\n

File: 11254_COR_20190904
\nEnd time: 10
\nLabel: nocall
\nComment: There’s bird sound, and it sure sounds like the same rubwre1 that is singing in the segments before and after. (Same file, time ending at 30 sec also seems to be a bird, presumably the same rubwre1, that is listed as nocall, and it seems to keep going to about the 35 second mark.)

\n

File: 11254_COR_20190904
\nEnd time: 195
\nLabel: nocall
\nComment: There’s a sharp chip that I’m pretty sure is a bird (if I had to guess, I’d say flycatcher), but it is listed as nocall here. It repeats within segments ending at 215, 220, 225, 230, 235, and 245. Through this section, there’s also something intermittently doing a really awesome song with a clear whistle, and it is not at all jay-like. (The Brown Jay is the only other species listed in this broader window of time).

\n

This isn’t a comprehensive list, just some things that came up from manually poking around at the data; I certainly haven’t gone over everything in great detail. But the more I look at the spectrograms, the more questions seem to come up.

\n

Thanks in advance.

", "rawMarkdown": "I’ve been looking in more detail at the training soundscapes, and I have some questions about the “answer” annotations in the train_soundscape_labels.csv file. There’s inherently a some gray area in making annotations, whether a human or computer is doing it, and I hope to understand a bit more about the existing annotations so the two approaches have the best chance at meshing. \n\n**Threshold of detectability?** \n\nIs there an objective threshold for determining when a bird sound is detectable vs. “nocall”? There are a variety of places that have a “nocall” label; however, my ear can detect what is, I’m pretty sure, some kind of bird. Some of these might not be definitively identifiable.\n\nOne specific example: Between the 1:50 and 2:00 marks (end times 115 and 120) of 10534_SSW_20170429, there’s a faint but pretty definite Black-capped Chickadee, plus some downslurs of what could be a Northern Cardinal, and possibly even a couple more species that I can’t quite distinguish. \n\nOn the other hand, there are some barely discernible sounds with a species listing. \nFor an example this, see 54955_SSW_20170617 at the 6:35 mark (segment end 395); there’s supposed to be a Gray Catbird, and I think it is right at the end of the segment and spilling into the next. I *think* I can convince myself that I can just hear it, but it is barely a smudge in my spectrogram view. \n\n**Sound that spans two 5-second windows**\n\n> For each row_id/time window, you need to provide a space delimited list of the set of unique birds that made a call beginning or ending in that time window. \n\nThis implies that if a sound spans the boundary between two five second windows, then it counts in both. Is that true even if it is a very brief bit of a call that spills over? In the above example (54955_SSW_20170617), the hard-to-see “mew” call of the catbird (grycat) appears to span the 5-second mark, but it is only listed in the first time period. (I thought I had found a better example, but I can’t find my notes on it right now.) \n\n**More examples** \nHere are a few more specific examples for which I would appreciate some explanation. I think some of these might be errors or omissions. \n\nFile: 14473_SSW_20170701\nEnd time: 590\nLabel: norwat\nComment: The last 30 seconds of this soundscape includes consistent “mewing” of a catbird (grycat), but if there's actually a Northern Waterthrush, I can't find it. Should this be grycat? \n\nFile: 11254_COR_20190904\nEnd time: 10\nLabel: nocall\nComment: There’s bird sound, and it sure sounds like the same rubwre1 that is singing in the segments before and after. (Same file, time ending at 30 sec also seems to be a bird, presumably the same rubwre1, that is listed as nocall, and it seems to keep going to about the 35 second mark.)\n\nFile: 11254_COR_20190904\nEnd time: 195\nLabel: nocall\nComment: There’s a sharp chip that I’m pretty sure is a bird (if I had to guess, I’d say flycatcher), but it is listed as nocall here. It repeats within segments ending at 215, 220, 225, 230, 235, and 245. Through this section, there’s also something intermittently doing a really awesome song with a clear whistle, and it is not at all jay-like. (The Brown Jay is the only other species listed in this broader window of time). \n\nThis isn’t a comprehensive list, just some things that came up from manually poking around at the data; I certainly haven’t gone over everything in great detail. But the more I look at the spectrograms, the more questions seem to come up. \n\nThanks in advance.", "votes": null }, { "id": "1273825", "postDate": "04/14/2021 17:22:45", "content": "
\n

There are a variety of places that have a “nocall” label; however, my ear can detect what is, I’m pretty sure, some kind of bird

\n
\n

I don't know for the specifics of what you refer too, and you may be perfectly right, but I think that 'nocall' means 'no call from one of the primary label species from train short audio'. A call from another species would be labelled as no call IMHO.

\n

If my interpretation is wrong then I welcome feedback and further explanations.

", "rawMarkdown": "> There are a variety of places that have a “nocall” label; however, my ear can detect what is, I’m pretty sure, some kind of bird\n\nI don't know for the specifics of what you refer too, and you may be perfectly right, but I think that 'nocall' means 'no call from one of the primary label species from train short audio'. A call from another species would be labelled as no call IMHO.\n\nIf my interpretation is wrong then I welcome feedback and further explanations.", "votes": null }, { "id": "1274546", "postDate": "04/15/2021 11:54:16", "content": "

Manually annotating soundscapes is a demanding task. We asked our expert annotators to label every call they could identify. Yet, we cannot guarantee that every label is 100% correct, some might be missing. However, all annotators were experts in their field and knew local bird species well. They didn't annotate in 5-second segments, but in bounding boxes around calls. We tried to employ a conversion process (bounding boxes to segment labels) that provides meaningful lists of species for 5-second segments based on the annotations. Yet, there is no such thing a \"Gold Standard\" for bioacoustic datasets, this year's dataset is no exception. As a result, some \"nocall\" labels might be the result of either the annotation or segmentation process.

", "rawMarkdown": "Manually annotating soundscapes is a demanding task. We asked our expert annotators to label every call they could identify. Yet, we cannot guarantee that every label is 100% correct, some might be missing. However, all annotators were experts in their field and knew local bird species well. They didn't annotate in 5-second segments, but in bounding boxes around calls. We tried to employ a conversion process (bounding boxes to segment labels) that provides meaningful lists of species for 5-second segments based on the annotations. Yet, there is no such thing a \"Gold Standard\" for bioacoustic datasets, this year's dataset is no exception. As a result, some \"nocall\" labels might be the result of either the annotation or segmentation process.", "votes": null } ], "comments": [ { "id": 1273825, "author_name": "cpmpml", "author_url": "", "post_date": "04/14/2021 17:22:45", "content": "
\n

There are a variety of places that have a “nocall” label; however, my ear can detect what is, I’m pretty sure, some kind of bird

\n
\n

I don't know for the specifics of what you refer too, and you may be perfectly right, but I think that 'nocall' means 'no call from one of the primary label species from train short audio'. A call from another species would be labelled as no call IMHO.

\n

If my interpretation is wrong then I welcome feedback and further explanations.

", "votes": null, "replies": [] }, { "id": 1274546, "author_name": "stefankahl", "author_url": "", "post_date": "04/15/2021 11:54:16", "content": "

Manually annotating soundscapes is a demanding task. We asked our expert annotators to label every call they could identify. Yet, we cannot guarantee that every label is 100% correct, some might be missing. However, all annotators were experts in their field and knew local bird species well. They didn't annotate in 5-second segments, but in bounding boxes around calls. We tried to employ a conversion process (bounding boxes to segment labels) that provides meaningful lists of species for 5-second segments based on the annotations. Yet, there is no such thing a \"Gold Standard\" for bioacoustic datasets, this year's dataset is no exception. As a result, some \"nocall\" labels might be the result of either the annotation or segmentation process.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1273805": "I’ve been looking in more detail at the training soundscapes, and I have some questions about the “answer” annotations in the train_soundscape_labels.csv file. There’s inherently a some gray area in making annotations, whether a human or computer is doing it, and I hope to understand a bit more about the existing annotations so the two approaches have the best chance at meshing. \n\n**Threshold of detectability?** \n\nIs there an objective threshold for determining when a bird sound is detectable vs. “nocall”? There are a variety of places that have a “nocall” label; however, my ear can detect what is, I’m pretty sure, some kind of bird. Some of these might not be definitively identifiable.\n\nOne specific example: Between the 1:50 and 2:00 marks (end times 115 and 120) of 10534_SSW_20170429, there’s a faint but pretty definite Black-capped Chickadee, plus some downslurs of what could be a Northern Cardinal, and possibly even a couple more species that I can’t quite distinguish. \n\nOn the other hand, there are some barely discernible sounds with a species listing. \nFor an example this, see 54955_SSW_20170617 at the 6:35 mark (segment end 395); there’s supposed to be a Gray Catbird, and I think it is right at the end of the segment and spilling into the next. I *think* I can convince myself that I can just hear it, but it is barely a smudge in my spectrogram view. \n\n**Sound that spans two 5-second windows**\n\n> For each row_id/time window, you need to provide a space delimited list of the set of unique birds that made a call beginning or ending in that time window. \n\nThis implies that if a sound spans the boundary between two five second windows, then it counts in both. Is that true even if it is a very brief bit of a call that spills over? In the above example (54955_SSW_20170617), the hard-to-see “mew” call of the catbird (grycat) appears to span the 5-second mark, but it is only listed in the first time period. (I thought I had found a better example, but I can’t find my notes on it right now.) \n\n**More examples** \nHere are a few more specific examples for which I would appreciate some explanation. I think some of these might be errors or omissions. \n\nFile: 14473_SSW_20170701\nEnd time: 590\nLabel: norwat\nComment: The last 30 seconds of this soundscape includes consistent “mewing” of a catbird (grycat), but if there's actually a Northern Waterthrush, I can't find it. Should this be grycat? \n\nFile: 11254_COR_20190904\nEnd time: 10\nLabel: nocall\nComment: There’s bird sound, and it sure sounds like the same rubwre1 that is singing in the segments before and after. (Same file, time ending at 30 sec also seems to be a bird, presumably the same rubwre1, that is listed as nocall, and it seems to keep going to about the 35 second mark.)\n\nFile: 11254_COR_20190904\nEnd time: 195\nLabel: nocall\nComment: There’s a sharp chip that I’m pretty sure is a bird (if I had to guess, I’d say flycatcher), but it is listed as nocall here. It repeats within segments ending at 215, 220, 225, 230, 235, and 245. Through this section, there’s also something intermittently doing a really awesome song with a clear whistle, and it is not at all jay-like. (The Brown Jay is the only other species listed in this broader window of time). \n\nThis isn’t a comprehensive list, just some things that came up from manually poking around at the data; I certainly haven’t gone over everything in great detail. But the more I look at the spectrograms, the more questions seem to come up. \n\nThanks in advance.", "1273825": "> There are a variety of places that have a “nocall” label; however, my ear can detect what is, I’m pretty sure, some kind of bird\n\nI don't know for the specifics of what you refer too, and you may be perfectly right, but I think that 'nocall' means 'no call from one of the primary label species from train short audio'. A call from another species would be labelled as no call IMHO.\n\nIf my interpretation is wrong then I welcome feedback and further explanations.", "1274546": "Manually annotating soundscapes is a demanding task. We asked our expert annotators to label every call they could identify. Yet, we cannot guarantee that every label is 100% correct, some might be missing. However, all annotators were experts in their field and knew local bird species well. They didn't annotate in 5-second segments, but in bounding boxes around calls. We tried to employ a conversion process (bounding boxes to segment labels) that provides meaningful lists of species for 5-second segments based on the annotations. Yet, there is no such thing a \"Gold Standard\" for bioacoustic datasets, this year's dataset is no exception. As a result, some \"nocall\" labels might be the result of either the annotation or segmentation process." }, "source": "meta" }