{ "id": 239911, "title": "Hand labeling is small impact", "url": "/competitions/birdclef-2021/discussion/239911", "author_name": "", "post_date": "2021-05-18T03:19:48.176491100Z", "votes": 11, "comment_count": 28, "views": 0, "content": "

Hi Kaggler,

\n

I know that hand labeling has been effective in the past two sound competition.

\n\n

In this competition, I tried hand labeling for a one month.
\nI made 4300 hand labeling data, but it is small impact(LB 0.04 improvement).

\n

Even if you try hand labeling, you cannot expect big impact.

", "messages": [ { "id": "1312401", "postDate": "05/18/2021 03:19:48", "content": "

Hi Kaggler,

\n

I know that hand labeling has been effective in the past two sound competition.

\n\n

In this competition, I tried hand labeling for a one month.
\nI made 4300 hand labeling data, but it is small impact(LB 0.04 improvement).

\n

Even if you try hand labeling, you cannot expect big impact.

", "rawMarkdown": "Hi Kaggler,\n\nI know that hand labeling has been effective in the past two sound competition.\n\n+ Rainforest Connection Species Audio Detection\n[1st place solution](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220563)\n[3rd place solution](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220522)\n[7th place solution](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220443)\n[8th place solution](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220803)\n\n+ Cornell Birdcall Identification\n[2nd place solution](https://www.kaggle.com/c/birdsong-recognition/discussion/183269)\n\nIn this competition, I tried hand labeling for a one month.\nI made **4300 hand labeling data**, but it is small impact(LB 0.04 improvement).\n\nEven if you try hand labeling, you cannot expect big impact.", "votes": null }, { "id": "1312561", "postDate": "05/18/2021 05:49:35", "content": "

Thank you for sharing!
\nLB 0.04 improvement is a big impact on me…..

", "rawMarkdown": "Thank you for sharing!\nLB 0.04 improvement is a big impact on me.....", "votes": null }, { "id": "1312573", "postDate": "05/18/2021 05:57:55", "content": "
\n

I made 4300 hand labeling data, but it is small impact(LB 0.04 improvement).

\n
\n

4300 individual examples or 4300 annotated sound files?

", "rawMarkdown": "> I made 4300 hand labeling data, but it is small impact(LB 0.04 improvement).\n\n4300 individual examples or 4300 annotated sound files?", "votes": null }, { "id": "1312699", "postDate": "05/18/2021 07:39:56", "content": "

By the way, does hand labeling break the rule of Kaggle competitions? As I have understood from Rules 7.C (However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants.), hand labeling provides you with some kind of external data, which is not available for other participants and can be produced with some automated algorithm.

\n

P.S. I do not accuse anyone. I just want to make this issue clear for me and other participants

", "rawMarkdown": "By the way, does hand labeling break the rule of Kaggle competitions? As I have understood from Rules 7.C (`However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants.`), hand labeling provides you with some kind of external data, which is not available for other participants and can be produced with some automated algorithm.\n\nP.S. I do not accuse anyone. I just want to make this issue clear for me and other participants", "votes": null }, { "id": "1312712", "postDate": "05/18/2021 07:46:53", "content": "

https://www.kaggle.com/c/birdclef-2021/discussion/230298

\n

Allowed in this competition. However, this is not an universal rule, and it might be banned in other competitions, it is up to the host to decide.

", "rawMarkdown": "https://www.kaggle.com/c/birdclef-2021/discussion/230298\n\nAllowed in this competition. However, this is not an universal rule, and it might be banned in other competitions, it is up to the host to decide.", "votes": null }, { "id": "1312747", "postDate": "05/18/2021 08:15:17", "content": "

If you team merge with us, you can use our hand labeling data.
\nBut you will not agree. Because you are a big chance to get solo gold!

", "rawMarkdown": "If you team merge with us, you can use our hand labeling data.\nBut you will not agree. Because you are a big chance to get solo gold!", "votes": null }, { "id": "1312749", "postDate": "05/18/2021 08:15:38", "content": "

I created one annotation per a sound file. These annotation contain only primary label birdcall.
\nTherefore, 4300 sound files and 4300 annotations were created.

", "rawMarkdown": "I created one annotation per a sound file. These annotation contain only primary label birdcall.\nTherefore, 4300 sound files and 4300 annotations were created.", "votes": null }, { "id": "1312763", "postDate": "05/18/2021 08:25:51", "content": "

Thanks for reply!

", "rawMarkdown": "Thanks for reply!", "votes": null }, { "id": "1312799", "postDate": "05/18/2021 08:53:34", "content": "

Hand labeling training data has always been allowed.

", "rawMarkdown": "Hand labeling training data has always been allowed.", "votes": null }, { "id": "1312827", "postDate": "05/18/2021 09:20:53", "content": "

That's right.
\nBesides, I'm not sure if hand labeling will work for teams in the gold zone (maybe they've already dealt with it explicitly or implicitly).
\nI look forward to seeing their strategy after the competition is over:)

", "rawMarkdown": "That's right.\nBesides, I'm not sure if hand labeling will work for teams in the gold zone (maybe they've already dealt with it explicitly or implicitly).\nI look forward to seeing their strategy after the competition is over:)", "votes": null }, { "id": "1313084", "postDate": "05/18/2021 11:59:31", "content": "

This time I don't use Hand labeling

", "rawMarkdown": "This time I don't use Hand labeling", "votes": null }, { "id": "1313312", "postDate": "05/18/2021 14:01:42", "content": "

@philippsinger did you use it in Rainforest? I thought you did not.

", "rawMarkdown": "philippsinger did you use it in Rainforest? I thought you did not.", "votes": null }, { "id": "1313314", "postDate": "05/18/2021 14:02:37", "content": "
\n

I look forward to seeing their strategy after the competition is over:)

\n
\n

Me too. I'm sure we will see a wide range of methods.

", "rawMarkdown": "> I look forward to seeing their strategy after the competition is over:)\n\nMe too. I'm sure we will see a wide range of methods.", "votes": null }, { "id": "1313318", "postDate": "05/18/2021 14:05:59", "content": "

64k files are too many, right? haha

", "rawMarkdown": "64k files are too many, right? haha", "votes": null }, { "id": "1313328", "postDate": "05/18/2021 14:10:00", "content": "

0.04 is huge

", "rawMarkdown": "0.04 is huge", "votes": null }, { "id": "1313335", "postDate": "05/18/2021 14:12:37", "content": "

20,000 of them I cleaned up for the last contest, but, for now, I don't use them

", "rawMarkdown": "20,000 of them I cleaned up for the last contest, but, for now, I don't use them", "votes": null }, { "id": "1313347", "postDate": "05/18/2021 14:15:02", "content": "

you must improve your method a lot, looking forward to read your write up after competition😃

", "rawMarkdown": "you must improve your method a lot, looking forward to read your write up after competition😃", "votes": null }, { "id": "1313374", "postDate": "05/18/2021 14:26:58", "content": "

I did not improve the result in two weeks =(

", "rawMarkdown": "I did not improve the result in two weeks =(", "votes": null }, { "id": "1313411", "postDate": "05/18/2021 14:46:36", "content": "

@vlomme, Have you open sourced your previous markup? Maybe I can give a try 😃

", "rawMarkdown": "vlomme, Have you open sourced your previous markup? Maybe I can give a try 😃", "votes": null }, { "id": "1313423", "postDate": "05/18/2021 14:58:32", "content": "

The competition is too strong, and my graphics card is working on another competition, so I'm afraid to fly out of the top ten

", "rawMarkdown": "The competition is too strong, and my graphics card is working on another competition, so I'm afraid to fly out of the top ten", "votes": null }, { "id": "1313751", "postDate": "05/18/2021 17:34:41", "content": "

@vlomme SETI will run for another two months. Waiting 2 weeks and using your GPU here in the meantime may be a better choice. That's mine anyway.

\n

Whatever your choice, I hope for you that you'll stay in top 10 given how well you did so far.

", "rawMarkdown": "vlomme SETI will run for another two months. Waiting 2 weeks and using your GPU here in the meantime may be a better choice. That's mine anyway.\n\nWhatever your choice, I hope for you that you'll stay in top 10 given how well you did so far.", "votes": null }, { "id": "1313775", "postDate": "05/18/2021 17:53:25", "content": "

You can read in our solution post, iirc we only tried adding them a bit but didnt get much out of it.

", "rawMarkdown": "You can read in our solution post, iirc we only tried adding them a bit but didnt get much out of it.", "votes": null }, { "id": "1313798", "postDate": "05/18/2021 18:11:36", "content": "

Tx, I read your post but missed that.

", "rawMarkdown": "Tx, I read your post but missed that.", "votes": null }, { "id": "1313803", "postDate": "05/18/2021 18:17:10", "content": "

Thank you, but there is also the CLRP and the Coleridge Initiative competitions.
\nHere I have tried many different models and methods of preprocessing data and all techniques get a similar rating. I just don't have any new ideas other than small changes to an existing method, which is boring.

", "rawMarkdown": "Thank you, but there is also the CLRP and the Coleridge Initiative competitions.\nHere I have tried many different models and methods of preprocessing data and all techniques get a similar rating. I just don't have any new ideas other than small changes to an existing method, which is boring.", "votes": null }, { "id": "1314025", "postDate": "05/18/2021 22:09:51", "content": "

Got only a slight boost out of it, but it may have helped with diversity in the ensemble

", "rawMarkdown": "Got only a slight boost out of it, but it may have helped with diversity in the ensemble", "votes": null }, { "id": "1314044", "postDate": "05/18/2021 22:55:32", "content": "

Is Coleridge metric mess over? If it is then I'll probably join.

", "rawMarkdown": "Is Coleridge metric mess over? If it is then I'll probably join.", "votes": null }, { "id": "1314821", "postDate": "05/19/2021 11:59:36", "content": "

Hand labeling training data is allowed. But even if it does not improve the score, having strong labels for a weakly labeled dataset can be very valuable in the future. Would you consider releasing your labels after the deadline?

", "rawMarkdown": "Hand labeling training data is allowed. But even if it does not improve the score, having strong labels for a weakly labeled dataset can be very valuable in the future. Would you consider releasing your labels after the deadline?", "votes": null }, { "id": "1316335", "postDate": "05/20/2021 13:18:47", "content": "

I was thinking to do that for identifying no signal in the audio spectrogram chunks( some of the specs don't have any signal in them. I am using 5-sec chunks for generating specs). Would that not help??

", "rawMarkdown": "I was thinking to do that for identifying no signal in the audio spectrogram chunks( some of the specs don't have any signal in them. I am using 5-sec chunks for generating specs). Would that not help??", "votes": null }, { "id": "1316456", "postDate": "05/20/2021 15:13:43", "content": "

You must be a good ornithologist cuz I can't even recognize the birds by myself 😂

", "rawMarkdown": "You must be a good ornithologist cuz I can't even recognize the birds by myself 😂", "votes": null } ], "comments": [ { "id": 1312561, "author_name": "tattaka", "author_url": "", "post_date": "05/18/2021 05:49:35", "content": "

Thank you for sharing!
\nLB 0.04 improvement is a big impact on me…..

", "votes": null, "replies": [ { "id": 1312747, "author_name": "shinmurashinmura", "author_url": "", "post_date": "05/18/2021 08:15:17", "content": "

If you team merge with us, you can use our hand labeling data.
\nBut you will not agree. Because you are a big chance to get solo gold!

", "votes": null, "replies": [] }, { "id": 1312827, "author_name": "tattaka", "author_url": "", "post_date": "05/18/2021 09:20:53", "content": "

That's right.
\nBesides, I'm not sure if hand labeling will work for teams in the gold zone (maybe they've already dealt with it explicitly or implicitly).
\nI look forward to seeing their strategy after the competition is over:)

", "votes": null, "replies": [] }, { "id": 1313314, "author_name": "cpmpml", "author_url": "", "post_date": "05/18/2021 14:02:37", "content": "
\n

I look forward to seeing their strategy after the competition is over:)

\n
\n

Me too. I'm sure we will see a wide range of methods.

", "votes": null, "replies": [] } ] }, { "id": 1312573, "author_name": "fffrrt", "author_url": "", "post_date": "05/18/2021 05:57:55", "content": "
\n

I made 4300 hand labeling data, but it is small impact(LB 0.04 improvement).

\n
\n

4300 individual examples or 4300 annotated sound files?

", "votes": null, "replies": [ { "id": 1312749, "author_name": "shinmurashinmura", "author_url": "", "post_date": "05/18/2021 08:15:38", "content": "

I created one annotation per a sound file. These annotation contain only primary label birdcall.
\nTherefore, 4300 sound files and 4300 annotations were created.

", "votes": null, "replies": [] } ] }, { "id": 1312699, "author_name": "vladimirsydor", "author_url": "", "post_date": "05/18/2021 07:39:56", "content": "

By the way, does hand labeling break the rule of Kaggle competitions? As I have understood from Rules 7.C (However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants.), hand labeling provides you with some kind of external data, which is not available for other participants and can be produced with some automated algorithm.

\n

P.S. I do not accuse anyone. I just want to make this issue clear for me and other participants

", "votes": null, "replies": [ { "id": 1312712, "author_name": "fffrrt", "author_url": "", "post_date": "05/18/2021 07:46:53", "content": "

https://www.kaggle.com/c/birdclef-2021/discussion/230298

\n

Allowed in this competition. However, this is not an universal rule, and it might be banned in other competitions, it is up to the host to decide.

", "votes": null, "replies": [] }, { "id": 1312763, "author_name": "vladimirsydor", "author_url": "", "post_date": "05/18/2021 08:25:51", "content": "

Thanks for reply!

", "votes": null, "replies": [] }, { "id": 1312799, "author_name": "philippsinger", "author_url": "", "post_date": "05/18/2021 08:53:34", "content": "

Hand labeling training data has always been allowed.

", "votes": null, "replies": [] }, { "id": 1313312, "author_name": "cpmpml", "author_url": "", "post_date": "05/18/2021 14:01:42", "content": "

@philippsinger did you use it in Rainforest? I thought you did not.

", "votes": null, "replies": [] }, { "id": 1313775, "author_name": "philippsinger", "author_url": "", "post_date": "05/18/2021 17:53:25", "content": "

You can read in our solution post, iirc we only tried adding them a bit but didnt get much out of it.

", "votes": null, "replies": [] }, { "id": 1313798, "author_name": "cpmpml", "author_url": "", "post_date": "05/18/2021 18:11:36", "content": "

Tx, I read your post but missed that.

", "votes": null, "replies": [] }, { "id": 1314025, "author_name": "ilu000", "author_url": "", "post_date": "05/18/2021 22:09:51", "content": "

Got only a slight boost out of it, but it may have helped with diversity in the ensemble

", "votes": null, "replies": [] } ] }, { "id": 1313084, "author_name": "vlomme", "author_url": "", "post_date": "05/18/2021 11:59:31", "content": "

This time I don't use Hand labeling

", "votes": null, "replies": [ { "id": 1313318, "author_name": "superchenhao", "author_url": "", "post_date": "05/18/2021 14:05:59", "content": "

64k files are too many, right? haha

", "votes": null, "replies": [] }, { "id": 1313335, "author_name": "vlomme", "author_url": "", "post_date": "05/18/2021 14:12:37", "content": "

20,000 of them I cleaned up for the last contest, but, for now, I don't use them

", "votes": null, "replies": [] }, { "id": 1313347, "author_name": "superchenhao", "author_url": "", "post_date": "05/18/2021 14:15:02", "content": "

you must improve your method a lot, looking forward to read your write up after competition😃

", "votes": null, "replies": [] }, { "id": 1313374, "author_name": "vlomme", "author_url": "", "post_date": "05/18/2021 14:26:58", "content": "

I did not improve the result in two weeks =(

", "votes": null, "replies": [] }, { "id": 1313411, "author_name": "vladimirsydor", "author_url": "", "post_date": "05/18/2021 14:46:36", "content": "

@vlomme, Have you open sourced your previous markup? Maybe I can give a try 😃

", "votes": null, "replies": [] }, { "id": 1313423, "author_name": "vlomme", "author_url": "", "post_date": "05/18/2021 14:58:32", "content": "

The competition is too strong, and my graphics card is working on another competition, so I'm afraid to fly out of the top ten

", "votes": null, "replies": [] }, { "id": 1313751, "author_name": "cpmpml", "author_url": "", "post_date": "05/18/2021 17:34:41", "content": "

@vlomme SETI will run for another two months. Waiting 2 weeks and using your GPU here in the meantime may be a better choice. That's mine anyway.

\n

Whatever your choice, I hope for you that you'll stay in top 10 given how well you did so far.

", "votes": null, "replies": [] }, { "id": 1313803, "author_name": "vlomme", "author_url": "", "post_date": "05/18/2021 18:17:10", "content": "

Thank you, but there is also the CLRP and the Coleridge Initiative competitions.
\nHere I have tried many different models and methods of preprocessing data and all techniques get a similar rating. I just don't have any new ideas other than small changes to an existing method, which is boring.

", "votes": null, "replies": [] }, { "id": 1314044, "author_name": "cpmpml", "author_url": "", "post_date": "05/18/2021 22:55:32", "content": "

Is Coleridge metric mess over? If it is then I'll probably join.

", "votes": null, "replies": [] } ] }, { "id": 1313328, "author_name": "superchenhao", "author_url": "", "post_date": "05/18/2021 14:10:00", "content": "

0.04 is huge

", "votes": null, "replies": [] }, { "id": 1314821, "author_name": "stefankahl", "author_url": "", "post_date": "05/19/2021 11:59:36", "content": "

Hand labeling training data is allowed. But even if it does not improve the score, having strong labels for a weakly labeled dataset can be very valuable in the future. Would you consider releasing your labels after the deadline?

", "votes": null, "replies": [] }, { "id": 1316335, "author_name": "virajkadam", "author_url": "", "post_date": "05/20/2021 13:18:47", "content": "

I was thinking to do that for identifying no signal in the audio spectrogram chunks( some of the specs don't have any signal in them. I am using 5-sec chunks for generating specs). Would that not help??

", "votes": null, "replies": [] }, { "id": 1316456, "author_name": "kneroma", "author_url": "", "post_date": "05/20/2021 15:13:43", "content": "

You must be a good ornithologist cuz I can't even recognize the birds by myself 😂

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1312401": "Hi Kaggler,\n\nI know that hand labeling has been effective in the past two sound competition.\n\n+ Rainforest Connection Species Audio Detection\n[1st place solution](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220563)\n[3rd place solution](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220522)\n[7th place solution](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220443)\n[8th place solution](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220803)\n\n+ Cornell Birdcall Identification\n[2nd place solution](https://www.kaggle.com/c/birdsong-recognition/discussion/183269)\n\nIn this competition, I tried hand labeling for a one month.\nI made **4300 hand labeling data**, but it is small impact(LB 0.04 improvement).\n\nEven if you try hand labeling, you cannot expect big impact.", "1312561": "Thank you for sharing!\nLB 0.04 improvement is a big impact on me.....", "1312573": "> I made 4300 hand labeling data, but it is small impact(LB 0.04 improvement).\n\n4300 individual examples or 4300 annotated sound files?", "1312699": "By the way, does hand labeling break the rule of Kaggle competitions? As I have understood from Rules 7.C (`However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants.`), hand labeling provides you with some kind of external data, which is not available for other participants and can be produced with some automated algorithm.\n\nP.S. I do not accuse anyone. I just want to make this issue clear for me and other participants", "1312712": "https://www.kaggle.com/c/birdclef-2021/discussion/230298\n\nAllowed in this competition. However, this is not an universal rule, and it might be banned in other competitions, it is up to the host to decide.", "1312747": "If you team merge with us, you can use our hand labeling data.\nBut you will not agree. Because you are a big chance to get solo gold!", "1312749": "I created one annotation per a sound file. These annotation contain only primary label birdcall.\nTherefore, 4300 sound files and 4300 annotations were created.", "1312763": "Thanks for reply!", "1312799": "Hand labeling training data has always been allowed.", "1312827": "That's right.\nBesides, I'm not sure if hand labeling will work for teams in the gold zone (maybe they've already dealt with it explicitly or implicitly).\nI look forward to seeing their strategy after the competition is over:)", "1313084": "This time I don't use Hand labeling", "1313312": "philippsinger did you use it in Rainforest? I thought you did not.", "1313314": "> I look forward to seeing their strategy after the competition is over:)\n\nMe too. I'm sure we will see a wide range of methods.", "1313318": "64k files are too many, right? haha", "1313328": "0.04 is huge", "1313335": "20,000 of them I cleaned up for the last contest, but, for now, I don't use them", "1313347": "you must improve your method a lot, looking forward to read your write up after competition😃", "1313374": "I did not improve the result in two weeks =(", "1313411": "vlomme, Have you open sourced your previous markup? Maybe I can give a try 😃", "1313423": "The competition is too strong, and my graphics card is working on another competition, so I'm afraid to fly out of the top ten", "1313751": "vlomme SETI will run for another two months. Waiting 2 weeks and using your GPU here in the meantime may be a better choice. That's mine anyway.\n\nWhatever your choice, I hope for you that you'll stay in top 10 given how well you did so far.", "1313775": "You can read in our solution post, iirc we only tried adding them a bit but didnt get much out of it.", "1313798": "Tx, I read your post but missed that.", "1313803": "Thank you, but there is also the CLRP and the Coleridge Initiative competitions.\nHere I have tried many different models and methods of preprocessing data and all techniques get a similar rating. I just don't have any new ideas other than small changes to an existing method, which is boring.", "1314025": "Got only a slight boost out of it, but it may have helped with diversity in the ensemble", "1314044": "Is Coleridge metric mess over? If it is then I'll probably join.", "1314821": "Hand labeling training data is allowed. But even if it does not improve the score, having strong labels for a weakly labeled dataset can be very valuable in the future. Would you consider releasing your labels after the deadline?", "1316335": "I was thinking to do that for identifying no signal in the audio spectrogram chunks( some of the specs don't have any signal in them. I am using 5-sec chunks for generating specs). Would that not help??", "1316456": "You must be a good ornithologist cuz I can't even recognize the birds by myself 😂" }, "source": "meta" }