As I promised, I will publish my solution right after the competition. This will be a long topic as the two solutions are totally different. As kaggle says > You've been sharing this link too often. We prevent redundant posts to reduce spam. So I put some link in the code block.
\nBefore the solutions, thanks to my teammate @jionie, our second gold medal, and first-time money zone!!!!!!!!!! Cheers!!!!!!!!!!
\nI will start with our [Private #7/8 (potential)] Solution first since I **promise **you will say the \"F\" word after seeing the other solution.
\n[Private #7/8 (potential)] Solution
\nTraining:
\nOur first solution is based on @kaerunantoka 's public notebook and last year's second-place solution. The training code is pretty the mixture of the two pipelines mentioned above.
\nSome key points to mention:
data augmentation:
\nOneOf([\n Gain(min_gain_in_db=-15, max_gain_in_db=15, p=0.8),\n GainTransition(min_gain_in_db=-15, max_gain_in_db=15, p=0.8),\n ]),\n OneOf(\n [\n NoiseInjection(p=1, max_noise_level=0.04),\n GaussianNoise(p=1, min_snr=5, max_snr=20),\n PinkNoise(p=1, min_snr=5, max_snr=20),\n AddGaussianNoise(min_amplitude=0.0001, max_amplitude=0.03, p=0.5),\n AddGaussianSNR(min_snr_in_db=5, max_snr_in_db=15, p=0.5),\n ],\n p=0.3,\n ),\n AddBackgroundNoise(\n sounds_path=self.config.BACKGROUND_PATH, min_snr_in_db=0, max_snr_in_db=2, p=0.5\n ),\n Normalize(p=1),
cut mix + mix up
loss function: BCEWithLogits + BCEFocal2WayLoss
hypers setting (n_fft, n_mels, hop_length) refer to our inference notebook.
Our inference kernel and all trained models are [publicly available]
\nOnly one key point to mention: if a bird is detected in the previous or next 5s, we will rank the probs and add the top5 bird species to the current detected birds.
Because this solution was not selected as our final submission, we don't know the actual rank, but as it scores 0.79 in private, it should rank 7-8.
\n[#1 Private #2] Solution
\nActually, I am curious that no one found it, the solution is provided by the host and finally allowed by the host.
\nwe won't enforce the \"non-commercial\" clause in the license for BirdCLEF, and you can use BirdNET in your submissions. (I don't know if they also benefit from this repo, we can wait for their solution)Some Notes:
\nHonestly, we didn't originally intend to use this model as a final submission until we notice the issue mentioned above, and actually, the ensemble of our own model could also reach the gold zone. The answer is provided by the host so that's why I say it is a weird solution. I guess the BirdNet was trained on the data for the public leaderboard but not the private one since there's a huge drop between the public lb to private (0.91 --> 0.84; 0.85 --> 0.78)
The BirdNet is running on CPU less than 2h inference time, while ours cost about 8h to run on GPU. So the host wins, my hope is the first place team doesn't benefit from that model and can beat the host. Looking forward to their solution.
\nFinally, a big thanks to @stefankahl for hosting this amazing competition and providing the answer as well xD.
", "messages": [ { "id": 1800539, "postDate": "2022-05-25T02:51:49.700Z", "content": "As I promised, I will publish my solution right after the competition. This will be a long topic as the two solutions are totally different. As kaggle says > You've been sharing this link too often. We prevent redundant posts to reduce spam. So I put some link in the code block.
\nBefore the solutions, thanks to my teammate @jionie, our second gold medal, and first-time money zone!!!!!!!!!! Cheers!!!!!!!!!!
\nI will start with our [Private #7/8 (potential)] Solution first since I **promise **you will say the \"F\" word after seeing the other solution.
\n[Private #7/8 (potential)] Solution
\nTraining:
\nOur first solution is based on @kaerunantoka 's public notebook and last year's second-place solution. The training code is pretty the mixture of the two pipelines mentioned above.
\nSome key points to mention:
data augmentation:
\nOneOf([\n Gain(min_gain_in_db=-15, max_gain_in_db=15, p=0.8),\n GainTransition(min_gain_in_db=-15, max_gain_in_db=15, p=0.8),\n ]),\n OneOf(\n [\n NoiseInjection(p=1, max_noise_level=0.04),\n GaussianNoise(p=1, min_snr=5, max_snr=20),\n PinkNoise(p=1, min_snr=5, max_snr=20),\n AddGaussianNoise(min_amplitude=0.0001, max_amplitude=0.03, p=0.5),\n AddGaussianSNR(min_snr_in_db=5, max_snr_in_db=15, p=0.5),\n ],\n p=0.3,\n ),\n AddBackgroundNoise(\n sounds_path=self.config.BACKGROUND_PATH, min_snr_in_db=0, max_snr_in_db=2, p=0.5\n ),\n Normalize(p=1),
cut mix + mix up
loss function: BCEWithLogits + BCEFocal2WayLoss
hypers setting (n_fft, n_mels, hop_length) refer to our inference notebook.
Our inference kernel and all trained models are [publicly available]
\nOnly one key point to mention: if a bird is detected in the previous or next 5s, we will rank the probs and add the top5 bird species to the current detected birds.
Because this solution was not selected as our final submission, we don't know the actual rank, but as it scores 0.79 in private, it should rank 7-8.
\n[#1 Private #2] Solution
\nActually, I am curious that no one found it, the solution is provided by the host and finally allowed by the host.
\nwe won't enforce the \"non-commercial\" clause in the license for BirdCLEF, and you can use BirdNET in your submissions. (I don't know if they also benefit from this repo, we can wait for their solution)Some Notes:
\nHonestly, we didn't originally intend to use this model as a final submission until we notice the issue mentioned above, and actually, the ensemble of our own model could also reach the gold zone. The answer is provided by the host so that's why I say it is a weird solution. I guess the BirdNet was trained on the data for the public leaderboard but not the private one since there's a huge drop between the public lb to private (0.91 --> 0.84; 0.85 --> 0.78)
The BirdNet is running on CPU less than 2h inference time, while ours cost about 8h to run on GPU. So the host wins, my hope is the first place team doesn't benefit from that model and can beat the host. Looking forward to their solution.
\nFinally, a big thanks to @stefankahl for hosting this amazing competition and providing the answer as well xD.
", "rawMarkdown": "As I promised, I will publish my solution right after the competition. This will be a long topic as the two solutions are totally different. As kaggle says > You've been sharing this link too often. We prevent redundant posts to reduce spam. So I put some link in the code block.\n\nBefore the solutions, thanks to my teammate @jionie, our second gold medal, and first-time money zone!!!!!!!!!! Cheers!!!!!!!!!!\n\nI will start with our [Private #7/8 (potential)] Solution first since I **promise **you will say the \"F\" word after seeing the other solution.\n\n**[Private #7/8 (potential)] Solution**\n\nTraining:\nOur first solution is based on @kaerunantoka 's [public notebook](https://www.kaggle.com/code/kaerunantoka/birdclef2022-use-2nd-label-f0) and last year's second-place solution. The training code is pretty the mixture of the two pipelines mentioned above. \nSome key points to mention:\n1. data augmentation:\n`OneOf([\n Gain(min_gain_in_db=-15, max_gain_in_db=15, p=0.8),\n GainTransition(min_gain_in_db=-15, max_gain_in_db=15, p=0.8),\n ]),\n OneOf(\n [\n NoiseInjection(p=1, max_noise_level=0.04),\n GaussianNoise(p=1, min_snr=5, max_snr=20),\n PinkNoise(p=1, min_snr=5, max_snr=20),\n AddGaussianNoise(min_amplitude=0.0001, max_amplitude=0.03, p=0.5),\n AddGaussianSNR(min_snr_in_db=5, max_snr_in_db=15, p=0.5),\n ],\n p=0.3,\n ),\n AddBackgroundNoise(\n sounds_path=self.config.BACKGROUND_PATH, min_snr_in_db=0, max_snr_in_db=2, p=0.5\n ),\n Normalize(p=1),`\n\n2. cut mix + mix up\n3. loss function: BCEWithLogits + BCEFocal2WayLoss\n4. hypers setting (n_fft, n_mels, hop_length) refer to our inference notebook.\n\nOur inference kernel and all trained models are [[publicly available](https://www.kaggle.com/code/leonshangguan/private-7-8-final-of-submission)] \nOnly one key point to mention: if a bird is detected in the previous or next 5s, we will rank the probs and add the top5 bird species to the current detected birds.\n\nBecause this solution was not selected as our final submission, we don't know the actual rank, but as it scores 0.79 in private, it should rank 7-8.\n\n\n**[#1 Private #2] Solution**\n\nActually, I am curious that no one found it, the solution is provided by the host and finally allowed by the host.\n\n1. If you look at the discussion [Meet the host](https://www.kaggle.com/competitions/birdclef-2022/discussion/307941), you will find the host mentioned [BirdNet project](https://birdnet.cornell.edu/) in the post.\n2. Click the link, and it is easy to find the host's [github repo](https://github.com/kahst/BirdNET-Analyzer).\n3. Then, find the overlap classes between the [host's model](https://github.com/kahst/BirdNET-Analyzer/blob/main/checkpoints/V2.1/BirdNET_GLOBAL_2K_V2.1_Labels.txt) and the scored birds, 20 out of 21 scored birds are the same (except aniani).\n4. Modify species_list.txt under example folder accordingly, I have uploaded the modified repo to kaggle [here](https://www.kaggle.com/datasets/leonshangguan/birdnet.).\n5. Do some post-processing as in our [[inference notebook](https://www.kaggle.com/code/leonshangguan/birdnet-inference)]\n6. Thanks to @ivanpan who opened [this issue](https://github.com/kahst/BirdNET-Analyzer/issues/40), the host says `we won't enforce the \"non-commercial\" clause in the license for BirdCLEF, and you can use BirdNET in your submissions.` (I don't know if they also benefit from this repo, we can wait for their solution)\n\nSome Notes:\nHonestly, we didn't originally intend to use this model as a final submission until we notice the issue mentioned above, and actually, the ensemble of our own model could also reach the gold zone. The answer is provided by the host so that's why I say it is a weird solution. I guess the BirdNet was trained on the data for the public leaderboard but not the private one since there's a huge drop between the public lb to private (0.91 --> 0.84; 0.85 --> 0.78)\n\nThe BirdNet is running on CPU less than 2h inference time, while ours cost about 8h to run on GPU. So the host wins, my hope is the first place team doesn't benefit from that model and can beat the host. Looking forward to their solution.\n\n**Finally, a big thanks to @stefankahl for hosting this amazing competition and providing the answer as well xD.**", "votes": 59 }, { "id": 1800598, "postDate": "2022-05-25T04:25:22.333Z", "content": "Congratulations!
\nI do feel this should have been disclosed by @stefankahl as it seems like the BirdNET model has been trained at least partially on the test data used in this competition.
", "rawMarkdown": "Congratulations!\n\nI do feel this should have been disclosed by @stefankahl as it seems like the BirdNET model has been trained at least partially on the test data used in this competition.", "votes": 10, "replies": [ { "id": 1801102, "postDate": "2022-05-25T12:54:06Z", "content": "No, it wasn't. Yet, we used other soundscape data and I never realized that this might be an issue for this competition. My guess is, with competition data only, BirdNET would not have scored in the top 10.
", "rawMarkdown": "No, it wasn't. Yet, we used other soundscape data and I never realized that this might be an issue for this competition. My guess is, with competition data only, BirdNET would not have scored in the top 10.", "votes": 4 } ] }, { "id": 1800551, "postDate": "2022-05-25T03:07:21.853Z", "content": "thanks for the story. it is very interesting whether the host used more training records to train the model or not
", "rawMarkdown": "thanks for the story. it is very interesting whether the host used more training records to train the model or not", "votes": 7, "replies": [ { "id": 1800565, "postDate": "2022-05-25T03:27:03.180Z", "content": "The host used many more. I think it was somewhere around 2434 species in the train set.
", "rawMarkdown": "The host used many more. I think it was somewhere around 2434 species in the train set.", "votes": 2 }, { "id": 1800634, "postDate": "2022-05-25T05:32:27.873Z", "content": "I'm talking about our species. did they use, for example, the test data of this competition for training
", "rawMarkdown": "I'm talking about our species. did they use, for example, the test data of this competition for training", "votes": 3 } ] }, { "id": 1801322, "postDate": "2022-05-25T16:02:52.193Z", "content": "Someone asked me why didn't you contact the host already those 20 days ago when you found it or at least 12 days ago when you wrote your comment? -- Because, as I said, we didn't manage to select BirdNet as our final submission so I didn't think that's an issue and did not contact the host. We keep training our own model until the last minute and our own one is ranking 9 which is also in the gold zone. To be honest, we didn't think the BirdNet is allowed until the host said it is allowed. I understand our solution is weird under some gray area, but I don't understand why someone downvotes my comments (and downvotes the 1st place team members comments). This situation happened in Cassava Leaf Disease Classification\n as well where someone use pre-trained model and reached top places. Kaggle should manage to develop more clear rules and we should keep looking forward to helping avoid this case happen again.
BirdNet was a huge revelation to me in this competition. I couldn't believe how well it works, so decided to test on last year competition where we got 8th place. Easily managed to get top-1 on private there, but I believe public scores were around silver zone, that's why I wasn't 100% sure it would perform that well on private here.
\nGlad I was wrong and now we can see that BirdNet gets top-1 on private in both years.
\nFor the sake of honesty, we actually ensembled BirdNet with our models to get some additional boost and spent a couple of months trying to perfect those models (if I'm not mistaken, they score 0.78-0.79 on private). I believe we will be sharing results soon. But anyway, thanks @stefankahl for this competition. I mentioned this last year, I'm gonna mention this again - great orgs! One of the very few on Kaggle who provide great resources here (including notebooks), and simply answer questions to participants without abandoning a competition 1 week in.
\nThanks!
", "rawMarkdown": "BirdNet was a huge revelation to me in this competition. I couldn't believe how well it works, so decided to test on last year competition where we got 8th place. Easily managed to get top-1 on private there, but I believe public scores were around silver zone, that's why I wasn't 100% sure it would perform that well on private here.\n\nGlad I was wrong and now we can see that BirdNet gets top-1 on private in both years. \n\nFor the sake of honesty, we actually ensembled BirdNet with our models to get some additional boost and spent a couple of months trying to perfect those models (if I'm not mistaken, they score 0.78-0.79 on private). I believe we will be sharing results soon. But anyway, thanks @stefankahl for this competition. I mentioned this last year, I'm gonna mention this again - great orgs! One of the very few on Kaggle who provide great resources here (including notebooks), and simply answer questions to participants without abandoning a competition 1 week in. \n\nThanks! ", "votes": 7, "replies": [ { "id": 1800653, "postDate": "2022-05-25T05:51:43.273Z", "content": "I think this network works better because they had more data. not a few records, as we have.
", "rawMarkdown": "I think this network works better because they had more data. not a few records, as we have.", "votes": 7 }, { "id": 1800661, "postDate": "2022-05-25T06:05:31.307Z", "content": "Probably BirdNet also trained on data without noisy labels and maybe even on data which used for eval here.
", "rawMarkdown": "Probably BirdNet also trained on data without noisy labels and maybe even on data which used for eval here.", "votes": 1 }, { "id": 1800663, "postDate": "2022-05-25T06:06:38.027Z", "content": "@vlomme I think you're right. The model itself is quite quite good, but combine it with a lot of data - and that's a recipe for SOTA.
", "rawMarkdown": "@vlomme I think you're right. The model itself is quite quite good, but combine it with a lot of data - and that's a recipe for SOTA. " }, { "id": 1800677, "postDate": "2022-05-25T06:11:14.007Z", "content": "It's just that if non-public data and, possibly, test data were used for it training, then it was pointless to compete with such a model. It's very strange and I'm sad
", "rawMarkdown": "It's just that if non-public data and, possibly, test data were used for it training, then it was pointless to compete with such a model. It's very strange and I'm sad", "votes": 7 }, { "id": 1800679, "postDate": "2022-05-25T06:14:01.277Z", "content": "I think that the conclusion of this competition is
\n\"Let's increase the number of data without training with a few amount of data\"
\nIt wasn't what I expected.
Hold up, let me get this straight. We do not allowed to use Macaulay Library data, but allowed to use BirdNet, trained on Macaulay Library data? Can I train my own public GlebNet on Macaulay Library data (with all agreement that needed), and use it?
", "rawMarkdown": "Hold up, let me get this straight. We do not allowed to use Macaulay Library data, but allowed to use BirdNet, trained on Macaulay Library data? Can I train my own public GlebNet on Macaulay Library data (with all agreement that needed), and use it?", "votes": 11, "replies": [ { "id": 1801090, "postDate": "2022-05-25T12:27:40.717Z", "content": "If it's okay to use BirdNET it should be alright to use your own model as long as it's publicly available, no matter what data it's been trained on. Is this correct @stefankahl ?
", "rawMarkdown": "If it's okay to use BirdNET it should be alright to use your own model as long as it's publicly available, no matter what data it's been trained on. Is this correct @stefankahl ?", "votes": 1 } ] }, { "id": 1800690, "postDate": "2022-05-25T06:29:22.980Z", "content": "No you can't.
", "rawMarkdown": "No you can't. ", "votes": 1 }, { "id": 1800699, "postDate": "2022-05-25T06:35:24.997Z", "content": "@bakeryproducts couldn't agree more, in the forum there were countless discussions that explicitly prohibited the use of Macaulay Library and audios from xeno-canto w/o appropriate license (e.g. see this post)
", "rawMarkdown": "@bakeryproducts couldn't agree more, in the forum there were countless discussions that explicitly prohibited the use of Macaulay Library and audios from xeno-canto w/o appropriate license (e.g. see this [post](https://www.kaggle.com/competitions/birdclef-2022/discussion/315361))", "votes": 4 }, { "id": 1800723, "postDate": "2022-05-25T06:59:33.683Z", "content": "To fellow participants, imagine scenario: someone creating topic \"BirdNet is kinda good\" two days ago. Month(s) of time -> poof -> 'i would like to thank kaggle for this competition and great orgs, …'
", "rawMarkdown": "To fellow participants, imagine scenario: someone creating topic \"BirdNet is kinda good\" two days ago. Month(s) of time -> poof -> 'i would like to thank kaggle for this competition and great orgs, ...'", "votes": 4 }, { "id": 1800724, "postDate": "2022-05-25T07:00:44.813Z", "content": "Around which point in the competition did you discover BirdNET?
", "rawMarkdown": "Around which point in the competition did you discover BirdNET?", "votes": 1 }, { "id": 1800728, "postDate": "2022-05-25T07:05:28.450Z", "content": "no questions to the participants, it is strange that the organizers allowed the use of this model
", "rawMarkdown": "no questions to the participants, it is strange that the organizers allowed the use of this model", "votes": 2 }, { "id": 1800730, "postDate": "2022-05-25T07:09:08.577Z", "content": "@vlomme thats my point, rules were made in such a way that it all could end disastrously
", "rawMarkdown": "@vlomme thats my point, rules were made in such a way that it all could end disastrously", "votes": 6 }, { "id": 1800732, "postDate": "2022-05-25T07:12:27.660Z", "content": "I agree
", "rawMarkdown": "I agree ", "votes": 1 }, { "id": 1800793, "postDate": "2022-05-25T08:15:55.780Z", "content": "I am very confident (although I have no proof) that BirdNet only works so well because it was either trained on more data, or even the test data itself. So I am not sure if the host really wins here… It looks like a big leak.
", "rawMarkdown": "I am very confident (although I have no proof) that BirdNet only works so well because it was either trained on more data, or even the test data itself. So I am not sure if the host really wins here... It looks like a big leak.", "votes": 19 }, { "id": 1800809, "postDate": "2022-05-25T08:26:26.350Z", "content": "It is a fact that BirdNet is trained on a larger amount of data (but hardly on a test). In a competition where there were few examples (several per class) this is a non-competitive advantage
", "rawMarkdown": "It is a fact that BirdNet is trained on a larger amount of data (but hardly on a test). In a competition where there were few examples (several per class) this is a non-competitive advantage", "votes": 3 }, { "id": 1800830, "postDate": "2022-05-25T08:44:38.680Z", "content": "And this extra data Macaulay Library was not allowed to be used here in this competition but the BirdNet model was trained on it? Wouldn't that mean that BirdNet model should not be allowed?
", "rawMarkdown": "And this extra data Macaulay Library was not allowed to be used here in this competition but the BirdNet model was trained on it? Wouldn't that mean that BirdNet model should not be allowed?", "votes": 13 }, { "id": 1800841, "postDate": "2022-05-25T08:55:22.787Z", "content": "Correct. But unfortunately, permission seems to have been given by the organizers to use this in a github issue 2 days ago.
", "rawMarkdown": "Correct. But unfortunately, permission seems to have been given by the organizers to use this in a github issue 2 days ago. ", "votes": 9 }, { "id": 1800842, "postDate": "2022-05-25T08:55:45.820Z", "content": "Yes, that's right. For some reason the organizer allowed the model to be used
", "rawMarkdown": "Yes, that's right. For some reason the organizer allowed the model to be used", "votes": 2 }, { "id": 1800937, "postDate": "2022-05-25T10:06:31.677Z", "content": "I think the organizers should clarify whether the Macaulay Library or competition test data were used or not in the BirdNet. In any case, I feel empty.
", "rawMarkdown": "I think the organizers should clarify whether the Macaulay Library or competition test data were used or not in the BirdNet. In any case, I feel empty.", "votes": 6 }, { "id": 1800946, "postDate": "2022-05-25T10:09:57.753Z", "content": "They mention it explicitly in the linked issue that they use Macaulay data!
", "rawMarkdown": "They mention it explicitly in the linked issue that they use Macaulay data!", "votes": 4 }, { "id": 1800962, "postDate": "2022-05-25T10:16:34.937Z", "content": "Wow, I feel for you. Crazy to make such a ground-breaking decision in a github issue one day before end of competition…
", "rawMarkdown": "Wow, I feel for you. Crazy to make such a ground-breaking decision in a github issue one day before end of competition...", "votes": 10 }, { "id": 1801094, "postDate": "2022-05-25T12:32:09.127Z", "content": "Yes, I also believe that BirdNET works so well because of additional training data (we indeed used Macaulay Library and another soundscape set from Hawaii). It would probably not score in the top 10 with only the competition data. Unfortunately, we only realized that people would use BirdNET one day before the deadline, and it was too late to implement any last-minute rule changes. This is just something that slipped our minds when organizing the competition.
", "rawMarkdown": "Yes, I also believe that BirdNET works so well because of additional training data (we indeed used Macaulay Library and another soundscape set from Hawaii). It would probably not score in the top 10 with only the competition data. Unfortunately, we only realized that people would use BirdNET one day before the deadline, and it was too late to implement any last-minute rule changes. This is just something that slipped our minds when organizing the competition.", "votes": -4 }, { "id": 1801096, "postDate": "2022-05-25T12:34:42.807Z", "content": "I am not sure I am following @stefankahl - but you explicitly allowed it one day before deadline, there was no reason to as it was disallowed before that. So you actually implemented a last-minute rule change.
", "rawMarkdown": "I am not sure I am following @stefankahl - but you explicitly **allowed** it one day before deadline, there was no reason to as it was disallowed before that. So you actually implemented a last-minute rule change.", "votes": 5 }, { "id": 1801100, "postDate": "2022-05-25T12:47:53.713Z", "content": "No, it was not disallowed before that, it's a public repo and anyone could have used it. Even the fact that it uses Macaulay data doesn't change that. If we had been aware earlier, we would have implemented a rule change at the beginning of the competition. It's like a vision model trained on the non-public Google JFT-300M dataset used for an ImageNet competition. However, we are interested in alternative approaches and thus would have limited the use of BirdNET - it just never occurred to us that this might be a useful rule addition.
", "rawMarkdown": "No, it was not disallowed before that, it's a public repo and anyone could have used it. Even the fact that it uses Macaulay data doesn't change that. If we had been aware earlier, we would have implemented a rule change at the beginning of the competition. It's like a vision model trained on the non-public Google JFT-300M dataset used for an ImageNet competition. However, we are interested in alternative approaches and thus would have limited the use of BirdNET - it just never occurred to us that this might be a useful rule addition.", "votes": -2 }, { "id": 1801129, "postDate": "2022-05-25T13:34:12.047Z", "content": "Your first statement in this github repo was very definite that it was not allowed. Also, according to reports of others, the Macaulay dataset was explicitly forbidden to use, so what sense does it make to allow a separate model that has been trained on this dataset?
\nIf a competition only allows external data that can be commercially used, which is the default, then also models trained on non-commercial-only data are obviously not allowed, this also applies to any of the standard pretrained models. If that is actually always enforced, I am not sure.
\nIf you really suddenly wanted to allow this dataset, then you should have posted it here for everyone to see and extend the deadline by 1-2 weeks.
", "rawMarkdown": "Your first statement in this github repo was very definite that it was not allowed. Also, according to reports of others, the Macaulay dataset was explicitly forbidden to use, so what sense does it make to allow a separate model that has been trained on this dataset?\n\nIf a competition only allows external data that can be commercially used, which is the default, then also models trained on non-commercial-only data are obviously not allowed, this also applies to any of the standard pretrained models. If that is actually always enforced, I am not sure. \n\nIf you really suddenly wanted to allow this dataset, then you should have posted it here for everyone to see and extend the deadline by 1-2 weeks.", "votes": 7 }, { "id": 1801132, "postDate": "2022-05-25T13:37:35.927Z", "content": "But here is where it gets complicated. BirdNET is not a dataset, it's a trained model which is publicly available, so the Macaulay Library rule does not apply. It was trained on ML data, but it is not the same as ML data. Yet, we should have anticipated the implications when publishing the model which is simply something that we missed. Otherwise, we would have been able to implement a \"no BirdNET\" rule.
", "rawMarkdown": "But here is where it gets complicated. BirdNET is not a dataset, it's a trained model which is publicly available, so the Macaulay Library rule does not apply. It was trained on ML data, but it is not the same as ML data. Yet, we should have anticipated the implications when publishing the model which is simply something that we missed. Otherwise, we would have been able to implement a \"no BirdNET\" rule.", "votes": -1 }, { "id": 1801140, "postDate": "2022-05-25T13:50:52.350Z", "content": "\n\nBirdNET is not a dataset, it's a trained model which is publicly available, so the Macaulay Library rule does not apply
\n
How do you come to that conclusion? A model always has the same license as the data it was trained on. At least that is my knowledge. Otherwise you could always train a model on unlicensed data and then say, well, I only use the model from now on, but not the data.
\nBut anyways, you have the model published as non-commercial only, so it was fine to not be used here in this competition.
", "rawMarkdown": "> BirdNET is not a dataset, it's a trained model which is publicly available, so the Macaulay Library rule does not apply\n\nHow do you come to that conclusion? A model always has the same license as the data it was trained on. At least that is my knowledge. Otherwise you could always train a model on unlicensed data and then say, well, I only use the model from now on, but not the data.\n\nBut anyways, you have the model published as non-commercial only, so it was fine to not be used here in this competition.", "votes": 4 }, { "id": 1801149, "postDate": "2022-05-25T13:55:45.447Z", "content": "The confusion is understandable, and our first response to the request to use BirdNET for this competition was to not allow it due to the non-commercial clause in the license. Yet, we realized that parts of the training data have the exact same license and that we cannot restrict the use of BirdNET based on the license without being inconsistent. We will do a better job next time - that's definitively a lesson learned.
", "rawMarkdown": "The confusion is understandable, and our first response to the request to use BirdNET for this competition was to not allow it due to the non-commercial clause in the license. Yet, we realized that parts of the training data have the exact same license and that we cannot restrict the use of BirdNET based on the license without being inconsistent. We will do a better job next time - that's definitively a lesson learned.", "votes": -1 }, { "id": 1801160, "postDate": "2022-05-25T13:58:57.260Z", "content": "But, correct me if I am wrong, as a host you can always disallow specific models and data, regardless of the circumstances.
", "rawMarkdown": "But, correct me if I am wrong, as a host you can always disallow specific models and data, regardless of the circumstances. ", "votes": 4 }, { "id": 1801165, "postDate": "2022-05-25T14:00:32.907Z", "content": "Guys, I think there's no need to quarrel over this topic. I think the main problem is I found the solution too late otherwise I would make it publicly available, and no one asked about the BirdNet. I don't know how many teams found the solution (perhaps only top-1 and us). If this solution impact too many teams, those teams should stand out to say it. It's a learning process for us as well as the host, things beyond expectation always happen. The best option is just never to allow external data, and I believe the host will organize better in the future.
", "rawMarkdown": "Guys, I think there's no need to quarrel over this topic. I think the main problem is I found the solution too late otherwise I would make it publicly available, and no one asked about the BirdNet. I don't know how many teams found the solution (perhaps only top-1 and us). If this solution impact too many teams, those teams should stand out to say it. It's a learning process for us as well as the host, things beyond expectation always happen. The best option is just never to allow external data, and I believe the host will organize better in the future.", "votes": -3 }, { "id": 1801174, "postDate": "2022-05-25T14:09:08.993Z", "content": "Yeah, someone could have used BirdNet last year and still win. But nobody found it (including us), until this competition. So live and learn. Now we can prepare better for the next competition(s)
", "rawMarkdown": "Yeah, someone could have used BirdNet last year and still win. But nobody found it (including us), until this competition. So live and learn. Now we can prepare better for the next competition(s)", "votes": -6 }, { "id": 1801176, "postDate": "2022-05-25T14:11:42.353Z", "content": "Did you use last year's version of BirdNet?
", "rawMarkdown": "Did you use last year's version of BirdNet?", "votes": 5 }, { "id": 1801177, "postDate": "2022-05-25T14:12:23.540Z", "content": "BirdNet in this form only exists since way after last year's competition. It is only useful because it has been trained on extra data that was not available to other participants.
\nAnd how does this help me to prepare better for the next competition?
", "rawMarkdown": "BirdNet in this form only exists since way after last year's competition. It is only useful because it has been trained on extra data that was not available to other participants.\n\nAnd how does this help me to prepare better for the next competition?", "votes": 6 }, { "id": 1801178, "postDate": "2022-05-25T14:13:33.700Z", "content": "@ivanpan Did you run your \"test\" on new edition of BirdNet, or correct edition (dated before 2021 competition) ?
", "rawMarkdown": "@ivanpan Did you run your \"test\" on new edition of BirdNet, or correct edition (dated before 2021 competition) ? ", "votes": 2 }, { "id": 1801182, "postDate": "2022-05-25T14:13:44.970Z", "content": "We will make sure that the competition data equals the data we used for BirdNET. This way, using BirdNET will not result in an advantage.
", "rawMarkdown": "We will make sure that the competition data equals the data we used for BirdNET. This way, using BirdNET will not result in an advantage.", "votes": 2 }, { "id": 1801189, "postDate": "2022-05-25T14:17:07.047Z", "content": "The first commit on the Birdnet repo (not including the lasagne code one) is from September 2021, so I doubt that it was useful in a competition that ended in June 2021.
\nThis just sets an extremely dangerous precedent for all future Kaggle competitions. What I learned from this is that I cannot use data that is not licensed CC-BY(-SA), but I am allowed to use a model trained on data that does not have that license as long as it is publicly available? I can buy a dataset, give it a friend, ask him to train a model on it and put it on a repo, and then I am allowed to use that in a competition. Clearly this is circumventing a restriction that is put onto competitors.
", "rawMarkdown": "The first commit on the Birdnet repo (not including the lasagne code one) is from September 2021, so I doubt that it was useful in a competition that ended in June 2021. \n\nThis just sets an extremely dangerous precedent for all future Kaggle competitions. What I learned from this is that I cannot use data that is not licensed CC-BY(-SA), but I am allowed to use a model trained on data that does not have that license as long as it is publicly available? I can buy a dataset, give it a friend, ask him to train a model on it and put it on a repo, and then I am allowed to use that in a competition. Clearly this is circumventing a restriction that is put onto competitors.", "votes": 10 }, { "id": 1801194, "postDate": "2022-05-25T14:18:57.163Z", "content": "BirdNet in this form only exists since way after last year's competition -- anyone even tried this repo and this for the last year's competition? This is before 2021 (even 3 years ago), I don't know how they perform but it seems no one even tried it.
\nprepare better for the next competition -- I think this is for both the host and us
\"But here is where it gets complicated. BirdNET is not a dataset, it's a trained model which is publicly available, so the Macaulay Library rule does not apply. It was trained on ML data, but it is not the same as ML data.\"
\n@stefankahl I think that last year I did ask if Macaulay data was usable and the answer was no. Now you are saying that using a model trained on that data is fine. This is weird. What is the difference between a model trained by you and a model trained by a competition participant?
\nI did find data for the birds in this comp on macalulay library, then remembered I could not use it and gave up on the competition. I'm glad I did not spent time on it.
@stefankahl at least, hundreds of competitors got hurt because of this pretrained model using data not allowed in the competition. This model should not be allowed, even in the last day of the competiton, because it simply hurt the rules. I got disqualified from other competition for much less than that.
", "rawMarkdown": "@stefankahl at least, hundreds of competitors got hurt because of this pretrained model using data not allowed in the competition. This model should not be allowed, even in the last day of the competiton, because it simply hurt the rules. I got disqualified from other competition for much less than that.", "votes": 12 }, { "id": 1801285, "postDate": "2022-05-25T15:31:19.117Z", "content": "@cpmpml the way I understand, a model trained on some data does not necessarily need to have the same license as the data it was trained on. So considering this, if someone (e.g. us as hosts) trains a model and publishes it so that everyone can use it, it doesn't matter which license the training data had. Yet, I fully understand that this is a weird situation and at least a legal gray area, and we have to be more thoughtful next year to avoid another situation like this. We probably would have had time to figure out a good solution if we had been aware earlier during the competition.
", "rawMarkdown": "@cpmpml the way I understand, a model trained on some data does not necessarily need to have the same license as the data it was trained on. So considering this, if someone (e.g. us as hosts) trains a model and publishes it so that everyone can use it, it doesn't matter which license the training data had. Yet, I fully understand that this is a weird situation and at least a legal gray area, and we have to be more thoughtful next year to avoid another situation like this. We probably would have had time to figure out a good solution if we had been aware earlier during the competition." }, { "id": 1801305, "postDate": "2022-05-25T15:50:44.137Z", "content": "Here you go future BirdClef2023, no need to thank me:
\nclass GlebNet:\n def __init__(self, macaulay_data):\n self.data = macaulay_data\n self.important_parameter = True\n def forward(self, idx):\n return self.data[idx]\nIt has already been said many times that a mistake was made. Now it can not be corrected and the 1st and 2nd place is not to blame for this. And in principle nothing will change (except for 3-5 places).
", "rawMarkdown": "It has already been said many times that a mistake was made. Now it can not be corrected and the 1st and 2nd place is not to blame for this. And in principle nothing will change (except for 3-5 places). ", "votes": 6 }, { "id": 1801483, "postDate": "2022-05-25T19:43:38.267Z", "content": "I fully agree. The LB cannot be changed, as organizers gave permission explicitly. However, this should be put into rules for future competitions, bcs as I mentioned before, this sets a dangerous precedent.
", "rawMarkdown": "I fully agree. The LB cannot be changed, as organizers gave permission explicitly. However, this should be put into rules for future competitions, bcs as I mentioned before, this sets a dangerous precedent.", "votes": 7 }, { "id": 1805103, "postDate": "2022-05-29T19:34:58.483Z", "content": "\"a model trained on some data does not necessarily need to have the same license as the data it was trained on\"
\n@stefankahl You should check with a lawyer. Intuition is not a good predictor of the meaning of legal terms.
\nIn a similar case I know about (different dataset and different pretrained model though, but similar data license), lawyers considered that the license limits on training data were considered to apply to the model trained on the data.
\nAnyway, my point was not on the legal side as I am no more a lawyer than you are. My point was about the distortion in training data. Using your model means using Macaulay Library as training data. Something we were told not to do.
\nI feel it is unfair, and this is why i am happy i did not spend time in this competition.
\nOn a related note, I don't blame people who used your model once you allowed them to. That's not the issue I am discussing.
\nI waited a bit to make sure there was no hard feelings, but I think it had to be said. For your own safety (from a legal point of view).
", "rawMarkdown": "\"a model trained on some data does not necessarily need to have the same license as the data it was trained on\"\n\n@stefankahl You should check with a lawyer. Intuition is not a good predictor of the meaning of legal terms.\n\nIn a similar case I know about (different dataset and different pretrained model though, but similar data license), lawyers considered that the license limits on training data were considered to apply to the model trained on the data.\n\nAnyway, my point was not on the legal side as I am no more a lawyer than you are. My point was about the distortion in training data. Using your model means using Macaulay Library as training data. Something we were told not to do.\n\nI feel it is unfair, and this is why i am happy i did not spend time in this competition.\n\nOn a related note, I don't blame people who used your model once you allowed them to. That's not the issue I am discussing.\n\nI waited a bit to make sure there was no hard feelings, but I think it had to be said. For your own safety (from a legal point of view).", "votes": 1 } ] }, { "id": 1801566, "postDate": "2022-05-25T22:07:45Z", "content": "Despite the fact that the host mistakenly allow the BirdNet model, it's execrable that some Kagglers can find a leak-like situation and secretly exploit it for gold and money prize. This is not a glorious habit and anyone doing it should not be exalted.
\nCheers for @maxhalford who could win the RIID 100k worth competition but chooses to timely disclose the leakage.
", "rawMarkdown": "Despite the fact that the host mistakenly allow the BirdNet model, it's execrable that some Kagglers can find a leak-like situation and secretly exploit it for gold and money prize. This is not a glorious habit and anyone doing it should not be exalted.\n\nCheers for @maxhalford who could win the [RIID 100k worth competition](https://www.kaggle.com/competitions/riiid-test-answer-prediction/discussion/189437) but chooses to timely disclose the leakage.", "votes": 5 }, { "id": 1800559, "postDate": "2022-05-25T03:18:34.827Z", "content": "It reminds me of the 2nd place solution of Cassava competition.
", "rawMarkdown": "It reminds me of the [2nd place solution of Cassava competition](https://www.kaggle.com/competitions/cassava-leaf-disease-classification/discussion/220898).", "votes": 3, "replies": [ { "id": 1800569, "postDate": "2022-05-25T03:38:31.283Z", "content": "Yes exactly,
", "rawMarkdown": "Yes exactly, ", "votes": 1 } ] }, { "id": 1800666, "postDate": "2022-05-25T06:08:01.630Z", "content": "Seems like this competition was useless:( Best results achieved by existed model which also can be used in real world instead of ensembles which we train.
", "rawMarkdown": "Seems like this competition was useless:( Best results achieved by existed model which also can be used in real world instead of ensembles which we train.", "votes": 2, "replies": [ { "id": 1801103, "postDate": "2022-05-25T12:56:26.457Z", "content": "No, it was not useless. We will investigate all write-ups, notebooks and working notes to add to our existing model, drawing ideas from everyone's effort. We want to improve bird sound detection and with your participation in this competition, we can do that.
", "rawMarkdown": "No, it was not useless. We will investigate all write-ups, notebooks and working notes to add to our existing model, drawing ideas from everyone's effort. We want to improve bird sound detection and with your participation in this competition, we can do that.", "votes": 2 } ] }, { "id": 1805062, "postDate": "2022-05-29T18:26:16.937Z", "content": "\n\nif a bird is detected in the previous or next 5s, we will rank the probs and add the top5 bird species to the current detected birds.
\n
It's really amazing that the information of adjacent time segments can bring up such a boost.
", "rawMarkdown": "> if a bird is detected in the previous or next 5s, we will rank the probs and add the top5 bird species to the current detected birds.\n\nIt's really amazing that the information of adjacent time segments can bring up such a boost. ", "votes": 1, "replies": [ { "id": 1805079, "postDate": "2022-05-29T19:00:41.153Z", "content": "I've also noticed in some recent work that averaging adjacent time windows can be quite helpful. This might in part be due to calls cut off at the window; you may get a misclassification with a partial-call which becomes easier with a small time-shift.
\nI've also noticed that this averaging tends to improve precision at the expense of recall. If any time-shift creates more ambiguity, then the model's score will decrease with time-shift averaging.
", "rawMarkdown": "I've also noticed in some recent work that averaging adjacent time windows can be quite helpful. This might in part be due to calls cut off at the window; you may get a misclassification with a partial-call which becomes easier with a small time-shift.\n\nI've also noticed that this averaging tends to improve precision at the expense of recall. If any time-shift creates more ambiguity, then the model's score will decrease with time-shift averaging.", "votes": 3 } ] }, { "id": 1800572, "postDate": "2022-05-25T03:44:07.690Z", "content": "Congrats and thanks for sharing the solution!
\nI can't always find this kind of information😂
Congratulations with your score
\nAnd I'm happy that you are the only team who found and used it :)
the first place also found this
", "rawMarkdown": "the first place also found this", "votes": 3 } ] }, { "id": 1801227, "postDate": "2022-05-25T14:34:07.357Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1801507, "postDate": "2022-05-25T20:08:57.670Z", "content": "Thanks for sharing this
", "rawMarkdown": "Thanks for sharing this" }, { "id": 1801395, "postDate": "2022-05-25T17:26:08.443Z", "content": "Thanks for sharing the solution.
", "rawMarkdown": "Thanks for sharing the solution." }, { "id": 1801208, "postDate": "2022-05-25T14:26:13.567Z", "content": "Thanks for sharing and Congrats!
", "rawMarkdown": "Thanks for sharing and Congrats!" } ], "comments": [ { "id": 1800598, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2022-05-25T04:25:22.333000", "content": "Congratulations!
\nI do feel this should have been disclosed by @stefankahl as it seems like the BirdNET model has been trained at least partially on the test data used in this competition.
", "votes": 10, "replies": [ { "id": 1801102, "author_name": "Stefan Kahl", "author_url": "", "post_date": "2022-05-25T12:54:06", "content": "No, it wasn't. Yet, we used other soundscape data and I never realized that this might be an issue for this competition. My guess is, with competition data only, BirdNET would not have scored in the top 10.
", "votes": 4, "replies": [] } ] }, { "id": 1800551, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2022-05-25T03:07:21.853000", "content": "thanks for the story. it is very interesting whether the host used more training records to train the model or not
", "votes": 7, "replies": [ { "id": 1800565, "author_name": "Ari", "author_url": "", "post_date": "2022-05-25T03:27:03.180000", "content": "The host used many more. I think it was somewhere around 2434 species in the train set.
", "votes": 2, "replies": [] }, { "id": 1800634, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2022-05-25T05:32:27.873000", "content": "I'm talking about our species. did they use, for example, the test data of this competition for training
", "votes": 3, "replies": [] } ] }, { "id": 1801322, "author_name": "Zhongkai Shangguan", "author_url": "", "post_date": "2022-05-25T16:02:52.193000", "content": "Someone asked me why didn't you contact the host already those 20 days ago when you found it or at least 12 days ago when you wrote your comment? -- Because, as I said, we didn't manage to select BirdNet as our final submission so I didn't think that's an issue and did not contact the host. We keep training our own model until the last minute and our own one is ranking 9 which is also in the gold zone. To be honest, we didn't think the BirdNet is allowed until the host said it is allowed. I understand our solution is weird under some gray area, but I don't understand why someone downvotes my comments (and downvotes the 1st place team members comments). This situation happened in Cassava Leaf Disease Classification\n as well where someone use pre-trained model and reached top places. Kaggle should manage to develop more clear rules and we should keep looking forward to helping avoid this case happen again.
BirdNet was a huge revelation to me in this competition. I couldn't believe how well it works, so decided to test on last year competition where we got 8th place. Easily managed to get top-1 on private there, but I believe public scores were around silver zone, that's why I wasn't 100% sure it would perform that well on private here.
\nGlad I was wrong and now we can see that BirdNet gets top-1 on private in both years.
\nFor the sake of honesty, we actually ensembled BirdNet with our models to get some additional boost and spent a couple of months trying to perfect those models (if I'm not mistaken, they score 0.78-0.79 on private). I believe we will be sharing results soon. But anyway, thanks @stefankahl for this competition. I mentioned this last year, I'm gonna mention this again - great orgs! One of the very few on Kaggle who provide great resources here (including notebooks), and simply answer questions to participants without abandoning a competition 1 week in.
\nThanks!
", "votes": 7, "replies": [ { "id": 1800653, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2022-05-25T05:51:43.273000", "content": "I think this network works better because they had more data. not a few records, as we have.
", "votes": 7, "replies": [] }, { "id": 1800661, "author_name": "anthony", "author_url": "", "post_date": "2022-05-25T06:05:31.307000", "content": "Probably BirdNet also trained on data without noisy labels and maybe even on data which used for eval here.
", "votes": 1, "replies": [] }, { "id": 1800663, "author_name": "Ivan Panshin", "author_url": "", "post_date": "2022-05-25T06:06:38.027000", "content": "@vlomme I think you're right. The model itself is quite quite good, but combine it with a lot of data - and that's a recipe for SOTA.
", "votes": 0, "replies": [] }, { "id": 1800677, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2022-05-25T06:11:14.007000", "content": "It's just that if non-public data and, possibly, test data were used for it training, then it was pointless to compete with such a model. It's very strange and I'm sad
", "votes": 7, "replies": [] }, { "id": 1800679, "author_name": "UEMU", "author_url": "", "post_date": "2022-05-25T06:14:01.277000", "content": "I think that the conclusion of this competition is
\n\"Let's increase the number of data without training with a few amount of data\"
\nIt wasn't what I expected.
Hold up, let me get this straight. We do not allowed to use Macaulay Library data, but allowed to use BirdNet, trained on Macaulay Library data? Can I train my own public GlebNet on Macaulay Library data (with all agreement that needed), and use it?
", "votes": 11, "replies": [ { "id": 1801090, "author_name": "Markus Vogelbacher", "author_url": "", "post_date": "2022-05-25T12:27:40.717000", "content": "If it's okay to use BirdNET it should be alright to use your own model as long as it's publicly available, no matter what data it's been trained on. Is this correct @stefankahl ?
", "votes": 1, "replies": [] } ] }, { "id": 1800690, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2022-05-25T06:29:22.980000", "content": "No you can't.
", "votes": 1, "replies": [] }, { "id": 1800699, "author_name": "slime", "author_url": "", "post_date": "2022-05-25T06:35:24.997000", "content": "@bakeryproducts couldn't agree more, in the forum there were countless discussions that explicitly prohibited the use of Macaulay Library and audios from xeno-canto w/o appropriate license (e.g. see this post)
", "votes": 4, "replies": [] }, { "id": 1800723, "author_name": "Gleb", "author_url": "", "post_date": "2022-05-25T06:59:33.683000", "content": "To fellow participants, imagine scenario: someone creating topic \"BirdNet is kinda good\" two days ago. Month(s) of time -> poof -> 'i would like to thank kaggle for this competition and great orgs, …'
", "votes": 4, "replies": [] }, { "id": 1800724, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2022-05-25T07:00:44.813000", "content": "Around which point in the competition did you discover BirdNET?
", "votes": 1, "replies": [] }, { "id": 1800728, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2022-05-25T07:05:28.450000", "content": "no questions to the participants, it is strange that the organizers allowed the use of this model
", "votes": 2, "replies": [] }, { "id": 1800730, "author_name": "Gleb", "author_url": "", "post_date": "2022-05-25T07:09:08.577000", "content": "@vlomme thats my point, rules were made in such a way that it all could end disastrously
", "votes": 6, "replies": [] }, { "id": 1800732, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2022-05-25T07:12:27.660000", "content": "I agree
", "votes": 1, "replies": [] }, { "id": 1800793, "author_name": "Psi", "author_url": "", "post_date": "2022-05-25T08:15:55.780000", "content": "I am very confident (although I have no proof) that BirdNet only works so well because it was either trained on more data, or even the test data itself. So I am not sure if the host really wins here… It looks like a big leak.
", "votes": 19, "replies": [] }, { "id": 1800809, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2022-05-25T08:26:26.350000", "content": "It is a fact that BirdNet is trained on a larger amount of data (but hardly on a test). In a competition where there were few examples (several per class) this is a non-competitive advantage
", "votes": 3, "replies": [] }, { "id": 1800830, "author_name": "Psi", "author_url": "", "post_date": "2022-05-25T08:44:38.680000", "content": "And this extra data Macaulay Library was not allowed to be used here in this competition but the BirdNet model was trained on it? Wouldn't that mean that BirdNet model should not be allowed?
", "votes": 13, "replies": [] }, { "id": 1800841, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2022-05-25T08:55:22.787000", "content": "Correct. But unfortunately, permission seems to have been given by the organizers to use this in a github issue 2 days ago.
", "votes": 9, "replies": [] }, { "id": 1800842, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2022-05-25T08:55:45.820000", "content": "Yes, that's right. For some reason the organizer allowed the model to be used
", "votes": 2, "replies": [] }, { "id": 1800937, "author_name": "S. Tomizawa", "author_url": "", "post_date": "2022-05-25T10:06:31.677000", "content": "I think the organizers should clarify whether the Macaulay Library or competition test data were used or not in the BirdNet. In any case, I feel empty.
", "votes": 6, "replies": [] }, { "id": 1800946, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2022-05-25T10:09:57.753000", "content": "They mention it explicitly in the linked issue that they use Macaulay data!
", "votes": 4, "replies": [] }, { "id": 1800962, "author_name": "Psi", "author_url": "", "post_date": "2022-05-25T10:16:34.937000", "content": "Wow, I feel for you. Crazy to make such a ground-breaking decision in a github issue one day before end of competition…
", "votes": 10, "replies": [] }, { "id": 1801094, "author_name": "Stefan Kahl", "author_url": "", "post_date": "2022-05-25T12:32:09.127000", "content": "Yes, I also believe that BirdNET works so well because of additional training data (we indeed used Macaulay Library and another soundscape set from Hawaii). It would probably not score in the top 10 with only the competition data. Unfortunately, we only realized that people would use BirdNET one day before the deadline, and it was too late to implement any last-minute rule changes. This is just something that slipped our minds when organizing the competition.
", "votes": -4, "replies": [] }, { "id": 1801096, "author_name": "Psi", "author_url": "", "post_date": "2022-05-25T12:34:42.807000", "content": "I am not sure I am following @stefankahl - but you explicitly allowed it one day before deadline, there was no reason to as it was disallowed before that. So you actually implemented a last-minute rule change.
", "votes": 5, "replies": [] }, { "id": 1801100, "author_name": "Stefan Kahl", "author_url": "", "post_date": "2022-05-25T12:47:53.713000", "content": "No, it was not disallowed before that, it's a public repo and anyone could have used it. Even the fact that it uses Macaulay data doesn't change that. If we had been aware earlier, we would have implemented a rule change at the beginning of the competition. It's like a vision model trained on the non-public Google JFT-300M dataset used for an ImageNet competition. However, we are interested in alternative approaches and thus would have limited the use of BirdNET - it just never occurred to us that this might be a useful rule addition.
", "votes": -2, "replies": [] }, { "id": 1801129, "author_name": "Psi", "author_url": "", "post_date": "2022-05-25T13:34:12.047000", "content": "Your first statement in this github repo was very definite that it was not allowed. Also, according to reports of others, the Macaulay dataset was explicitly forbidden to use, so what sense does it make to allow a separate model that has been trained on this dataset?
\nIf a competition only allows external data that can be commercially used, which is the default, then also models trained on non-commercial-only data are obviously not allowed, this also applies to any of the standard pretrained models. If that is actually always enforced, I am not sure.
\nIf you really suddenly wanted to allow this dataset, then you should have posted it here for everyone to see and extend the deadline by 1-2 weeks.
", "votes": 7, "replies": [] }, { "id": 1801132, "author_name": "Stefan Kahl", "author_url": "", "post_date": "2022-05-25T13:37:35.927000", "content": "But here is where it gets complicated. BirdNET is not a dataset, it's a trained model which is publicly available, so the Macaulay Library rule does not apply. It was trained on ML data, but it is not the same as ML data. Yet, we should have anticipated the implications when publishing the model which is simply something that we missed. Otherwise, we would have been able to implement a \"no BirdNET\" rule.
", "votes": -1, "replies": [] }, { "id": 1801140, "author_name": "Psi", "author_url": "", "post_date": "2022-05-25T13:50:52.350000", "content": "\n\nBirdNET is not a dataset, it's a trained model which is publicly available, so the Macaulay Library rule does not apply
\n
How do you come to that conclusion? A model always has the same license as the data it was trained on. At least that is my knowledge. Otherwise you could always train a model on unlicensed data and then say, well, I only use the model from now on, but not the data.
\nBut anyways, you have the model published as non-commercial only, so it was fine to not be used here in this competition.
", "votes": 4, "replies": [] }, { "id": 1801149, "author_name": "Stefan Kahl", "author_url": "", "post_date": "2022-05-25T13:55:45.447000", "content": "The confusion is understandable, and our first response to the request to use BirdNET for this competition was to not allow it due to the non-commercial clause in the license. Yet, we realized that parts of the training data have the exact same license and that we cannot restrict the use of BirdNET based on the license without being inconsistent. We will do a better job next time - that's definitively a lesson learned.
", "votes": -1, "replies": [] }, { "id": 1801160, "author_name": "Psi", "author_url": "", "post_date": "2022-05-25T13:58:57.260000", "content": "But, correct me if I am wrong, as a host you can always disallow specific models and data, regardless of the circumstances.
", "votes": 4, "replies": [] }, { "id": 1801165, "author_name": "Zhongkai Shangguan", "author_url": "", "post_date": "2022-05-25T14:00:32.907000", "content": "Guys, I think there's no need to quarrel over this topic. I think the main problem is I found the solution too late otherwise I would make it publicly available, and no one asked about the BirdNet. I don't know how many teams found the solution (perhaps only top-1 and us). If this solution impact too many teams, those teams should stand out to say it. It's a learning process for us as well as the host, things beyond expectation always happen. The best option is just never to allow external data, and I believe the host will organize better in the future.
", "votes": -3, "replies": [] }, { "id": 1801174, "author_name": "Ivan Panshin", "author_url": "", "post_date": "2022-05-25T14:09:08.993000", "content": "Yeah, someone could have used BirdNet last year and still win. But nobody found it (including us), until this competition. So live and learn. Now we can prepare better for the next competition(s)
", "votes": -6, "replies": [] }, { "id": 1801176, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2022-05-25T14:11:42.353000", "content": "Did you use last year's version of BirdNet?
", "votes": 5, "replies": [] }, { "id": 1801177, "author_name": "Psi", "author_url": "", "post_date": "2022-05-25T14:12:23.540000", "content": "BirdNet in this form only exists since way after last year's competition. It is only useful because it has been trained on extra data that was not available to other participants.
\nAnd how does this help me to prepare better for the next competition?
", "votes": 6, "replies": [] }, { "id": 1801178, "author_name": "Gleb", "author_url": "", "post_date": "2022-05-25T14:13:33.700000", "content": "@ivanpan Did you run your \"test\" on new edition of BirdNet, or correct edition (dated before 2021 competition) ?
", "votes": 2, "replies": [] }, { "id": 1801182, "author_name": "Stefan Kahl", "author_url": "", "post_date": "2022-05-25T14:13:44.970000", "content": "We will make sure that the competition data equals the data we used for BirdNET. This way, using BirdNET will not result in an advantage.
", "votes": 2, "replies": [] }, { "id": 1801189, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2022-05-25T14:17:07.047000", "content": "The first commit on the Birdnet repo (not including the lasagne code one) is from September 2021, so I doubt that it was useful in a competition that ended in June 2021.
\nThis just sets an extremely dangerous precedent for all future Kaggle competitions. What I learned from this is that I cannot use data that is not licensed CC-BY(-SA), but I am allowed to use a model trained on data that does not have that license as long as it is publicly available? I can buy a dataset, give it a friend, ask him to train a model on it and put it on a repo, and then I am allowed to use that in a competition. Clearly this is circumventing a restriction that is put onto competitors.
", "votes": 10, "replies": [] }, { "id": 1801194, "author_name": "Zhongkai Shangguan", "author_url": "", "post_date": "2022-05-25T14:18:57.163000", "content": "BirdNet in this form only exists since way after last year's competition -- anyone even tried this repo and this for the last year's competition? This is before 2021 (even 3 years ago), I don't know how they perform but it seems no one even tried it.
\nprepare better for the next competition -- I think this is for both the host and us
\"But here is where it gets complicated. BirdNET is not a dataset, it's a trained model which is publicly available, so the Macaulay Library rule does not apply. It was trained on ML data, but it is not the same as ML data.\"
\n@stefankahl I think that last year I did ask if Macaulay data was usable and the answer was no. Now you are saying that using a model trained on that data is fine. This is weird. What is the difference between a model trained by you and a model trained by a competition participant?
\nI did find data for the birds in this comp on macalulay library, then remembered I could not use it and gave up on the competition. I'm glad I did not spent time on it.
@stefankahl at least, hundreds of competitors got hurt because of this pretrained model using data not allowed in the competition. This model should not be allowed, even in the last day of the competiton, because it simply hurt the rules. I got disqualified from other competition for much less than that.
", "votes": 12, "replies": [] }, { "id": 1801285, "author_name": "Stefan Kahl", "author_url": "", "post_date": "2022-05-25T15:31:19.117000", "content": "@cpmpml the way I understand, a model trained on some data does not necessarily need to have the same license as the data it was trained on. So considering this, if someone (e.g. us as hosts) trains a model and publishes it so that everyone can use it, it doesn't matter which license the training data had. Yet, I fully understand that this is a weird situation and at least a legal gray area, and we have to be more thoughtful next year to avoid another situation like this. We probably would have had time to figure out a good solution if we had been aware earlier during the competition.
", "votes": 0, "replies": [] }, { "id": 1801305, "author_name": "Gleb", "author_url": "", "post_date": "2022-05-25T15:50:44.137000", "content": "Here you go future BirdClef2023, no need to thank me:
\nclass GlebNet:\n def __init__(self, macaulay_data):\n self.data = macaulay_data\n self.important_parameter = True\n def forward(self, idx):\n return self.data[idx]\nIt has already been said many times that a mistake was made. Now it can not be corrected and the 1st and 2nd place is not to blame for this. And in principle nothing will change (except for 3-5 places).
", "votes": 6, "replies": [] }, { "id": 1801483, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2022-05-25T19:43:38.267000", "content": "I fully agree. The LB cannot be changed, as organizers gave permission explicitly. However, this should be put into rules for future competitions, bcs as I mentioned before, this sets a dangerous precedent.
", "votes": 7, "replies": [] }, { "id": 1805103, "author_name": "CPMP", "author_url": "", "post_date": "2022-05-29T19:34:58.483000", "content": "\"a model trained on some data does not necessarily need to have the same license as the data it was trained on\"
\n@stefankahl You should check with a lawyer. Intuition is not a good predictor of the meaning of legal terms.
\nIn a similar case I know about (different dataset and different pretrained model though, but similar data license), lawyers considered that the license limits on training data were considered to apply to the model trained on the data.
\nAnyway, my point was not on the legal side as I am no more a lawyer than you are. My point was about the distortion in training data. Using your model means using Macaulay Library as training data. Something we were told not to do.
\nI feel it is unfair, and this is why i am happy i did not spend time in this competition.
\nOn a related note, I don't blame people who used your model once you allowed them to. That's not the issue I am discussing.
\nI waited a bit to make sure there was no hard feelings, but I think it had to be said. For your own safety (from a legal point of view).
", "votes": 1, "replies": [] } ] }, { "id": 1801566, "author_name": "kkiller", "author_url": "", "post_date": "2022-05-25T22:07:45", "content": "Despite the fact that the host mistakenly allow the BirdNet model, it's execrable that some Kagglers can find a leak-like situation and secretly exploit it for gold and money prize. This is not a glorious habit and anyone doing it should not be exalted.
\nCheers for @maxhalford who could win the RIID 100k worth competition but chooses to timely disclose the leakage.
", "votes": 5, "replies": [] }, { "id": 1800559, "author_name": "S. Tomizawa", "author_url": "", "post_date": "2022-05-25T03:18:34.827000", "content": "It reminds me of the 2nd place solution of Cassava competition.
", "votes": 3, "replies": [ { "id": 1800569, "author_name": "Zhongkai Shangguan", "author_url": "", "post_date": "2022-05-25T03:38:31.283000", "content": "Yes exactly,
", "votes": 1, "replies": [] } ] }, { "id": 1800666, "author_name": "anthony", "author_url": "", "post_date": "2022-05-25T06:08:01.630000", "content": "Seems like this competition was useless:( Best results achieved by existed model which also can be used in real world instead of ensembles which we train.
", "votes": 2, "replies": [ { "id": 1801103, "author_name": "Stefan Kahl", "author_url": "", "post_date": "2022-05-25T12:56:26.457000", "content": "No, it was not useless. We will investigate all write-ups, notebooks and working notes to add to our existing model, drawing ideas from everyone's effort. We want to improve bird sound detection and with your participation in this competition, we can do that.
", "votes": 2, "replies": [] } ] }, { "id": 1805062, "author_name": "WANG XIN", "author_url": "", "post_date": "2022-05-29T18:26:16.937000", "content": "\n\nif a bird is detected in the previous or next 5s, we will rank the probs and add the top5 bird species to the current detected birds.
\n
It's really amazing that the information of adjacent time segments can bring up such a boost.
", "votes": 1, "replies": [ { "id": 1805079, "author_name": "Tom Denton", "author_url": "", "post_date": "2022-05-29T19:00:41.153000", "content": "I've also noticed in some recent work that averaging adjacent time windows can be quite helpful. This might in part be due to calls cut off at the window; you may get a misclassification with a partial-call which becomes easier with a small time-shift.
\nI've also noticed that this averaging tends to improve precision at the expense of recall. If any time-shift creates more ambiguity, then the model's score will decrease with time-shift averaging.
", "votes": 3, "replies": [] } ] }, { "id": 1800572, "author_name": "atfujita", "author_url": "", "post_date": "2022-05-25T03:44:07.690000", "content": "Congrats and thanks for sharing the solution!
\nI can't always find this kind of information😂
Congratulations with your score
\nAnd I'm happy that you are the only team who found and used it :)
the first place also found this
", "votes": 3, "replies": [] } ] }, { "id": 1801227, "author_name": "", "author_url": "", "post_date": "2022-05-25T14:34:07.357000", "content": "", "votes": 0, "replies": [] }, { "id": 1801507, "author_name": "gilty15", "author_url": "", "post_date": "2022-05-25T20:08:57.670000", "content": "Thanks for sharing this
", "votes": 0, "replies": [] }, { "id": 1801395, "author_name": "jackmtlee", "author_url": "", "post_date": "2022-05-25T17:26:08.443000", "content": "Thanks for sharing the solution.
", "votes": 0, "replies": [] }, { "id": 1801208, "author_name": "Thịnh Lâm", "author_url": "", "post_date": "2022-05-25T14:26:13.567000", "content": "Thanks for sharing and Congrats!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1800539": "As I promised, I will publish my solution right after the competition. This will be a long topic as the two solutions are totally different. As kaggle says > You've been sharing this link too often. We prevent redundant posts to reduce spam. So I put some link in the code block.\n\nBefore the solutions, thanks to my teammate @jionie, our second gold medal, and first-time money zone!!!!!!!!!! Cheers!!!!!!!!!!\n\nI will start with our [Private #7/8 (potential)] Solution first since I **promise **you will say the \"F\" word after seeing the other solution.\n\n**[Private #7/8 (potential)] Solution**\n\nTraining:\nOur first solution is based on @kaerunantoka 's [public notebook](https://www.kaggle.com/code/kaerunantoka/birdclef2022-use-2nd-label-f0) and last year's second-place solution. The training code is pretty the mixture of the two pipelines mentioned above. \nSome key points to mention:\n1. data augmentation:\n`OneOf([\n Gain(min_gain_in_db=-15, max_gain_in_db=15, p=0.8),\n GainTransition(min_gain_in_db=-15, max_gain_in_db=15, p=0.8),\n ]),\n OneOf(\n [\n NoiseInjection(p=1, max_noise_level=0.04),\n GaussianNoise(p=1, min_snr=5, max_snr=20),\n PinkNoise(p=1, min_snr=5, max_snr=20),\n AddGaussianNoise(min_amplitude=0.0001, max_amplitude=0.03, p=0.5),\n AddGaussianSNR(min_snr_in_db=5, max_snr_in_db=15, p=0.5),\n ],\n p=0.3,\n ),\n AddBackgroundNoise(\n sounds_path=self.config.BACKGROUND_PATH, min_snr_in_db=0, max_snr_in_db=2, p=0.5\n ),\n Normalize(p=1),`\n\n2. cut mix + mix up\n3. loss function: BCEWithLogits + BCEFocal2WayLoss\n4. hypers setting (n_fft, n_mels, hop_length) refer to our inference notebook.\n\nOur inference kernel and all trained models are [[publicly available](https://www.kaggle.com/code/leonshangguan/private-7-8-final-of-submission)] \nOnly one key point to mention: if a bird is detected in the previous or next 5s, we will rank the probs and add the top5 bird species to the current detected birds.\n\nBecause this solution was not selected as our final submission, we don't know the actual rank, but as it scores 0.79 in private, it should rank 7-8.\n\n\n**[#1 Private #2] Solution**\n\nActually, I am curious that no one found it, the solution is provided by the host and finally allowed by the host.\n\n1. If you look at the discussion [Meet the host](https://www.kaggle.com/competitions/birdclef-2022/discussion/307941), you will find the host mentioned [BirdNet project](https://birdnet.cornell.edu/) in the post.\n2. Click the link, and it is easy to find the host's [github repo](https://github.com/kahst/BirdNET-Analyzer).\n3. Then, find the overlap classes between the [host's model](https://github.com/kahst/BirdNET-Analyzer/blob/main/checkpoints/V2.1/BirdNET_GLOBAL_2K_V2.1_Labels.txt) and the scored birds, 20 out of 21 scored birds are the same (except aniani).\n4. Modify species_list.txt under example folder accordingly, I have uploaded the modified repo to kaggle [here](https://www.kaggle.com/datasets/leonshangguan/birdnet.).\n5. Do some post-processing as in our [[inference notebook](https://www.kaggle.com/code/leonshangguan/birdnet-inference)]\n6. Thanks to @ivanpan who opened [this issue](https://github.com/kahst/BirdNET-Analyzer/issues/40), the host says `we won't enforce the \"non-commercial\" clause in the license for BirdCLEF, and you can use BirdNET in your submissions.` (I don't know if they also benefit from this repo, we can wait for their solution)\n\nSome Notes:\nHonestly, we didn't originally intend to use this model as a final submission until we notice the issue mentioned above, and actually, the ensemble of our own model could also reach the gold zone. The answer is provided by the host so that's why I say it is a weird solution. I guess the BirdNet was trained on the data for the public leaderboard but not the private one since there's a huge drop between the public lb to private (0.91 --> 0.84; 0.85 --> 0.78)\n\nThe BirdNet is running on CPU less than 2h inference time, while ours cost about 8h to run on GPU. So the host wins, my hope is the first place team doesn't benefit from that model and can beat the host. Looking forward to their solution.\n\n**Finally, a big thanks to @stefankahl for hosting this amazing competition and providing the answer as well xD.**", "1800598": "Congratulations!\n\nI do feel this should have been disclosed by @stefankahl as it seems like the BirdNET model has been trained at least partially on the test data used in this competition.", "1800551": "thanks for the story. it is very interesting whether the host used more training records to train the model or not", "1801322": "Someone asked me `why didn't you contact the host already those 20 days ago when you found it or at least 12 days ago when you wrote your comment?` -- Because, as I said, we didn't manage to select BirdNet as our final submission so I didn't think that's an issue and did not contact the host. We keep training our own model until the last minute and our own one is ranking 9 which is also in the gold zone. To be honest, we didn't think the BirdNet is allowed until the host said it is allowed. I understand our solution is weird under some gray area, but I don't understand why someone downvotes my comments (and downvotes the 1st place team members comments). This situation happened in [Cassava Leaf Disease Classification\n](https://www.kaggle.com/competitions/cassava-leaf-disease-classification/overview) as well where someone use pre-trained model and reached top places. Kaggle should manage to develop more clear rules and we should keep looking forward to helping avoid this case happen again.", "1800641": "BirdNet was a huge revelation to me in this competition. I couldn't believe how well it works, so decided to test on last year competition where we got 8th place. Easily managed to get top-1 on private there, but I believe public scores were around silver zone, that's why I wasn't 100% sure it would perform that well on private here.\n\nGlad I was wrong and now we can see that BirdNet gets top-1 on private in both years. \n\nFor the sake of honesty, we actually ensembled BirdNet with our models to get some additional boost and spent a couple of months trying to perfect those models (if I'm not mistaken, they score 0.78-0.79 on private). I believe we will be sharing results soon. But anyway, thanks @stefankahl for this competition. I mentioned this last year, I'm gonna mention this again - great orgs! One of the very few on Kaggle who provide great resources here (including notebooks), and simply answer questions to participants without abandoning a competition 1 week in. \n\nThanks! ", "1801566": "Despite the fact that the host mistakenly allow the BirdNet model, it's execrable that some Kagglers can find a leak-like situation and secretly exploit it for gold and money prize. This is not a glorious habit and anyone doing it should not be exalted.\n\nCheers for @maxhalford who could win the [RIID 100k worth competition](https://www.kaggle.com/competitions/riiid-test-answer-prediction/discussion/189437) but chooses to timely disclose the leakage.", "1800559": "It reminds me of the [2nd place solution of Cassava competition](https://www.kaggle.com/competitions/cassava-leaf-disease-classification/discussion/220898).", "1800666": "Seems like this competition was useless:( Best results achieved by existed model which also can be used in real world instead of ensembles which we train.", "1805062": "> if a bird is detected in the previous or next 5s, we will rank the probs and add the top5 bird species to the current detected birds.\n\nIt's really amazing that the information of adjacent time segments can bring up such a boost. ", "1800572": "Congrats and thanks for sharing the solution!\nI can't always find this kind of information😂", "1800554": "Congratulations with your score\nAnd I'm happy that you are the only team who found and used it :)", "1801227": "", "1801507": "Thanks for sharing this", "1801395": "Thanks for sharing the solution.", "1801208": "Thanks for sharing and Congrats!" } }