My teammate @kami634 has already posted a quick version of our solution. Our solution pipeline is composed of three stages and you can overview the stages info here.
\nhttps://www.kaggle.com/c/birdclef-2021/discussion/243304
\nAlso, this is an image of our pipeline. (only inference part, so some of the steps including the first stage are omitted.)
The complete code used in this competition has been uploaded to the following github. All of the ipynb files there have been confirmed to work properly in the current kaggle notebook environment (2021/6/4).
\nhttps://github.com/namakemono/kaggle-birdclef-2021
The first thing we did when we joined this competition was to find out the percentage of nocalls in train_soundscapes and test_soundscapes. In train_soundscapes, it is easy to find out, and in test_soundscapes (Public LB), it can be found by submitting all lines as nocall. The results were 0.637 for train_soundscapes and 0.54 for Public LB. Similarly, in the 2020 competition, we submitted all lines as nocall as late submission, and the results were 0.577 for Public LB and 0.544 for Private LB. In all cases, the majority of the targets were nocalls, and we speculated that the difference between birdcalls by bird species and the difference between birds singing and nocall might be qualitatively different. That’s why we considered nocall detection and bird identification as separate tasks. This was the origin of the binary nocall detector from melspectrograms(1). Initially, we came up with the flow to use the nocall detector to eliminate nocalls for sure, and then predict some birds with nocall labels disabled for the rest, but this did not work. The next idea was to modify the weak labels when building a multilabel classifier, in other words, multiplying the call probabilities obtained from the nocall detector by the labels of train_short_audio. This worked well.
\nNext, we built a multilabel classifier (backbone was resnest) for the melspectrogram as the second stage, but there were following four problems at this stage.
\nA. The labels of train_short_audio were weak, so it was not clear whether primary or secondary labeled birds were sounding in each frame.
\nB. It was possible that the labels itself were wrong.
\nC. The output was 397-dimensional probability vectors, and the measure of whether some bird was singing or not was unclear.
\nD. Information before and after the current frame might be meaningful.
To deal with these problems, we created another table competition by ourselves, using the results of the nocall detector, train_metadata, and time series information before and after the current frame. The procedure for creating this table competition data was as follows:
\nⅠ. Extract the top N candidates in each frame as rows from the results of the multilabel classifier.
\nⅡ. Among the N candidates, assigned label 1 to the common set with primary or secondary label, and 0 to the others. However, if the output of the nocall detector indicated that the possibilites of birds singing were low, all the candidates were set to 0. This would be the target variable.
\nⅢ. For each candidate, the meta data (time and location information) of the frame and the probabilities that the candidate bird was singing before and after the frame (output of multilabel classifier) were assigned. In addition, more features were added by feature engineering (2), and the table was completed. Then It was analyzed by lightGBM.
The above steps were expected to have the following effects on the issues A to D mentioned above.
\nA. Even if the given data has only a weak label, it is possible to reassign the label for each frame.
\nB. The noisy labels can be somewhat cancelled.
\nC. The output of the nocall detector can be taken into account.
\nD. Information before and after the current frame can be incorporated.
In addition, by incorporating metadata into the table, it was not necessary to create multiple melspectrogram multilabel classifiers for each region or season, which saved time and computational resources. In fact, in our first-place submission, we used only 15 minutes out of the 3-hour time limit. The fact that we were able to compete based on colab pro might be due to the fact that we succeeded in making this audio competition come down to the table competition.
\nThe output of our table competition was one probability value per candidate (5 candidates x number of frames in total). The next task was to find an appropriate threshold value for these. In this case, we used train_soundscapes as a reference and searched for the value that maximizes the F1 score by the ternary search. In this process, we hypothesized that the threshold would be different depending on the percentage of nocalls in train_soundscapes, and experimented by reducing the percentage of nocalls, but the conclusion was that the threshold did not change much.
\nFinally, we would like to propose a technique we call “nocall injection.” In the current algorithm, rows predicted as “nocall” does not intersected with rows in which some birds were predicted, and vice versa. However, due to the nature of the F1 score, if you predict only one out of the two correct answers, you get 2/3 of the score, and if you miss the only one label in some row, you get 0. Both are the same in that only one label is wrong, but the size of the penalty is different. Therefore, in order to make sure that nocalls are caught, we added “nocall” to test frames with a high possibility of being nocalls (That would be determined from the output of lightgbm), even if they had been already labeled as some birds. We called this nocall injection. As a result, the nocall and the bird appeared in the same row, and that row could not get the full F1 score, but the benefit of capturing the nocall outweighed it, so the score increased.
\nSorry for writing such a long thread but that's our 3-week activities. It was our first time to compete in a sound recognition competition and we didn't know anything about it at first, but @kneroma, @hidehisaarai1213, @cpmpml and many others helped us a lot with their comments and code. I would like to take this opportunity to thank them again.
\n(1) The nocall detector took the external data (freefield1010) as input because the data from train_soundscapes was not enough. The accuracy was also improved by using data augmentation for images.
\n(2) The features we created were as follows.
\n \"year\": The year when the audio was recorded.
\n \"month\": The month in which the audio was recorded.
\n \"sum_prob\": The sum of the probability values of the 397 dimensions for the frame.
\n \"mean_prob\": Mean of the 397 dimensional probability values for the frame.
\n \"max_prob\": Maximum 397-dimensional probability values for the frame.
\n \"prev_prob\" ~ \"prev6_prob\": Probability values for the candidate birds in the previous n frames.
\n \"prob\": Probability value for the candidate bird in the target frame.
\n \"next_prob\" ~ \"next6_prob\": Probability values of the candidate bird in the next n frames.
\n \"rank\": The probability that the bird of interest has the highest probability value out of 397 in the frame.
\n \"latitude\":
\n \"longitude\":
\n \"bird_id\": Which of the 397 birds is the target.
\n \"seconds\": The number of seconds in the frame.
\n \"num_appear\": The number of files where the bird is the primary label in train_short_audio
\n \"site_num_appear\": how many times the bird in train_short_audio appeared in the region
\n \"site_appear_ratio\": site_num_appear/num_appear
\n \"prob_diff\": Difference between prob and average of prob values for 3 frames
My teammate @kami634 has already posted a quick version of our solution. Our solution pipeline is composed of three stages and you can overview the stages info here.
\nhttps://www.kaggle.com/c/birdclef-2021/discussion/243304
\nAlso, this is an image of our pipeline. (only inference part, so some of the steps including the first stage are omitted.)
The complete code used in this competition has been uploaded to the following github. All of the ipynb files there have been confirmed to work properly in the current kaggle notebook environment (2021/6/4).
\nhttps://github.com/namakemono/kaggle-birdclef-2021
The first thing we did when we joined this competition was to find out the percentage of nocalls in train_soundscapes and test_soundscapes. In train_soundscapes, it is easy to find out, and in test_soundscapes (Public LB), it can be found by submitting all lines as nocall. The results were 0.637 for train_soundscapes and 0.54 for Public LB. Similarly, in the 2020 competition, we submitted all lines as nocall as late submission, and the results were 0.577 for Public LB and 0.544 for Private LB. In all cases, the majority of the targets were nocalls, and we speculated that the difference between birdcalls by bird species and the difference between birds singing and nocall might be qualitatively different. That’s why we considered nocall detection and bird identification as separate tasks. This was the origin of the binary nocall detector from melspectrograms(1). Initially, we came up with the flow to use the nocall detector to eliminate nocalls for sure, and then predict some birds with nocall labels disabled for the rest, but this did not work. The next idea was to modify the weak labels when building a multilabel classifier, in other words, multiplying the call probabilities obtained from the nocall detector by the labels of train_short_audio. This worked well.
\nNext, we built a multilabel classifier (backbone was resnest) for the melspectrogram as the second stage, but there were following four problems at this stage.
\nA. The labels of train_short_audio were weak, so it was not clear whether primary or secondary labeled birds were sounding in each frame.
\nB. It was possible that the labels itself were wrong.
\nC. The output was 397-dimensional probability vectors, and the measure of whether some bird was singing or not was unclear.
\nD. Information before and after the current frame might be meaningful.
To deal with these problems, we created another table competition by ourselves, using the results of the nocall detector, train_metadata, and time series information before and after the current frame. The procedure for creating this table competition data was as follows:
\nⅠ. Extract the top N candidates in each frame as rows from the results of the multilabel classifier.
\nⅡ. Among the N candidates, assigned label 1 to the common set with primary or secondary label, and 0 to the others. However, if the output of the nocall detector indicated that the possibilites of birds singing were low, all the candidates were set to 0. This would be the target variable.
\nⅢ. For each candidate, the meta data (time and location information) of the frame and the probabilities that the candidate bird was singing before and after the frame (output of multilabel classifier) were assigned. In addition, more features were added by feature engineering (2), and the table was completed. Then It was analyzed by lightGBM.
The above steps were expected to have the following effects on the issues A to D mentioned above.
\nA. Even if the given data has only a weak label, it is possible to reassign the label for each frame.
\nB. The noisy labels can be somewhat cancelled.
\nC. The output of the nocall detector can be taken into account.
\nD. Information before and after the current frame can be incorporated.
In addition, by incorporating metadata into the table, it was not necessary to create multiple melspectrogram multilabel classifiers for each region or season, which saved time and computational resources. In fact, in our first-place submission, we used only 15 minutes out of the 3-hour time limit. The fact that we were able to compete based on colab pro might be due to the fact that we succeeded in making this audio competition come down to the table competition.
\nThe output of our table competition was one probability value per candidate (5 candidates x number of frames in total). The next task was to find an appropriate threshold value for these. In this case, we used train_soundscapes as a reference and searched for the value that maximizes the F1 score by the ternary search. In this process, we hypothesized that the threshold would be different depending on the percentage of nocalls in train_soundscapes, and experimented by reducing the percentage of nocalls, but the conclusion was that the threshold did not change much.
\nFinally, we would like to propose a technique we call “nocall injection.” In the current algorithm, rows predicted as “nocall” does not intersected with rows in which some birds were predicted, and vice versa. However, due to the nature of the F1 score, if you predict only one out of the two correct answers, you get 2/3 of the score, and if you miss the only one label in some row, you get 0. Both are the same in that only one label is wrong, but the size of the penalty is different. Therefore, in order to make sure that nocalls are caught, we added “nocall” to test frames with a high possibility of being nocalls (That would be determined from the output of lightgbm), even if they had been already labeled as some birds. We called this nocall injection. As a result, the nocall and the bird appeared in the same row, and that row could not get the full F1 score, but the benefit of capturing the nocall outweighed it, so the score increased.
\nSorry for writing such a long thread but that's our 3-week activities. It was our first time to compete in a sound recognition competition and we didn't know anything about it at first, but @kneroma, @hidehisaarai1213, @cpmpml and many others helped us a lot with their comments and code. I would like to take this opportunity to thank them again.
\n(1) The nocall detector took the external data (freefield1010) as input because the data from train_soundscapes was not enough. The accuracy was also improved by using data augmentation for images.
\n(2) The features we created were as follows.
\n \"year\": The year when the audio was recorded.
\n \"month\": The month in which the audio was recorded.
\n \"sum_prob\": The sum of the probability values of the 397 dimensions for the frame.
\n \"mean_prob\": Mean of the 397 dimensional probability values for the frame.
\n \"max_prob\": Maximum 397-dimensional probability values for the frame.
\n \"prev_prob\" ~ \"prev6_prob\": Probability values for the candidate birds in the previous n frames.
\n \"prob\": Probability value for the candidate bird in the target frame.
\n \"next_prob\" ~ \"next6_prob\": Probability values of the candidate bird in the next n frames.
\n \"rank\": The probability that the bird of interest has the highest probability value out of 397 in the frame.
\n \"latitude\":
\n \"longitude\":
\n \"bird_id\": Which of the 397 birds is the target.
\n \"seconds\": The number of seconds in the frame.
\n \"num_appear\": The number of files where the bird is the primary label in train_short_audio
\n \"site_num_appear\": how many times the bird in train_short_audio appeared in the region
\n \"site_appear_ratio\": site_num_appear/num_appear
\n \"prob_diff\": Difference between prob and average of prob values for 3 frames
We mainly used the Colab Pro as the machine, and were lucky to be assigned a V100 occasionally, but usually a P100.
\nA lot of computational resources were needed for the second stage of training. We had converted the images into melspectrograms in advance, so the cost during training was kept low.
\nIt took a few minutes per epoch, so a total of 2-5 hours per model to train.
We had multiple notebooks open at the same time for training and inference(for validation), so even ‘Colab Pro’ sometimes stopped if we used too much.
\nIf you get suspended for using the regular ‘Colab’ too much, Google will recommend that you subscribe to ‘Colab Pro’.
\nIf you use ‘Colab Pro’ too much and get suspended, Google will recommend that you subscribe to ‘Colab Pro’! If only they offered ‘Colab Pro Pro’, I would subscribe!
Congrats. I became convinced that a stacking model like yours could be useful, only too late. It is good to see it worked well.
", "rawMarkdown": "Congrats. I became convinced that a stacking model like yours could be useful, only too late. It is good to see it worked well.", "votes": null }, { "id": "1335917", "postDate": "06/04/2021 14:06:41", "content": "XD haha, Colab Pro Pro
", "rawMarkdown": "XD haha, **Colab Pro Pro**", "votes": null }, { "id": "1335973", "postDate": "06/04/2021 14:59:17", "content": "Thank you.
\nDuring the competition, we suffered from the feeling that your score was endlessly far away, actually.
Congrats once again!! Seems that the limitation on resources make you explore other ways to use the data and pays you back at the end - Well done!!
\nps: I have some questions but I'll look in more detail to your code first… thanks for sharing
", "rawMarkdown": "Congrats once again!! Seems that the limitation on resources make you explore other ways to use the data and pays you back at the end - Well done!! \n\nps: I have some questions but I'll look in more detail to your code first... thanks for sharing", "votes": null }, { "id": "1336203", "postDate": "06/04/2021 17:59:40", "content": "Thank you for the detailed write-up, and congrats on your win. I think your solution was very innovative overall, but I especially liked the idea of the no-call injection, which I think might have been the difference-maker in the end.
", "rawMarkdown": "Thank you for the detailed write-up, and congrats on your win. I think your solution was very innovative overall, but I especially liked the idea of the no-call injection, which I think might have been the difference-maker in the end.", "votes": null }, { "id": "1337714", "postDate": "06/05/2021 20:16:59", "content": "Thanks for sharing your solution and congratulations on the first place. As for uploading images, sadly it has been disabled for now. One workaround, is to upload to google drive, share the file with everyone and then use the following URL template:
\n
where the img_id is the id of your image: you can find it by clicking on the image.
I hope this helps!
", "rawMarkdown": "Thanks for sharing your solution and congratulations on the first place. As for uploading images, sadly it has been disabled for now. One workaround, is to upload to google drive, share the file with everyone and then use the following URL template: \n\n``\n\nwhere the `img_id` is the id of your image: you can find it by clicking on the image.\n\nI hope this helps!", "votes": null }, { "id": "1338514", "postDate": "06/06/2021 14:04:04", "content": "Is it possible to download Academic Torrents data like original freefield1010 without torrent software? Torrent software is blocked by my security software.
", "rawMarkdown": "Is it possible to download Academic Torrents data like original freefield1010 without torrent software? Torrent software is blocked by my security software.", "votes": null }, { "id": "1339003", "postDate": "06/07/2021 00:04:52", "content": "Thank you so much for your help!
\nI didn't know that way. I succeeded in uploading our pipeline image.
Hi @shigemitsutomizawa, sorry for the late reply.
\nActually, we did not download the original freefield1010 audio data, but used the one that was already uploaded as a kaggle dataset. Please check the below dataset.
\nhttps://www.kaggle.com/rlmwang/ff1010bird
We apologize for not being able to meet your expectations.
", "rawMarkdown": "Hi @shigemitsutomizawa, sorry for the late reply.\n\nActually, we did not download the original freefield1010 audio data, but used the one that was already uploaded as a kaggle dataset. Please check the below dataset.\nhttps://www.kaggle.com/rlmwang/ff1010bird\n\nWe apologize for not being able to meet your expectations.", "votes": null }, { "id": "1341295", "postDate": "06/08/2021 15:02:15", "content": "Awesome! The graph looks neat. 👌
", "rawMarkdown": "Awesome! The graph looks neat. 👌", "votes": null }, { "id": "1345189", "postDate": "06/11/2021 11:43:12", "content": "Very usefull solution, thank's for sharing. Congrats once again !
", "rawMarkdown": "Very usefull solution, thank's for sharing. Congrats once again !", "votes": null }, { "id": "1375829", "postDate": "07/04/2021 13:59:26", "content": "Thanks for sharing your detailed solution and congratulations on the first place.
\nI would like to ask you two questions, even though it has been a long time since the competition ended.
Hi @naoism, thank you for your questions.
\nAs you pointed out, the y-axis of the mel spectrogram does indeed refer to frequency, and the vertical flip seems to be nonsense. However, I remember that we adopted it because it actually increased the CV score. This is just a hypothesis, but I think that the CV score increased because the vertical flip could preserve the shape of signals (or can we call it \"waveform\" here?) that appear on the mel spectrogram, even though it twisted the sound information itself.
LightGBM can accept NaN as input.
I hope these will help…
", "rawMarkdown": "Hi @naoism, thank you for your questions.\n\n1. As you pointed out, the y-axis of the mel spectrogram does indeed refer to frequency, and the vertical flip seems to be nonsense. However, I remember that we adopted it because it actually increased the CV score. This is just a hypothesis, but I think that the CV score increased because the vertical flip could preserve the shape of signals (or can we call it \"waveform\" here?) that appear on the mel spectrogram, even though it twisted the sound information itself.\n\n2. LightGBM can accept NaN as input.\n\nI hope these will help...", "votes": null }, { "id": "1378353", "postDate": "07/06/2021 13:26:14", "content": "Thank you for your quick reply.
\nIf the CV got better, then the new information appeared by flipping it. It is also possible that by flipping the spectrogram, one bird's call became a different bird's call (with a different main frequency). On the other hand, I think this is only better because of the nocall detector. Of coarse, that's just my hypothesis.
Thanks for your explanations. I couldn't figure out the repository description for producing same results. You are mentioning about imitating the directory structure like Kaggle, but if I want to use Kaggle itself, can I create such a directory? Or it is expected to create this directory on my personal sources? I am confused..
", "rawMarkdown": "Thanks for your explanations. I couldn't figure out the repository description for producing same results. You are mentioning about imitating the directory structure like Kaggle, but if I want to use Kaggle itself, can I create such a directory? Or it is expected to create this directory on my personal sources? I am confused..", "votes": null } ], "comments": [ { "id": 1335890, "author_name": "kami634", "author_url": "", "post_date": "06/04/2021 13:50:39", "content": "We mainly used the Colab Pro as the machine, and were lucky to be assigned a V100 occasionally, but usually a P100.
\nA lot of computational resources were needed for the second stage of training. We had converted the images into melspectrograms in advance, so the cost during training was kept low.
\nIt took a few minutes per epoch, so a total of 2-5 hours per model to train.
We had multiple notebooks open at the same time for training and inference(for validation), so even ‘Colab Pro’ sometimes stopped if we used too much.
\nIf you get suspended for using the regular ‘Colab’ too much, Google will recommend that you subscribe to ‘Colab Pro’.
\nIf you use ‘Colab Pro’ too much and get suspended, Google will recommend that you subscribe to ‘Colab Pro’! If only they offered ‘Colab Pro Pro’, I would subscribe!
XD haha, Colab Pro Pro
", "votes": null, "replies": [] } ] }, { "id": 1335891, "author_name": "cpmpml", "author_url": "", "post_date": "06/04/2021 13:51:32", "content": "Congrats. I became convinced that a stacking model like yours could be useful, only too late. It is good to see it worked well.
", "votes": null, "replies": [ { "id": 1335973, "author_name": "startjapan", "author_url": "", "post_date": "06/04/2021 14:59:17", "content": "Thank you.
\nDuring the competition, we suffered from the feeling that your score was endlessly far away, actually.
Congrats once again!! Seems that the limitation on resources make you explore other ways to use the data and pays you back at the end - Well done!!
\nps: I have some questions but I'll look in more detail to your code first… thanks for sharing
", "votes": null, "replies": [] }, { "id": 1336203, "author_name": "agneev", "author_url": "", "post_date": "06/04/2021 17:59:40", "content": "Thank you for the detailed write-up, and congrats on your win. I think your solution was very innovative overall, but I especially liked the idea of the no-call injection, which I think might have been the difference-maker in the end.
", "votes": null, "replies": [] }, { "id": 1337714, "author_name": "yassinealouini", "author_url": "", "post_date": "06/05/2021 20:16:59", "content": "Thanks for sharing your solution and congratulations on the first place. As for uploading images, sadly it has been disabled for now. One workaround, is to upload to google drive, share the file with everyone and then use the following URL template:
\n
where the img_id is the id of your image: you can find it by clicking on the image.
I hope this helps!
", "votes": null, "replies": [ { "id": 1339003, "author_name": "startjapan", "author_url": "", "post_date": "06/07/2021 00:04:52", "content": "Thank you so much for your help!
\nI didn't know that way. I succeeded in uploading our pipeline image.
Awesome! The graph looks neat. 👌
", "votes": null, "replies": [] } ] }, { "id": 1338514, "author_name": "shigemitsutomizawa", "author_url": "", "post_date": "06/06/2021 14:04:04", "content": "Is it possible to download Academic Torrents data like original freefield1010 without torrent software? Torrent software is blocked by my security software.
", "votes": null, "replies": [ { "id": 1341174, "author_name": "startjapan", "author_url": "", "post_date": "06/08/2021 13:33:49", "content": "Hi @shigemitsutomizawa, sorry for the late reply.
\nActually, we did not download the original freefield1010 audio data, but used the one that was already uploaded as a kaggle dataset. Please check the below dataset.
\nhttps://www.kaggle.com/rlmwang/ff1010bird
We apologize for not being able to meet your expectations.
", "votes": null, "replies": [] } ] }, { "id": 1345189, "author_name": "raphaelbordeau", "author_url": "", "post_date": "06/11/2021 11:43:12", "content": "Very usefull solution, thank's for sharing. Congrats once again !
", "votes": null, "replies": [] }, { "id": 1375829, "author_name": "naoism", "author_url": "", "post_date": "07/04/2021 13:59:26", "content": "Thanks for sharing your detailed solution and congratulations on the first place.
\nI would like to ask you two questions, even though it has been a long time since the competition ended.
Hi @naoism, thank you for your questions.
\nAs you pointed out, the y-axis of the mel spectrogram does indeed refer to frequency, and the vertical flip seems to be nonsense. However, I remember that we adopted it because it actually increased the CV score. This is just a hypothesis, but I think that the CV score increased because the vertical flip could preserve the shape of signals (or can we call it \"waveform\" here?) that appear on the mel spectrogram, even though it twisted the sound information itself.
LightGBM can accept NaN as input.
I hope these will help…
", "votes": null, "replies": [] }, { "id": 1378353, "author_name": "naoism", "author_url": "", "post_date": "07/06/2021 13:26:14", "content": "Thank you for your quick reply.
\nIf the CV got better, then the new information appeared by flipping it. It is also possible that by flipping the spectrogram, one bird's call became a different bird's call (with a different main frequency). On the other hand, I think this is only better because of the nocall detector. Of coarse, that's just my hypothesis.
Thanks for your explanations. I couldn't figure out the repository description for producing same results. You are mentioning about imitating the directory structure like Kaggle, but if I want to use Kaggle itself, can I create such a directory? Or it is expected to create this directory on my personal sources? I am confused..
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1335887": "# tl;dr\nMy teammate @kami634 has already posted a quick version of our solution. Our solution pipeline is composed of three stages and you can overview the stages info here.\nhttps://www.kaggle.com/c/birdclef-2021/discussion/243304\nAlso, this is an image of our pipeline. (only inference part, so some of the steps including the first stage are omitted.)\n\n\n\nThe complete code used in this competition has been uploaded to the following github. All of the ipynb files there have been confirmed to work properly in the current kaggle notebook environment (2021/6/4).\nhttps://github.com/namakemono/kaggle-birdclef-2021\n\n