Thanks to all the organizers for hosting such a challenging competition. Also, thanks to all other competitors for sharing valuable information in forums and notebooks. Special thanks to @vlomme for sharing his code on github from last competition.
\nPrayers for all the folks and their families suffering from Covid.
\nFor most of my models, I used 5-7 second clips for training. I gave lower weight to secondary predictions and used modified mixup just likes last year's 1st and 3rd place solutions.
\nMostly, the strategy was to get as much diversity possible into the ensemble to make predictions more robust.
\nThis was the trickiest part. We had following information at hand in addition to raw probabilities for each 5s clip:
\ntrain_metadata.csv within +/- x deg of soundscape locationInitially I tried fitting a meta estimator (LightGBM and RandomForest, LogisticRegression) over above variable but results using custom rule were better on both validation and public LB (Thinking retrospectively, I think this is where I overfit).
\nCustom rule was something like: p1 * a1 + np.clip(p2, 0, t1) * a2 + ...
parakeet from Orange billed parakeet) as an additional label during training.hofwoo1). So, I split them into 10s clips and hand labelled them to increase their sample size but this didnt help. Also, class balancing gave lower validation loss to ditched it as well.Thanks to all the organizers for hosting such a challenging competition. Also, thanks to all other competitors for sharing valuable information in forums and notebooks. Special thanks to @vlomme for sharing his code on github from last competition.
\nPrayers for all the folks and their families suffering from Covid.
\nFor most of my models, I used 5-7 second clips for training. I gave lower weight to secondary predictions and used modified mixup just likes last year's 1st and 3rd place solutions.
\nMostly, the strategy was to get as much diversity possible into the ensemble to make predictions more robust.
\nThis was the trickiest part. We had following information at hand in addition to raw probabilities for each 5s clip:
\ntrain_metadata.csv within +/- x deg of soundscape locationInitially I tried fitting a meta estimator (LightGBM and RandomForest, LogisticRegression) over above variable but results using custom rule were better on both validation and public LB (Thinking retrospectively, I think this is where I overfit).
\nCustom rule was something like: p1 * a1 + np.clip(p2, 0, t1) * a2 + ...
parakeet from Orange billed parakeet) as an additional label during training.hofwoo1). So, I split them into 10s clips and hand labelled them to increase their sample size but this didnt help. Also, class balancing gave lower validation loss to ditched it as well.Hi @tezdhar congrats on the solo gold , thanks for sharing the solution , I had two questions , it would be great if you can answer them :
\nIt's nice that my decision was useful to someone. And I'm even more pleased to see that it was the basis for the first place decision in the public rankings.
", "rawMarkdown": "It's nice that my decision was useful to someone. And I'm even more pleased to see that it was the basis for the first place decision in the public rankings.", "votes": null }, { "id": "1332569", "postDate": "06/02/2021 07:34:48", "content": "Thanks @tanulsingh077 !
\ndef post_process2(probas, test, bird_loc_cnts, max_w=0.5, mean_w=0.5, cnt_w=1, max_w2=0.1):\n proba = probas.copy()\n dates = test.filepaths.str.split('/').str.get(-1).str.slice(-12,-4).values\n audio_ids = test.audio_id.values\n proba_date_max = groupby_np(dates, probas, np.max)\n proba_audio_max = groupby_np(audio_ids, probas, np.max)\n proba_audio_mean = groupby_np(audio_ids, probas, np.mean)\n\n for audio_id in test.audio_id.unique():\n rows = test.audio_id.values == audio_id\n site = test.loc[rows, 'site'].iloc[0]\n bird_cnts = bird_loc_cnts[site]\n cnt_prob1 = np.log1p(np.repeat(np.array([np.clip(bird_cnts.get(INT2CODE[i], 0), 0.0, 1) for i in range(398)]).reshape(1, -1), sum(rows), 0))\n cnt_prob2 = np.log1p(np.repeat(np.array([np.clip(bird_cnts.get(INT2CODE[i], 0), 0.0, 100) for i in range(398)]).reshape(1, -1), sum(rows), 0))/100\n # max_probs = np.max(proba[rows], axis=0, keepdims=True)\n # thresh_cnt = np.sum(proba[rows] > thresh, axis=0, keepdims=True)\n # mean_probs = np.mean(proba[rows], axis=0, keepdims=True)\n max_probs = proba_audio_max[rows]\n mean_probs = proba_audio_mean[rows]\n max_probs2 = proba_date_max[rows]\n if cnt_w > 0:\n proba[rows] *= cnt_prob1*1.0\n proba[rows] = proba[rows] + max_w*np.clip(max_probs, 0, 0.67) + max_w2*np.clip(max_probs2, 0, 0.67) + mean_w*np.clip(mean_probs, 0, 0.1) + cnt_w*np.clip(cnt_prob2, 0, 0.01) # + thresh_w*np.clip(thresh_cnt, 0, 5)/5.0\n proba[:, 397] = probas[:, 397]\n return proba\nCongrats on your result. I was watching you climb the LB and thought you would finish well.
\n\n\nresults using custom rule were better on both validation and public LB (Thinking retrospectively, I think this is where I overfit).
\n
I did the same mistake probably. My post processing helped a lot, maybe too much…
", "rawMarkdown": "Congrats on your result. I was watching you climb the LB and thought you would finish well.\n\n> results using custom rule were better on both validation and public LB (Thinking retrospectively, I think this is where I overfit).\n\nI did the same mistake probably. My post processing helped a lot, maybe too much...", "votes": null }, { "id": "1332617", "postDate": "06/02/2021 08:07:35", "content": "Congrats on solo gold, great effort!
\nI am a bit surprised that both you and @cpmpml only worked on 5sec crops if I understand that correctly. It appeared to us that there is way too much noise if you only pick such a short sub-sample.
", "rawMarkdown": "Congrats on solo gold, great effort!\n\nI am a bit surprised that both you and @cpmpml only worked on 5sec crops if I understand that correctly. It appeared to us that there is way too much noise if you only pick such a short sub-sample.", "votes": null }, { "id": "1332624", "postDate": "06/02/2021 08:11:12", "content": "Well, it matches what we have to predict. But we cannot capture longer clip dynamics and have to rely on post processing for it. Mine is similar to what is described here. The trick is that this worked very well on CV and public LB. Maybe we were fooled by it.
\nWhen I saw how much tuning I had ot do on postprocessing I thought it would be better to have it learned by a model. This leads to SED like model. I only started on it the last day, which was hopeless of course.
", "rawMarkdown": "Well, it matches what we have to predict. But we cannot capture longer clip dynamics and have to rely on post processing for it. Mine is similar to what is described here. The trick is that this worked very well on CV and public LB. Maybe we were fooled by it.\n\nWhen I saw how much tuning I had ot do on postprocessing I thought it would be better to have it learned by a model. This leads to SED like model. I only started on it the last day, which was hopeless of course.", "votes": null }, { "id": "1332627", "postDate": "06/02/2021 08:12:15", "content": "But if you crop only 5 seconds you have a lot more wrong labels while training because we only have weak labels.
", "rawMarkdown": "But if you crop only 5 seconds you have a lot more wrong labels while training because we only have weak labels.", "votes": null }, { "id": "1332660", "postDate": "06/02/2021 08:30:00", "content": "@psi sorry I am coming in between your discussions , we have train soundscapes which are 30secs-3min long , and I think @cpmp is cropping the start of the audio and the end of the audio from these based on his hypothesis that the train soundscapes were cropped to the start of birdcall start and hence this would reduce noise.
\nIn last year's comp people used a lot of things to counter the issue mentioned by you , theo used better cropping based on oof's score , vladmir used RMS for better cropping
", "rawMarkdown": "psi sorry I am coming in between your discussions , we have train soundscapes which are 30secs-3min long , and I think @cpmp is cropping the start of the audio and the end of the audio from these based on his hypothesis that the train soundscapes were cropped to the start of birdcall start and hence this would reduce noise.\n\nIn last year's comp people used a lot of things to counter the issue mentioned by you , theo used better cropping based on oof's score , vladmir used RMS for better cropping", "votes": null }, { "id": "1332662", "postDate": "06/02/2021 08:30:15", "content": "@philippsinger As in Cornell I used first 5 seconds and last 5 seconds. Rationale is given in this post:
\nhttps://www.kaggle.com/c/birdclef-2021/discussion/230733#1265442
\nI commented I used the same in Cornell competition ;) And I reused it here as well.
", "rawMarkdown": "philippsinger As in Cornell I used first 5 seconds and last 5 seconds. Rationale is given in this post:\n\nhttps://www.kaggle.com/c/birdclef-2021/discussion/230733#1265442\n\nI commented I used the same in Cornell competition ;) And I reused it here as well.", "votes": null }, { "id": "1332680", "postDate": "06/02/2021 08:38:59", "content": "Makes sense to reduce the noise. Did you still try larger crop lengths?
", "rawMarkdown": "Makes sense to reduce the noise. Did you still try larger crop lengths?", "votes": null }, { "id": "1332698", "postDate": "06/02/2021 08:48:05", "content": "Thanks for providing the function 😊
", "rawMarkdown": "Thanks for providing the function 😊", "votes": null }, { "id": "1332707", "postDate": "06/02/2021 08:52:31", "content": "I looked into many train short audios and noticed in many files the 1st or last seconds are people speaking, some language I dont understand, so we dont use assumption to crop 1st or last 5 seconds. Instead use energy + pseudo based probs cropping.
Thanks @philippsinger! Yes, I did try training on 30s clips few days before deadline with same noise setup. But that did not give me good results on train soundscapes (~0.68 f1) which was too low to be even included in ensemble.
\nIt could be I had some bug in my pipeline or using too much augmentation was causing some issues. Anyway, I guess I decided to drop it too early without much experimentation.
Thank you :).
\nWe fell for the mirage created by public LB. Hopefully, I am able to do better in my next competitions
One more thing, you could use different thresholds for bird and nocall. For example, you could make prediction like this chswar nocall. This helped slightly.
Hi @tezdhar congrats on the solo gold , thanks for sharing the solution , I had two questions , it would be great if you can answer them :
\nThanks @tanulsingh077 !
\ndef post_process2(probas, test, bird_loc_cnts, max_w=0.5, mean_w=0.5, cnt_w=1, max_w2=0.1):\n proba = probas.copy()\n dates = test.filepaths.str.split('/').str.get(-1).str.slice(-12,-4).values\n audio_ids = test.audio_id.values\n proba_date_max = groupby_np(dates, probas, np.max)\n proba_audio_max = groupby_np(audio_ids, probas, np.max)\n proba_audio_mean = groupby_np(audio_ids, probas, np.mean)\n\n for audio_id in test.audio_id.unique():\n rows = test.audio_id.values == audio_id\n site = test.loc[rows, 'site'].iloc[0]\n bird_cnts = bird_loc_cnts[site]\n cnt_prob1 = np.log1p(np.repeat(np.array([np.clip(bird_cnts.get(INT2CODE[i], 0), 0.0, 1) for i in range(398)]).reshape(1, -1), sum(rows), 0))\n cnt_prob2 = np.log1p(np.repeat(np.array([np.clip(bird_cnts.get(INT2CODE[i], 0), 0.0, 100) for i in range(398)]).reshape(1, -1), sum(rows), 0))/100\n # max_probs = np.max(proba[rows], axis=0, keepdims=True)\n # thresh_cnt = np.sum(proba[rows] > thresh, axis=0, keepdims=True)\n # mean_probs = np.mean(proba[rows], axis=0, keepdims=True)\n max_probs = proba_audio_max[rows]\n mean_probs = proba_audio_mean[rows]\n max_probs2 = proba_date_max[rows]\n if cnt_w > 0:\n proba[rows] *= cnt_prob1*1.0\n proba[rows] = proba[rows] + max_w*np.clip(max_probs, 0, 0.67) + max_w2*np.clip(max_probs2, 0, 0.67) + mean_w*np.clip(mean_probs, 0, 0.1) + cnt_w*np.clip(cnt_prob2, 0, 0.01) # + thresh_w*np.clip(thresh_cnt, 0, 5)/5.0\n proba[:, 397] = probas[:, 397]\n return proba\nThanks for providing the function 😊
", "votes": null, "replies": [] } ] }, { "id": 1332428, "author_name": "vlomme", "author_url": "", "post_date": "06/02/2021 05:55:21", "content": "It's nice that my decision was useful to someone. And I'm even more pleased to see that it was the basis for the first place decision in the public rankings.
", "votes": null, "replies": [] }, { "id": 1332613, "author_name": "cpmpml", "author_url": "", "post_date": "06/02/2021 08:01:22", "content": "Congrats on your result. I was watching you climb the LB and thought you would finish well.
\n\n\nresults using custom rule were better on both validation and public LB (Thinking retrospectively, I think this is where I overfit).
\n
I did the same mistake probably. My post processing helped a lot, maybe too much…
", "votes": null, "replies": [ { "id": 1332725, "author_name": "tezdhar", "author_url": "", "post_date": "06/02/2021 09:05:30", "content": "Thank you :).
\nWe fell for the mirage created by public LB. Hopefully, I am able to do better in my next competitions
Congrats on solo gold, great effort!
\nI am a bit surprised that both you and @cpmpml only worked on 5sec crops if I understand that correctly. It appeared to us that there is way too much noise if you only pick such a short sub-sample.
", "votes": null, "replies": [ { "id": 1332624, "author_name": "cpmpml", "author_url": "", "post_date": "06/02/2021 08:11:12", "content": "Well, it matches what we have to predict. But we cannot capture longer clip dynamics and have to rely on post processing for it. Mine is similar to what is described here. The trick is that this worked very well on CV and public LB. Maybe we were fooled by it.
\nWhen I saw how much tuning I had ot do on postprocessing I thought it would be better to have it learned by a model. This leads to SED like model. I only started on it the last day, which was hopeless of course.
", "votes": null, "replies": [] }, { "id": 1332627, "author_name": "philippsinger", "author_url": "", "post_date": "06/02/2021 08:12:15", "content": "But if you crop only 5 seconds you have a lot more wrong labels while training because we only have weak labels.
", "votes": null, "replies": [] }, { "id": 1332660, "author_name": "tanulsingh077", "author_url": "", "post_date": "06/02/2021 08:30:00", "content": "@psi sorry I am coming in between your discussions , we have train soundscapes which are 30secs-3min long , and I think @cpmp is cropping the start of the audio and the end of the audio from these based on his hypothesis that the train soundscapes were cropped to the start of birdcall start and hence this would reduce noise.
\nIn last year's comp people used a lot of things to counter the issue mentioned by you , theo used better cropping based on oof's score , vladmir used RMS for better cropping
", "votes": null, "replies": [] }, { "id": 1332662, "author_name": "cpmpml", "author_url": "", "post_date": "06/02/2021 08:30:15", "content": "@philippsinger As in Cornell I used first 5 seconds and last 5 seconds. Rationale is given in this post:
\nhttps://www.kaggle.com/c/birdclef-2021/discussion/230733#1265442
\nI commented I used the same in Cornell competition ;) And I reused it here as well.
", "votes": null, "replies": [] }, { "id": 1332680, "author_name": "philippsinger", "author_url": "", "post_date": "06/02/2021 08:38:59", "content": "Makes sense to reduce the noise. Did you still try larger crop lengths?
", "votes": null, "replies": [] }, { "id": 1332707, "author_name": "superchenhao", "author_url": "", "post_date": "06/02/2021 08:52:31", "content": "I looked into many train short audios and noticed in many files the 1st or last seconds are people speaking, some language I dont understand, so we dont use assumption to crop 1st or last 5 seconds. Instead use energy + pseudo based probs cropping.
Thanks @philippsinger! Yes, I did try training on 30s clips few days before deadline with same noise setup. But that did not give me good results on train soundscapes (~0.68 f1) which was too low to be even included in ensemble.
\nIt could be I had some bug in my pipeline or using too much augmentation was causing some issues. Anyway, I guess I decided to drop it too early without much experimentation.
One more thing, you could use different thresholds for bird and nocall. For example, you could make prediction like this chswar nocall. This helped slightly.