First of all, thank you Kaggle and the hosts for this interesting competition! Congrats to all the winners and people who have participated! While my team is working on the writeup, I'd like to share the post processing we used in this competition.
\nJust by adding the post processing alone, the kkiller's public inference notebook can achieve 0.62(in bronze medal range) on private leaderboard. Just for reference, the original inference notebook is 0.55. So that's a +0.07!
Back to the main topic, our post processing idea is from @triplex 's Cornell writeup(https://www.kaggle.com/c/birdsong-recognition/discussion/183255).
\nTo quote his words:
\n\"\"\"
\nif model gets a confident prediction of any bird, then lower threshold for this bird in the same audio file
\nuse thr_median as initial threshold
\nuse thr_high for confident prediction
\nif any bird with probability higher than thr_high in any clip, lower threshold to thr_low for this specific bird in the same audio file
\n\"\"\"
\nHis post processing code can be found here: https://github.com/xins-yao/Kaggle_Birdcall_17th_solution/blob/master/inference.py
I adopted his idea and used optuna to tune the thr_dict to get the optimal threshold according to cv.
\nBesides this, I've also tried using location information to adjust threshold. We did not end up using location information in our submission because it consistently gives us -0.01 on public leaderboard but in fact gives us a big boost on the private leaderboard(our best submission is 0.68 with location processing).
Here's the general idea for adding location:
\nIn the train audios, I computed the distance between the location of the recording and the 4 sites(COL,COR,SNE,SSW) based on their coordinates. I also extracted the month information from the metadata. During test time, I use the location and month data of each test audio and get a list of birds that are not present near that location(I set the radius to 360km but with bird domain knowledge you can definitely change this number) during that month based on the location and month. information I extracted from the train audio data. I then increase the threshold for non-present birds. This threshold is also tunable!
\nI will do more experiment on location once I get more gpu time.
Thank you for reading!
", "messages": [ { "id": "1333391", "postDate": "06/02/2021 17:21:41", "content": "First of all, thank you Kaggle and the hosts for this interesting competition! Congrats to all the winners and people who have participated! While my team is working on the writeup, I'd like to share the post processing we used in this competition.
\nJust by adding the post processing alone, the kkiller's public inference notebook can achieve 0.62(in bronze medal range) on private leaderboard. Just for reference, the original inference notebook is 0.55. So that's a +0.07!
Back to the main topic, our post processing idea is from @triplex 's Cornell writeup(https://www.kaggle.com/c/birdsong-recognition/discussion/183255).
\nTo quote his words:
\n\"\"\"
\nif model gets a confident prediction of any bird, then lower threshold for this bird in the same audio file
\nuse thr_median as initial threshold
\nuse thr_high for confident prediction
\nif any bird with probability higher than thr_high in any clip, lower threshold to thr_low for this specific bird in the same audio file
\n\"\"\"
\nHis post processing code can be found here: https://github.com/xins-yao/Kaggle_Birdcall_17th_solution/blob/master/inference.py
I adopted his idea and used optuna to tune the thr_dict to get the optimal threshold according to cv.
\nBesides this, I've also tried using location information to adjust threshold. We did not end up using location information in our submission because it consistently gives us -0.01 on public leaderboard but in fact gives us a big boost on the private leaderboard(our best submission is 0.68 with location processing).
Here's the general idea for adding location:
\nIn the train audios, I computed the distance between the location of the recording and the 4 sites(COL,COR,SNE,SSW) based on their coordinates. I also extracted the month information from the metadata. During test time, I use the location and month data of each test audio and get a list of birds that are not present near that location(I set the radius to 360km but with bird domain knowledge you can definitely change this number) during that month based on the location and month. information I extracted from the train audio data. I then increase the threshold for non-present birds. This threshold is also tunable!
\nI will do more experiment on location once I get more gpu time.
Thank you for reading!
", "rawMarkdown": "First of all, thank you Kaggle and the hosts for this interesting competition! Congrats to all the winners and people who have participated! While my team is working on the writeup, I'd like to share the post processing we used in this competition.\nJust by adding the post processing alone, the kkiller's public inference notebook can achieve 0.62(in bronze medal range) on private leaderboard. Just for reference, the original inference notebook is 0.55. So that's a +0.07!\n\nBack to the main topic, our post processing idea is from @triplex 's Cornell writeup(https://www.kaggle.com/c/birdsong-recognition/discussion/183255).\nTo quote his words:\n\"\"\"\nif model gets a confident prediction of any bird, then lower threshold for this bird in the same audio file\nuse thr_median as initial threshold\nuse thr_high for confident prediction\nif any bird with probability higher than thr_high in any clip, lower threshold to thr_low for this specific bird in the same audio file\n\"\"\"\nHis post processing code can be found here: https://github.com/xins-yao/Kaggle_Birdcall_17th_solution/blob/master/inference.py\n\nI adopted his idea and used optuna to tune the thr_dict to get the optimal threshold according to cv. \nBesides this, I've also tried using location information to adjust threshold. We did not end up using location information in our submission because it consistently gives us -0.01 on public leaderboard but in fact gives us a big boost on the private leaderboard(our best submission is 0.68 with location processing). \n\nHere's the general idea for adding location:\nIn the train audios, I computed the distance between the location of the recording and the 4 sites(COL,COR,SNE,SSW) based on their coordinates. I also extracted the month information from the metadata. During test time, I use the location and month data of each test audio and get a list of birds that are not present near that location(I set the radius to 360km but with bird domain knowledge you can definitely change this number) during that month based on the location and month. information I extracted from the train audio data. I then increase the threshold for non-present birds. This threshold is also tunable!\nI will do more experiment on location once I get more gpu time. \n\nThank you for reading!", "votes": null }, { "id": "1333450", "postDate": "06/02/2021 18:16:10", "content": "Looks like we had similar ideas for thresholding ;)
", "rawMarkdown": "Looks like we had similar ideas for thresholding ;)", "votes": null }, { "id": "1335292", "postDate": "06/04/2021 06:17:20", "content": "Thank you for sharing this tip! I went to try out with the public notebook and i saw increase with CV, but LB (priv/pub) doesnt change. I was wondering if you used thr_bottom as well.
", "rawMarkdown": "Thank you for sharing this tip! I went to try out with the public notebook and i saw increase with CV, but LB (priv/pub) doesnt change. I was wondering if you used thr_bottom as well.", "votes": null }, { "id": "1335950", "postDate": "06/04/2021 14:36:20", "content": "I also used thr_bottom as well! Here's the thr_dict that I used in one of my notebooks(public:0.6766 private: 0.6162): thr_dict = {'high': 0.6219136066238313, 'median': 0.21652254656576478, 'low': 0.16546842334609826, 'bottom': 0.021796972061225543}. With some further finetuning, I believe you can get the score higher for sure. Hope that helps!
", "rawMarkdown": "I also used thr_bottom as well! Here's the thr_dict that I used in one of my notebooks(public:0.6766 private: 0.6162): thr_dict = {'high': 0.6219136066238313, 'median': 0.21652254656576478, 'low': 0.16546842334609826, 'bottom': 0.021796972061225543}. With some further finetuning, I believe you can get the score higher for sure. Hope that helps!", "votes": null }, { "id": "1337099", "postDate": "06/05/2021 12:16:24", "content": "Thank you! Will you be able to share your notebook? I would love to see how it is adopted as i was struggling to understand the threshold shape and matrix shape :)
", "rawMarkdown": "Thank you! Will you be able to share your notebook? I would love to see how it is adopted as i was struggling to understand the threshold shape and matrix shape :)", "votes": null } ], "comments": [ { "id": 1333450, "author_name": "cpmpml", "author_url": "", "post_date": "06/02/2021 18:16:10", "content": "Looks like we had similar ideas for thresholding ;)
", "votes": null, "replies": [] }, { "id": 1335292, "author_name": "colaboratory", "author_url": "", "post_date": "06/04/2021 06:17:20", "content": "Thank you for sharing this tip! I went to try out with the public notebook and i saw increase with CV, but LB (priv/pub) doesnt change. I was wondering if you used thr_bottom as well.
", "votes": null, "replies": [ { "id": 1335950, "author_name": "derekcai", "author_url": "", "post_date": "06/04/2021 14:36:20", "content": "I also used thr_bottom as well! Here's the thr_dict that I used in one of my notebooks(public:0.6766 private: 0.6162): thr_dict = {'high': 0.6219136066238313, 'median': 0.21652254656576478, 'low': 0.16546842334609826, 'bottom': 0.021796972061225543}. With some further finetuning, I believe you can get the score higher for sure. Hope that helps!
", "votes": null, "replies": [] }, { "id": 1337099, "author_name": "colaboratory", "author_url": "", "post_date": "06/05/2021 12:16:24", "content": "Thank you! Will you be able to share your notebook? I would love to see how it is adopted as i was struggling to understand the threshold shape and matrix shape :)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1333391": "First of all, thank you Kaggle and the hosts for this interesting competition! Congrats to all the winners and people who have participated! While my team is working on the writeup, I'd like to share the post processing we used in this competition.\nJust by adding the post processing alone, the kkiller's public inference notebook can achieve 0.62(in bronze medal range) on private leaderboard. Just for reference, the original inference notebook is 0.55. So that's a +0.07!\n\nBack to the main topic, our post processing idea is from @triplex 's Cornell writeup(https://www.kaggle.com/c/birdsong-recognition/discussion/183255).\nTo quote his words:\n\"\"\"\nif model gets a confident prediction of any bird, then lower threshold for this bird in the same audio file\nuse thr_median as initial threshold\nuse thr_high for confident prediction\nif any bird with probability higher than thr_high in any clip, lower threshold to thr_low for this specific bird in the same audio file\n\"\"\"\nHis post processing code can be found here: https://github.com/xins-yao/Kaggle_Birdcall_17th_solution/blob/master/inference.py\n\nI adopted his idea and used optuna to tune the thr_dict to get the optimal threshold according to cv. \nBesides this, I've also tried using location information to adjust threshold. We did not end up using location information in our submission because it consistently gives us -0.01 on public leaderboard but in fact gives us a big boost on the private leaderboard(our best submission is 0.68 with location processing). \n\nHere's the general idea for adding location:\nIn the train audios, I computed the distance between the location of the recording and the 4 sites(COL,COR,SNE,SSW) based on their coordinates. I also extracted the month information from the metadata. During test time, I use the location and month data of each test audio and get a list of birds that are not present near that location(I set the radius to 360km but with bird domain knowledge you can definitely change this number) during that month based on the location and month. information I extracted from the train audio data. I then increase the threshold for non-present birds. This threshold is also tunable!\nI will do more experiment on location once I get more gpu time. \n\nThank you for reading!", "1333450": "Looks like we had similar ideas for thresholding ;)", "1335292": "Thank you for sharing this tip! I went to try out with the public notebook and i saw increase with CV, but LB (priv/pub) doesnt change. I was wondering if you used thr_bottom as well.", "1335950": "I also used thr_bottom as well! Here's the thr_dict that I used in one of my notebooks(public:0.6766 private: 0.6162): thr_dict = {'high': 0.6219136066238313, 'median': 0.21652254656576478, 'low': 0.16546842334609826, 'bottom': 0.021796972061225543}. With some further finetuning, I believe you can get the score higher for sure. Hope that helps!", "1337099": "Thank you! Will you be able to share your notebook? I would love to see how it is adopted as i was struggling to understand the threshold shape and matrix shape :)" }, "source": "meta" }