{ "id": 398229, "title": "Identifying Audio Duplications", "url": "/competitions/birdclef-2023/discussion/398229", "author_name": "", "post_date": "2023-03-29T04:57:36.739220500Z", "votes": 27, "comment_count": 8, "views": 0, "content": "

- Update 2023/04/11: Added extra pairs found by @robbynevels by comparing the distance between google bird classifier embeddings.
\n- Update 2023/03/31: Added extra pairs that were found by @mattop and not detected by DTW.
\nPlease give their original notebooks upvotes since the identical samples wouldn't be found without their effort:
\nMatt OP's EDA and comparing identical samples
\nThacrobatheskis's 3D embedding EDA
\nThacrobatheskis's identifying duplicate samples in embedding space

\n

TL;DR

\n

My notebook can be found here.

\n

The duplicated audios pairs are

\n
[['afrgos1/XC400373.ogg', 'afrgos1/XC418707.ogg'],\n ['brubru1/XC364486.ogg', 'brubru1/XC364487.ogg'],\n ['grewoo2/XC527938.ogg', 'grewoo2/XC527939.ogg'],\n ['spepig1/XC589785.ogg', 'spepig1/XC590084.ogg'],\n ['yenspu1/XC756890.ogg', 'yenspu1/XC756891.ogg'],\n ['abhori1/XC748240.ogg', 'ccbeat1/XC748235.ogg'],\n ['wlwwar/XC478705.ogg', 'wlwwar/XC478767.ogg'],\n ['combul2/XC447669.ogg', 'combul2/XC650878.ogg'],\n ['combul2/XC401370.ogg', 'combul2/XC401371.ogg'],\n ['litegr/XC447850.ogg', 'litegr/XC535552.ogg'],\n ['carcha1/XC324665.ogg', 'carcha1/XC324666.ogg'],\n ['litegr/XC448898.ogg', 'litegr/XC535540.ogg'],\n ['cabgre1/XC115976.ogg', 'cabgre1/XC116119.ogg'],\n ['cibwar1/XC395511.ogg', 'cibwar1/XC432840.ogg'],\n ['piecro1/XC514153.ogg', 'piecro1/XC605859.ogg'],\n ['reccuc1/XC514156.ogg', 'reccuc1/XC605854.ogg'],\n ['combuz1/XC647786.ogg', 'combuz1/XC647787.ogg'],\n ['sccsun2/XC609477.ogg', 'sccsun2/XC609478.ogg'],\n ['piecro1/XC514160.ogg', 'trobou1/XC608026.ogg'],\n ['somgre1/XC440353.ogg', 'somgre1/XC443103.ogg'],\n ['hoopoe/XC252584.ogg', 'hoopoe/XC631304.ogg'],\n ['woosan/XC740798.ogg', 'woosan/XC742927.ogg'],\n ['grbcam1/XC247923.ogg', 'grbcam1/XC247925.ogg'],\n ['reccuc1/XC118219.ogg', 'reccuc1/XC118431.ogg'],\n ['kerspa2/XC524173.ogg', 'kerspa2/XC524174.ogg'],\n ['barswa/XC575747.ogg', 'barswa/XC575749.ogg'],\n ['hadibi1/XC359010.ogg', 'hadibi1/XC360552.ogg'],\n ['didcuc1/XC403193.ogg', 'didcuc1/XC409433.ogg'],\n ['combuz1/XC144257.ogg', 'combuz1/XC144258.ogg'],\n ['somgre1/XC443513.ogg', 'somgre1/XC443747.ogg'],\n ['cohmar1/XC564020.ogg', 'cohmar1/XC564021.ogg'],\n ['fotdro5/XC619811.ogg', 'fotdro5/XC664384.ogg'],\n ['litegr/XC411319.ogg', 'litegr/XC411320.ogg'],\n ['blaplo1/XC416393.ogg', 'blaplo1/XC418717.ogg'],\n ['fotdro5/XC619817.ogg', 'fotdro5/XC664389.ogg'],\n ['tafpri1/XC443724.ogg', 'tafpri1/XC443725.ogg'],\n ['litswi1/XC443712.ogg', 'litswi1/XC443713.ogg'],\n ['cibwar1/XC316684.ogg', 'gnbcam2/XC316684.ogg'],\n ['cohmar1/XC553166.ogg', 'cohmar1/XC558316.ogg'],\n ['fotdro5/XC619815.ogg', 'fotdro5/XC664387.ogg'],\n ['comsan/XC613127.ogg', 'comsan/XC613128.ogg'],\n ['gyhbus1/XC251906.ogg', 'gyhbus1/XC251907.ogg'],\n ['somgre1/XC608053.ogg', 'somgre1/XC609998.ogg'],\n ['egygoo/XC358927.ogg', 'egygoo/XC528135.ogg'],\n ['strher/XC453981.ogg', 'strher/XC550529.ogg'],\n ['btweye2/XC530963.ogg', 'btweye2/XC532754.ogg'],\n ['wtbeat1/XC234928.ogg', 'wtbeat1/XC234929.ogg'],\n ['abhori1/XC430723.ogg', 'crheag1/XC430724.ogg'],\n ['cohmar1/XC658833.ogg', 'cohmar1/XC729379.ogg'],\n ['combul2/XC748220.ogg', 'combul2/XC748221.ogg'],\n ['piecro1/XC401557.ogg', 'trobou1/XC401556.ogg'],\n ['subbus1/XC603421.ogg', 'subbus1/XC603426.ogg'],\n ['woosan/XC578599.ogg', 'woosan/XC587076.ogg'],\n ['colsun2/XC755891.ogg', 'colsun2/XC755892.ogg'],\n ['hoopoe/XC365530.ogg', 'hoopoe/XC631301.ogg'],\n ['wookin1/XC704764.ogg', 'wookin1/XC741180.ogg'],\n ['fotdro5/XC619813.ogg', 'fotdro5/XC664386.ogg'],\n ['barswa/XC664976.ogg', 'barswa/XC664977.ogg'],\n ['combul2/XC447668.ogg', 'combul2/XC650877.ogg'],\n ['rerswa1/XC194484.ogg', 'rerswa1/XC753215.ogg'],\n ['gobbun1/XC394478.ogg', 'gobbun1/XC395111.ogg'],\n ['scthon1/XC200766.ogg', 'scthon1/XC200767.ogg'],\n ['woosan/XC494020.ogg', 'woosan/XC496525.ogg'],\n ['categr/XC197438.ogg', 'greegr/XC247286.ogg'],\n ['fotdro5/XC538792.ogg', 'fotdro5/XC540732.ogg'],\n ['thrnig1/XC728003.ogg', 'thrnig1/XC728016.ogg'],\n ['subbus1/XC406665.ogg', 'subbus1/XC418769.ogg'],\n ['wbrcha2/XC613380.ogg', 'wbrcha2/XC613384.ogg'],\n ['fotdro5/XC619812.ogg', 'fotdro5/XC664385.ogg'],\n ['grecor/XC215126.ogg', 'grecor/XC753201.ogg'],\n ['hadibi1/XC492329.ogg', 'hadibi1/XC492330.ogg'],\n ['litswi1/XC440301.ogg', 'litswi1/XC443070.ogg'],\n ['laudov1/XC405374.ogg', 'laudov1/XC405375.ogg'],\n ['gnbcam2/XC530150.ogg', 'gnbcam2/XC530151.ogg'],\n ['sichor1/XC446028.ogg', 'sichor1/XC446029.ogg'],\n ['afrthr1/XC652880.ogg', 'afrthr1/XC652884.ogg'],\n ['fislov1/XC503794.ogg', 'fislov1/XC526237.ogg'],\n ['afpfly1/XC418708.ogg', 'afpfly1/XC454899.ogg'],\n ['crohor1/XC194761.ogg', 'crohor1/XC194762.ogg']]\n
\n
\n

Motivation
\nThanks to Matt OP's great observation posted here, he discovered that there are duplicates in the dataset with different XC ID. My initial approach to find all duplicates was to hash the first five seconds of the audio and see if they are identical or not, but surprisingly none of the audios have the same hash. It turned out that even if they're identical, their values still differ slightly (maybe due to some compression reasons?).

\n

Solutions
\nThe solution is to use dynamic time warping (DTW) distance as the similarity metric. DTW is an common metric used to measure time series similarity since it is robust to time shift.
\nManual validation is still needed since there exists false positives, for example, being completely silent in the first five seconds. This can be done be observing the spectrogram after subtracting the two signals.

\n
\n

True positive: the subtracted spectrogram is black
\n \"True

\n

False positive: the subtracted spectrogram shows parts of respective signals
\n \"False

\n
\n

In the end, there are 18 duplicates that are discovered. They all seem to be reuploads from the same author, but some of them have differences in call types, ratings, and secondary labels. If you are using those fields maybe it is worth thinking about which row to keep.

\n

Do note that this is not guaranteed to find all duplicates. Adjusting the threshold of similarity may find more duplicates.

", "messages": [ { "id": "2201148", "postDate": "03/29/2023 04:57:36", "content": "

- Update 2023/04/11: Added extra pairs found by @robbynevels by comparing the distance between google bird classifier embeddings.
\n- Update 2023/03/31: Added extra pairs that were found by @mattop and not detected by DTW.
\nPlease give their original notebooks upvotes since the identical samples wouldn't be found without their effort:
\nMatt OP's EDA and comparing identical samples
\nThacrobatheskis's 3D embedding EDA
\nThacrobatheskis's identifying duplicate samples in embedding space

\n

TL;DR

\n

My notebook can be found here.

\n

The duplicated audios pairs are

\n
[['afrgos1/XC400373.ogg', 'afrgos1/XC418707.ogg'],\n ['brubru1/XC364486.ogg', 'brubru1/XC364487.ogg'],\n ['grewoo2/XC527938.ogg', 'grewoo2/XC527939.ogg'],\n ['spepig1/XC589785.ogg', 'spepig1/XC590084.ogg'],\n ['yenspu1/XC756890.ogg', 'yenspu1/XC756891.ogg'],\n ['abhori1/XC748240.ogg', 'ccbeat1/XC748235.ogg'],\n ['wlwwar/XC478705.ogg', 'wlwwar/XC478767.ogg'],\n ['combul2/XC447669.ogg', 'combul2/XC650878.ogg'],\n ['combul2/XC401370.ogg', 'combul2/XC401371.ogg'],\n ['litegr/XC447850.ogg', 'litegr/XC535552.ogg'],\n ['carcha1/XC324665.ogg', 'carcha1/XC324666.ogg'],\n ['litegr/XC448898.ogg', 'litegr/XC535540.ogg'],\n ['cabgre1/XC115976.ogg', 'cabgre1/XC116119.ogg'],\n ['cibwar1/XC395511.ogg', 'cibwar1/XC432840.ogg'],\n ['piecro1/XC514153.ogg', 'piecro1/XC605859.ogg'],\n ['reccuc1/XC514156.ogg', 'reccuc1/XC605854.ogg'],\n ['combuz1/XC647786.ogg', 'combuz1/XC647787.ogg'],\n ['sccsun2/XC609477.ogg', 'sccsun2/XC609478.ogg'],\n ['piecro1/XC514160.ogg', 'trobou1/XC608026.ogg'],\n ['somgre1/XC440353.ogg', 'somgre1/XC443103.ogg'],\n ['hoopoe/XC252584.ogg', 'hoopoe/XC631304.ogg'],\n ['woosan/XC740798.ogg', 'woosan/XC742927.ogg'],\n ['grbcam1/XC247923.ogg', 'grbcam1/XC247925.ogg'],\n ['reccuc1/XC118219.ogg', 'reccuc1/XC118431.ogg'],\n ['kerspa2/XC524173.ogg', 'kerspa2/XC524174.ogg'],\n ['barswa/XC575747.ogg', 'barswa/XC575749.ogg'],\n ['hadibi1/XC359010.ogg', 'hadibi1/XC360552.ogg'],\n ['didcuc1/XC403193.ogg', 'didcuc1/XC409433.ogg'],\n ['combuz1/XC144257.ogg', 'combuz1/XC144258.ogg'],\n ['somgre1/XC443513.ogg', 'somgre1/XC443747.ogg'],\n ['cohmar1/XC564020.ogg', 'cohmar1/XC564021.ogg'],\n ['fotdro5/XC619811.ogg', 'fotdro5/XC664384.ogg'],\n ['litegr/XC411319.ogg', 'litegr/XC411320.ogg'],\n ['blaplo1/XC416393.ogg', 'blaplo1/XC418717.ogg'],\n ['fotdro5/XC619817.ogg', 'fotdro5/XC664389.ogg'],\n ['tafpri1/XC443724.ogg', 'tafpri1/XC443725.ogg'],\n ['litswi1/XC443712.ogg', 'litswi1/XC443713.ogg'],\n ['cibwar1/XC316684.ogg', 'gnbcam2/XC316684.ogg'],\n ['cohmar1/XC553166.ogg', 'cohmar1/XC558316.ogg'],\n ['fotdro5/XC619815.ogg', 'fotdro5/XC664387.ogg'],\n ['comsan/XC613127.ogg', 'comsan/XC613128.ogg'],\n ['gyhbus1/XC251906.ogg', 'gyhbus1/XC251907.ogg'],\n ['somgre1/XC608053.ogg', 'somgre1/XC609998.ogg'],\n ['egygoo/XC358927.ogg', 'egygoo/XC528135.ogg'],\n ['strher/XC453981.ogg', 'strher/XC550529.ogg'],\n ['btweye2/XC530963.ogg', 'btweye2/XC532754.ogg'],\n ['wtbeat1/XC234928.ogg', 'wtbeat1/XC234929.ogg'],\n ['abhori1/XC430723.ogg', 'crheag1/XC430724.ogg'],\n ['cohmar1/XC658833.ogg', 'cohmar1/XC729379.ogg'],\n ['combul2/XC748220.ogg', 'combul2/XC748221.ogg'],\n ['piecro1/XC401557.ogg', 'trobou1/XC401556.ogg'],\n ['subbus1/XC603421.ogg', 'subbus1/XC603426.ogg'],\n ['woosan/XC578599.ogg', 'woosan/XC587076.ogg'],\n ['colsun2/XC755891.ogg', 'colsun2/XC755892.ogg'],\n ['hoopoe/XC365530.ogg', 'hoopoe/XC631301.ogg'],\n ['wookin1/XC704764.ogg', 'wookin1/XC741180.ogg'],\n ['fotdro5/XC619813.ogg', 'fotdro5/XC664386.ogg'],\n ['barswa/XC664976.ogg', 'barswa/XC664977.ogg'],\n ['combul2/XC447668.ogg', 'combul2/XC650877.ogg'],\n ['rerswa1/XC194484.ogg', 'rerswa1/XC753215.ogg'],\n ['gobbun1/XC394478.ogg', 'gobbun1/XC395111.ogg'],\n ['scthon1/XC200766.ogg', 'scthon1/XC200767.ogg'],\n ['woosan/XC494020.ogg', 'woosan/XC496525.ogg'],\n ['categr/XC197438.ogg', 'greegr/XC247286.ogg'],\n ['fotdro5/XC538792.ogg', 'fotdro5/XC540732.ogg'],\n ['thrnig1/XC728003.ogg', 'thrnig1/XC728016.ogg'],\n ['subbus1/XC406665.ogg', 'subbus1/XC418769.ogg'],\n ['wbrcha2/XC613380.ogg', 'wbrcha2/XC613384.ogg'],\n ['fotdro5/XC619812.ogg', 'fotdro5/XC664385.ogg'],\n ['grecor/XC215126.ogg', 'grecor/XC753201.ogg'],\n ['hadibi1/XC492329.ogg', 'hadibi1/XC492330.ogg'],\n ['litswi1/XC440301.ogg', 'litswi1/XC443070.ogg'],\n ['laudov1/XC405374.ogg', 'laudov1/XC405375.ogg'],\n ['gnbcam2/XC530150.ogg', 'gnbcam2/XC530151.ogg'],\n ['sichor1/XC446028.ogg', 'sichor1/XC446029.ogg'],\n ['afrthr1/XC652880.ogg', 'afrthr1/XC652884.ogg'],\n ['fislov1/XC503794.ogg', 'fislov1/XC526237.ogg'],\n ['afpfly1/XC418708.ogg', 'afpfly1/XC454899.ogg'],\n ['crohor1/XC194761.ogg', 'crohor1/XC194762.ogg']]\n
\n
\n

Motivation
\nThanks to Matt OP's great observation posted here, he discovered that there are duplicates in the dataset with different XC ID. My initial approach to find all duplicates was to hash the first five seconds of the audio and see if they are identical or not, but surprisingly none of the audios have the same hash. It turned out that even if they're identical, their values still differ slightly (maybe due to some compression reasons?).

\n

Solutions
\nThe solution is to use dynamic time warping (DTW) distance as the similarity metric. DTW is an common metric used to measure time series similarity since it is robust to time shift.
\nManual validation is still needed since there exists false positives, for example, being completely silent in the first five seconds. This can be done be observing the spectrogram after subtracting the two signals.

\n
\n

True positive: the subtracted spectrogram is black
\n \"True

\n

False positive: the subtracted spectrogram shows parts of respective signals
\n \"False

\n
\n

In the end, there are 18 duplicates that are discovered. They all seem to be reuploads from the same author, but some of them have differences in call types, ratings, and secondary labels. If you are using those fields maybe it is worth thinking about which row to keep.

\n

Do note that this is not guaranteed to find all duplicates. Adjusting the threshold of similarity may find more duplicates.

", "rawMarkdown": "**- Update 2023/04/11**: Added extra pairs found by @robbynevels by comparing the distance between google bird classifier embeddings.\n**- Update 2023/03/31**: Added extra pairs that were found by @mattop and not detected by DTW. \nPlease give their original notebooks upvotes since the identical samples wouldn't be found without their effort:\n[Matt OP's EDA and comparing identical samples](https://www.kaggle.com/code/mattop/birdclef-2023-eda)\n[Thacrobatheskis's 3D embedding EDA](https://www.kaggle.com/code/robbynevels/birdclef2023-eda-with-3d-embeddings)\n[Thacrobatheskis's identifying duplicate samples in embedding space](https://www.kaggle.com/code/robbynevels/identifying-duplicates-with-embeddings)\n\n**TL;DR**\n\nMy notebook can be [found here](https://www.kaggle.com/code/lhanhsin/birdclef-2023-identifying-duplicates). \n\nThe duplicated audios pairs are\n```\n[['afrgos1/XC400373.ogg', 'afrgos1/XC418707.ogg'],\n ['brubru1/XC364486.ogg', 'brubru1/XC364487.ogg'],\n ['grewoo2/XC527938.ogg', 'grewoo2/XC527939.ogg'],\n ['spepig1/XC589785.ogg', 'spepig1/XC590084.ogg'],\n ['yenspu1/XC756890.ogg', 'yenspu1/XC756891.ogg'],\n ['abhori1/XC748240.ogg', 'ccbeat1/XC748235.ogg'],\n ['wlwwar/XC478705.ogg', 'wlwwar/XC478767.ogg'],\n ['combul2/XC447669.ogg', 'combul2/XC650878.ogg'],\n ['combul2/XC401370.ogg', 'combul2/XC401371.ogg'],\n ['litegr/XC447850.ogg', 'litegr/XC535552.ogg'],\n ['carcha1/XC324665.ogg', 'carcha1/XC324666.ogg'],\n ['litegr/XC448898.ogg', 'litegr/XC535540.ogg'],\n ['cabgre1/XC115976.ogg', 'cabgre1/XC116119.ogg'],\n ['cibwar1/XC395511.ogg', 'cibwar1/XC432840.ogg'],\n ['piecro1/XC514153.ogg', 'piecro1/XC605859.ogg'],\n ['reccuc1/XC514156.ogg', 'reccuc1/XC605854.ogg'],\n ['combuz1/XC647786.ogg', 'combuz1/XC647787.ogg'],\n ['sccsun2/XC609477.ogg', 'sccsun2/XC609478.ogg'],\n ['piecro1/XC514160.ogg', 'trobou1/XC608026.ogg'],\n ['somgre1/XC440353.ogg', 'somgre1/XC443103.ogg'],\n ['hoopoe/XC252584.ogg', 'hoopoe/XC631304.ogg'],\n ['woosan/XC740798.ogg', 'woosan/XC742927.ogg'],\n ['grbcam1/XC247923.ogg', 'grbcam1/XC247925.ogg'],\n ['reccuc1/XC118219.ogg', 'reccuc1/XC118431.ogg'],\n ['kerspa2/XC524173.ogg', 'kerspa2/XC524174.ogg'],\n ['barswa/XC575747.ogg', 'barswa/XC575749.ogg'],\n ['hadibi1/XC359010.ogg', 'hadibi1/XC360552.ogg'],\n ['didcuc1/XC403193.ogg', 'didcuc1/XC409433.ogg'],\n ['combuz1/XC144257.ogg', 'combuz1/XC144258.ogg'],\n ['somgre1/XC443513.ogg', 'somgre1/XC443747.ogg'],\n ['cohmar1/XC564020.ogg', 'cohmar1/XC564021.ogg'],\n ['fotdro5/XC619811.ogg', 'fotdro5/XC664384.ogg'],\n ['litegr/XC411319.ogg', 'litegr/XC411320.ogg'],\n ['blaplo1/XC416393.ogg', 'blaplo1/XC418717.ogg'],\n ['fotdro5/XC619817.ogg', 'fotdro5/XC664389.ogg'],\n ['tafpri1/XC443724.ogg', 'tafpri1/XC443725.ogg'],\n ['litswi1/XC443712.ogg', 'litswi1/XC443713.ogg'],\n ['cibwar1/XC316684.ogg', 'gnbcam2/XC316684.ogg'],\n ['cohmar1/XC553166.ogg', 'cohmar1/XC558316.ogg'],\n ['fotdro5/XC619815.ogg', 'fotdro5/XC664387.ogg'],\n ['comsan/XC613127.ogg', 'comsan/XC613128.ogg'],\n ['gyhbus1/XC251906.ogg', 'gyhbus1/XC251907.ogg'],\n ['somgre1/XC608053.ogg', 'somgre1/XC609998.ogg'],\n ['egygoo/XC358927.ogg', 'egygoo/XC528135.ogg'],\n ['strher/XC453981.ogg', 'strher/XC550529.ogg'],\n ['btweye2/XC530963.ogg', 'btweye2/XC532754.ogg'],\n ['wtbeat1/XC234928.ogg', 'wtbeat1/XC234929.ogg'],\n ['abhori1/XC430723.ogg', 'crheag1/XC430724.ogg'],\n ['cohmar1/XC658833.ogg', 'cohmar1/XC729379.ogg'],\n ['combul2/XC748220.ogg', 'combul2/XC748221.ogg'],\n ['piecro1/XC401557.ogg', 'trobou1/XC401556.ogg'],\n ['subbus1/XC603421.ogg', 'subbus1/XC603426.ogg'],\n ['woosan/XC578599.ogg', 'woosan/XC587076.ogg'],\n ['colsun2/XC755891.ogg', 'colsun2/XC755892.ogg'],\n ['hoopoe/XC365530.ogg', 'hoopoe/XC631301.ogg'],\n ['wookin1/XC704764.ogg', 'wookin1/XC741180.ogg'],\n ['fotdro5/XC619813.ogg', 'fotdro5/XC664386.ogg'],\n ['barswa/XC664976.ogg', 'barswa/XC664977.ogg'],\n ['combul2/XC447668.ogg', 'combul2/XC650877.ogg'],\n ['rerswa1/XC194484.ogg', 'rerswa1/XC753215.ogg'],\n ['gobbun1/XC394478.ogg', 'gobbun1/XC395111.ogg'],\n ['scthon1/XC200766.ogg', 'scthon1/XC200767.ogg'],\n ['woosan/XC494020.ogg', 'woosan/XC496525.ogg'],\n ['categr/XC197438.ogg', 'greegr/XC247286.ogg'],\n ['fotdro5/XC538792.ogg', 'fotdro5/XC540732.ogg'],\n ['thrnig1/XC728003.ogg', 'thrnig1/XC728016.ogg'],\n ['subbus1/XC406665.ogg', 'subbus1/XC418769.ogg'],\n ['wbrcha2/XC613380.ogg', 'wbrcha2/XC613384.ogg'],\n ['fotdro5/XC619812.ogg', 'fotdro5/XC664385.ogg'],\n ['grecor/XC215126.ogg', 'grecor/XC753201.ogg'],\n ['hadibi1/XC492329.ogg', 'hadibi1/XC492330.ogg'],\n ['litswi1/XC440301.ogg', 'litswi1/XC443070.ogg'],\n ['laudov1/XC405374.ogg', 'laudov1/XC405375.ogg'],\n ['gnbcam2/XC530150.ogg', 'gnbcam2/XC530151.ogg'],\n ['sichor1/XC446028.ogg', 'sichor1/XC446029.ogg'],\n ['afrthr1/XC652880.ogg', 'afrthr1/XC652884.ogg'],\n ['fislov1/XC503794.ogg', 'fislov1/XC526237.ogg'],\n ['afpfly1/XC418708.ogg', 'afpfly1/XC454899.ogg'],\n ['crohor1/XC194761.ogg', 'crohor1/XC194762.ogg']]\n```\n\n----\n\n**Motivation**\nThanks to Matt OP's great observation [posted here](https://www.kaggle.com/competitions/birdclef-2023/discussion/396506), he discovered that there are duplicates in the dataset with different XC ID. My initial approach to find all duplicates was to hash the first five seconds of the audio and see if they are identical or not, but surprisingly none of the audios have the same hash. It turned out that even if they're identical, their values still differ slightly (maybe due to some compression reasons?).\n\n**Solutions**\nThe solution is to use dynamic time warping (DTW) distance as the similarity metric. DTW is an common metric used to measure time series similarity since it is robust to time shift.\nManual validation is still needed since there exists false positives, for example, being completely silent in the first five seconds. This can be done be observing the spectrogram after subtracting the two signals.\n\n>**True positive**: the subtracted spectrogram is black\n![True positive: the subtracted spectrogram is black](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2F8282338570c4c34e66dbbfc3dc236093%2F__results___11_80.png?generation=1680065468372086&alt=media)\n\n>**False positive**: the subtracted spectrogram shows parts of respective signals\n![False positive: the subtracted spectrogram shows parts of respective signals](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2F45767965ae6517dbaa1d2fddfff63872%2F__results___11_48.png?generation=1680065526057087&alt=media)\n\nIn the end, there are 18 duplicates that are discovered. They all seem to be reuploads from the same author, but some of them have differences in call types, ratings, and secondary labels. If you are using those fields maybe it is worth thinking about which row to keep.\n\n*Do note that this is not guaranteed to find all duplicates. Adjusting the threshold of similarity may find more duplicates.*", "votes": null }, { "id": "2202789", "postDate": "03/30/2023 10:31:17", "content": "

nice shaređź‘Ť

", "rawMarkdown": "nice shaređź‘Ť", "votes": null }, { "id": "2203384", "postDate": "03/30/2023 18:57:07", "content": "

Very nice work @lhanhsin, thank you for expanding on the duplicate identification. Using dynamic time warping is a clever idea.

\n

I noticed that the list you provided does not include some of the duplicate pairs I found in my EDA notebook where I used the method of audio durations that match to identify the duplicates. Here is the full list duplicate pairs that have been found up to this point (26 in total):

\n
[['carcha1/XC324665.ogg', 'carcha1/XC324666.ogg'],\n ['wbrcha2/XC613380.ogg', 'wbrcha2/XC613384.ogg'],\n ['comsan/XC613127.ogg', 'comsan/XC613128.ogg'],\n ['tafpri1/XC443724.ogg', 'tafpri1/XC443725.ogg'],\n ['colsun2/XC755891.ogg', 'colsun2/XC755892.ogg'],\n ['grewoo2/XC527938.ogg', 'grewoo2/XC527939.ogg'],\n ['sccsun2/XC609477.ogg', 'sccsun2/XC609478.ogg'],\n ['subbus1/XC603421.ogg', 'subbus1/XC603426.ogg'],\n ['combul2/XC748220.ogg', 'combul2/XC748221.ogg'],\n ['wtbeat1/XC234928.ogg', 'wtbeat1/XC234929.ogg'],\n ['afrthr1/XC652880.ogg', 'afrthr1/XC652884.ogg'],\n ['laudov1/XC405374.ogg', 'laudov1/XC405375.ogg'],\n ['litegr/XC411319.ogg', 'litegr/XC411320.ogg'],\n ['cohmar1/XC564020.ogg', 'cohmar1/XC564021.ogg'],\n ['egygoo/XC358927.ogg', 'egygoo/XC528135.ogg'],\n ['combuz1/XC647786.ogg', 'combuz1/XC647787.ogg'],\n ['combuz1/XC144257.ogg', 'combuz1/XC144258.ogg'],\n ['wlwwar/XC478705.ogg', 'wlwwar/XC478767.ogg'],\n\n# Duplicate audio pairs not found using dynamic time warping (found using audio durations)\n[“woosan/XC740798.ogg\", \"woosan/XC742927.ogg”],\n[“gnbcam2/XC530150.ogg\", \"gnbcam2/XC530151.ogg”], # plotted below\n[“litswi1/XC443712.ogg\", \"litswi1/XC443713.ogg”],\n[“combul2/XC650878.ogg\", \"combul2/XC447669.ogg”],\n[“gobbun1/XC394478.ogg\", \"gobbun1/XC395111.ogg”],\n[“fislov1/XC503794.ogg\", \"fislov1/XC526237.ogg”],\n[“cibwar1/XC395511.ogg\", \"cibwar1/XC432840.ogg”],\n[“combul2/XC447668.ogg\", \"combul2/XC650877.ogg”]]\n
\n

Here are the Mel-Spectrograms side by side for /gnbcam2/XC530150.ogg and /gnbcam2/XC530151.ogg which could answer why DTW could not identify these files as duplicates:

\n

\"\"

\n

Also, have you tried to identify duplicates in the BirdCLEF 2021 & 2022 data using DTW? It would be interesting to see the results as @awsaf49 figured out that pretraining on the 2021 & 2022 data improves performance.

", "rawMarkdown": "Very nice work @lhanhsin, thank you for expanding on the duplicate identification. Using dynamic time warping is a clever idea. \n\nI noticed that the list you provided does not include some of the duplicate pairs I found in my EDA notebook where I used the method of audio durations that match to identify the duplicates. Here is the full list duplicate pairs that have been found up to this point (26 in total):\n\n```\n[['carcha1/XC324665.ogg', 'carcha1/XC324666.ogg'],\n ['wbrcha2/XC613380.ogg', 'wbrcha2/XC613384.ogg'],\n ['comsan/XC613127.ogg', 'comsan/XC613128.ogg'],\n ['tafpri1/XC443724.ogg', 'tafpri1/XC443725.ogg'],\n ['colsun2/XC755891.ogg', 'colsun2/XC755892.ogg'],\n ['grewoo2/XC527938.ogg', 'grewoo2/XC527939.ogg'],\n ['sccsun2/XC609477.ogg', 'sccsun2/XC609478.ogg'],\n ['subbus1/XC603421.ogg', 'subbus1/XC603426.ogg'],\n ['combul2/XC748220.ogg', 'combul2/XC748221.ogg'],\n ['wtbeat1/XC234928.ogg', 'wtbeat1/XC234929.ogg'],\n ['afrthr1/XC652880.ogg', 'afrthr1/XC652884.ogg'],\n ['laudov1/XC405374.ogg', 'laudov1/XC405375.ogg'],\n ['litegr/XC411319.ogg', 'litegr/XC411320.ogg'],\n ['cohmar1/XC564020.ogg', 'cohmar1/XC564021.ogg'],\n ['egygoo/XC358927.ogg', 'egygoo/XC528135.ogg'],\n ['combuz1/XC647786.ogg', 'combuz1/XC647787.ogg'],\n ['combuz1/XC144257.ogg', 'combuz1/XC144258.ogg'],\n ['wlwwar/XC478705.ogg', 'wlwwar/XC478767.ogg'],\n\n# Duplicate audio pairs not found using dynamic time warping (found using audio durations)\n[“woosan/XC740798.ogg\", \"woosan/XC742927.ogg”],\n[“gnbcam2/XC530150.ogg\", \"gnbcam2/XC530151.ogg”], # plotted below\n[“litswi1/XC443712.ogg\", \"litswi1/XC443713.ogg”],\n[“combul2/XC650878.ogg\", \"combul2/XC447669.ogg”],\n[“gobbun1/XC394478.ogg\", \"gobbun1/XC395111.ogg”],\n[“fislov1/XC503794.ogg\", \"fislov1/XC526237.ogg”],\n[“cibwar1/XC395511.ogg\", \"cibwar1/XC432840.ogg”],\n[“combul2/XC447668.ogg\", \"combul2/XC650877.ogg”]]\n```\n\nHere are the Mel-Spectrograms side by side for `/gnbcam2/XC530150.ogg` and `/gnbcam2/XC530151.ogg` which could answer why DTW could not identify these files as duplicates:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10590800%2F77678a8222d0e43b97a43c4244a27d01%2Fmel-spect-db-shift.png?generation=1680201585698872&alt=media)\n\nAlso, have you tried to identify duplicates in the BirdCLEF 2021 & 2022 data using DTW? It would be interesting to see the results as @awsaf49 figured out that pretraining on the 2021 & 2022 data improves performance.", "votes": null }, { "id": "2203622", "postDate": "03/31/2023 01:50:50", "content": "

Thank you for your input! I'll update the list above.

\n

I dug a bit deeper into the first two pairs you provided.
\nIn the first example there is a slight time shift, but DTW wasn't able to detect it. The info of the audio says that they have a slightly different bitrate which may be the reason. Maybe a possible solution to this is to run DTW on the spectrogram instead of the raw audio signal it self.

\n
\n

[“woosan/XC740798.ogg\", \"woosan/XC742927.ogg”] A slight time shift
\n \"First

\n
\n

The second example doesn't have a time shift, but seems to be denoised. The difference in the low frequency range is obvious.
\nI'm not sure what a good solution would be though, if we do some sort of denoising too before comparing them would work but will also generate a lot of false positives. If we change to using a image similarity detection, the number of false positives will still be an issue.

\n
\n

[“gnbcam2/XC530150.ogg\", \"gnbcam2/XC530151.ogg”] No time shift, but seems to be denoised.
\n \"Second

\n
\n

The only solution that I can think of right now is to:

\n
    \n
  1. Trim the audio, trimming out the silence will lower the number of false positives.
    2. \n
  2. Run STFT
    3. \n
  3. Denoise the audio, something simple like clipping the magnitude in the spectrogram might work.
    4. \n
  4. Run DTW across different frequencies in the spectrogram, this should be more robust compared to running on the raw signal.
    5. \n
\n

This will not cover situations like different sound intensity (Normalizing the audio should work?) or slight shifts in the frequency. Maybe treating the spectrogram as an image and use some similarity model will be a solution for all the issues above, but requires some experiments.

\n

And no I haven't applied it to BirdClef 2021&2022 yet! I wrote the code with that in mind but it's now a low priority task.
\nIdentifying duplicates in the BirdClef 2023 is important because I want to avoid contaminating the validation set. However, in the pretraining part it's not that much of an issue since the hidden test set is released so there is no contamination there, even if there was contamination it wouldn't do much harm since after all, it's only for pretraining. The only thing to watch out for is if the 2023 validation set has duplicates in the 2021&2022.

", "rawMarkdown": "Thank you for your input! I'll update the list above.\n\nI dug a bit deeper into the first two pairs you provided.\nIn the first example there is a slight time shift, but DTW wasn't able to detect it. The info of the audio says that they have a slightly different bitrate which may be the reason. Maybe a possible solution to this is to run DTW on the spectrogram instead of the raw audio signal it self.\n>**[“woosan/XC740798.ogg\", \"woosan/XC742927.ogg”]** A slight time shift\n![First pair](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2F26d660baf8ce27cc92488fd68bee2f84%2Ffirst_pair.png?generation=1680225249642445&alt=media)\n\nThe second example doesn't have a time shift, but seems to be denoised. The difference in the low frequency range is obvious. \nI'm not sure what a good solution would be though, if we do some sort of denoising too before comparing them would work but will also generate a lot of false positives. If we change to using a image similarity detection, the number of false positives will still be an issue.\n>**[“gnbcam2/XC530150.ogg\", \"gnbcam2/XC530151.ogg”]** No time shift, but seems to be denoised.\n![Second pair](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2F057d7e76cffc8e92e46128c35f12731b%2Fsecond_image.png?generation=1680225628408916&alt=media)\n\nThe only solution that I can think of right now is to:\n1. Trim the audio, trimming out the silence will lower the number of false positives.\n2. Run STFT\n3. Denoise the audio, something simple like clipping the magnitude in the spectrogram might work.\n4. Run DTW across different frequencies in the spectrogram, this should be more robust compared to running on the raw signal.\n\nThis will not cover situations like different sound intensity (Normalizing the audio should work?) or slight shifts in the frequency. Maybe treating the spectrogram as an image and use some similarity model will be a solution for all the issues above, but requires some experiments.\n\nAnd no I haven't applied it to BirdClef 2021&2022 yet! I wrote the code with that in mind but it's now a low priority task. \nIdentifying duplicates in the BirdClef 2023 is important because I want to avoid contaminating the validation set. However, in the pretraining part it's not that much of an issue since the [hidden test set is released](https://www.kaggle.com/competitions/birdclef-2023/discussion/394358) so there is no contamination there, even if there was contamination it wouldn't do much harm since after all, it's only for pretraining. The only thing to watch out for is if the 2023 validation set has duplicates in the 2021&2022.", "votes": null }, { "id": "2215845", "postDate": "04/09/2023 16:21:20", "content": "

Thanks for finding this! I found 54 more duplicates by comparing the google bird classifier embeddings of the first and last 5 seconds of each recording + some thresholds on the audio magnitude + some manual filtering. My notebook is here. I also explored visualizing embeddings in 3D here.

\n

This technique also uncovered all the duplicates you and @mattop found except for 1. I think there's probably several more that could be found by adjusting thresholds or combining our techniques.

\n

Here's a full list now (also listed in the notebook) [edited to remove one pair, based on the convo below]:

\n
[['afrgos1/XC400373.ogg', 'afrgos1/XC418707.ogg'],\n ['brubru1/XC364486.ogg', 'brubru1/XC364487.ogg'],\n ['grewoo2/XC527938.ogg', 'grewoo2/XC527939.ogg'],\n ['spepig1/XC589785.ogg', 'spepig1/XC590084.ogg'],\n ['yenspu1/XC756890.ogg', 'yenspu1/XC756891.ogg'],\n ['abhori1/XC748240.ogg', 'ccbeat1/XC748235.ogg'],\n ['wlwwar/XC478705.ogg', 'wlwwar/XC478767.ogg'],\n ['combul2/XC447669.ogg', 'combul2/XC650878.ogg'],\n ['combul2/XC401370.ogg', 'combul2/XC401371.ogg'],\n ['litegr/XC447850.ogg', 'litegr/XC535552.ogg'],\n ['carcha1/XC324665.ogg', 'carcha1/XC324666.ogg'],\n ['litegr/XC448898.ogg', 'litegr/XC535540.ogg'],\n ['cabgre1/XC115976.ogg', 'cabgre1/XC116119.ogg'],\n ['cibwar1/XC395511.ogg', 'cibwar1/XC432840.ogg'],\n ['piecro1/XC514153.ogg', 'piecro1/XC605859.ogg'],\n ['reccuc1/XC514156.ogg', 'reccuc1/XC605854.ogg'],\n ['combuz1/XC647786.ogg', 'combuz1/XC647787.ogg'],\n ['sccsun2/XC609477.ogg', 'sccsun2/XC609478.ogg'],\n ['piecro1/XC514160.ogg', 'trobou1/XC608026.ogg'],\n ['somgre1/XC440353.ogg', 'somgre1/XC443103.ogg'],\n ['hoopoe/XC252584.ogg', 'hoopoe/XC631304.ogg'],\n ['woosan/XC740798.ogg', 'woosan/XC742927.ogg'],\n ['grbcam1/XC247923.ogg', 'grbcam1/XC247925.ogg'],\n ['reccuc1/XC118219.ogg', 'reccuc1/XC118431.ogg'],\n ['kerspa2/XC524173.ogg', 'kerspa2/XC524174.ogg'],\n ['barswa/XC575747.ogg', 'barswa/XC575749.ogg'],\n ['hadibi1/XC359010.ogg', 'hadibi1/XC360552.ogg'],\n ['didcuc1/XC403193.ogg', 'didcuc1/XC409433.ogg'],\n ['combuz1/XC144257.ogg', 'combuz1/XC144258.ogg'],\n ['somgre1/XC443513.ogg', 'somgre1/XC443747.ogg'],\n ['cohmar1/XC564020.ogg', 'cohmar1/XC564021.ogg'],\n ['fotdro5/XC619811.ogg', 'fotdro5/XC664384.ogg'],\n ['litegr/XC411319.ogg', 'litegr/XC411320.ogg'],\n ['blaplo1/XC416393.ogg', 'blaplo1/XC418717.ogg'],\n ['fotdro5/XC619817.ogg', 'fotdro5/XC664389.ogg'],\n ['tafpri1/XC443724.ogg', 'tafpri1/XC443725.ogg'],\n ['litswi1/XC443712.ogg', 'litswi1/XC443713.ogg'],\n ['cibwar1/XC316684.ogg', 'gnbcam2/XC316684.ogg'],\n ['cohmar1/XC553166.ogg', 'cohmar1/XC558316.ogg'],\n ['fotdro5/XC619815.ogg', 'fotdro5/XC664387.ogg'],\n ['comsan/XC613127.ogg', 'comsan/XC613128.ogg'],\n ['gyhbus1/XC251906.ogg', 'gyhbus1/XC251907.ogg'],\n ['somgre1/XC608053.ogg', 'somgre1/XC609998.ogg'],\n ['egygoo/XC358927.ogg', 'egygoo/XC528135.ogg'],\n ['strher/XC453981.ogg', 'strher/XC550529.ogg'],\n ['btweye2/XC530963.ogg', 'btweye2/XC532754.ogg'],\n ['wtbeat1/XC234928.ogg', 'wtbeat1/XC234929.ogg'],\n ['abhori1/XC430723.ogg', 'crheag1/XC430724.ogg'],\n ['cohmar1/XC658833.ogg', 'cohmar1/XC729379.ogg'],\n ['combul2/XC748220.ogg', 'combul2/XC748221.ogg'],\n ['piecro1/XC401557.ogg', 'trobou1/XC401556.ogg'],\n ['subbus1/XC603421.ogg', 'subbus1/XC603426.ogg'],\n ['woosan/XC578599.ogg', 'woosan/XC587076.ogg'],\n ['colsun2/XC755891.ogg', 'colsun2/XC755892.ogg'],\n ['hoopoe/XC365530.ogg', 'hoopoe/XC631301.ogg'],\n ['wookin1/XC704764.ogg', 'wookin1/XC741180.ogg'],\n ['fotdro5/XC619813.ogg', 'fotdro5/XC664386.ogg'],\n ['barswa/XC664976.ogg', 'barswa/XC664977.ogg'],\n ['combul2/XC447668.ogg', 'combul2/XC650877.ogg'],\n ['rerswa1/XC194484.ogg', 'rerswa1/XC753215.ogg'],\n ['gobbun1/XC394478.ogg', 'gobbun1/XC395111.ogg'],\n ['scthon1/XC200766.ogg', 'scthon1/XC200767.ogg'],\n ['woosan/XC494020.ogg', 'woosan/XC496525.ogg'],\n ['categr/XC197438.ogg', 'greegr/XC247286.ogg'],\n ['fotdro5/XC538792.ogg', 'fotdro5/XC540732.ogg'],\n ['thrnig1/XC728003.ogg', 'thrnig1/XC728016.ogg'],\n ['subbus1/XC406665.ogg', 'subbus1/XC418769.ogg'],\n ['wbrcha2/XC613380.ogg', 'wbrcha2/XC613384.ogg'],\n ['fotdro5/XC619812.ogg', 'fotdro5/XC664385.ogg'],\n ['grecor/XC215126.ogg', 'grecor/XC753201.ogg'],\n ['hadibi1/XC492329.ogg', 'hadibi1/XC492330.ogg'],\n ['litswi1/XC440301.ogg', 'litswi1/XC443070.ogg'],\n ['laudov1/XC405374.ogg', 'laudov1/XC405375.ogg'],\n ['gnbcam2/XC530150.ogg', 'gnbcam2/XC530151.ogg'],\n ['sichor1/XC446028.ogg', 'sichor1/XC446029.ogg'],\n ['afrthr1/XC652880.ogg', 'afrthr1/XC652884.ogg'],\n ['fislov1/XC503794.ogg', 'fislov1/XC526237.ogg'],\n ['afpfly1/XC418708.ogg', 'afpfly1/XC454899.ogg'],\n ['crohor1/XC194761.ogg', 'crohor1/XC194762.ogg']]\n
", "rawMarkdown": "Thanks for finding this! I found 54 more duplicates by comparing the google bird classifier embeddings of the first and last 5 seconds of each recording + some thresholds on the audio magnitude + some manual filtering. My notebook is [here](https://www.kaggle.com/code/robbynevels/identifying-duplicates-with-embeddings). I also explored visualizing embeddings in 3D [here](https://www.kaggle.com/code/robbynevels/birdclef2023-eda-with-3d-embeddings).\n\nThis technique also uncovered all the duplicates you and @mattop found except for 1. I think there's probably several more that could be found by adjusting thresholds or combining our techniques.\n\nHere's a full list now (also listed in the notebook) [edited to remove one pair, based on the convo below]:\n\n```\n[['afrgos1/XC400373.ogg', 'afrgos1/XC418707.ogg'],\n ['brubru1/XC364486.ogg', 'brubru1/XC364487.ogg'],\n ['grewoo2/XC527938.ogg', 'grewoo2/XC527939.ogg'],\n ['spepig1/XC589785.ogg', 'spepig1/XC590084.ogg'],\n ['yenspu1/XC756890.ogg', 'yenspu1/XC756891.ogg'],\n ['abhori1/XC748240.ogg', 'ccbeat1/XC748235.ogg'],\n ['wlwwar/XC478705.ogg', 'wlwwar/XC478767.ogg'],\n ['combul2/XC447669.ogg', 'combul2/XC650878.ogg'],\n ['combul2/XC401370.ogg', 'combul2/XC401371.ogg'],\n ['litegr/XC447850.ogg', 'litegr/XC535552.ogg'],\n ['carcha1/XC324665.ogg', 'carcha1/XC324666.ogg'],\n ['litegr/XC448898.ogg', 'litegr/XC535540.ogg'],\n ['cabgre1/XC115976.ogg', 'cabgre1/XC116119.ogg'],\n ['cibwar1/XC395511.ogg', 'cibwar1/XC432840.ogg'],\n ['piecro1/XC514153.ogg', 'piecro1/XC605859.ogg'],\n ['reccuc1/XC514156.ogg', 'reccuc1/XC605854.ogg'],\n ['combuz1/XC647786.ogg', 'combuz1/XC647787.ogg'],\n ['sccsun2/XC609477.ogg', 'sccsun2/XC609478.ogg'],\n ['piecro1/XC514160.ogg', 'trobou1/XC608026.ogg'],\n ['somgre1/XC440353.ogg', 'somgre1/XC443103.ogg'],\n ['hoopoe/XC252584.ogg', 'hoopoe/XC631304.ogg'],\n ['woosan/XC740798.ogg', 'woosan/XC742927.ogg'],\n ['grbcam1/XC247923.ogg', 'grbcam1/XC247925.ogg'],\n ['reccuc1/XC118219.ogg', 'reccuc1/XC118431.ogg'],\n ['kerspa2/XC524173.ogg', 'kerspa2/XC524174.ogg'],\n ['barswa/XC575747.ogg', 'barswa/XC575749.ogg'],\n ['hadibi1/XC359010.ogg', 'hadibi1/XC360552.ogg'],\n ['didcuc1/XC403193.ogg', 'didcuc1/XC409433.ogg'],\n ['combuz1/XC144257.ogg', 'combuz1/XC144258.ogg'],\n ['somgre1/XC443513.ogg', 'somgre1/XC443747.ogg'],\n ['cohmar1/XC564020.ogg', 'cohmar1/XC564021.ogg'],\n ['fotdro5/XC619811.ogg', 'fotdro5/XC664384.ogg'],\n ['litegr/XC411319.ogg', 'litegr/XC411320.ogg'],\n ['blaplo1/XC416393.ogg', 'blaplo1/XC418717.ogg'],\n ['fotdro5/XC619817.ogg', 'fotdro5/XC664389.ogg'],\n ['tafpri1/XC443724.ogg', 'tafpri1/XC443725.ogg'],\n ['litswi1/XC443712.ogg', 'litswi1/XC443713.ogg'],\n ['cibwar1/XC316684.ogg', 'gnbcam2/XC316684.ogg'],\n ['cohmar1/XC553166.ogg', 'cohmar1/XC558316.ogg'],\n ['fotdro5/XC619815.ogg', 'fotdro5/XC664387.ogg'],\n ['comsan/XC613127.ogg', 'comsan/XC613128.ogg'],\n ['gyhbus1/XC251906.ogg', 'gyhbus1/XC251907.ogg'],\n ['somgre1/XC608053.ogg', 'somgre1/XC609998.ogg'],\n ['egygoo/XC358927.ogg', 'egygoo/XC528135.ogg'],\n ['strher/XC453981.ogg', 'strher/XC550529.ogg'],\n ['btweye2/XC530963.ogg', 'btweye2/XC532754.ogg'],\n ['wtbeat1/XC234928.ogg', 'wtbeat1/XC234929.ogg'],\n ['abhori1/XC430723.ogg', 'crheag1/XC430724.ogg'],\n ['cohmar1/XC658833.ogg', 'cohmar1/XC729379.ogg'],\n ['combul2/XC748220.ogg', 'combul2/XC748221.ogg'],\n ['piecro1/XC401557.ogg', 'trobou1/XC401556.ogg'],\n ['subbus1/XC603421.ogg', 'subbus1/XC603426.ogg'],\n ['woosan/XC578599.ogg', 'woosan/XC587076.ogg'],\n ['colsun2/XC755891.ogg', 'colsun2/XC755892.ogg'],\n ['hoopoe/XC365530.ogg', 'hoopoe/XC631301.ogg'],\n ['wookin1/XC704764.ogg', 'wookin1/XC741180.ogg'],\n ['fotdro5/XC619813.ogg', 'fotdro5/XC664386.ogg'],\n ['barswa/XC664976.ogg', 'barswa/XC664977.ogg'],\n ['combul2/XC447668.ogg', 'combul2/XC650877.ogg'],\n ['rerswa1/XC194484.ogg', 'rerswa1/XC753215.ogg'],\n ['gobbun1/XC394478.ogg', 'gobbun1/XC395111.ogg'],\n ['scthon1/XC200766.ogg', 'scthon1/XC200767.ogg'],\n ['woosan/XC494020.ogg', 'woosan/XC496525.ogg'],\n ['categr/XC197438.ogg', 'greegr/XC247286.ogg'],\n ['fotdro5/XC538792.ogg', 'fotdro5/XC540732.ogg'],\n ['thrnig1/XC728003.ogg', 'thrnig1/XC728016.ogg'],\n ['subbus1/XC406665.ogg', 'subbus1/XC418769.ogg'],\n ['wbrcha2/XC613380.ogg', 'wbrcha2/XC613384.ogg'],\n ['fotdro5/XC619812.ogg', 'fotdro5/XC664385.ogg'],\n ['grecor/XC215126.ogg', 'grecor/XC753201.ogg'],\n ['hadibi1/XC492329.ogg', 'hadibi1/XC492330.ogg'],\n ['litswi1/XC440301.ogg', 'litswi1/XC443070.ogg'],\n ['laudov1/XC405374.ogg', 'laudov1/XC405375.ogg'],\n ['gnbcam2/XC530150.ogg', 'gnbcam2/XC530151.ogg'],\n ['sichor1/XC446028.ogg', 'sichor1/XC446029.ogg'],\n ['afrthr1/XC652880.ogg', 'afrthr1/XC652884.ogg'],\n ['fislov1/XC503794.ogg', 'fislov1/XC526237.ogg'],\n ['afpfly1/XC418708.ogg', 'afpfly1/XC454899.ogg'],\n ['crohor1/XC194761.ogg', 'crohor1/XC194762.ogg']]\n```", "votes": null }, { "id": "2216273", "postDate": "04/10/2023 00:15:15", "content": "

This is awesome! The embedding visualizations look great and provides some extra insight over the data.
\nUsing embeddings for discovering duplicates is indeed a clever way to get rid of all the noise and nuances in the data. Unfortunately, your first link doesn't work so I can't see the notebook.

", "rawMarkdown": "This is awesome! The embedding visualizations look great and provides some extra insight over the data.\nUsing embeddings for discovering duplicates is indeed a clever way to get rid of all the noise and nuances in the data. Unfortunately, your first link doesn't work so I can't see the notebook.", "votes": null }, { "id": "2216491", "postDate": "04/10/2023 05:50:29", "content": "

Whoops, sorry! Forgot to make it public. It should be viewable now.

", "rawMarkdown": "Whoops, sorry! Forgot to make it public. It should be viewable now.", "votes": null }, { "id": "2216618", "postDate": "04/10/2023 07:51:00", "content": "

Nice! Would you mind double checking these four pairs below? I think they are distinct.
\nI'll update the list above after you confirm.

\n
7359 gnbcam2/XC195528.ogg 6995 fotdro5/XC195989.ogg 0.666067898273468\n16073 woosan/XC587076.ogg 16049 woosan/XC578599.ogg 1.6366488933563232\n12077 rerswa1/XC753215.ogg 11896 rerswa1/XC194484.ogg 2.4423563480377197\n5570 crohor1/XC194762.ogg 5569 crohor1/XC194761.ogg 3.9656319618225098\n
", "rawMarkdown": "Nice! Would you mind double checking these four pairs below? I think they are distinct.\nI'll update the list above after you confirm.\n```\n7359 gnbcam2/XC195528.ogg 6995 fotdro5/XC195989.ogg 0.666067898273468\n16073 woosan/XC587076.ogg 16049 woosan/XC578599.ogg 1.6366488933563232\n12077 rerswa1/XC753215.ogg 11896 rerswa1/XC194484.ogg 2.4423563480377197\n5570 crohor1/XC194762.ogg 5569 crohor1/XC194761.ogg 3.9656319618225098\n```", "votes": null }, { "id": "2216912", "postDate": "04/10/2023 13:17:25", "content": "

Ah, you're 100% right about that first one. But for the other 3, one of the pair is a crop of the other one. This is more obvious when zooming out to see the first 10 seconds: https://www.kaggle.com/robbynevels/zooming-out-on-some-duplicate-recordings

", "rawMarkdown": "Ah, you're 100% right about that first one. But for the other 3, one of the pair is a crop of the other one. This is more obvious when zooming out to see the first 10 seconds: https://www.kaggle.com/robbynevels/zooming-out-on-some-duplicate-recordings", "votes": null } ], "comments": [ { "id": 2202789, "author_name": "chongleongchan", "author_url": "", "post_date": "03/30/2023 10:31:17", "content": "

nice shaređź‘Ť

", "votes": null, "replies": [] }, { "id": 2203384, "author_name": "mattop", "author_url": "", "post_date": "03/30/2023 18:57:07", "content": "

Very nice work @lhanhsin, thank you for expanding on the duplicate identification. Using dynamic time warping is a clever idea.

\n

I noticed that the list you provided does not include some of the duplicate pairs I found in my EDA notebook where I used the method of audio durations that match to identify the duplicates. Here is the full list duplicate pairs that have been found up to this point (26 in total):

\n
[['carcha1/XC324665.ogg', 'carcha1/XC324666.ogg'],\n ['wbrcha2/XC613380.ogg', 'wbrcha2/XC613384.ogg'],\n ['comsan/XC613127.ogg', 'comsan/XC613128.ogg'],\n ['tafpri1/XC443724.ogg', 'tafpri1/XC443725.ogg'],\n ['colsun2/XC755891.ogg', 'colsun2/XC755892.ogg'],\n ['grewoo2/XC527938.ogg', 'grewoo2/XC527939.ogg'],\n ['sccsun2/XC609477.ogg', 'sccsun2/XC609478.ogg'],\n ['subbus1/XC603421.ogg', 'subbus1/XC603426.ogg'],\n ['combul2/XC748220.ogg', 'combul2/XC748221.ogg'],\n ['wtbeat1/XC234928.ogg', 'wtbeat1/XC234929.ogg'],\n ['afrthr1/XC652880.ogg', 'afrthr1/XC652884.ogg'],\n ['laudov1/XC405374.ogg', 'laudov1/XC405375.ogg'],\n ['litegr/XC411319.ogg', 'litegr/XC411320.ogg'],\n ['cohmar1/XC564020.ogg', 'cohmar1/XC564021.ogg'],\n ['egygoo/XC358927.ogg', 'egygoo/XC528135.ogg'],\n ['combuz1/XC647786.ogg', 'combuz1/XC647787.ogg'],\n ['combuz1/XC144257.ogg', 'combuz1/XC144258.ogg'],\n ['wlwwar/XC478705.ogg', 'wlwwar/XC478767.ogg'],\n\n# Duplicate audio pairs not found using dynamic time warping (found using audio durations)\n[“woosan/XC740798.ogg\", \"woosan/XC742927.ogg”],\n[“gnbcam2/XC530150.ogg\", \"gnbcam2/XC530151.ogg”], # plotted below\n[“litswi1/XC443712.ogg\", \"litswi1/XC443713.ogg”],\n[“combul2/XC650878.ogg\", \"combul2/XC447669.ogg”],\n[“gobbun1/XC394478.ogg\", \"gobbun1/XC395111.ogg”],\n[“fislov1/XC503794.ogg\", \"fislov1/XC526237.ogg”],\n[“cibwar1/XC395511.ogg\", \"cibwar1/XC432840.ogg”],\n[“combul2/XC447668.ogg\", \"combul2/XC650877.ogg”]]\n
\n

Here are the Mel-Spectrograms side by side for /gnbcam2/XC530150.ogg and /gnbcam2/XC530151.ogg which could answer why DTW could not identify these files as duplicates:

\n

\"\"

\n

Also, have you tried to identify duplicates in the BirdCLEF 2021 & 2022 data using DTW? It would be interesting to see the results as @awsaf49 figured out that pretraining on the 2021 & 2022 data improves performance.

", "votes": null, "replies": [ { "id": 2203622, "author_name": "lhanhsin", "author_url": "", "post_date": "03/31/2023 01:50:50", "content": "

Thank you for your input! I'll update the list above.

\n

I dug a bit deeper into the first two pairs you provided.
\nIn the first example there is a slight time shift, but DTW wasn't able to detect it. The info of the audio says that they have a slightly different bitrate which may be the reason. Maybe a possible solution to this is to run DTW on the spectrogram instead of the raw audio signal it self.

\n
\n

[“woosan/XC740798.ogg\", \"woosan/XC742927.ogg”] A slight time shift
\n \"First

\n
\n

The second example doesn't have a time shift, but seems to be denoised. The difference in the low frequency range is obvious.
\nI'm not sure what a good solution would be though, if we do some sort of denoising too before comparing them would work but will also generate a lot of false positives. If we change to using a image similarity detection, the number of false positives will still be an issue.

\n
\n

[“gnbcam2/XC530150.ogg\", \"gnbcam2/XC530151.ogg”] No time shift, but seems to be denoised.
\n \"Second

\n
\n

The only solution that I can think of right now is to:

\n
    \n
  1. Trim the audio, trimming out the silence will lower the number of false positives.
    2. \n
  2. Run STFT
    3. \n
  3. Denoise the audio, something simple like clipping the magnitude in the spectrogram might work.
    4. \n
  4. Run DTW across different frequencies in the spectrogram, this should be more robust compared to running on the raw signal.
    5. \n
\n

This will not cover situations like different sound intensity (Normalizing the audio should work?) or slight shifts in the frequency. Maybe treating the spectrogram as an image and use some similarity model will be a solution for all the issues above, but requires some experiments.

\n

And no I haven't applied it to BirdClef 2021&2022 yet! I wrote the code with that in mind but it's now a low priority task.
\nIdentifying duplicates in the BirdClef 2023 is important because I want to avoid contaminating the validation set. However, in the pretraining part it's not that much of an issue since the hidden test set is released so there is no contamination there, even if there was contamination it wouldn't do much harm since after all, it's only for pretraining. The only thing to watch out for is if the 2023 validation set has duplicates in the 2021&2022.

", "votes": null, "replies": [] } ] }, { "id": 2215845, "author_name": "robbynevels", "author_url": "", "post_date": "04/09/2023 16:21:20", "content": "

Thanks for finding this! I found 54 more duplicates by comparing the google bird classifier embeddings of the first and last 5 seconds of each recording + some thresholds on the audio magnitude + some manual filtering. My notebook is here. I also explored visualizing embeddings in 3D here.

\n

This technique also uncovered all the duplicates you and @mattop found except for 1. I think there's probably several more that could be found by adjusting thresholds or combining our techniques.

\n

Here's a full list now (also listed in the notebook) [edited to remove one pair, based on the convo below]:

\n
[['afrgos1/XC400373.ogg', 'afrgos1/XC418707.ogg'],\n ['brubru1/XC364486.ogg', 'brubru1/XC364487.ogg'],\n ['grewoo2/XC527938.ogg', 'grewoo2/XC527939.ogg'],\n ['spepig1/XC589785.ogg', 'spepig1/XC590084.ogg'],\n ['yenspu1/XC756890.ogg', 'yenspu1/XC756891.ogg'],\n ['abhori1/XC748240.ogg', 'ccbeat1/XC748235.ogg'],\n ['wlwwar/XC478705.ogg', 'wlwwar/XC478767.ogg'],\n ['combul2/XC447669.ogg', 'combul2/XC650878.ogg'],\n ['combul2/XC401370.ogg', 'combul2/XC401371.ogg'],\n ['litegr/XC447850.ogg', 'litegr/XC535552.ogg'],\n ['carcha1/XC324665.ogg', 'carcha1/XC324666.ogg'],\n ['litegr/XC448898.ogg', 'litegr/XC535540.ogg'],\n ['cabgre1/XC115976.ogg', 'cabgre1/XC116119.ogg'],\n ['cibwar1/XC395511.ogg', 'cibwar1/XC432840.ogg'],\n ['piecro1/XC514153.ogg', 'piecro1/XC605859.ogg'],\n ['reccuc1/XC514156.ogg', 'reccuc1/XC605854.ogg'],\n ['combuz1/XC647786.ogg', 'combuz1/XC647787.ogg'],\n ['sccsun2/XC609477.ogg', 'sccsun2/XC609478.ogg'],\n ['piecro1/XC514160.ogg', 'trobou1/XC608026.ogg'],\n ['somgre1/XC440353.ogg', 'somgre1/XC443103.ogg'],\n ['hoopoe/XC252584.ogg', 'hoopoe/XC631304.ogg'],\n ['woosan/XC740798.ogg', 'woosan/XC742927.ogg'],\n ['grbcam1/XC247923.ogg', 'grbcam1/XC247925.ogg'],\n ['reccuc1/XC118219.ogg', 'reccuc1/XC118431.ogg'],\n ['kerspa2/XC524173.ogg', 'kerspa2/XC524174.ogg'],\n ['barswa/XC575747.ogg', 'barswa/XC575749.ogg'],\n ['hadibi1/XC359010.ogg', 'hadibi1/XC360552.ogg'],\n ['didcuc1/XC403193.ogg', 'didcuc1/XC409433.ogg'],\n ['combuz1/XC144257.ogg', 'combuz1/XC144258.ogg'],\n ['somgre1/XC443513.ogg', 'somgre1/XC443747.ogg'],\n ['cohmar1/XC564020.ogg', 'cohmar1/XC564021.ogg'],\n ['fotdro5/XC619811.ogg', 'fotdro5/XC664384.ogg'],\n ['litegr/XC411319.ogg', 'litegr/XC411320.ogg'],\n ['blaplo1/XC416393.ogg', 'blaplo1/XC418717.ogg'],\n ['fotdro5/XC619817.ogg', 'fotdro5/XC664389.ogg'],\n ['tafpri1/XC443724.ogg', 'tafpri1/XC443725.ogg'],\n ['litswi1/XC443712.ogg', 'litswi1/XC443713.ogg'],\n ['cibwar1/XC316684.ogg', 'gnbcam2/XC316684.ogg'],\n ['cohmar1/XC553166.ogg', 'cohmar1/XC558316.ogg'],\n ['fotdro5/XC619815.ogg', 'fotdro5/XC664387.ogg'],\n ['comsan/XC613127.ogg', 'comsan/XC613128.ogg'],\n ['gyhbus1/XC251906.ogg', 'gyhbus1/XC251907.ogg'],\n ['somgre1/XC608053.ogg', 'somgre1/XC609998.ogg'],\n ['egygoo/XC358927.ogg', 'egygoo/XC528135.ogg'],\n ['strher/XC453981.ogg', 'strher/XC550529.ogg'],\n ['btweye2/XC530963.ogg', 'btweye2/XC532754.ogg'],\n ['wtbeat1/XC234928.ogg', 'wtbeat1/XC234929.ogg'],\n ['abhori1/XC430723.ogg', 'crheag1/XC430724.ogg'],\n ['cohmar1/XC658833.ogg', 'cohmar1/XC729379.ogg'],\n ['combul2/XC748220.ogg', 'combul2/XC748221.ogg'],\n ['piecro1/XC401557.ogg', 'trobou1/XC401556.ogg'],\n ['subbus1/XC603421.ogg', 'subbus1/XC603426.ogg'],\n ['woosan/XC578599.ogg', 'woosan/XC587076.ogg'],\n ['colsun2/XC755891.ogg', 'colsun2/XC755892.ogg'],\n ['hoopoe/XC365530.ogg', 'hoopoe/XC631301.ogg'],\n ['wookin1/XC704764.ogg', 'wookin1/XC741180.ogg'],\n ['fotdro5/XC619813.ogg', 'fotdro5/XC664386.ogg'],\n ['barswa/XC664976.ogg', 'barswa/XC664977.ogg'],\n ['combul2/XC447668.ogg', 'combul2/XC650877.ogg'],\n ['rerswa1/XC194484.ogg', 'rerswa1/XC753215.ogg'],\n ['gobbun1/XC394478.ogg', 'gobbun1/XC395111.ogg'],\n ['scthon1/XC200766.ogg', 'scthon1/XC200767.ogg'],\n ['woosan/XC494020.ogg', 'woosan/XC496525.ogg'],\n ['categr/XC197438.ogg', 'greegr/XC247286.ogg'],\n ['fotdro5/XC538792.ogg', 'fotdro5/XC540732.ogg'],\n ['thrnig1/XC728003.ogg', 'thrnig1/XC728016.ogg'],\n ['subbus1/XC406665.ogg', 'subbus1/XC418769.ogg'],\n ['wbrcha2/XC613380.ogg', 'wbrcha2/XC613384.ogg'],\n ['fotdro5/XC619812.ogg', 'fotdro5/XC664385.ogg'],\n ['grecor/XC215126.ogg', 'grecor/XC753201.ogg'],\n ['hadibi1/XC492329.ogg', 'hadibi1/XC492330.ogg'],\n ['litswi1/XC440301.ogg', 'litswi1/XC443070.ogg'],\n ['laudov1/XC405374.ogg', 'laudov1/XC405375.ogg'],\n ['gnbcam2/XC530150.ogg', 'gnbcam2/XC530151.ogg'],\n ['sichor1/XC446028.ogg', 'sichor1/XC446029.ogg'],\n ['afrthr1/XC652880.ogg', 'afrthr1/XC652884.ogg'],\n ['fislov1/XC503794.ogg', 'fislov1/XC526237.ogg'],\n ['afpfly1/XC418708.ogg', 'afpfly1/XC454899.ogg'],\n ['crohor1/XC194761.ogg', 'crohor1/XC194762.ogg']]\n
", "votes": null, "replies": [ { "id": 2216273, "author_name": "lhanhsin", "author_url": "", "post_date": "04/10/2023 00:15:15", "content": "

This is awesome! The embedding visualizations look great and provides some extra insight over the data.
\nUsing embeddings for discovering duplicates is indeed a clever way to get rid of all the noise and nuances in the data. Unfortunately, your first link doesn't work so I can't see the notebook.

", "votes": null, "replies": [ { "id": 2216491, "author_name": "robbynevels", "author_url": "", "post_date": "04/10/2023 05:50:29", "content": "

Whoops, sorry! Forgot to make it public. It should be viewable now.

", "votes": null, "replies": [ { "id": 2216618, "author_name": "lhanhsin", "author_url": "", "post_date": "04/10/2023 07:51:00", "content": "

Nice! Would you mind double checking these four pairs below? I think they are distinct.
\nI'll update the list above after you confirm.

\n
7359 gnbcam2/XC195528.ogg 6995 fotdro5/XC195989.ogg 0.666067898273468\n16073 woosan/XC587076.ogg 16049 woosan/XC578599.ogg 1.6366488933563232\n12077 rerswa1/XC753215.ogg 11896 rerswa1/XC194484.ogg 2.4423563480377197\n5570 crohor1/XC194762.ogg 5569 crohor1/XC194761.ogg 3.9656319618225098\n
", "votes": null, "replies": [ { "id": 2216912, "author_name": "robbynevels", "author_url": "", "post_date": "04/10/2023 13:17:25", "content": "

Ah, you're 100% right about that first one. But for the other 3, one of the pair is a crop of the other one. This is more obvious when zooming out to see the first 10 seconds: https://www.kaggle.com/robbynevels/zooming-out-on-some-duplicate-recordings

", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2201148": "**- Update 2023/04/11**: Added extra pairs found by @robbynevels by comparing the distance between google bird classifier embeddings.\n**- Update 2023/03/31**: Added extra pairs that were found by @mattop and not detected by DTW. \nPlease give their original notebooks upvotes since the identical samples wouldn't be found without their effort:\n[Matt OP's EDA and comparing identical samples](https://www.kaggle.com/code/mattop/birdclef-2023-eda)\n[Thacrobatheskis's 3D embedding EDA](https://www.kaggle.com/code/robbynevels/birdclef2023-eda-with-3d-embeddings)\n[Thacrobatheskis's identifying duplicate samples in embedding space](https://www.kaggle.com/code/robbynevels/identifying-duplicates-with-embeddings)\n\n**TL;DR**\n\nMy notebook can be [found here](https://www.kaggle.com/code/lhanhsin/birdclef-2023-identifying-duplicates). \n\nThe duplicated audios pairs are\n```\n[['afrgos1/XC400373.ogg', 'afrgos1/XC418707.ogg'],\n ['brubru1/XC364486.ogg', 'brubru1/XC364487.ogg'],\n ['grewoo2/XC527938.ogg', 'grewoo2/XC527939.ogg'],\n ['spepig1/XC589785.ogg', 'spepig1/XC590084.ogg'],\n ['yenspu1/XC756890.ogg', 'yenspu1/XC756891.ogg'],\n ['abhori1/XC748240.ogg', 'ccbeat1/XC748235.ogg'],\n ['wlwwar/XC478705.ogg', 'wlwwar/XC478767.ogg'],\n ['combul2/XC447669.ogg', 'combul2/XC650878.ogg'],\n ['combul2/XC401370.ogg', 'combul2/XC401371.ogg'],\n ['litegr/XC447850.ogg', 'litegr/XC535552.ogg'],\n ['carcha1/XC324665.ogg', 'carcha1/XC324666.ogg'],\n ['litegr/XC448898.ogg', 'litegr/XC535540.ogg'],\n ['cabgre1/XC115976.ogg', 'cabgre1/XC116119.ogg'],\n ['cibwar1/XC395511.ogg', 'cibwar1/XC432840.ogg'],\n ['piecro1/XC514153.ogg', 'piecro1/XC605859.ogg'],\n ['reccuc1/XC514156.ogg', 'reccuc1/XC605854.ogg'],\n ['combuz1/XC647786.ogg', 'combuz1/XC647787.ogg'],\n ['sccsun2/XC609477.ogg', 'sccsun2/XC609478.ogg'],\n ['piecro1/XC514160.ogg', 'trobou1/XC608026.ogg'],\n ['somgre1/XC440353.ogg', 'somgre1/XC443103.ogg'],\n ['hoopoe/XC252584.ogg', 'hoopoe/XC631304.ogg'],\n ['woosan/XC740798.ogg', 'woosan/XC742927.ogg'],\n ['grbcam1/XC247923.ogg', 'grbcam1/XC247925.ogg'],\n ['reccuc1/XC118219.ogg', 'reccuc1/XC118431.ogg'],\n ['kerspa2/XC524173.ogg', 'kerspa2/XC524174.ogg'],\n ['barswa/XC575747.ogg', 'barswa/XC575749.ogg'],\n ['hadibi1/XC359010.ogg', 'hadibi1/XC360552.ogg'],\n ['didcuc1/XC403193.ogg', 'didcuc1/XC409433.ogg'],\n ['combuz1/XC144257.ogg', 'combuz1/XC144258.ogg'],\n ['somgre1/XC443513.ogg', 'somgre1/XC443747.ogg'],\n ['cohmar1/XC564020.ogg', 'cohmar1/XC564021.ogg'],\n ['fotdro5/XC619811.ogg', 'fotdro5/XC664384.ogg'],\n ['litegr/XC411319.ogg', 'litegr/XC411320.ogg'],\n ['blaplo1/XC416393.ogg', 'blaplo1/XC418717.ogg'],\n ['fotdro5/XC619817.ogg', 'fotdro5/XC664389.ogg'],\n ['tafpri1/XC443724.ogg', 'tafpri1/XC443725.ogg'],\n ['litswi1/XC443712.ogg', 'litswi1/XC443713.ogg'],\n ['cibwar1/XC316684.ogg', 'gnbcam2/XC316684.ogg'],\n ['cohmar1/XC553166.ogg', 'cohmar1/XC558316.ogg'],\n ['fotdro5/XC619815.ogg', 'fotdro5/XC664387.ogg'],\n ['comsan/XC613127.ogg', 'comsan/XC613128.ogg'],\n ['gyhbus1/XC251906.ogg', 'gyhbus1/XC251907.ogg'],\n ['somgre1/XC608053.ogg', 'somgre1/XC609998.ogg'],\n ['egygoo/XC358927.ogg', 'egygoo/XC528135.ogg'],\n ['strher/XC453981.ogg', 'strher/XC550529.ogg'],\n ['btweye2/XC530963.ogg', 'btweye2/XC532754.ogg'],\n ['wtbeat1/XC234928.ogg', 'wtbeat1/XC234929.ogg'],\n ['abhori1/XC430723.ogg', 'crheag1/XC430724.ogg'],\n ['cohmar1/XC658833.ogg', 'cohmar1/XC729379.ogg'],\n ['combul2/XC748220.ogg', 'combul2/XC748221.ogg'],\n ['piecro1/XC401557.ogg', 'trobou1/XC401556.ogg'],\n ['subbus1/XC603421.ogg', 'subbus1/XC603426.ogg'],\n ['woosan/XC578599.ogg', 'woosan/XC587076.ogg'],\n ['colsun2/XC755891.ogg', 'colsun2/XC755892.ogg'],\n ['hoopoe/XC365530.ogg', 'hoopoe/XC631301.ogg'],\n ['wookin1/XC704764.ogg', 'wookin1/XC741180.ogg'],\n ['fotdro5/XC619813.ogg', 'fotdro5/XC664386.ogg'],\n ['barswa/XC664976.ogg', 'barswa/XC664977.ogg'],\n ['combul2/XC447668.ogg', 'combul2/XC650877.ogg'],\n ['rerswa1/XC194484.ogg', 'rerswa1/XC753215.ogg'],\n ['gobbun1/XC394478.ogg', 'gobbun1/XC395111.ogg'],\n ['scthon1/XC200766.ogg', 'scthon1/XC200767.ogg'],\n ['woosan/XC494020.ogg', 'woosan/XC496525.ogg'],\n ['categr/XC197438.ogg', 'greegr/XC247286.ogg'],\n ['fotdro5/XC538792.ogg', 'fotdro5/XC540732.ogg'],\n ['thrnig1/XC728003.ogg', 'thrnig1/XC728016.ogg'],\n ['subbus1/XC406665.ogg', 'subbus1/XC418769.ogg'],\n ['wbrcha2/XC613380.ogg', 'wbrcha2/XC613384.ogg'],\n ['fotdro5/XC619812.ogg', 'fotdro5/XC664385.ogg'],\n ['grecor/XC215126.ogg', 'grecor/XC753201.ogg'],\n ['hadibi1/XC492329.ogg', 'hadibi1/XC492330.ogg'],\n ['litswi1/XC440301.ogg', 'litswi1/XC443070.ogg'],\n ['laudov1/XC405374.ogg', 'laudov1/XC405375.ogg'],\n ['gnbcam2/XC530150.ogg', 'gnbcam2/XC530151.ogg'],\n ['sichor1/XC446028.ogg', 'sichor1/XC446029.ogg'],\n ['afrthr1/XC652880.ogg', 'afrthr1/XC652884.ogg'],\n ['fislov1/XC503794.ogg', 'fislov1/XC526237.ogg'],\n ['afpfly1/XC418708.ogg', 'afpfly1/XC454899.ogg'],\n ['crohor1/XC194761.ogg', 'crohor1/XC194762.ogg']]\n```\n\n----\n\n**Motivation**\nThanks to Matt OP's great observation [posted here](https://www.kaggle.com/competitions/birdclef-2023/discussion/396506), he discovered that there are duplicates in the dataset with different XC ID. My initial approach to find all duplicates was to hash the first five seconds of the audio and see if they are identical or not, but surprisingly none of the audios have the same hash. It turned out that even if they're identical, their values still differ slightly (maybe due to some compression reasons?).\n\n**Solutions**\nThe solution is to use dynamic time warping (DTW) distance as the similarity metric. DTW is an common metric used to measure time series similarity since it is robust to time shift.\nManual validation is still needed since there exists false positives, for example, being completely silent in the first five seconds. This can be done be observing the spectrogram after subtracting the two signals.\n\n>**True positive**: the subtracted spectrogram is black\n![True positive: the subtracted spectrogram is black](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2F8282338570c4c34e66dbbfc3dc236093%2F__results___11_80.png?generation=1680065468372086&alt=media)\n\n>**False positive**: the subtracted spectrogram shows parts of respective signals\n![False positive: the subtracted spectrogram shows parts of respective signals](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2F45767965ae6517dbaa1d2fddfff63872%2F__results___11_48.png?generation=1680065526057087&alt=media)\n\nIn the end, there are 18 duplicates that are discovered. They all seem to be reuploads from the same author, but some of them have differences in call types, ratings, and secondary labels. If you are using those fields maybe it is worth thinking about which row to keep.\n\n*Do note that this is not guaranteed to find all duplicates. Adjusting the threshold of similarity may find more duplicates.*", "2202789": "nice share👍", "2203384": "Very nice work @lhanhsin, thank you for expanding on the duplicate identification. Using dynamic time warping is a clever idea. \n\nI noticed that the list you provided does not include some of the duplicate pairs I found in my EDA notebook where I used the method of audio durations that match to identify the duplicates. Here is the full list duplicate pairs that have been found up to this point (26 in total):\n\n```\n[['carcha1/XC324665.ogg', 'carcha1/XC324666.ogg'],\n ['wbrcha2/XC613380.ogg', 'wbrcha2/XC613384.ogg'],\n ['comsan/XC613127.ogg', 'comsan/XC613128.ogg'],\n ['tafpri1/XC443724.ogg', 'tafpri1/XC443725.ogg'],\n ['colsun2/XC755891.ogg', 'colsun2/XC755892.ogg'],\n ['grewoo2/XC527938.ogg', 'grewoo2/XC527939.ogg'],\n ['sccsun2/XC609477.ogg', 'sccsun2/XC609478.ogg'],\n ['subbus1/XC603421.ogg', 'subbus1/XC603426.ogg'],\n ['combul2/XC748220.ogg', 'combul2/XC748221.ogg'],\n ['wtbeat1/XC234928.ogg', 'wtbeat1/XC234929.ogg'],\n ['afrthr1/XC652880.ogg', 'afrthr1/XC652884.ogg'],\n ['laudov1/XC405374.ogg', 'laudov1/XC405375.ogg'],\n ['litegr/XC411319.ogg', 'litegr/XC411320.ogg'],\n ['cohmar1/XC564020.ogg', 'cohmar1/XC564021.ogg'],\n ['egygoo/XC358927.ogg', 'egygoo/XC528135.ogg'],\n ['combuz1/XC647786.ogg', 'combuz1/XC647787.ogg'],\n ['combuz1/XC144257.ogg', 'combuz1/XC144258.ogg'],\n ['wlwwar/XC478705.ogg', 'wlwwar/XC478767.ogg'],\n\n# Duplicate audio pairs not found using dynamic time warping (found using audio durations)\n[“woosan/XC740798.ogg\", \"woosan/XC742927.ogg”],\n[“gnbcam2/XC530150.ogg\", \"gnbcam2/XC530151.ogg”], # plotted below\n[“litswi1/XC443712.ogg\", \"litswi1/XC443713.ogg”],\n[“combul2/XC650878.ogg\", \"combul2/XC447669.ogg”],\n[“gobbun1/XC394478.ogg\", \"gobbun1/XC395111.ogg”],\n[“fislov1/XC503794.ogg\", \"fislov1/XC526237.ogg”],\n[“cibwar1/XC395511.ogg\", \"cibwar1/XC432840.ogg”],\n[“combul2/XC447668.ogg\", \"combul2/XC650877.ogg”]]\n```\n\nHere are the Mel-Spectrograms side by side for `/gnbcam2/XC530150.ogg` and `/gnbcam2/XC530151.ogg` which could answer why DTW could not identify these files as duplicates:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10590800%2F77678a8222d0e43b97a43c4244a27d01%2Fmel-spect-db-shift.png?generation=1680201585698872&alt=media)\n\nAlso, have you tried to identify duplicates in the BirdCLEF 2021 & 2022 data using DTW? It would be interesting to see the results as @awsaf49 figured out that pretraining on the 2021 & 2022 data improves performance.", "2203622": "Thank you for your input! I'll update the list above.\n\nI dug a bit deeper into the first two pairs you provided.\nIn the first example there is a slight time shift, but DTW wasn't able to detect it. The info of the audio says that they have a slightly different bitrate which may be the reason. Maybe a possible solution to this is to run DTW on the spectrogram instead of the raw audio signal it self.\n>**[“woosan/XC740798.ogg\", \"woosan/XC742927.ogg”]** A slight time shift\n![First pair](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2F26d660baf8ce27cc92488fd68bee2f84%2Ffirst_pair.png?generation=1680225249642445&alt=media)\n\nThe second example doesn't have a time shift, but seems to be denoised. The difference in the low frequency range is obvious. \nI'm not sure what a good solution would be though, if we do some sort of denoising too before comparing them would work but will also generate a lot of false positives. If we change to using a image similarity detection, the number of false positives will still be an issue.\n>**[“gnbcam2/XC530150.ogg\", \"gnbcam2/XC530151.ogg”]** No time shift, but seems to be denoised.\n![Second pair](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2F057d7e76cffc8e92e46128c35f12731b%2Fsecond_image.png?generation=1680225628408916&alt=media)\n\nThe only solution that I can think of right now is to:\n1. Trim the audio, trimming out the silence will lower the number of false positives.\n2. Run STFT\n3. Denoise the audio, something simple like clipping the magnitude in the spectrogram might work.\n4. Run DTW across different frequencies in the spectrogram, this should be more robust compared to running on the raw signal.\n\nThis will not cover situations like different sound intensity (Normalizing the audio should work?) or slight shifts in the frequency. Maybe treating the spectrogram as an image and use some similarity model will be a solution for all the issues above, but requires some experiments.\n\nAnd no I haven't applied it to BirdClef 2021&2022 yet! I wrote the code with that in mind but it's now a low priority task. \nIdentifying duplicates in the BirdClef 2023 is important because I want to avoid contaminating the validation set. However, in the pretraining part it's not that much of an issue since the [hidden test set is released](https://www.kaggle.com/competitions/birdclef-2023/discussion/394358) so there is no contamination there, even if there was contamination it wouldn't do much harm since after all, it's only for pretraining. The only thing to watch out for is if the 2023 validation set has duplicates in the 2021&2022.", "2215845": "Thanks for finding this! I found 54 more duplicates by comparing the google bird classifier embeddings of the first and last 5 seconds of each recording + some thresholds on the audio magnitude + some manual filtering. My notebook is [here](https://www.kaggle.com/code/robbynevels/identifying-duplicates-with-embeddings). I also explored visualizing embeddings in 3D [here](https://www.kaggle.com/code/robbynevels/birdclef2023-eda-with-3d-embeddings).\n\nThis technique also uncovered all the duplicates you and @mattop found except for 1. I think there's probably several more that could be found by adjusting thresholds or combining our techniques.\n\nHere's a full list now (also listed in the notebook) [edited to remove one pair, based on the convo below]:\n\n```\n[['afrgos1/XC400373.ogg', 'afrgos1/XC418707.ogg'],\n ['brubru1/XC364486.ogg', 'brubru1/XC364487.ogg'],\n ['grewoo2/XC527938.ogg', 'grewoo2/XC527939.ogg'],\n ['spepig1/XC589785.ogg', 'spepig1/XC590084.ogg'],\n ['yenspu1/XC756890.ogg', 'yenspu1/XC756891.ogg'],\n ['abhori1/XC748240.ogg', 'ccbeat1/XC748235.ogg'],\n ['wlwwar/XC478705.ogg', 'wlwwar/XC478767.ogg'],\n ['combul2/XC447669.ogg', 'combul2/XC650878.ogg'],\n ['combul2/XC401370.ogg', 'combul2/XC401371.ogg'],\n ['litegr/XC447850.ogg', 'litegr/XC535552.ogg'],\n ['carcha1/XC324665.ogg', 'carcha1/XC324666.ogg'],\n ['litegr/XC448898.ogg', 'litegr/XC535540.ogg'],\n ['cabgre1/XC115976.ogg', 'cabgre1/XC116119.ogg'],\n ['cibwar1/XC395511.ogg', 'cibwar1/XC432840.ogg'],\n ['piecro1/XC514153.ogg', 'piecro1/XC605859.ogg'],\n ['reccuc1/XC514156.ogg', 'reccuc1/XC605854.ogg'],\n ['combuz1/XC647786.ogg', 'combuz1/XC647787.ogg'],\n ['sccsun2/XC609477.ogg', 'sccsun2/XC609478.ogg'],\n ['piecro1/XC514160.ogg', 'trobou1/XC608026.ogg'],\n ['somgre1/XC440353.ogg', 'somgre1/XC443103.ogg'],\n ['hoopoe/XC252584.ogg', 'hoopoe/XC631304.ogg'],\n ['woosan/XC740798.ogg', 'woosan/XC742927.ogg'],\n ['grbcam1/XC247923.ogg', 'grbcam1/XC247925.ogg'],\n ['reccuc1/XC118219.ogg', 'reccuc1/XC118431.ogg'],\n ['kerspa2/XC524173.ogg', 'kerspa2/XC524174.ogg'],\n ['barswa/XC575747.ogg', 'barswa/XC575749.ogg'],\n ['hadibi1/XC359010.ogg', 'hadibi1/XC360552.ogg'],\n ['didcuc1/XC403193.ogg', 'didcuc1/XC409433.ogg'],\n ['combuz1/XC144257.ogg', 'combuz1/XC144258.ogg'],\n ['somgre1/XC443513.ogg', 'somgre1/XC443747.ogg'],\n ['cohmar1/XC564020.ogg', 'cohmar1/XC564021.ogg'],\n ['fotdro5/XC619811.ogg', 'fotdro5/XC664384.ogg'],\n ['litegr/XC411319.ogg', 'litegr/XC411320.ogg'],\n ['blaplo1/XC416393.ogg', 'blaplo1/XC418717.ogg'],\n ['fotdro5/XC619817.ogg', 'fotdro5/XC664389.ogg'],\n ['tafpri1/XC443724.ogg', 'tafpri1/XC443725.ogg'],\n ['litswi1/XC443712.ogg', 'litswi1/XC443713.ogg'],\n ['cibwar1/XC316684.ogg', 'gnbcam2/XC316684.ogg'],\n ['cohmar1/XC553166.ogg', 'cohmar1/XC558316.ogg'],\n ['fotdro5/XC619815.ogg', 'fotdro5/XC664387.ogg'],\n ['comsan/XC613127.ogg', 'comsan/XC613128.ogg'],\n ['gyhbus1/XC251906.ogg', 'gyhbus1/XC251907.ogg'],\n ['somgre1/XC608053.ogg', 'somgre1/XC609998.ogg'],\n ['egygoo/XC358927.ogg', 'egygoo/XC528135.ogg'],\n ['strher/XC453981.ogg', 'strher/XC550529.ogg'],\n ['btweye2/XC530963.ogg', 'btweye2/XC532754.ogg'],\n ['wtbeat1/XC234928.ogg', 'wtbeat1/XC234929.ogg'],\n ['abhori1/XC430723.ogg', 'crheag1/XC430724.ogg'],\n ['cohmar1/XC658833.ogg', 'cohmar1/XC729379.ogg'],\n ['combul2/XC748220.ogg', 'combul2/XC748221.ogg'],\n ['piecro1/XC401557.ogg', 'trobou1/XC401556.ogg'],\n ['subbus1/XC603421.ogg', 'subbus1/XC603426.ogg'],\n ['woosan/XC578599.ogg', 'woosan/XC587076.ogg'],\n ['colsun2/XC755891.ogg', 'colsun2/XC755892.ogg'],\n ['hoopoe/XC365530.ogg', 'hoopoe/XC631301.ogg'],\n ['wookin1/XC704764.ogg', 'wookin1/XC741180.ogg'],\n ['fotdro5/XC619813.ogg', 'fotdro5/XC664386.ogg'],\n ['barswa/XC664976.ogg', 'barswa/XC664977.ogg'],\n ['combul2/XC447668.ogg', 'combul2/XC650877.ogg'],\n ['rerswa1/XC194484.ogg', 'rerswa1/XC753215.ogg'],\n ['gobbun1/XC394478.ogg', 'gobbun1/XC395111.ogg'],\n ['scthon1/XC200766.ogg', 'scthon1/XC200767.ogg'],\n ['woosan/XC494020.ogg', 'woosan/XC496525.ogg'],\n ['categr/XC197438.ogg', 'greegr/XC247286.ogg'],\n ['fotdro5/XC538792.ogg', 'fotdro5/XC540732.ogg'],\n ['thrnig1/XC728003.ogg', 'thrnig1/XC728016.ogg'],\n ['subbus1/XC406665.ogg', 'subbus1/XC418769.ogg'],\n ['wbrcha2/XC613380.ogg', 'wbrcha2/XC613384.ogg'],\n ['fotdro5/XC619812.ogg', 'fotdro5/XC664385.ogg'],\n ['grecor/XC215126.ogg', 'grecor/XC753201.ogg'],\n ['hadibi1/XC492329.ogg', 'hadibi1/XC492330.ogg'],\n ['litswi1/XC440301.ogg', 'litswi1/XC443070.ogg'],\n ['laudov1/XC405374.ogg', 'laudov1/XC405375.ogg'],\n ['gnbcam2/XC530150.ogg', 'gnbcam2/XC530151.ogg'],\n ['sichor1/XC446028.ogg', 'sichor1/XC446029.ogg'],\n ['afrthr1/XC652880.ogg', 'afrthr1/XC652884.ogg'],\n ['fislov1/XC503794.ogg', 'fislov1/XC526237.ogg'],\n ['afpfly1/XC418708.ogg', 'afpfly1/XC454899.ogg'],\n ['crohor1/XC194761.ogg', 'crohor1/XC194762.ogg']]\n```", "2216273": "This is awesome! The embedding visualizations look great and provides some extra insight over the data.\nUsing embeddings for discovering duplicates is indeed a clever way to get rid of all the noise and nuances in the data. Unfortunately, your first link doesn't work so I can't see the notebook.", "2216491": "Whoops, sorry! Forgot to make it public. It should be viewable now.", "2216618": "Nice! Would you mind double checking these four pairs below? I think they are distinct.\nI'll update the list above after you confirm.\n```\n7359 gnbcam2/XC195528.ogg 6995 fotdro5/XC195989.ogg 0.666067898273468\n16073 woosan/XC587076.ogg 16049 woosan/XC578599.ogg 1.6366488933563232\n12077 rerswa1/XC753215.ogg 11896 rerswa1/XC194484.ogg 2.4423563480377197\n5570 crohor1/XC194762.ogg 5569 crohor1/XC194761.ogg 3.9656319618225098\n```", "2216912": "Ah, you're 100% right about that first one. But for the other 3, one of the pair is a crop of the other one. This is more obvious when zooming out to see the first 10 seconds: https://www.kaggle.com/robbynevels/zooming-out-on-some-duplicate-recordings" }, "source": "meta" }