Hi there, I have a few questions for the competition hosts and thought I could pose them as a discussion here.
\nThere are quite a few audio files for under-represented bird species (and scored bird species) available on xenocanto.org that are not part of our original dataset.
\nFor example, here are the IDs of audio clips that exist on xenocanto for scored bird species that ARE NOT present in our training dataset.
\nAkiapolaau - 14 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC91359', 'XC58926', 'XC91229'}\nAnianiau - 12 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC59046', 'XC59047', 'XC59045', 'XC59049', 'XC59048'}\nApapane - 47 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC58981', 'XC58984', 'XC58979', 'XC59065', 'XC58980', 'XC58982', 'XC59066', 'XC59064', 'XC58983', 'XC91413', 'XC59067'}\nBand-rumped Storm Petrel - 15 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC18562', 'XC18561', 'XC18563', 'XC374996'}\nAkohekohe - 2 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> set()\nHawaii Elepaio - 14 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC58931'}\nErckel's Francolin - 6 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC66313', 'XC300639', 'XC300640', 'XC83661', 'XC300641'}\nHawaii Amakihi - 21 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC59059', 'XC59068', 'XC91477', 'XC58936', 'XC91357', 'XC59060', 'XC105455', 'XC59062', 'XC59061'}\nHawaii Creeper - 20 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC91233'}\nNene - 9 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC58929', 'XC58930', 'XC58927', 'XC58928'}\nHawaiian Hawk - 3 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> set()\nHawaiian Petrel - 3 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC89638'}\nHouse Finch - 322 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC1295', 'XC10166', 'XC71455', 'XC669436', 'XC56856', 'XC704622', 'XC103197', 'XC14517', 'XC89858', 'XC703917', 'XC101439', 'XC159129', 'XC474138', 'XC21398', 'XC158930', 'XC73496', 'XC159121', 'XC220526', 'XC691600', 'XC595829', 'XC227310', 'XC10167', 'XC220519', 'XC72282', 'XC602449', 'XC110709', 'XC220518', 'XC624381', 'XC362519', 'XC71842', 'XC220520', 'XC220525', 'XC34860', 'XC691598', 'XC159075', 'XC698521', 'XC1296', 'XC599221', 'XC701679', 'XC55229', 'XC220523', 'XC71841', 'XC220521', 'XC104710', 'XC679337', 'XC160394', 'XC160255', 'XC687813', 'XC409334', 'XC227312', 'XC696320', 'XC594090', 'XC220524', 'XC89177', 'XC220527', 'XC623936', 'XC21399', 'XC682570', 'XC73495', 'XC160192', 'XC73494', 'XC698317', 'XC89178', 'XC10165', 'XC474137', 'XC220522', 'XC55557', 'XC71030', 'XC701527', 'XC159717', 'XC159913', 'XC689212', 'XC241148', 'XC56159', 'XC159883', 'XC159970', 'XC13700', 'XC412509', 'XC59063', 'XC255739', 'XC73499', 'XC70992', 'XC71840', 'XC548267', 'XC602450', 'XC679994'}\nIiwi - 37 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC58937', 'XC91397', 'XC58938', 'XC453296', 'XC58939'}\nJapanese Bush Warbler - 78 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC191526', 'XC192386', 'XC285848', 'XC156050', 'XC285821', 'XC217325', 'XC59057', 'XC155764', 'XC191997', 'XC155676', 'XC685882', 'XC286124', 'XC59055', 'XC664691', 'XC617272', 'XC479120', 'XC155854', 'XC217774', 'XC286121', 'XC217773', 'XC478466', 'XC707574', 'XC194292', 'XC285789', 'XC155871', 'XC286120', 'XC59056', 'XC286119', 'XC201553', 'XC285619', 'XC217503', 'XC480105', 'XC191996', 'XC155858', 'XC200516', 'XC671476', 'XC286152', 'XC191995', 'XC150437', 'XC697521', 'XC191822', 'XC190381', 'XC34405', 'XC155855', 'XC190418', 'XC286336', 'XC59054', 'XC200213', 'XC285699', 'XC90807', 'XC217327', 'XC301082', 'XC199655', 'XC193566', 'XC286125', 'XC285822', 'XC285823', 'XC285694', 'XC192390', 'XC286235', 'XC200517', 'XC193558', 'XC59058', 'XC34404', 'XC155856', 'XC195518', 'XC155870', 'XC215546'}\nMaui Parrotbill - 1 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC58971'}\nOmao - 21 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC58935', 'XC91250', 'XC91358', 'XC58933', 'XC58932', 'XC58934', 'XC91475', 'XC91396'}\nPuaiohi - 3 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> set()\nEurasian Skylark - 500 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> LOTS\nWarbling White-eye - 71 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC156342', 'XC617437', 'XC155883', 'XC192602', 'XC496994', 'XC191117', 'XC191539', 'XC555895', 'XC69775', 'XC475646', 'XC539043', 'XC308147', 'XC286099', 'XC193575', 'XC193166', 'XC474648', 'XC615470', 'XC285855', 'XC617446', 'XC617440', 'XC155881', 'XC201291', 'XC660334', 'XC386755', 'XC617439', 'XC617433', 'XC201916', 'XC679118', 'XC509438', 'XC617438', 'XC478035', 'XC194949', 'XC496874', 'XC476239', 'XC69780', 'XC191023', 'XC617441', 'XC200843', 'XC286144', 'XC201915', 'XC469413', 'XC194950', 'XC651657', 'XC476238', 'XC194489', 'XC617448', 'XC69777', 'XC411211', 'XC617435', 'XC617442', 'XC617434', 'XC543755', 'XC680985', 'XC192368', 'XC285915', 'XC155880', 'XC409644', 'XC421243', 'XC198192', 'XC617444', 'XC680984', 'XC79215', 'XC202880', 'XC285872', 'XC216691', 'XC156343', 'XC193880', 'XC69776', 'XC194040', 'XC233552', 'XC155882', 'XC285853', 'XC639450', 'XC285947', 'XC193576', 'XC286102', 'XC192270', 'XC156111', 'XC285712', 'XC194291', 'XC194493', 'XC413569', 'XC193174', 'XC216690', 'XC191958', 'XC336464', 'XC194494', 'XC285856', 'XC285946', 'XC192563', 'XC194492', 'XC155884', 'XC197357', 'XC660335', 'XC468447', 'XC464957', 'XC194370', 'XC193455', 'XC69774', 'XC285900', 'XC193175', 'XC200841', 'XC190697', 'XC193012', 'XC59071', 'XC192562', 'XC286126', 'XC539042', 'XC617346', 'XC617445', 'XC191960', 'XC659635', 'XC69779', 'XC69778', 'XC196346'}\nYellow-fronted Canary - 67 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC398014', 'XC425312', 'XC268716', 'XC73416', 'XC280191', 'XC339125', 'XC425353', 'XC701608', 'XC104550', 'XC268717', 'XC92214', 'XC425354', 'XC64973', 'XC701607', 'XC47815', 'XC430233', 'XC99869', 'XC99872', 'XC100536', 'XC62325', 'XC425355', 'XC268715', 'XC99870', 'XC280202', 'XC280201', 'XC365040', 'XC701606', 'XC280192', 'XC425676', 'XC690680', 'XC602126', 'XC99871', 'XC280203', 'XC43946', 'XC510116', 'XC510117'}\nAs you can see, there are lots of available audio clips not present in our training data. I'm in the process of using these to create an auxiliary dataset.
\nIn addition to this, I may bolster the under-represented non-scored species as well (up to a max of 50 or something).
\nThat being said, there are still some problem areas. Some scored bird species don't have additional audio clips and are still severely underrepresented. Specifically:
\nI looked around and I noticed that there are some other sites similar to xenocanto.org that have clips from these species. Sites like https://search.macaulaylibrary.org/ and others. Some of these sites only allow usage of the data for particular purposes.
\nI was wondering if the hosts knew of any other audio data banks that could have potential audio samples for rare birds that we could use to bolster our training data? Also, are there certain limitations preventing us from using any particular additional data sources?
\nThank you in advance!
", "messages": [ { "id": "1736786", "postDate": "03/27/2022 18:26:51", "content": "Hi there, I have a few questions for the competition hosts and thought I could pose them as a discussion here.
\nThere are quite a few audio files for under-represented bird species (and scored bird species) available on xenocanto.org that are not part of our original dataset.
\nFor example, here are the IDs of audio clips that exist on xenocanto for scored bird species that ARE NOT present in our training dataset.
\nAkiapolaau - 14 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC91359', 'XC58926', 'XC91229'}\nAnianiau - 12 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC59046', 'XC59047', 'XC59045', 'XC59049', 'XC59048'}\nApapane - 47 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC58981', 'XC58984', 'XC58979', 'XC59065', 'XC58980', 'XC58982', 'XC59066', 'XC59064', 'XC58983', 'XC91413', 'XC59067'}\nBand-rumped Storm Petrel - 15 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC18562', 'XC18561', 'XC18563', 'XC374996'}\nAkohekohe - 2 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> set()\nHawaii Elepaio - 14 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC58931'}\nErckel's Francolin - 6 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC66313', 'XC300639', 'XC300640', 'XC83661', 'XC300641'}\nHawaii Amakihi - 21 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC59059', 'XC59068', 'XC91477', 'XC58936', 'XC91357', 'XC59060', 'XC105455', 'XC59062', 'XC59061'}\nHawaii Creeper - 20 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC91233'}\nNene - 9 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC58929', 'XC58930', 'XC58927', 'XC58928'}\nHawaiian Hawk - 3 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> set()\nHawaiian Petrel - 3 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC89638'}\nHouse Finch - 322 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC1295', 'XC10166', 'XC71455', 'XC669436', 'XC56856', 'XC704622', 'XC103197', 'XC14517', 'XC89858', 'XC703917', 'XC101439', 'XC159129', 'XC474138', 'XC21398', 'XC158930', 'XC73496', 'XC159121', 'XC220526', 'XC691600', 'XC595829', 'XC227310', 'XC10167', 'XC220519', 'XC72282', 'XC602449', 'XC110709', 'XC220518', 'XC624381', 'XC362519', 'XC71842', 'XC220520', 'XC220525', 'XC34860', 'XC691598', 'XC159075', 'XC698521', 'XC1296', 'XC599221', 'XC701679', 'XC55229', 'XC220523', 'XC71841', 'XC220521', 'XC104710', 'XC679337', 'XC160394', 'XC160255', 'XC687813', 'XC409334', 'XC227312', 'XC696320', 'XC594090', 'XC220524', 'XC89177', 'XC220527', 'XC623936', 'XC21399', 'XC682570', 'XC73495', 'XC160192', 'XC73494', 'XC698317', 'XC89178', 'XC10165', 'XC474137', 'XC220522', 'XC55557', 'XC71030', 'XC701527', 'XC159717', 'XC159913', 'XC689212', 'XC241148', 'XC56159', 'XC159883', 'XC159970', 'XC13700', 'XC412509', 'XC59063', 'XC255739', 'XC73499', 'XC70992', 'XC71840', 'XC548267', 'XC602450', 'XC679994'}\nIiwi - 37 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC58937', 'XC91397', 'XC58938', 'XC453296', 'XC58939'}\nJapanese Bush Warbler - 78 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC191526', 'XC192386', 'XC285848', 'XC156050', 'XC285821', 'XC217325', 'XC59057', 'XC155764', 'XC191997', 'XC155676', 'XC685882', 'XC286124', 'XC59055', 'XC664691', 'XC617272', 'XC479120', 'XC155854', 'XC217774', 'XC286121', 'XC217773', 'XC478466', 'XC707574', 'XC194292', 'XC285789', 'XC155871', 'XC286120', 'XC59056', 'XC286119', 'XC201553', 'XC285619', 'XC217503', 'XC480105', 'XC191996', 'XC155858', 'XC200516', 'XC671476', 'XC286152', 'XC191995', 'XC150437', 'XC697521', 'XC191822', 'XC190381', 'XC34405', 'XC155855', 'XC190418', 'XC286336', 'XC59054', 'XC200213', 'XC285699', 'XC90807', 'XC217327', 'XC301082', 'XC199655', 'XC193566', 'XC286125', 'XC285822', 'XC285823', 'XC285694', 'XC192390', 'XC286235', 'XC200517', 'XC193558', 'XC59058', 'XC34404', 'XC155856', 'XC195518', 'XC155870', 'XC215546'}\nMaui Parrotbill - 1 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC58971'}\nOmao - 21 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC58935', 'XC91250', 'XC91358', 'XC58933', 'XC58932', 'XC58934', 'XC91475', 'XC91396'}\nPuaiohi - 3 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> set()\nEurasian Skylark - 500 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> LOTS\nWarbling White-eye - 71 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC156342', 'XC617437', 'XC155883', 'XC192602', 'XC496994', 'XC191117', 'XC191539', 'XC555895', 'XC69775', 'XC475646', 'XC539043', 'XC308147', 'XC286099', 'XC193575', 'XC193166', 'XC474648', 'XC615470', 'XC285855', 'XC617446', 'XC617440', 'XC155881', 'XC201291', 'XC660334', 'XC386755', 'XC617439', 'XC617433', 'XC201916', 'XC679118', 'XC509438', 'XC617438', 'XC478035', 'XC194949', 'XC496874', 'XC476239', 'XC69780', 'XC191023', 'XC617441', 'XC200843', 'XC286144', 'XC201915', 'XC469413', 'XC194950', 'XC651657', 'XC476238', 'XC194489', 'XC617448', 'XC69777', 'XC411211', 'XC617435', 'XC617442', 'XC617434', 'XC543755', 'XC680985', 'XC192368', 'XC285915', 'XC155880', 'XC409644', 'XC421243', 'XC198192', 'XC617444', 'XC680984', 'XC79215', 'XC202880', 'XC285872', 'XC216691', 'XC156343', 'XC193880', 'XC69776', 'XC194040', 'XC233552', 'XC155882', 'XC285853', 'XC639450', 'XC285947', 'XC193576', 'XC286102', 'XC192270', 'XC156111', 'XC285712', 'XC194291', 'XC194493', 'XC413569', 'XC193174', 'XC216690', 'XC191958', 'XC336464', 'XC194494', 'XC285856', 'XC285946', 'XC192563', 'XC194492', 'XC155884', 'XC197357', 'XC660335', 'XC468447', 'XC464957', 'XC194370', 'XC193455', 'XC69774', 'XC285900', 'XC193175', 'XC200841', 'XC190697', 'XC193012', 'XC59071', 'XC192562', 'XC286126', 'XC539042', 'XC617346', 'XC617445', 'XC191960', 'XC659635', 'XC69779', 'XC69778', 'XC196346'}\nYellow-fronted Canary - 67 Clips In Training Data
\nXC IDS NOT PRESENT IN OUR DATA\n --> {'XC398014', 'XC425312', 'XC268716', 'XC73416', 'XC280191', 'XC339125', 'XC425353', 'XC701608', 'XC104550', 'XC268717', 'XC92214', 'XC425354', 'XC64973', 'XC701607', 'XC47815', 'XC430233', 'XC99869', 'XC99872', 'XC100536', 'XC62325', 'XC425355', 'XC268715', 'XC99870', 'XC280202', 'XC280201', 'XC365040', 'XC701606', 'XC280192', 'XC425676', 'XC690680', 'XC602126', 'XC99871', 'XC280203', 'XC43946', 'XC510116', 'XC510117'}\nAs you can see, there are lots of available audio clips not present in our training data. I'm in the process of using these to create an auxiliary dataset.
\nIn addition to this, I may bolster the under-represented non-scored species as well (up to a max of 50 or something).
\nThat being said, there are still some problem areas. Some scored bird species don't have additional audio clips and are still severely underrepresented. Specifically:
\nI looked around and I noticed that there are some other sites similar to xenocanto.org that have clips from these species. Sites like https://search.macaulaylibrary.org/ and others. Some of these sites only allow usage of the data for particular purposes.
\nI was wondering if the hosts knew of any other audio data banks that could have potential audio samples for rare birds that we could use to bolster our training data? Also, are there certain limitations preventing us from using any particular additional data sources?
\nThank you in advance!
", "rawMarkdown": "Hi there, I have a few questions for the competition hosts and thought I could pose them as a discussion here.\n\nThere are quite a few audio files for under-represented bird species (and scored bird species) available on xenocanto.org that are not part of our original dataset.\n\nFor example, here are the IDs of audio clips that exist on xenocanto for scored bird species that ARE NOT present in our training dataset.\n\n---\n\nAkiapolaau - 14 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC91359', 'XC58926', 'XC91229'}\n\n---\n\nAnianiau - 12 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC59046', 'XC59047', 'XC59045', 'XC59049', 'XC59048'}\n\n---\n\nApapane - 47 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC58981', 'XC58984', 'XC58979', 'XC59065', 'XC58980', 'XC58982', 'XC59066', 'XC59064', 'XC58983', 'XC91413', 'XC59067'}\n\n---\n\nBand-rumped Storm Petrel - 15 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC18562', 'XC18561', 'XC18563', 'XC374996'}\n\n---\n\nAkohekohe - 2 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> set()\n\n---\n\nHawaii Elepaio - 14 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC58931'}\n\n---\n\nErckel's Francolin - 6 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC66313', 'XC300639', 'XC300640', 'XC83661', 'XC300641'}\n\n---\n\nHawaii Amakihi - 21 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC59059', 'XC59068', 'XC91477', 'XC58936', 'XC91357', 'XC59060', 'XC105455', 'XC59062', 'XC59061'}\n\n---\n\nHawaii Creeper - 20 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC91233'}\n\n---\n\nNene - 9 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC58929', 'XC58930', 'XC58927', 'XC58928'}\n\n---\n\nHawaiian Hawk - 3 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> set()\n\n---\n\nHawaiian Petrel - 3 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC89638'}\n\n---\n\nHouse Finch - 322 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC1295', 'XC10166', 'XC71455', 'XC669436', 'XC56856', 'XC704622', 'XC103197', 'XC14517', 'XC89858', 'XC703917', 'XC101439', 'XC159129', 'XC474138', 'XC21398', 'XC158930', 'XC73496', 'XC159121', 'XC220526', 'XC691600', 'XC595829', 'XC227310', 'XC10167', 'XC220519', 'XC72282', 'XC602449', 'XC110709', 'XC220518', 'XC624381', 'XC362519', 'XC71842', 'XC220520', 'XC220525', 'XC34860', 'XC691598', 'XC159075', 'XC698521', 'XC1296', 'XC599221', 'XC701679', 'XC55229', 'XC220523', 'XC71841', 'XC220521', 'XC104710', 'XC679337', 'XC160394', 'XC160255', 'XC687813', 'XC409334', 'XC227312', 'XC696320', 'XC594090', 'XC220524', 'XC89177', 'XC220527', 'XC623936', 'XC21399', 'XC682570', 'XC73495', 'XC160192', 'XC73494', 'XC698317', 'XC89178', 'XC10165', 'XC474137', 'XC220522', 'XC55557', 'XC71030', 'XC701527', 'XC159717', 'XC159913', 'XC689212', 'XC241148', 'XC56159', 'XC159883', 'XC159970', 'XC13700', 'XC412509', 'XC59063', 'XC255739', 'XC73499', 'XC70992', 'XC71840', 'XC548267', 'XC602450', 'XC679994'}\n\n---\n\nIiwi - 37 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC58937', 'XC91397', 'XC58938', 'XC453296', 'XC58939'}\n\n---\n\nJapanese Bush Warbler - 78 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC191526', 'XC192386', 'XC285848', 'XC156050', 'XC285821', 'XC217325', 'XC59057', 'XC155764', 'XC191997', 'XC155676', 'XC685882', 'XC286124', 'XC59055', 'XC664691', 'XC617272', 'XC479120', 'XC155854', 'XC217774', 'XC286121', 'XC217773', 'XC478466', 'XC707574', 'XC194292', 'XC285789', 'XC155871', 'XC286120', 'XC59056', 'XC286119', 'XC201553', 'XC285619', 'XC217503', 'XC480105', 'XC191996', 'XC155858', 'XC200516', 'XC671476', 'XC286152', 'XC191995', 'XC150437', 'XC697521', 'XC191822', 'XC190381', 'XC34405', 'XC155855', 'XC190418', 'XC286336', 'XC59054', 'XC200213', 'XC285699', 'XC90807', 'XC217327', 'XC301082', 'XC199655', 'XC193566', 'XC286125', 'XC285822', 'XC285823', 'XC285694', 'XC192390', 'XC286235', 'XC200517', 'XC193558', 'XC59058', 'XC34404', 'XC155856', 'XC195518', 'XC155870', 'XC215546'}\n\n---\n\nMaui Parrotbill - 1 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC58971'}\n\n---\n\nOmao - 21 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC58935', 'XC91250', 'XC91358', 'XC58933', 'XC58932', 'XC58934', 'XC91475', 'XC91396'}\n\n---\n\nPuaiohi - 3 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> set()\n\n---\n\nEurasian Skylark - 500 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> LOTS\n\n---\n\nWarbling White-eye - 71 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC156342', 'XC617437', 'XC155883', 'XC192602', 'XC496994', 'XC191117', 'XC191539', 'XC555895', 'XC69775', 'XC475646', 'XC539043', 'XC308147', 'XC286099', 'XC193575', 'XC193166', 'XC474648', 'XC615470', 'XC285855', 'XC617446', 'XC617440', 'XC155881', 'XC201291', 'XC660334', 'XC386755', 'XC617439', 'XC617433', 'XC201916', 'XC679118', 'XC509438', 'XC617438', 'XC478035', 'XC194949', 'XC496874', 'XC476239', 'XC69780', 'XC191023', 'XC617441', 'XC200843', 'XC286144', 'XC201915', 'XC469413', 'XC194950', 'XC651657', 'XC476238', 'XC194489', 'XC617448', 'XC69777', 'XC411211', 'XC617435', 'XC617442', 'XC617434', 'XC543755', 'XC680985', 'XC192368', 'XC285915', 'XC155880', 'XC409644', 'XC421243', 'XC198192', 'XC617444', 'XC680984', 'XC79215', 'XC202880', 'XC285872', 'XC216691', 'XC156343', 'XC193880', 'XC69776', 'XC194040', 'XC233552', 'XC155882', 'XC285853', 'XC639450', 'XC285947', 'XC193576', 'XC286102', 'XC192270', 'XC156111', 'XC285712', 'XC194291', 'XC194493', 'XC413569', 'XC193174', 'XC216690', 'XC191958', 'XC336464', 'XC194494', 'XC285856', 'XC285946', 'XC192563', 'XC194492', 'XC155884', 'XC197357', 'XC660335', 'XC468447', 'XC464957', 'XC194370', 'XC193455', 'XC69774', 'XC285900', 'XC193175', 'XC200841', 'XC190697', 'XC193012', 'XC59071', 'XC192562', 'XC286126', 'XC539042', 'XC617346', 'XC617445', 'XC191960', 'XC659635', 'XC69779', 'XC69778', 'XC196346'}\n\n---\n\nYellow-fronted Canary - 67 Clips In Training Data\n\n\tXC IDS NOT PRESENT IN OUR DATA\n\t\t--> {'XC398014', 'XC425312', 'XC268716', 'XC73416', 'XC280191', 'XC339125', 'XC425353', 'XC701608', 'XC104550', 'XC268717', 'XC92214', 'XC425354', 'XC64973', 'XC701607', 'XC47815', 'XC430233', 'XC99869', 'XC99872', 'XC100536', 'XC62325', 'XC425355', 'XC268715', 'XC99870', 'XC280202', 'XC280201', 'XC365040', 'XC701606', 'XC280192', 'XC425676', 'XC690680', 'XC602126', 'XC99871', 'XC280203', 'XC43946', 'XC510116', 'XC510117'}\n\n---\n\nAs you can see, there are lots of available audio clips not present in our training data. I'm in the process of using these to create an auxiliary dataset.\n\nIn addition to this, I may bolster the under-represented non-scored species as well (up to a max of 50 or something).\n\nThat being said, there are still some problem areas. Some scored bird species don't have additional audio clips and are still severely underrepresented. Specifically:\n* Puaiohi (1 in training and 0 additional clips)\n* Hawaiian Hawk (3 in training and 1 additional clip(s))\n* Hawaiian Petrel (3 in training and 1 additional clip(s))\n* Maui Parrotbill (1 in training and 1 additional clip(s))\n* Akohekohe (2 in training and 0 additional clips)\n\n---\n\nI looked around and I noticed that there are some other sites similar to xenocanto.org that have clips from these species. Sites like https://search.macaulaylibrary.org/ and others. Some of these sites only allow usage of the data for particular purposes.\n\nI was wondering if the hosts knew of any other audio data banks that could have potential audio samples for rare birds that we could use to bolster our training data? Also, are there certain limitations preventing us from using any particular additional data sources?\n\n* @stefankahl (https://www.kaggle.com/stefankahl)\n* @tomdenton (https://www.kaggle.com/tomdenton)\n* @amandanavine (https://www.kaggle.com/amandanavine)\n* @holgerklinck (https://www.kaggle.com/holgerklinck)\n\nThank you in advance!", "votes": null }, { "id": "1736865", "postDate": "03/27/2022 20:07:11", "content": "Hi, thanks for the thoroughly researched question!
\nAs in previous years, we've truncated training sets for very well-represented species, and also skipped recordings with \"No Derivatives\" licenses; I believe all of the rare-species recordings missing from the training set will have ND licenses. The competition dataset also be missing any recordings uploaded in the last few months.
\nPlease read section 7 of the competition rules carefully for outside data restrictions. The most important factor (from a science perspective) is reproducibility, which includes respecting the license agreements which data is provided under. Limiting to CC-BY-SA licensed data should do the trick, and all data sources will need to be declared for any winning entry.
", "rawMarkdown": "Hi, thanks for the thoroughly researched question!\n\nAs in previous years, we've truncated training sets for very well-represented species, and also skipped recordings with \"No Derivatives\" licenses; I believe all of the rare-species recordings missing from the training set will have ND licenses. The competition dataset also be missing any recordings uploaded in the last few months.\n\nPlease read section 7 of the competition rules carefully for outside data restrictions. The most important factor (from a science perspective) is reproducibility, which includes respecting the license agreements which data is provided under. Limiting to CC-BY-SA licensed data should do the trick, and all data sources will need to be declared for any winning entry.", "votes": null }, { "id": "1736908", "postDate": "03/27/2022 22:45:21", "content": "Is this competition's wined models are planned to be used for commercially?
\nAccording to the Macaulay Library's media usage policy[1], we can download media for free only if it is for education or research.
\nIf it isn't, I think we can use this recordings for external data, right?
No, models won't be used commercially, but may very well be used to aid Hawaiian researchers in their studies.
\nExternal data is tricky this year, like Tom mentioned, we excluded files from the training data that didn't have an appropriate license. This also applies to Macaulay Library because of the strict (and sometimes closed-source) licensing.
\nPlease do not simply crawl recordings from Macaulay Library, doing so would violate the terms of use, which require you to officially request data from ML. But even if you do so, you can't use these data in this competition because you are not allowed to share and thus the data can't be considered publicly available. Unfortunately, this is how it is right now with Macaulay Library, if that changes in the next few years (which it might, because ML wants to share data), we'll include ML data in the training data for these competitions.
", "rawMarkdown": "No, models won't be used commercially, but may very well be used to aid Hawaiian researchers in their studies. \n\nExternal data is tricky this year, like Tom mentioned, we excluded files from the training data that didn't have an appropriate license. This also applies to Macaulay Library because of the strict (and sometimes closed-source) licensing. \n\nPlease do not simply crawl recordings from Macaulay Library, doing so would violate the terms of use, which require you to officially request data from ML. But even if you do so, you can't use these data in this competition because you are not allowed to share and thus the data can't be considered publicly available. Unfortunately, this is how it is right now with Macaulay Library, if that changes in the next few years (which it might, because ML wants to share data), we'll include ML data in the training data for these competitions.", "votes": null }, { "id": "1737379", "postDate": "03/28/2022 11:45:29", "content": "@stefankahl I see.. It's a pity that we cannot use ML dataset because it have different sound source than xeno-canto especially for few-observed species.
\nBut thank you for the clarification.
", "rawMarkdown": "stefankahl I see.. It's a pity that we cannot use ML dataset because it have different sound source than xeno-canto especially for few-observed species.\n\nBut thank you for the clarification.", "votes": null }, { "id": "1738619", "postDate": "03/29/2022 12:07:59", "content": "Thank you for the detailed response. This makes sense. I will spend my time working with what we have. Thanks!
", "rawMarkdown": "Thank you for the detailed response. This makes sense. I will spend my time working with what we have. Thanks!", "votes": null }, { "id": "1739731", "postDate": "03/30/2022 09:48:22", "content": "@stefankahl @tomdenton @sohier
\nSorry to be talking about individual data, but could you clarify the licensing of the datasets (audio files and metadata) for the LifeCLEF 2020 Bird - Monophone competition at AIcround[1]? I checked the competition website but could not find any mention of a license for the dataset.
\nThanks in advance.
\n[1] LifeCLEF 2020 Bird - Monophone competition at AIcround
", "rawMarkdown": "stefankahl @tomdenton @sohier \n\nSorry to be talking about individual data, but could you clarify the licensing of the datasets (audio files and metadata) for the LifeCLEF 2020 Bird - Monophone competition at AIcround[1]? I checked the competition website but could not find any mention of a license for the dataset.\n\nThanks in advance.\n\n[1] [LifeCLEF 2020 Bird - Monophone competition at AIcround](https://www.aicrowd.com/clef_tasks/22/task_dataset_files?challenge_id=211)", "votes": null }, { "id": "1739754", "postDate": "03/30/2022 10:06:52", "content": "Also, the license for the BirdCLEF 2021 data[2] and 2019 competition data[3] are listed as BY-NC-SA, can we use it for this competition?
\n\n\nDATA ACCESS AND USE: Attribution-NonCommercial-ShareAlike (BY-NC-SA)
\n
[2] BirdCLEF 2021 - Birdcall Identification - Rules
\n[3] Cornell Birdcall Identification - Rules
Yes, absolutely, feel free to use data from previous competitions.
", "rawMarkdown": "Yes, absolutely, feel free to use data from previous competitions.", "votes": null }, { "id": "1740715", "postDate": "03/31/2022 08:00:51", "content": "Using the 2020 data should be fine. Not sure if it will help much though.
", "rawMarkdown": "Using the 2020 data should be fine. Not sure if it will help much though.", "votes": null }, { "id": "1740736", "postDate": "03/31/2022 08:16:57", "content": "I see. Thanks.
", "rawMarkdown": "I see. Thanks.", "votes": null }, { "id": "1740864", "postDate": "03/31/2022 10:42:16", "content": "\nwe plan to use external data from xeno-canto for pre-training our models.
\nTo our understanding all data published under cc-licenses can be used as external data for the competition as they are \"publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants\" (rule 7C).
Could you please confirm this?
\nThanks in advance.
", "rawMarkdown": "Hi @stefankahl @tomdenton,\n\nwe plan to use external data from xeno-canto for pre-training our models.\nTo our understanding all data published under cc-licenses can be used as external data for the competition as they are \"publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants\" (rule 7C).\n\nCould you please confirm this?\n\nThanks in advance.", "votes": null }, { "id": "1752671", "postDate": "04/12/2022 03:26:14", "content": "CC-BY-SA licensed data : we can use
\nCC-BY-ND licensed data : we can't use(rule violation)
@stefankahl @tomdenton
\nis this correct?
Hi, thanks for the thoroughly researched question!
\nAs in previous years, we've truncated training sets for very well-represented species, and also skipped recordings with \"No Derivatives\" licenses; I believe all of the rare-species recordings missing from the training set will have ND licenses. The competition dataset also be missing any recordings uploaded in the last few months.
\nPlease read section 7 of the competition rules carefully for outside data restrictions. The most important factor (from a science perspective) is reproducibility, which includes respecting the license agreements which data is provided under. Limiting to CC-BY-SA licensed data should do the trick, and all data sources will need to be declared for any winning entry.
", "votes": null, "replies": [ { "id": 1736908, "author_name": "tatamikenn", "author_url": "", "post_date": "03/27/2022 22:45:21", "content": "Is this competition's wined models are planned to be used for commercially?
\nAccording to the Macaulay Library's media usage policy[1], we can download media for free only if it is for education or research.
\nIf it isn't, I think we can use this recordings for external data, right?
No, models won't be used commercially, but may very well be used to aid Hawaiian researchers in their studies.
\nExternal data is tricky this year, like Tom mentioned, we excluded files from the training data that didn't have an appropriate license. This also applies to Macaulay Library because of the strict (and sometimes closed-source) licensing.
\nPlease do not simply crawl recordings from Macaulay Library, doing so would violate the terms of use, which require you to officially request data from ML. But even if you do so, you can't use these data in this competition because you are not allowed to share and thus the data can't be considered publicly available. Unfortunately, this is how it is right now with Macaulay Library, if that changes in the next few years (which it might, because ML wants to share data), we'll include ML data in the training data for these competitions.
", "votes": null, "replies": [] }, { "id": 1737379, "author_name": "tatamikenn", "author_url": "", "post_date": "03/28/2022 11:45:29", "content": "@stefankahl I see.. It's a pity that we cannot use ML dataset because it have different sound source than xeno-canto especially for few-observed species.
\nBut thank you for the clarification.
", "votes": null, "replies": [] }, { "id": 1738619, "author_name": "dschettler8845", "author_url": "", "post_date": "03/29/2022 12:07:59", "content": "Thank you for the detailed response. This makes sense. I will spend my time working with what we have. Thanks!
", "votes": null, "replies": [] } ] }, { "id": 1739731, "author_name": "tatamikenn", "author_url": "", "post_date": "03/30/2022 09:48:22", "content": "@stefankahl @tomdenton @sohier
\nSorry to be talking about individual data, but could you clarify the licensing of the datasets (audio files and metadata) for the LifeCLEF 2020 Bird - Monophone competition at AIcround[1]? I checked the competition website but could not find any mention of a license for the dataset.
\nThanks in advance.
\n[1] LifeCLEF 2020 Bird - Monophone competition at AIcround
", "votes": null, "replies": [ { "id": 1739754, "author_name": "tatamikenn", "author_url": "", "post_date": "03/30/2022 10:06:52", "content": "Also, the license for the BirdCLEF 2021 data[2] and 2019 competition data[3] are listed as BY-NC-SA, can we use it for this competition?
\n\n\nDATA ACCESS AND USE: Attribution-NonCommercial-ShareAlike (BY-NC-SA)
\n
[2] BirdCLEF 2021 - Birdcall Identification - Rules
\n[3] Cornell Birdcall Identification - Rules
Yes, absolutely, feel free to use data from previous competitions.
", "votes": null, "replies": [] }, { "id": 1740715, "author_name": "stefankahl", "author_url": "", "post_date": "03/31/2022 08:00:51", "content": "Using the 2020 data should be fine. Not sure if it will help much though.
", "votes": null, "replies": [] }, { "id": 1740736, "author_name": "tatamikenn", "author_url": "", "post_date": "03/31/2022 08:16:57", "content": "I see. Thanks.
", "votes": null, "replies": [] } ] }, { "id": 1740864, "author_name": "schneidermarburg", "author_url": "", "post_date": "03/31/2022 10:42:16", "content": "\nwe plan to use external data from xeno-canto for pre-training our models.
\nTo our understanding all data published under cc-licenses can be used as external data for the competition as they are \"publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants\" (rule 7C).
Could you please confirm this?
\nThanks in advance.
", "votes": null, "replies": [ { "id": 1752671, "author_name": "asaliquid1011", "author_url": "", "post_date": "04/12/2022 03:26:14", "content": "CC-BY-SA licensed data : we can use
\nCC-BY-ND licensed data : we can't use(rule violation)
@stefankahl @tomdenton
\nis this correct?