Dear participants,
\n\nThe competition has ended. On behalf of the organizers, thank you all for participating. It has been a few intense months where we guess both participants and organizers have learnt a lot. We have had 558 teams with a total of 5,685 submissions in the first edition of the competition! We would like to give special thanks to those who contributed in the Discussion forums sharing their knowledge, and also to those who submitted through the DCASE system. This will have a real impact on the research community.
\n\nWe would like to make a few final announcements:
\n\nWhile strictly speaking the competition is over, you all still have a few days (until Monday 6th August) to submit a scientific paper to the DCASE Workshop. We encourage you all to take advantage of the great work done in the competition and make a paper out of it!
The ground truth for the test set is now published on the Data tab. This also includes Freesound IDs and licenses for the sounds in the dataset (for both train and test). We encourage participants submitting to the DCASE Workshop to use this ground truth to report per-class mAP on your papers.
The organizers will soon begin the review of the Judges’ Award submissions. We’ll announce the winner in the coming weeks. Stay tuned!
We plan to do a publication reviewing the competition, including an analysis of the results, the major trends used, etc. But for this it is essential that we have reports describing the systems. If you'd like us to include your methodology and results in the competition review, either you submit a Workshop paper (which is encouraged), or at least it’d be great if you can send a write-up to eduardo.fonseca@upf.edu, and fill this form https://goo.gl/forms/H6NkZV07p7HIqKXz2 . This would be very useful for us.
If you use the FSDKaggle2018 dataset or baseline code provided in this competition please cite the following paper:
\n\n\nEduardo Fonseca, Manoj Plakal, Frederic Font, Daniel P. W. Ellis,\n Xavier Favory, Jordi Pons, Xavier Serra. General-purpose Tagging of\n Freesound Audio with AudioSet Labels: Task Description, Dataset, and\n Baseline. Submitted to DCASE2018 Workshop, 2018. URL:\n https://arxiv.org/abs/1807.09902
\n
You can also consider citing the Freesound Datasets platform, that was used to build the FSDKaggle2018 dataset:
\n\n\n\n\nEduardo Fonseca, Jordi Pons, Xavier Favory, Frederic Font, Dmitry\n Bogdanov, Andrés Ferraro, Sergio Oramas, Alastair Porter, and Xavier\n Serra. Freesound datasets: a platform for the creation of open audio\n datasets. In Proceedings of the 18th International Society for Music\n Information Retrieval Conference (ISMIR 2017), pp 486-493. Suzhou,\n China, 2017.\n https://repositori.upf.edu/bitstream/handle/10230/33299/fonseca_ismir17_freesound.pdf?sequence=1&isAllowed=y
\n
That’s all for now. Please feel free to share your comments and impressions about the competition in this thread. We’d love to hear your feedback. Thanks again! We hope you enjoyed participating in this competition. It’s been just great!
\n\n
\nSincerely,
Eduardo Fonseca, Frederic Font (Freesound, MTG-UPF)
\nDan Ellis, Manoj Plakal (Google Machine Perception)
Dear participants,
\n\nThe competition has ended. On behalf of the organizers, thank you all for participating. It has been a few intense months where we guess both participants and organizers have learnt a lot. We have had 558 teams with a total of 5,685 submissions in the first edition of the competition! We would like to give special thanks to those who contributed in the Discussion forums sharing their knowledge, and also to those who submitted through the DCASE system. This will have a real impact on the research community.
\n\nWe would like to make a few final announcements:
\n\nWhile strictly speaking the competition is over, you all still have a few days (until Monday 6th August) to submit a scientific paper to the DCASE Workshop. We encourage you all to take advantage of the great work done in the competition and make a paper out of it!
The ground truth for the test set is now published on the Data tab. This also includes Freesound IDs and licenses for the sounds in the dataset (for both train and test). We encourage participants submitting to the DCASE Workshop to use this ground truth to report per-class mAP on your papers.
The organizers will soon begin the review of the Judges’ Award submissions. We’ll announce the winner in the coming weeks. Stay tuned!
We plan to do a publication reviewing the competition, including an analysis of the results, the major trends used, etc. But for this it is essential that we have reports describing the systems. If you'd like us to include your methodology and results in the competition review, either you submit a Workshop paper (which is encouraged), or at least it’d be great if you can send a write-up to eduardo.fonseca@upf.edu, and fill this form https://goo.gl/forms/H6NkZV07p7HIqKXz2 . This would be very useful for us.
If you use the FSDKaggle2018 dataset or baseline code provided in this competition please cite the following paper:
\n\n\nEduardo Fonseca, Manoj Plakal, Frederic Font, Daniel P. W. Ellis,\n Xavier Favory, Jordi Pons, Xavier Serra. General-purpose Tagging of\n Freesound Audio with AudioSet Labels: Task Description, Dataset, and\n Baseline. Submitted to DCASE2018 Workshop, 2018. URL:\n https://arxiv.org/abs/1807.09902
\n
You can also consider citing the Freesound Datasets platform, that was used to build the FSDKaggle2018 dataset:
\n\n\n\n\nEduardo Fonseca, Jordi Pons, Xavier Favory, Frederic Font, Dmitry\n Bogdanov, Andrés Ferraro, Sergio Oramas, Alastair Porter, and Xavier\n Serra. Freesound datasets: a platform for the creation of open audio\n datasets. In Proceedings of the 18th International Society for Music\n Information Retrieval Conference (ISMIR 2017), pp 486-493. Suzhou,\n China, 2017.\n https://repositori.upf.edu/bitstream/handle/10230/33299/fonseca_ismir17_freesound.pdf?sequence=1&isAllowed=y
\n
That’s all for now. Please feel free to share your comments and impressions about the competition in this thread. We’d love to hear your feedback. Thanks again! We hope you enjoyed participating in this competition. It’s been just great!
\n\n
\nSincerely,
Eduardo Fonseca, Frederic Font (Freesound, MTG-UPF)
\nDan Ellis, Manoj Plakal (Google Machine Perception)
The updated files are now available on the Data tab.
", "rawMarkdown": "The updated files are now available on the Data tab.", "votes": null }, { "id": "365307", "postDate": "08/02/2018 10:07:49", "content": "Dear organizers,
\n\nthank you for hosting this challenge. It was a pleasure to participate!
\n\nI have one last minor question though. When reporting the per-class mAP@3s in our papers, would you prefer that we
\n\nPersonally, I prefer option a) since the dataset is larger than b). But in any case, it seems to be desirable that we all use the same option to be able to compare the results more precisely.
", "rawMarkdown": "Dear organizers,\n\nthank you for hosting this challenge. It was a pleasure to participate!\n\nI have one last minor question though. When reporting the per-class mAP@3s in our papers, would you prefer that we\n\n - a) use the manually-verified part of the test set (public+private leaderboard split, not using 'ignore') or\n - b) only use the split used to obtain the final results (private leaderboard) ?\n\nPersonally, I prefer option a) since the dataset is larger than b). But in any case, it seems to be desirable that we all use the same option to be able to compare the results more precisely.", "votes": null }, { "id": "365480", "postDate": "08/02/2018 18:42:35", "content": "I was also wondering this in terms of the results that will be published on the DCASE website. My thinking is that the public test set can be considered as a validation set, since we were able to get live feedback on scores, albeit with a limit on tries.
", "rawMarkdown": "I was also wondering this in terms of the results that will be published on the DCASE website. My thinking is that the public test set can be considered as a validation set, since we were able to get live feedback on scores, albeit with a limit on tries.", "votes": null }, { "id": "365545", "postDate": "08/02/2018 21:51:03", "content": "Hi all, congratulations to both for the good results and also for taking the effort of writing the paper.
\n\n@Kevin, this is a discussion that we've had for a few days. On the one hand, reporting per-class scores on the full test set (ie, private + public splits) can be thought of more meaningful as there is more data to report on. On the other hand, strictly speaking, the final ranking is done solely on the private LB, while the public LB was more of a validation set, as Turab points out.
\n\nAfter discussing, we decided the following (although we admit it can be controversial):
\n\nwe prefer per-class scores on the full test set (ie, private + public splits together). This can be useful in the future should other researchers want to use the FSDKaggle2018 dataset. That being said, if you want, you are welcome to report per-class scores also for the private LB only.
Then, for the global mAP, we think it is most appropriate to report on the triplet: entire test set / public LB / private LB.
The aforementioned criteria is what we followed for the publication describing the competition (see Table 2 and Section 4). https://arxiv.org/pdf/1807.09902.pdf
\n\n@Turab, we haven't decided how to display results on the DCASE web. I guess at least we'll provide mAP on public and private LB.
\n\nThank you!
", "rawMarkdown": "Hi all, congratulations to both for the good results and also for taking the effort of writing the paper.\n\n@Kevin, this is a discussion that we've had for a few days. On the one hand, reporting per-class scores on the full test set (ie, private + public splits) can be thought of more meaningful as there is more data to report on. On the other hand, strictly speaking, the final ranking is done solely on the private LB, while the public LB was more of a validation set, as Turab points out.\n\nAfter discussing, we decided the following (although we admit it can be controversial):\n\n- we prefer **per-class scores on the full test set (ie, private + public splits together)**. This can be useful in the future should other researchers want to use the FSDKaggle2018 dataset. That being said, if you want, you are welcome to report per-class scores also for the private LB only. \n\n- Then, **for the global mAP**, we think it is most appropriate to report on the **triplet: entire test set / public LB / private LB.** \n\nThe aforementioned criteria is what we followed for the publication describing the competition (see Table 2 and Section 4). https://arxiv.org/pdf/1807.09902.pdf\n\n@Turab, we haven't decided how to display results on the DCASE web. I guess at least we'll provide mAP on public and private LB. \n\nThank you!", "votes": null }, { "id": "366623", "postDate": "08/06/2018 04:07:48", "content": "Dear organizers and people here,\nThank you so much for sharing wonderful opportunity to learn to solve sound classification task.\nWhat's most impressive to me is that we could achieve around 95% accuracy for that 41 classes, though I personally will fail to do that for some of them like Double bass and Cello.\nIt was close to 90% in the early days, but participants improves their solutions and we see over 97% private LB eventually. This is awesome.\nThe experience earned here is now applied to a real problem in my work. This won't have happened without this confidence about what could be done with sounds.\nOnce again thank you, see you in the future competitions.
", "rawMarkdown": "Dear organizers and people here,\nThank you so much for sharing wonderful opportunity to learn to solve sound classification task.\nWhat's most impressive to me is that we could achieve around 95% accuracy for that 41 classes, though I personally will fail to do that for some of them like Double bass and Cello.\nIt was close to 90% in the early days, but participants improves their solutions and we see over 97% private LB eventually. This is awesome.\nThe experience earned here is now applied to a real problem in my work. This won't have happened without this confidence about what could be done with sounds.\nOnce again thank you, see you in the future competitions.", "votes": null }, { "id": "366734", "postDate": "08/06/2018 11:17:34", "content": "Dear organizers, thanks a lot for this challenge. \nA small question here. You mentioned to provide \"per-class mAP on your papers\" in the Workshop paper. It's a long table but the paper is limited to 4 + 1 pages. I wonder if the per-class mAP can be attached in the end as added page?
", "rawMarkdown": "Dear organizers, thanks a lot for this challenge. \nA small question here. You mentioned to provide \"per-class mAP on your papers\" in the Workshop paper. It's a long table but the paper is limited to 4 + 1 pages. I wonder if the per-class mAP can be attached in the end as added page?", "votes": null }, { "id": "366744", "postDate": "08/06/2018 11:46:45", "content": "Hi,
\n\nAccording to the Workshop rules, all the academic content should be within the 4 pages. So, if possible, try to fit the Table with per-class scores in those 4 pages.
\n\nIf that implies substantial changes in the paper, and considering that the deadline is in a few hours, maybe you can submit a version that uses part of the 5th page. Then, the reviewers will probably suggest to fit it into 4 pages for the camera ready version.
\n\nA suggestion to save space (that you might have already considered): don't list all the category names in text, and also on the results Table. Instead, include them only in the results Table and refer to it.
\n\nhope that helps
", "rawMarkdown": "Hi,\n\nAccording to the Workshop rules, all the academic content should be within the 4 pages. So, if possible, try to fit the Table with per-class scores in those 4 pages.\n\nIf that implies substantial changes in the paper, and considering that the deadline is in a few hours, maybe you can submit a version that uses part of the 5th page. Then, the reviewers will probably suggest to fit it into 4 pages for the camera ready version.\n\nA suggestion to save space (that you might have already considered): don't list all the category names in text, and also on the results Table. Instead, include them only in the results Table and refer to it.\n\nhope that helps", "votes": null }, { "id": "366747", "postDate": "08/06/2018 12:05:13", "content": "> the deadline is in a few hours
\n\nIsn't the deadline 6th Aug 11:59 PM AoE (almost 24 hours away)? Just checking.
", "rawMarkdown": "> the deadline is in a few hours\n\nIsn't the deadline 6th Aug 11:59 PM AoE (almost 24 hours away)? Just checking.", "votes": null }, { "id": "366752", "postDate": "08/06/2018 12:30:36", "content": "yes, deadline is 6th Aug 23:59 AoE
", "rawMarkdown": "yes, deadline is 6th Aug 23:59 AoE", "votes": null }, { "id": "367077", "postDate": "08/07/2018 02:59:28", "content": "Got it, thanks.
", "rawMarkdown": "Got it, thanks.", "votes": null }, { "id": "367368", "postDate": "08/07/2018 15:38:42", "content": "Hello. I haven't received any confirmation email regarding my submission to DCASE workshop. This wasn't the case with technical reports, since one has been sent to me couple of minutes after deadline. I wanted to check if this is a problem or we can expect emails later. Thanks.
", "rawMarkdown": "Hello. I haven't received any confirmation email regarding my submission to DCASE workshop. This wasn't the case with technical reports, since one has been sent to me couple of minutes after deadline. I wanted to check if this is a problem or we can expect emails later. Thanks.", "votes": null }, { "id": "367507", "postDate": "08/07/2018 23:28:03", "content": "I think it should be ok. We didn't receive an email either (when we submitted our paper about the Task Description)
", "rawMarkdown": "I think it should be ok. We didn't receive an email either (when we submitted our paper about the Task Description)", "votes": null }, { "id": "367509", "postDate": "08/07/2018 23:33:25", "content": "Ok, I suppose it should be fine then. Thank you.
", "rawMarkdown": "Ok, I suppose it should be fine then. Thank you.", "votes": null }, { "id": "390288", "postDate": "09/20/2018 01:27:15", "content": "Hi,
\n\nI came to know about this challenge today. For how much time the data will be available? I don't want to officially participate but would like to work on it. Is it possible?
", "rawMarkdown": "Hi,\n\nI came to know about this challenge today. For how much time the data will be available? I don't want to officially participate but would like to work on it. Is it possible?", "votes": null }, { "id": "390334", "postDate": "09/20/2018 03:43:42", "content": "Hi Ethan,
\n\nThis dataset will remain here for your submissions until further notice by the sponsors.
\n\nHappy Modeling!
", "rawMarkdown": "Hi Ethan,\n\nThis dataset will remain here for your submissions until further notice by the sponsors.\n\nHappy Modeling!", "votes": null } ], "comments": [ { "id": 365066, "author_name": "inversion", "author_url": "", "post_date": "08/01/2018 20:26:47", "content": "The updated files are now available on the Data tab.
", "votes": null, "replies": [] }, { "id": 365307, "author_name": "kevinwilkinghoff", "author_url": "", "post_date": "08/02/2018 10:07:49", "content": "Dear organizers,
\n\nthank you for hosting this challenge. It was a pleasure to participate!
\n\nI have one last minor question though. When reporting the per-class mAP@3s in our papers, would you prefer that we
\n\nPersonally, I prefer option a) since the dataset is larger than b). But in any case, it seems to be desirable that we all use the same option to be able to compare the results more precisely.
", "votes": null, "replies": [ { "id": 365480, "author_name": "tqbl95", "author_url": "", "post_date": "08/02/2018 18:42:35", "content": "I was also wondering this in terms of the results that will be published on the DCASE website. My thinking is that the public test set can be considered as a validation set, since we were able to get live feedback on scores, albeit with a limit on tries.
", "votes": null, "replies": [] }, { "id": 365545, "author_name": "eduardofonseca", "author_url": "", "post_date": "08/02/2018 21:51:03", "content": "Hi all, congratulations to both for the good results and also for taking the effort of writing the paper.
\n\n@Kevin, this is a discussion that we've had for a few days. On the one hand, reporting per-class scores on the full test set (ie, private + public splits) can be thought of more meaningful as there is more data to report on. On the other hand, strictly speaking, the final ranking is done solely on the private LB, while the public LB was more of a validation set, as Turab points out.
\n\nAfter discussing, we decided the following (although we admit it can be controversial):
\n\nwe prefer per-class scores on the full test set (ie, private + public splits together). This can be useful in the future should other researchers want to use the FSDKaggle2018 dataset. That being said, if you want, you are welcome to report per-class scores also for the private LB only.
Then, for the global mAP, we think it is most appropriate to report on the triplet: entire test set / public LB / private LB.
The aforementioned criteria is what we followed for the publication describing the competition (see Table 2 and Section 4). https://arxiv.org/pdf/1807.09902.pdf
\n\n@Turab, we haven't decided how to display results on the DCASE web. I guess at least we'll provide mAP on public and private LB.
\n\nThank you!
", "votes": null, "replies": [] } ] }, { "id": 366623, "author_name": "daisukelab", "author_url": "", "post_date": "08/06/2018 04:07:48", "content": "Dear organizers and people here,\nThank you so much for sharing wonderful opportunity to learn to solve sound classification task.\nWhat's most impressive to me is that we could achieve around 95% accuracy for that 41 classes, though I personally will fail to do that for some of them like Double bass and Cello.\nIt was close to 90% in the early days, but participants improves their solutions and we see over 97% private LB eventually. This is awesome.\nThe experience earned here is now applied to a real problem in my work. This won't have happened without this confidence about what could be done with sounds.\nOnce again thank you, see you in the future competitions.
", "votes": null, "replies": [] }, { "id": 366734, "author_name": "sailorwei", "author_url": "", "post_date": "08/06/2018 11:17:34", "content": "Dear organizers, thanks a lot for this challenge. \nA small question here. You mentioned to provide \"per-class mAP on your papers\" in the Workshop paper. It's a long table but the paper is limited to 4 + 1 pages. I wonder if the per-class mAP can be attached in the end as added page?
", "votes": null, "replies": [ { "id": 366744, "author_name": "eduardofonseca", "author_url": "", "post_date": "08/06/2018 11:46:45", "content": "Hi,
\n\nAccording to the Workshop rules, all the academic content should be within the 4 pages. So, if possible, try to fit the Table with per-class scores in those 4 pages.
\n\nIf that implies substantial changes in the paper, and considering that the deadline is in a few hours, maybe you can submit a version that uses part of the 5th page. Then, the reviewers will probably suggest to fit it into 4 pages for the camera ready version.
\n\nA suggestion to save space (that you might have already considered): don't list all the category names in text, and also on the results Table. Instead, include them only in the results Table and refer to it.
\n\nhope that helps
", "votes": null, "replies": [] }, { "id": 366747, "author_name": "tqbl95", "author_url": "", "post_date": "08/06/2018 12:05:13", "content": "> the deadline is in a few hours
\n\nIsn't the deadline 6th Aug 11:59 PM AoE (almost 24 hours away)? Just checking.
", "votes": null, "replies": [] }, { "id": 366752, "author_name": "eduardofonseca", "author_url": "", "post_date": "08/06/2018 12:30:36", "content": "yes, deadline is 6th Aug 23:59 AoE
", "votes": null, "replies": [] }, { "id": 367077, "author_name": "sailorwei", "author_url": "", "post_date": "08/07/2018 02:59:28", "content": "Got it, thanks.
", "votes": null, "replies": [] } ] }, { "id": 367368, "author_name": "kardecbp", "author_url": "", "post_date": "08/07/2018 15:38:42", "content": "Hello. I haven't received any confirmation email regarding my submission to DCASE workshop. This wasn't the case with technical reports, since one has been sent to me couple of minutes after deadline. I wanted to check if this is a problem or we can expect emails later. Thanks.
", "votes": null, "replies": [ { "id": 367507, "author_name": "eduardofonseca", "author_url": "", "post_date": "08/07/2018 23:28:03", "content": "I think it should be ok. We didn't receive an email either (when we submitted our paper about the Task Description)
", "votes": null, "replies": [] }, { "id": 367509, "author_name": "kardecbp", "author_url": "", "post_date": "08/07/2018 23:33:25", "content": "Ok, I suppose it should be fine then. Thank you.
", "votes": null, "replies": [] } ] }, { "id": 390288, "author_name": "ethan14567", "author_url": "", "post_date": "09/20/2018 01:27:15", "content": "Hi,
\n\nI came to know about this challenge today. For how much time the data will be available? I don't want to officially participate but would like to work on it. Is it possible?
", "votes": null, "replies": [ { "id": 390334, "author_name": "addisonhoward", "author_url": "", "post_date": "09/20/2018 03:43:42", "content": "Hi Ethan,
\n\nThis dataset will remain here for your submissions until further notice by the sponsors.
\n\nHappy Modeling!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "365024": "Dear participants,\n\nThe competition has ended. On behalf of the organizers, thank you all for participating. It has been a few intense months where we guess both participants and organizers have learnt a lot. We have had 558 teams with a total of 5,685 submissions in the first edition of the competition! We would like to give special thanks to those who contributed in the Discussion forums sharing their knowledge, and also to those who submitted through the DCASE system. This will have a real impact on the research community.\n\nWe would like to make a few final announcements:\n\n- While strictly speaking the competition is over, you all still have a few days (until Monday 6th August) to **submit a scientific paper** to the [DCASE Workshop][1]. We encourage you all to take advantage of the great work done in the competition and make a paper out of it!\n\n- The **ground truth for the test set is now published on the Data tab**. This also includes Freesound IDs and licenses for the sounds in the dataset (for both train and test). We encourage participants submitting to the DCASE Workshop to use this ground truth to report per-class mAP on your papers.\n\n- The organizers will soon begin the review of the **Judges’ Award** submissions. We’ll announce the winner in the coming weeks. Stay tuned!\n\n- We plan to do a **publication reviewing the competition**, including an analysis of the results, the major trends used, etc. But for this it is essential that we have reports describing the systems. If you'd like us to include your methodology and results in the competition review, either you submit a Workshop paper (which is encouraged), or at least it’d be great if you can send a write-up to eduardo.fonseca@upf.edu, and fill this form https://goo.gl/forms/H6NkZV07p7HIqKXz2 . This would be very useful for us.\n\n- If you use the FSDKaggle2018 dataset or baseline code provided in this competition please cite the following paper:\n\n> Eduardo Fonseca, Manoj Plakal, Frederic Font, Daniel P. W. Ellis,\n> Xavier Favory, Jordi Pons, Xavier Serra. *General-purpose Tagging of\n> Freesound Audio with AudioSet Labels: Task Description, Dataset, and\n> Baseline*. Submitted to DCASE2018 Workshop, 2018. URL:\n> https://arxiv.org/abs/1807.09902\n\nYou can also consider citing the Freesound Datasets platform, that was used to build the FSDKaggle2018 dataset:\n\n> Eduardo Fonseca, Jordi Pons, Xavier Favory, Frederic Font, Dmitry\n> Bogdanov, Andrés Ferraro, Sergio Oramas, Alastair Porter, and Xavier\n> Serra. *Freesound datasets: a platform for the creation of open audio\n> datasets*. In Proceedings of the 18th International Society for Music\n> Information Retrieval Conference (ISMIR 2017), pp 486-493. Suzhou,\n> China, 2017.\n> https://repositori.upf.edu/bitstream/handle/10230/33299/fonseca_ismir17_freesound.pdf?sequence=1&isAllowed=y\n\n- The dataset for this competition, FSDKaggle2018, is a reduced version of a much larger dataset (called [FSD][2]) that we are currently developing at the [Music Technology Group][3] in Barcelona. To this end, we are crowdsourcing annotations through [Freesound Datasets][4], a platform for the creation of open audio datasets. **You are welcome to visit the platform and [contribute with some annotations][5]**. You will be contributing to a large-scale dataset to foster sound recognition research and, who knows, maybe to have a larger and better dataset for a second edition of this competition!\n\nThat’s all for now. Please feel free to share your comments and impressions about the competition in this thread. We’d love to hear your feedback. Thanks again! We hope you enjoyed participating in this competition. It’s been just great!\n\n