In the Freesound General-Purpose Audio Tagging Challenge, contestants have three complementary options for submitting their work, each of them with its own submission requirements and implications. To avoid confusion, this post describes each of them explicitly.
\n\n
\n(1) Kaggle leaderboard
The usual leaderboard in Kaggle competitions. Apart from the system’s output, contestants can submit a write-up describing the submitted system and the code. The model upload occurs through the Team tab of the Kaggle competition page. You will be ranked according to the mAP score.
\n\n
\n(2) DCASE
DCASE is the academic competition in which the Freesound General-Purpose Audio Tagging Challenge is framed. You can find more info in https://www.kaggle.com/c/freesound-audio-tagging#DCASE
\n\nAll participants from the Kaggle leaderboard can also submit to DCASE. DCASE requires that a technical report is submitted along with a metainformation file to help the automatic handling of the submissions. Both of these have predefined templates and must be submitted using the DCASE submissions system. Technical reports will be published on the DCASE website in order to share the knowledge with the research community; an example from last year is here:\nhttp://www.cs.tut.fi/sgn/arg/dcase2017/challenge/task-acoustic-scene-classification-results
\nNOTE1: Systems in DCASE will be ranked according to their mAP score, but the DCASE leaderboard will only include the work of those participants who submit to DCASE (and not all the submissions in the Kaggle leaderboard).
\nNOTE2: while the systems' scoring is done in Kaggle, you can additionally submit your submission_file.csv to DCASE. System outputs submitted to DCASE will be published online on the DCASE2018 website later to allow future evaluations.
Additionally, participants can also submit the same report as a scientific paper to DCASE 2018 Workshop, for which it will be peer-reviewed. In this way you will get the opportunity to attend the workshop and present and discuss your results with other researchers.
\n\nWe strongly encourage all contestants to submit to DCASE in order to share your work with a greater research community. All the details, including templates and the DCASE submission system, can be found at the following urls:
\n\nDeadline for DCASE Technical report and Workshop paper submission is July 31st.
\n\n
\n(3) Judges’ Award
The Judges’ award is a special award that we have organized in order to encourage contestants to use innovative, computationally efficient and problem-specific approaches. To be eligible for the Judges’ Award, participants must submit a technical report and a metainformation file through the DCASE submission system using the DCASE template (submitting the code to DCASE is optional, but encouraged). In addition, participants must submit their code and the technical report to Kaggle. The award will be judged primarily on the description provided in the technical report.
\n\nIn the Judges’ Award, the ranking is not done using the Kaggle leaderboard but based on the subjective judgment of the organizers, according to a set of criteria specified in this thread along with the Judges’ Award Rules. Please check it out for more info.
\n\n
\nNOTE: All the three submission options are compatible. The Kaggle leaderboard is the default and main ranking for this competition. DCASE and Judges’ Award are optional. The DCASE submission allows participants to easily share their work with a greater research community and get further academic reputation. The Judges’ Award is a way to showcase model’s novelty and efficiency beyond the Kaggle leaderboard.
Please don’t hesitate to post your questions in this thread.
\n\nGood luck!
\n\nFrederic Font, Eduardo Fonseca (Freesound, MTG-UPF)
\nDan Ellis, Manoj Plakal (Google Machine Perception)
In the Freesound General-Purpose Audio Tagging Challenge, contestants have three complementary options for submitting their work, each of them with its own submission requirements and implications. To avoid confusion, this post describes each of them explicitly.
\n\n
\n(1) Kaggle leaderboard
The usual leaderboard in Kaggle competitions. Apart from the system’s output, contestants can submit a write-up describing the submitted system and the code. The model upload occurs through the Team tab of the Kaggle competition page. You will be ranked according to the mAP score.
\n\n
\n(2) DCASE
DCASE is the academic competition in which the Freesound General-Purpose Audio Tagging Challenge is framed. You can find more info in https://www.kaggle.com/c/freesound-audio-tagging#DCASE
\n\nAll participants from the Kaggle leaderboard can also submit to DCASE. DCASE requires that a technical report is submitted along with a metainformation file to help the automatic handling of the submissions. Both of these have predefined templates and must be submitted using the DCASE submissions system. Technical reports will be published on the DCASE website in order to share the knowledge with the research community; an example from last year is here:\nhttp://www.cs.tut.fi/sgn/arg/dcase2017/challenge/task-acoustic-scene-classification-results
\nNOTE1: Systems in DCASE will be ranked according to their mAP score, but the DCASE leaderboard will only include the work of those participants who submit to DCASE (and not all the submissions in the Kaggle leaderboard).
\nNOTE2: while the systems' scoring is done in Kaggle, you can additionally submit your submission_file.csv to DCASE. System outputs submitted to DCASE will be published online on the DCASE2018 website later to allow future evaluations.
Additionally, participants can also submit the same report as a scientific paper to DCASE 2018 Workshop, for which it will be peer-reviewed. In this way you will get the opportunity to attend the workshop and present and discuss your results with other researchers.
\n\nWe strongly encourage all contestants to submit to DCASE in order to share your work with a greater research community. All the details, including templates and the DCASE submission system, can be found at the following urls:
\n\nDeadline for DCASE Technical report and Workshop paper submission is July 31st.
\n\n
\n(3) Judges’ Award
The Judges’ award is a special award that we have organized in order to encourage contestants to use innovative, computationally efficient and problem-specific approaches. To be eligible for the Judges’ Award, participants must submit a technical report and a metainformation file through the DCASE submission system using the DCASE template (submitting the code to DCASE is optional, but encouraged). In addition, participants must submit their code and the technical report to Kaggle. The award will be judged primarily on the description provided in the technical report.
\n\nIn the Judges’ Award, the ranking is not done using the Kaggle leaderboard but based on the subjective judgment of the organizers, according to a set of criteria specified in this thread along with the Judges’ Award Rules. Please check it out for more info.
\n\n
\nNOTE: All the three submission options are compatible. The Kaggle leaderboard is the default and main ranking for this competition. DCASE and Judges’ Award are optional. The DCASE submission allows participants to easily share their work with a greater research community and get further academic reputation. The Judges’ Award is a way to showcase model’s novelty and efficiency beyond the Kaggle leaderboard.
Please don’t hesitate to post your questions in this thread.
\n\nGood luck!
\n\nFrederic Font, Eduardo Fonseca (Freesound, MTG-UPF)
\nDan Ellis, Manoj Plakal (Google Machine Perception)
@Eduardo,\nFirst of all, thank you and @Kaggle for organizing this wonderful competition here.
\n\nDo you think it is anyway possible to have the paper submission deadlines extended? I mean we can have the competitions end as scheduled, however, I was wondering if we can have a little more time to work on the papers. I joined quite late into the competition and the final LB scores can determine more confidently which approaches worked and which did not. \nWill it is possible?\nRegards,\nRC
", "rawMarkdown": "Eduardo,\nFirst of all, thank you and @Kaggle for organizing this wonderful competition here. \n\nDo you think it is anyway possible to have the paper submission deadlines extended? I mean we can have the competitions end as scheduled, however, I was wondering if we can have a little more time to work on the papers. I joined quite late into the competition and the final LB scores can determine more confidently which approaches worked and which did not. \nWill it is possible?\nRegards,\nRC", "votes": null }, { "id": "360864", "postDate": "07/23/2018 11:35:03", "content": "Hi, glad to hear that you like it!
\n\nThe technical report and workshop paper submission deadlines are established by the DCASE organization and are common to all the five tasks in DCASE (we are task 2).
\n\nAccording to the respective websites, the deadlines are July31st for:
\n\ntechnical report:
\nhttp://dcase.community/challenge2018/
\n(please find the Submission system here http://dcase.community/challenge2018/submission )
workshop paper:
\nhttp://dcase.community/workshop2018/submission
However, if a paper submission deadline extension were to happen, I will make sure to post it here ASAP.
\n\nPD. It also happens in the rest of the tasks that the participants submit their reports/papers without knowing their scores on the final LB.
\n\nhope this helps. Please do not hesitate to post your doubts!
", "rawMarkdown": "Hi, glad to hear that you like it!\n\nThe technical report and workshop paper submission deadlines are established by the DCASE organization and are common to all the five tasks in DCASE (we are task 2). \n\nAccording to the respective websites, the deadlines are July31st for:\n\n- technical report: \nhttp://dcase.community/challenge2018/ \n(please find the Submission system here http://dcase.community/challenge2018/submission )\n\n- workshop paper: \nhttp://dcase.community/workshop2018/submission\n\nHowever, if a paper submission deadline extension were to happen, I will make sure to post it here ASAP. \n\nPD. It also happens in the rest of the tasks that the participants submit their reports/papers without knowing their scores on the final LB. \n\nhope this helps. Please do not hesitate to post your doubts!", "votes": null }, { "id": "363017", "postDate": "07/27/2018 15:50:55", "content": "@Gyat,
\n\njust to let you know that DCASE organizers have just confirmed an extension to the scientific papers submission deadline, to Monday 6th August 2018.
\n\nHowever, the competition deadline is still the same (July 31st). Please find details on this thread:\nhttps://www.kaggle.com/c/freesound-audio-tagging/discussion/62100
", "rawMarkdown": "Gyat, \n\njust to let you know that DCASE organizers have just confirmed an **extension to the scientific papers submission deadline, to Monday 6th August 2018.**\n\nHowever, the competition deadline is still the same (July 31st). Please find details on this thread:\nhttps://www.kaggle.com/c/freesound-audio-tagging/discussion/62100", "votes": null }, { "id": "363026", "postDate": "07/27/2018 16:15:49", "content": "This is a good news indeed! Thank you :)
", "rawMarkdown": "This is a good news indeed! Thank you :)", "votes": null }, { "id": "363615", "postDate": "07/29/2018 17:13:33", "content": "Just to confirm, does this extension apply to the technical report as well?
", "rawMarkdown": "Just to confirm, does this extension apply to the technical report as well?", "votes": null }, { "id": "363702", "postDate": "07/29/2018 23:08:58", "content": "No it does not.\nDeadline for technical report and metainfo file through DCASE challenge is July 31st.
\n\nhttps://www.kaggle.com/c/freesound-audio-tagging/discussion/62100
", "rawMarkdown": "No it does not.\nDeadline for technical report and metainfo file through DCASE challenge is July 31st.\n\nhttps://www.kaggle.com/c/freesound-audio-tagging/discussion/62100", "votes": null }, { "id": "363927", "postDate": "07/30/2018 10:50:37", "content": "Eduardo,
\n\nKindly confirm if the following zip structure looks correct:
\n\nZip-package root
\n\n|__ Task 2
\n\n |___ technical_report.pdf\n\n |___ submission 1\n\n |____ submission 1.csv\n\n |____ submission 1.meta.yaml\n\n |___ submission 2\n\n |____ submission 2.csv\n\n |____ submission 2.meta.yaml\n\n |___ Judge's choice\n\n |____ judge's choice.csv\n\n |____ judge's choice.meta.yaml\nRequesting you to kindly ignore the names for now... The naming rules would be as per the website, except for the judge's choice, where the files would have a suffix as 'judges_choice'.
\n\nSo to sum up, zip would contain one technical report, two submission files along with their yamls and one extra file (with yaml) for judge's choice, right?
", "rawMarkdown": "Eduardo,\n\nKindly confirm if the following zip structure looks correct:\n\nZip-package root\n\n|__ Task 2\n\n |___ technical_report.pdf\n\n |___ submission 1\n\n |____ submission 1.csv\n\n |____ submission 1.meta.yaml\n\n |___ submission 2\n\n |____ submission 2.csv\n\n |____ submission 2.meta.yaml\n\n |___ Judge's choice\n\n |____ judge's choice.csv\n\n |____ judge's choice.meta.yaml\n\nRequesting you to kindly ignore the names for now... The naming rules would be as per the website, except for the judge's choice, where the files would have a suffix as 'judges_choice'.\n\nSo to sum up, zip would contain one technical report, two submission files along with their yamls and one extra file (with yaml) for judge's choice, right?", "votes": null }, { "id": "363949", "postDate": "07/30/2018 11:53:48", "content": "Hei Gyat,
\n\nyes, it seems correct at first sight.
\n\nNevertheless you can double-check the template here:\nhttp://dcase.community/challenge2018/submission#package-structure
\n\nAbout the judges' award submission, we are interested in knowing the original filename.csv that you submit to Kaggle. In this way, we can check the mAP score directly from the Kaggle UI. So please:
\n\nin your technical report, please indicate the name of the original csv, as per the Requirements established in https://www.kaggle.com/c/freesound-audio-tagging/discussion/59932
when you also submit the Judges' Award csv within the DCASE zip, please keep the original filename for the CSV
Finally, to avoid potential problems, it is better to use the string judges_award, without any '
thanks
", "rawMarkdown": "Hei Gyat, \n\nyes, it seems correct at first sight. \n\nNevertheless you can double-check the template here:\nhttp://dcase.community/challenge2018/submission#package-structure\n\nAbout the **judges' award submission**, we are interested in knowing **the original filename.csv** that you submit to Kaggle. In this way, we can check the mAP score directly from the Kaggle UI. So please:\n\n- in your technical report, please indicate the name of the original csv, as per the Requirements established in https://www.kaggle.com/c/freesound-audio-tagging/discussion/59932\n\n- when you also submit the Judges' Award csv within the DCASE zip, please keep the original filename for the CSV\n\nFinally, to avoid potential problems, it is better to use the string `judges_award`, without any `'`\n\nthanks", "votes": null } ], "comments": [ { "id": 360843, "author_name": "gyat2017", "author_url": "", "post_date": "07/23/2018 10:25:19", "content": "@Eduardo,\nFirst of all, thank you and @Kaggle for organizing this wonderful competition here.
\n\nDo you think it is anyway possible to have the paper submission deadlines extended? I mean we can have the competitions end as scheduled, however, I was wondering if we can have a little more time to work on the papers. I joined quite late into the competition and the final LB scores can determine more confidently which approaches worked and which did not. \nWill it is possible?\nRegards,\nRC
", "votes": null, "replies": [ { "id": 360864, "author_name": "eduardofonseca", "author_url": "", "post_date": "07/23/2018 11:35:03", "content": "Hi, glad to hear that you like it!
\n\nThe technical report and workshop paper submission deadlines are established by the DCASE organization and are common to all the five tasks in DCASE (we are task 2).
\n\nAccording to the respective websites, the deadlines are July31st for:
\n\ntechnical report:
\nhttp://dcase.community/challenge2018/
\n(please find the Submission system here http://dcase.community/challenge2018/submission )
workshop paper:
\nhttp://dcase.community/workshop2018/submission
However, if a paper submission deadline extension were to happen, I will make sure to post it here ASAP.
\n\nPD. It also happens in the rest of the tasks that the participants submit their reports/papers without knowing their scores on the final LB.
\n\nhope this helps. Please do not hesitate to post your doubts!
", "votes": null, "replies": [] }, { "id": 363017, "author_name": "eduardofonseca", "author_url": "", "post_date": "07/27/2018 15:50:55", "content": "@Gyat,
\n\njust to let you know that DCASE organizers have just confirmed an extension to the scientific papers submission deadline, to Monday 6th August 2018.
\n\nHowever, the competition deadline is still the same (July 31st). Please find details on this thread:\nhttps://www.kaggle.com/c/freesound-audio-tagging/discussion/62100
", "votes": null, "replies": [] }, { "id": 363026, "author_name": "gyat2017", "author_url": "", "post_date": "07/27/2018 16:15:49", "content": "This is a good news indeed! Thank you :)
", "votes": null, "replies": [] }, { "id": 363615, "author_name": "tqbl95", "author_url": "", "post_date": "07/29/2018 17:13:33", "content": "Just to confirm, does this extension apply to the technical report as well?
", "votes": null, "replies": [] }, { "id": 363702, "author_name": "eduardofonseca", "author_url": "", "post_date": "07/29/2018 23:08:58", "content": "No it does not.\nDeadline for technical report and metainfo file through DCASE challenge is July 31st.
\n\nhttps://www.kaggle.com/c/freesound-audio-tagging/discussion/62100
", "votes": null, "replies": [] }, { "id": 363927, "author_name": "gyat2017", "author_url": "", "post_date": "07/30/2018 10:50:37", "content": "Eduardo,
\n\nKindly confirm if the following zip structure looks correct:
\n\nZip-package root
\n\n|__ Task 2
\n\n |___ technical_report.pdf\n\n |___ submission 1\n\n |____ submission 1.csv\n\n |____ submission 1.meta.yaml\n\n |___ submission 2\n\n |____ submission 2.csv\n\n |____ submission 2.meta.yaml\n\n |___ Judge's choice\n\n |____ judge's choice.csv\n\n |____ judge's choice.meta.yaml\nRequesting you to kindly ignore the names for now... The naming rules would be as per the website, except for the judge's choice, where the files would have a suffix as 'judges_choice'.
\n\nSo to sum up, zip would contain one technical report, two submission files along with their yamls and one extra file (with yaml) for judge's choice, right?
", "votes": null, "replies": [] }, { "id": 363949, "author_name": "eduardofonseca", "author_url": "", "post_date": "07/30/2018 11:53:48", "content": "Hei Gyat,
\n\nyes, it seems correct at first sight.
\n\nNevertheless you can double-check the template here:\nhttp://dcase.community/challenge2018/submission#package-structure
\n\nAbout the judges' award submission, we are interested in knowing the original filename.csv that you submit to Kaggle. In this way, we can check the mAP score directly from the Kaggle UI. So please:
\n\nin your technical report, please indicate the name of the original csv, as per the Requirements established in https://www.kaggle.com/c/freesound-audio-tagging/discussion/59932
when you also submit the Judges' Award csv within the DCASE zip, please keep the original filename for the CSV
Finally, to avoid potential problems, it is better to use the string judges_award, without any '
thanks
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "349643": "In the Freesound General-Purpose Audio Tagging Challenge, contestants have three complementary options for submitting their work, each of them with its own submission requirements and implications. To avoid confusion, this post describes each of them explicitly.\n\n