The test set for this competition is somehow light (~2000 records of 60 seconds each) but still publicly available. Isn't something wrong ? What if someone just walk through it and listen to the records and manually label them accordingly. That is a concrete risk even if labeling hours of records may be a tough task !
\nA solution could consist in organizing this as Code competition, as pointed out by @theo. Hence, the organizers could hide the private test set. Perhaps, I'm missing something ?
", "messages": [ { "id": "1083260", "postDate": "11/18/2020 19:13:33", "content": "The test set for this competition is somehow light (~2000 records of 60 seconds each) but still publicly available. Isn't something wrong ? What if someone just walk through it and listen to the records and manually label them accordingly. That is a concrete risk even if labeling hours of records may be a tough task !
\nA solution could consist in organizing this as Code competition, as pointed out by @theo. Hence, the organizers could hide the private test set. Perhaps, I'm missing something ?
", "rawMarkdown": "The test set for this competition is somehow light (**~2000 records of 60 seconds each**) but still publicly available. Isn't something wrong ? What if someone just walk through it and listen to the records and manually label them accordingly. That is a concrete risk even if labeling hours of records may be a tough task !\n\nA solution could consist in organizing this as **Code competition**, as pointed out by *@theo*. Hence, the organizers could hide the private test set. Perhaps, I'm missing something ?", "votes": null }, { "id": "1083273", "postDate": "11/18/2020 19:32:21", "content": "I also tought about that but maybe that is one of the reasons why they are hiding the species_id mapping too. You would have to map it and understand also their songtype_id
But yes I would also like the private test set to be hidden.
", "rawMarkdown": "I also tought about that but maybe that is one of the reasons why they are hiding the `species_id` mapping too. You would have to map it and understand also their `songtype_id` \n\nBut yes I would also like the private test set to be hidden.", "votes": null }, { "id": "1083306", "postDate": "11/18/2020 20:28:31", "content": "I agree that this would have been better as a code competition but probably a bit late now maybe since many people have probably already downloaded the dataset.
", "rawMarkdown": "I agree that this would have been better as a code competition but probably a bit late now maybe since many people have probably already downloaded the dataset.", "votes": null }, { "id": "1083313", "postDate": "11/18/2020 20:38:48", "content": "A bit late yes !
", "rawMarkdown": "A bit late yes !", "votes": null }, { "id": "1083314", "postDate": "11/18/2020 20:41:07", "content": "Unfortunately, hiding species_id is not enough as a person with some domain knowledge could re-identify them by listening to the records.
", "rawMarkdown": "Unfortunately, hiding **species_id** is not enough as a person with some domain knowledge could re-identify them by listening to the records.", "votes": null }, { "id": "1083317", "postDate": "11/18/2020 20:55:51", "content": "Thanks for raising the question so we can address.
\nFrom a theoretical standpoint, it's always better to run competitions with a completely hidden test set. From a practical standpoint, though, they can create extra friction for community members who may prefer to do work using their own hardware with their own tools.
\nWe are always assessing these competing factors when deciding on how to launch a competition.
\nTo your point, this competition does have a smaller test set, but an important factor is the amount of other-species noise in each audio sample, as well as the number of species to be labeled. If this was, for example, a single-class-per-file format, with rather simple audio, 2,000 observations wouldn't have been enough, for sure.
\nSo that brings me to my obligatory reminder: hand labeling a test set is cheating and could result in a permanent account ban. So let's all be Kaggley and to the right thing. 😃
", "rawMarkdown": "Thanks for raising the question so we can address.\n\nFrom a theoretical standpoint, it's always better to run competitions with a completely hidden test set. From a practical standpoint, though, they can create extra friction for community members who may prefer to do work using their own hardware with their own tools.\n\nWe are always assessing these competing factors when deciding on how to launch a competition.\n\nTo your point, this competition does have a smaller test set, but an important factor is the amount of other-species noise in each audio sample, as well as the number of species to be labeled. If this was, for example, a single-class-per-file format, with rather simple audio, 2,000 observations wouldn't have been enough, for sure.\n\nSo that brings me to my obligatory reminder: **hand labeling a test set is cheating** and could result in a permanent account ban. So let's all be Kaggley and to the right thing. 😃", "votes": null }, { "id": "1083339", "postDate": "11/18/2020 21:59:40", "content": "Thanks for your quick and thorough answer @inversion . I hope all of us will be following your last recommendation :
\n\n", "rawMarkdown": "Thanks for your quick and thorough answer @inversion . I hope all of us will be following your last recommendation : \n> So let's all be Kaggley and do the right thing 😃", "votes": null }, { "id": "1085111", "postDate": "11/20/2020 17:13:41", "content": "So let's all be Kaggley and do the right thing 😃
\n
Open tests are much easier to work with than hidden test set. But this makes cheating possible. For the next competition it is possible to provide a full test 2 weeks before the end of the competition.
", "rawMarkdown": "Open tests are much easier to work with than hidden test set. But this makes cheating possible. For the next competition it is possible to provide a full test 2 weeks before the end of the competition.", "votes": null } ], "comments": [ { "id": 1083273, "author_name": "aliabdin1", "author_url": "", "post_date": "11/18/2020 19:32:21", "content": "I also tought about that but maybe that is one of the reasons why they are hiding the species_id mapping too. You would have to map it and understand also their songtype_id
But yes I would also like the private test set to be hidden.
", "votes": null, "replies": [ { "id": 1083314, "author_name": "kneroma", "author_url": "", "post_date": "11/18/2020 20:41:07", "content": "Unfortunately, hiding species_id is not enough as a person with some domain knowledge could re-identify them by listening to the records.
", "votes": null, "replies": [] } ] }, { "id": 1083306, "author_name": "jackvial", "author_url": "", "post_date": "11/18/2020 20:28:31", "content": "I agree that this would have been better as a code competition but probably a bit late now maybe since many people have probably already downloaded the dataset.
", "votes": null, "replies": [ { "id": 1083313, "author_name": "kneroma", "author_url": "", "post_date": "11/18/2020 20:38:48", "content": "A bit late yes !
", "votes": null, "replies": [] } ] }, { "id": 1083317, "author_name": "inversion", "author_url": "", "post_date": "11/18/2020 20:55:51", "content": "Thanks for raising the question so we can address.
\nFrom a theoretical standpoint, it's always better to run competitions with a completely hidden test set. From a practical standpoint, though, they can create extra friction for community members who may prefer to do work using their own hardware with their own tools.
\nWe are always assessing these competing factors when deciding on how to launch a competition.
\nTo your point, this competition does have a smaller test set, but an important factor is the amount of other-species noise in each audio sample, as well as the number of species to be labeled. If this was, for example, a single-class-per-file format, with rather simple audio, 2,000 observations wouldn't have been enough, for sure.
\nSo that brings me to my obligatory reminder: hand labeling a test set is cheating and could result in a permanent account ban. So let's all be Kaggley and to the right thing. 😃
", "votes": null, "replies": [ { "id": 1083339, "author_name": "kneroma", "author_url": "", "post_date": "11/18/2020 21:59:40", "content": "Thanks for your quick and thorough answer @inversion . I hope all of us will be following your last recommendation :
\n\n", "votes": null, "replies": [] } ] }, { "id": 1085111, "author_name": "alien308", "author_url": "", "post_date": "11/20/2020 17:13:41", "content": "So let's all be Kaggley and do the right thing 😃
\n
Open tests are much easier to work with than hidden test set. But this makes cheating possible. For the next competition it is possible to provide a full test 2 weeks before the end of the competition.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1083260": "The test set for this competition is somehow light (**~2000 records of 60 seconds each**) but still publicly available. Isn't something wrong ? What if someone just walk through it and listen to the records and manually label them accordingly. That is a concrete risk even if labeling hours of records may be a tough task !\n\nA solution could consist in organizing this as **Code competition**, as pointed out by *@theo*. Hence, the organizers could hide the private test set. Perhaps, I'm missing something ?", "1083273": "I also tought about that but maybe that is one of the reasons why they are hiding the `species_id` mapping too. You would have to map it and understand also their `songtype_id` \n\nBut yes I would also like the private test set to be hidden.", "1083306": "I agree that this would have been better as a code competition but probably a bit late now maybe since many people have probably already downloaded the dataset.", "1083313": "A bit late yes !", "1083314": "Unfortunately, hiding **species_id** is not enough as a person with some domain knowledge could re-identify them by listening to the records.", "1083317": "Thanks for raising the question so we can address.\n\nFrom a theoretical standpoint, it's always better to run competitions with a completely hidden test set. From a practical standpoint, though, they can create extra friction for community members who may prefer to do work using their own hardware with their own tools.\n\nWe are always assessing these competing factors when deciding on how to launch a competition.\n\nTo your point, this competition does have a smaller test set, but an important factor is the amount of other-species noise in each audio sample, as well as the number of species to be labeled. If this was, for example, a single-class-per-file format, with rather simple audio, 2,000 observations wouldn't have been enough, for sure.\n\nSo that brings me to my obligatory reminder: **hand labeling a test set is cheating** and could result in a permanent account ban. So let's all be Kaggley and to the right thing. 😃", "1083339": "Thanks for your quick and thorough answer @inversion . I hope all of us will be following your last recommendation : \n> So let's all be Kaggley and do the right thing 😃", "1085111": "Open tests are much easier to work with than hidden test set. But this makes cheating possible. For the next competition it is possible to provide a full test 2 weeks before the end of the competition." }, "source": "meta" }