While I haven't done a systematic check, it appears to me that the people in the Training set are distinct from the people in the Validation set. Could one of the competition admins tell us whether the Test set will contain:
\r\n\r\na) only people from the Training set,
\r\n\r\nb) only people from the validation set,
\r\n\r\nc) a roughly 50/50 mix of people from both the Training and Validation sets, or
\r\n\r\nd) something else?
\r\n\r\nKnowing this will affect the training strategy, without taking away from the challenge of the competition. thanks, wweight
\r\n", "messages": [ { "id": "27429", "postDate": "07/18/2013 23:16:36", "content": "While I haven't done a systematic check, it appears to me that the people in the Training set are distinct from the people in the Validation set. Could one of the competition admins tell us whether the Test set will contain:
\r\n\r\na) only people from the Training set,
\r\n\r\nb) only people from the validation set,
\r\n\r\nc) a roughly 50/50 mix of people from both the Training and Validation sets, or
\r\n\r\nd) something else?
\r\n\r\nKnowing this will affect the training strategy, without taking away from the challenge of the competition. thanks, wweight
\r\n", "rawMarkdown": "", "votes": null }, { "id": "27474", "postDate": "07/20/2013 15:30:23", "content": "Dear wweight,
\nthe final evaluation data (Test data) will contain people appeared in Training and Validation sets. Take into account that labels for the Validation data will be provided together with the final evaluation data, therefore, you will be able to use both, training and validation data sets in order to train your methods.
\nThanks.
\nXavier
", "rawMarkdown": "", "votes": null }, { "id": "27493", "postDate": "07/21/2013 18:53:19", "content": "Thanks for the response Xavier. It's probably too late to change, but here are some arguments against the Test set including only people that we've trained on.
\n\n
First, this will make the test phase quite different from the initial phase. In the initial phase, where the Training and Validation people are distinct, there is an incentive to create a model that will generalize well to people that are not in the Training set. In the second, final phase, however, there will be no such incentive, and the algorithms and the results might look quite different between these two phases. For example, I estimate that, if my current model were being tested on Training people, then my score would improve from ~0.5 to ~0.2. Such large changes in the incentives and scores between two phases of a competition are a little jarring and unpleasant, although I admit that you have given us plenty of warning time.
\n\n
The second, similar point is that this competition will reward the models that can best recognize gestures of people it has already seen, rather than new people, and will therefore have less "real-world" utility. If the current rules continue, I'd be very surprised if the winner doesn't score lower than 0.1, whereas the generalization error would probably be significantly higher.
\n\n
Two possibilities are (1) do not release the labels for the Validation set, and restrict the Test set to be only people from the Validation set, or (2) release the labels for the Validation set, but use a third, distinct set of people for the Test set (unlikely that such data exists, I know).
\n\n
I may have misunderstood the aim of the competition. Maybe recognizing the gestures of people that have already been seen, rather than new people, is exactly what you're after, in which case the current rules are perfect. Anyway, thanks for a fun competition. -wweight
\n", "rawMarkdown": "", "votes": null }, { "id": "27653", "postDate": "07/25/2013 16:02:36", "content": "
Hi wweight,
\nI am puzzled by your comment that a score of lower than 0.1 is expected by the winning team. After looking at the video and the gestures, my humble opinion is that accurate classification is not easy at all. This is because of two reasons: (1) many gestures are very similar visually, e.g. raising of the hand and small waving of the palm; (2) many gestures are done very quickly, e.g. less than 20 frames.
\nMy understanding of the normalized edit distance score is that it is a kind of error metric between the ground truth and the predicted labels. Given that there are 10 gestures in a test video, a score of 0.1 would mean that there is at most only 1 error (can be 1 insertion, 1 deletion, but no more than that). To me this is very difficult. Do I remember correctly that the state of art for an "easy" dataset such as the KTH (only 6 actions, and each input video is with frames and hence rich with features) is only around 95%?
\nThanks for any feedback and comments.
\ntelepoints
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 27474, "author_name": "xbaro100272", "author_url": "", "post_date": "07/20/2013 15:30:23", "content": "Dear wweight,
\nthe final evaluation data (Test data) will contain people appeared in Training and Validation sets. Take into account that labels for the Validation data will be provided together with the final evaluation data, therefore, you will be able to use both, training and validation data sets in order to train your methods.
\nThanks.
\nXavier
", "votes": null, "replies": [] }, { "id": 27493, "author_name": "rkeisler", "author_url": "", "post_date": "07/21/2013 18:53:19", "content": "Thanks for the response Xavier. It's probably too late to change, but here are some arguments against the Test set including only people that we've trained on.
\n\n
First, this will make the test phase quite different from the initial phase. In the initial phase, where the Training and Validation people are distinct, there is an incentive to create a model that will generalize well to people that are not in the Training set. In the second, final phase, however, there will be no such incentive, and the algorithms and the results might look quite different between these two phases. For example, I estimate that, if my current model were being tested on Training people, then my score would improve from ~0.5 to ~0.2. Such large changes in the incentives and scores between two phases of a competition are a little jarring and unpleasant, although I admit that you have given us plenty of warning time.
\n\n
The second, similar point is that this competition will reward the models that can best recognize gestures of people it has already seen, rather than new people, and will therefore have less "real-world" utility. If the current rules continue, I'd be very surprised if the winner doesn't score lower than 0.1, whereas the generalization error would probably be significantly higher.
\n\n
Two possibilities are (1) do not release the labels for the Validation set, and restrict the Test set to be only people from the Validation set, or (2) release the labels for the Validation set, but use a third, distinct set of people for the Test set (unlikely that such data exists, I know).
\n\n
I may have misunderstood the aim of the competition. Maybe recognizing the gestures of people that have already been seen, rather than new people, is exactly what you're after, in which case the current rules are perfect. Anyway, thanks for a fun competition. -wweight
\n", "votes": null, "replies": [] }, { "id": 27653, "author_name": "kongwah", "author_url": "", "post_date": "07/25/2013 16:02:36", "content": "
Hi wweight,
\nI am puzzled by your comment that a score of lower than 0.1 is expected by the winning team. After looking at the video and the gestures, my humble opinion is that accurate classification is not easy at all. This is because of two reasons: (1) many gestures are very similar visually, e.g. raising of the hand and small waving of the palm; (2) many gestures are done very quickly, e.g. less than 20 frames.
\nMy understanding of the normalized edit distance score is that it is a kind of error metric between the ground truth and the predicted labels. Given that there are 10 gestures in a test video, a score of 0.1 would mean that there is at most only 1 error (can be 1 insertion, 1 deletion, but no more than that). To me this is very difficult. Do I remember correctly that the state of art for an "easy" dataset such as the KTH (only 6 actions, and each input video is with frames and hence rich with features) is only around 95%?
\nThanks for any feedback and comments.
\ntelepoints
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "27429": "", "27474": "", "27493": "", "27653": "" }, "source": "meta" }