{ "id": 5193, "title": "request for edit distance source code", "url": "/competitions/multi-modal-gesture-recognition/discussion/5193", "author_name": "", "post_date": "2013-07-24T16:55:19.057Z", "votes": null, "comment_count": 5, "views": 1275, "content": "

Dear organizers,

\n

Is the source code for the edit distance metric available? I understand that we are able submit to the online website to get instant results. But having the edit distance source code will be useful for our internal tuning.

\n

thanks

\n

telepoints

\n

 

", "messages": [ { "id": "27603", "postDate": "07/24/2013 16:55:19", "content": "

Dear organizers,

\n

Is the source code for the edit distance metric available? I understand that we are able submit to the online website to get instant results. But having the edit distance source code will be useful for our internal tuning.

\n

thanks

\n

telepoints

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "27614", "postDate": "07/24/2013 18:32:46", "content": "

Hi kongwah,

\n

 I guess you can have the levenshtein_distance from here:

\n

http://en.wikibooks.org/wiki/Algorithm_Implementation/Strings/Levenshtein_distance#Octave_And_MATLAB

\n

Regards

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "27633", "postDate": "07/25/2013 00:47:06", "content": "

Thank you StevenWudi,

\n

I am trying to understand the levenshtein distance metric more.

\n

This is a metric measuring the distance between two input strings. In Chalearn evaluation, one of the two input strings is the ground truth label, e.g. [14, 8, 3, ....]; the other input string is the predicted label. The two input strings can be of different length.

\n

I submitted a random test.csv to the system, and the metric returns 1.46

\n

This is surprising because given that each evaluation ground truth label is expected to be of length 20, the edit distance of a random test string would be very high. Can you roughly explain to me why the return score is so low?

\n

thanks so much for your time.

\n

telepoints

", "rawMarkdown": "", "votes": null }, { "id": "27642", "postDate": "07/25/2013 09:04:30", "content": "

HI telepoint,

\n

  I am not quite sure when you said you submitted a test.csv for validation set or training set. For validation set, I think the number of gesture is not fixed to the length of 20. So that may be the problem.

\n

Cheers

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "27651", "postDate": "07/25/2013 15:18:56", "content": "

So sorry for not being clear.

\n

I generated a random test.csv file. This file is in the same format as the sample file to be uploaded to the submission system. I did this to ensure that I know how to submit my prediction to the system.

\n

In this test.csv, I generated 287 lines. Each line is prefix by the video ID, and then a string of 20 unique numbers (from 1 to 20). This simulates the prediction for all 287 validation videos.

\n

The system accepts my upload and returns a score of 1.46

\n

As you have said, even though the number of gesture in the Validation video is not fixed to 20, I think the edit distance for each video will be very high, because of the randomness of the predicted number.

\n

After reading more on edit distance, I now believe the system could be computing a normalized score. But still, why is the number not between [0,1] ?

\n

I think it is better for the organizers to send us the source code.

", "rawMarkdown": "", "votes": null }, { "id": "27652", "postDate": "07/25/2013 15:28:09", "content": "

Hi Stevenwudi,

\n

By the way, I understand that we are supposed to upload our code/scripts, etc, alongside with the predictions for the final evaluation data. Do you know what kind of code are we expected to upload? 

\n

For example, are we expected to upload code to generate the visual features (I am working on the video images) of an input video? There are two problems if this is the case

\n

First, I am using quite a few third party modules such as OpenCV, ffmpeg, imagemagick, etc. It can be tricky to reproduuce the exact runtime environment as my Linux machine.

\n

Second, the feature extraction I am using is very compute intensive. I had resorted to using mpi to parallelize the code so that it can spawn many threads.  I am not sure if the mpi framework is supported by the organizers.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 27614, "author_name": "stevenwudi", "author_url": "", "post_date": "07/24/2013 18:32:46", "content": "

Hi kongwah,

\n

 I guess you can have the levenshtein_distance from here:

\n

http://en.wikibooks.org/wiki/Algorithm_Implementation/Strings/Levenshtein_distance#Octave_And_MATLAB

\n

Regards

\n

 

", "votes": null, "replies": [] }, { "id": 27633, "author_name": "kongwah", "author_url": "", "post_date": "07/25/2013 00:47:06", "content": "

Thank you StevenWudi,

\n

I am trying to understand the levenshtein distance metric more.

\n

This is a metric measuring the distance between two input strings. In Chalearn evaluation, one of the two input strings is the ground truth label, e.g. [14, 8, 3, ....]; the other input string is the predicted label. The two input strings can be of different length.

\n

I submitted a random test.csv to the system, and the metric returns 1.46

\n

This is surprising because given that each evaluation ground truth label is expected to be of length 20, the edit distance of a random test string would be very high. Can you roughly explain to me why the return score is so low?

\n

thanks so much for your time.

\n

telepoints

", "votes": null, "replies": [] }, { "id": 27642, "author_name": "stevenwudi", "author_url": "", "post_date": "07/25/2013 09:04:30", "content": "

HI telepoint,

\n

  I am not quite sure when you said you submitted a test.csv for validation set or training set. For validation set, I think the number of gesture is not fixed to the length of 20. So that may be the problem.

\n

Cheers

\n

 

\n

 

", "votes": null, "replies": [] }, { "id": 27651, "author_name": "kongwah", "author_url": "", "post_date": "07/25/2013 15:18:56", "content": "

So sorry for not being clear.

\n

I generated a random test.csv file. This file is in the same format as the sample file to be uploaded to the submission system. I did this to ensure that I know how to submit my prediction to the system.

\n

In this test.csv, I generated 287 lines. Each line is prefix by the video ID, and then a string of 20 unique numbers (from 1 to 20). This simulates the prediction for all 287 validation videos.

\n

The system accepts my upload and returns a score of 1.46

\n

As you have said, even though the number of gesture in the Validation video is not fixed to 20, I think the edit distance for each video will be very high, because of the randomness of the predicted number.

\n

After reading more on edit distance, I now believe the system could be computing a normalized score. But still, why is the number not between [0,1] ?

\n

I think it is better for the organizers to send us the source code.

", "votes": null, "replies": [] }, { "id": 27652, "author_name": "kongwah", "author_url": "", "post_date": "07/25/2013 15:28:09", "content": "

Hi Stevenwudi,

\n

By the way, I understand that we are supposed to upload our code/scripts, etc, alongside with the predictions for the final evaluation data. Do you know what kind of code are we expected to upload? 

\n

For example, are we expected to upload code to generate the visual features (I am working on the video images) of an input video? There are two problems if this is the case

\n

First, I am using quite a few third party modules such as OpenCV, ffmpeg, imagemagick, etc. It can be tricky to reproduuce the exact runtime environment as my Linux machine.

\n

Second, the feature extraction I am using is very compute intensive. I had resorted to using mpi to parallelize the code so that it can spawn many threads.  I am not sure if the mpi framework is supported by the organizers.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "27603": "", "27614": "", "27633": "", "27642": "", "27651": "", "27652": "" }, "source": "meta" }