The goal of the challenge is to devise a fully automatic system of gesture recognition. When we release the final evaluation set, the participants will have to provide predictions of labels without making changes to their system that involves\r\n a human looking at the data (either to label it, perform temporal segmentation, identify body parts, or to provide any other type of human knowledge).
\r\nTo enforce that rule, we will ask the top ranking participants to cooperate to reproduce their results (see the\r\nverification procedure). The preferred method is that the participants submit their code together with their predictions. Submission of code will be open shortly. It is OPTIONAL but will\r\n facilitate the verification and reduce the risk of disqualification.
\r\nWe are showing below a DRAFT of instructions for people who want to submit code. Please give us feed-back BEFORE March 7, 2012, if you want your comments to be taken into account
\r\n=====================================================
\r\nCHALEARN Gesture Challenge", "messages": [ { "id": "8743", "postDate": "02/28/2012 19:26:42", "content": "
DRAFT instructions to prepare software submission (for verification purpose only)
For questions, contact: events@chalearn.org
==========================================================================
By submitting their software, the authors grant to the organizers a license to evaluate it for the purpose of verifying the challenge results only. The authors retain all rights to their software. The organizers will keep the software confidential.
==========================================================================
1) Libraries
The authors are responsible to include all the necessary libraries with their software. The software should be completely self-contained, unless agreed in advance with the organizers.
2) Platform
The authors should provide an executable for either
- Windows 7
- Mac OS X 10.5 or later
or interpretable code in either
- Matlab (release R2011a or higher)
- Java (latest release)
The authors should contact the organizers in advance if they want to submit compilable source code under Unix platforms or other versions of the OS, Matlab or Java, or if they want to use other interpreted languages such as Python of R. Matlab and Java libraries will not be included by the organizers, unless agreed in advance.
3) Put your software, including all libraries in a directory bearing the name of your team. Add a README file with installation and usage instructions. Zip the directory and upload it to xxxxxx.
4) Recommended usage:
> prepare_final_resu(input_dir, output_file)
The code should comprise a main command line function called prepare_final_resu taking two arguments: the input directory and the output file name. The input directory will contain the batches to be processed organized in sub-directories (e.g. valid01, valid02, ... final01, final02). The software should list these files and automatically process them all.
It is recommended that the code possess some kind of exception handling such that if it fails on a few files, it does not crash but runs on the remaining files. The organizers should be able to run the software on new data batches that were never released to the participants.
5) Output format:
The output file should be in the challenge submission format, see:
http://www.kaggle.com/c/GestureChallenge/details/SubmissionInstructions
such that the organizers can submit the file to the challenge website and obtain the same result that was submitted by the authors of the software on valid and final data.
The goal of the challenge is to devise a fully automatic system of gesture recognition. When we release the final evaluation set, the participants will have to provide predictions of labels without making changes to their system that involves\r\n a human looking at the data (either to label it, perform temporal segmentation, identify body parts, or to provide any other type of human knowledge).
\r\nTo enforce that rule, we will ask the top ranking participants to cooperate to reproduce their results (see the\r\nverification procedure). The preferred method is that the participants submit their code together with their predictions. Submission of code will be open shortly. It is OPTIONAL but will\r\n facilitate the verification and reduce the risk of disqualification.
\r\nWe are showing below a DRAFT of instructions for people who want to submit code. Please give us feed-back BEFORE March 7, 2012, if you want your comments to be taken into account
\r\n=====================================================
\r\nCHALEARN Gesture Challenge", "rawMarkdown": "", "votes": null }, { "id": "8744", "postDate": "02/28/2012 23:03:46", "content": "
DRAFT instructions to prepare software submission (for verification purpose only)
For questions, contact: events@chalearn.org
==========================================================================
By submitting their software, the authors grant to the organizers a license to evaluate it for the purpose of verifying the challenge results only. The authors retain all rights to their software. The organizers will keep the software confidential.
==========================================================================
1) Libraries
The authors are responsible to include all the necessary libraries with their software. The software should be completely self-contained, unless agreed in advance with the organizers.
2) Platform
The authors should provide an executable for either
- Windows 7
- Mac OS X 10.5 or later
or interpretable code in either
- Matlab (release R2011a or higher)
- Java (latest release)
The authors should contact the organizers in advance if they want to submit compilable source code under Unix platforms or other versions of the OS, Matlab or Java, or if they want to use other interpreted languages such as Python of R. Matlab and Java libraries will not be included by the organizers, unless agreed in advance.
3) Put your software, including all libraries in a directory bearing the name of your team. Add a README file with installation and usage instructions. Zip the directory and upload it to xxxxxx.
4) Recommended usage:
> prepare_final_resu(input_dir, output_file)
The code should comprise a main command line function called prepare_final_resu taking two arguments: the input directory and the output file name. The input directory will contain the batches to be processed organized in sub-directories (e.g. valid01, valid02, ... final01, final02). The software should list these files and automatically process them all.
It is recommended that the code possess some kind of exception handling such that if it fails on a few files, it does not crash but runs on the remaining files. The organizers should be able to run the software on new data batches that were never released to the participants.
5) Output format:
The output file should be in the challenge submission format, see:
http://www.kaggle.com/c/GestureChallenge/details/SubmissionInstructions
such that the organizers can submit the file to the challenge website and obtain the same result that was submitted by the authors of the software on valid and final data.
Okay, I'll start.
\r\nQ1) I'm using perl for development. This runs on any windows system, but would need to be installed. By compiling the perl into bytecode, I can have the "executable" not require any additional perl libraries. Is this OK?
\r\nQ2) I'm using FFMPEG and ImageMagick for the initial digest. These are open source, but would need to be installed on the target system. The perl executable calls the DLL's installed by the FFMPEG/ImageMagick package.
\r\nQ3) In order to access the information, I run a digest pass which extracts the pixels into a more easily read form. This takes a looong time - the video has to be rendered, frames have to be examined, and data has to be written. (Note: This is only to deal\r\n with the .AVI file encoding. The recognition algorithm is much faster.) Are there any time constraints for the validation step?
\r\nQ4) The digest step I am using breaks out the videos into individual images. This takes a fair amount of space - about 200GB for the combined devel/valid set. Are there any space constraints?
\r\nIn the spirit of the requirement, I'm considering taking a different path to demonstrate that my system is automatic. This will only come up if I win, so it may be a non-issue. There are already several more likely contenders on the leaderboard.
", "rawMarkdown": "", "votes": null }, { "id": "8746", "postDate": "02/29/2012 01:44:25", "content": "Hopefully you'll accept most platform (language/OS) requests (within reason) as some people have presumably invested a lot of time in this competition already and would be pretty disappointed to find out this late in the game that their platform is disallowed.\r\n To the best of my knowledge, there was no warning of this possibility up front.
\r\nIn my case, I've been developing in C++ on Linux (Ubuntu). I'm not positive if I personally will end up submitting anything, but I'm sure there are other Linux-based competitors out there.
", "rawMarkdown": "", "votes": null }, { "id": "8752", "postDate": "02/29/2012 08:25:41", "content": "I'm using Java but relying on ffmpeg to grab the frames out of the avi files. So I can't provide a simple out of the box command line that does all the magic, the frame grabbing must happen separately before my Java code runs. I hope this is ok and can be\r\n addressed in the procedure description. As Rajstennaj Barrabas said this step is slow and requires lots of storage place but as there were no restrictions on these I'm assuming this is fine.
", "rawMarkdown": "", "votes": null }, { "id": "8793", "postDate": "03/01/2012 21:52:40", "content": "We will try to accommodate as many platforms as possible, but:
\r\n- We cannot promise that we will support every platform. Your best chances to have your platform supported will be to submit an early version of your system as soon as we open the platform for software submission. You will be allowed to keep submitting new\r\n versions with all your new submissions and only your final submission will count towards your ranking. However, this will give us a change to identify ahead of time possible difficulties with running your code on our computers.
\r\n- We will try to support standard libraries like ffmpeg. It is important that people post requests on this forum because we will give priority to the most popular ones.
\r\n- If you do not submit code with your final submission or if for any reason we fail to run the code you provided with your final submission, we will require that you bring your system (hardware and software) for validation to the site of the CVPR workshop\r\n where we will run tests. See http://www.kaggle.com/c/GestureChallenge/details/Evaluation for more details. This may result in the unfair elimination of honest candidates, so we hope that it can be avoided by letting people submit code a month before the deadline.
", "rawMarkdown": "", "votes": null }, { "id": "8794", "postDate": "03/01/2012 22:04:25", "content": "[quote=Rajstennaj Barrabas;8744]
\r\nOkay, I'll start.
\r\nQ1) I'm using perl for development. This runs on any windows system, but would need to be installed. By compiling the perl into bytecode, I can have the "executable" not require any additional perl libraries. Is this OK?
\r\nQ2) I'm using FFMPEG and ImageMagick for the initial digest. These are open source, but would need to be installed on the target system. The perl executable calls the DLL's installed by the FFMPEG/ImageMagick package.
\r\nQ3) In order to access the information, I run a digest pass which extracts the pixels into a more easily read form. This takes a looong time - the video has to be rendered, frames have to be examined, and data has to be written. (Note: This is only to deal\r\n with the .AVI file encoding. The recognition algorithm is much faster.) Are there any time constraints for the validation step?
\r\nQ4) The digest step I am using breaks out the videos into individual images. This takes a fair amount of space - about 200GB for the combined devel/valid set. Are there any space constraints?
\r\nIn the spirit of the requirement, I'm considering taking a different path to demonstrate that my system is automatic. This will only come up if I win, so it may be a non-issue. There are already several more likely contenders on the leaderboard.
\r\n[/quote]
\r\nQ1: Yes, we can install and run Perl. It is a good idea to provide self contained code not requiring library installations on our side.
\r\nQ2: For libraries: we prefer self-contained code, but will make some exceptions. We will need you to submit early versions of your code in advance and detailed instructions on how to install the libraries. We don't want last minute surprises. Make sure that\r\n there are no licensing problems in the way.
\r\nQ3: Be careful about the time it takes you to run your code on the final evaluation set. You will be under time pressure to produce the results on the final evaluation set. How does it take you to process the entire validation set?
\r\nQ4: 200 GB should be OK.
\r\nDon't take the leaderboard order as a good indication of who will win. Frequently people overfit the validation set. Keep doing good work!
", "rawMarkdown": "", "votes": null }, { "id": "8795", "postDate": "03/01/2012 22:08:46", "content": "[quote=Eric Jackson;8746]
\r\nHopefully you'll accept most platform (language/OS) requests (within reason) as some people have presumably invested a lot of time in this competition already and would be pretty disappointed to find out this late in the game that their platform is disallowed.\r\n To the best of my knowledge, there was no warning of this possibility up front.
\r\nIn my case, I've been developing in C++ on Linux (Ubuntu). I'm not positive if I personally will end up submitting anything, but I'm sure there are other Linux-based competitors out there.
\r\n[/quote]
\r\nThe best way to ensure that we support your platform will be to submit an early version of your code. The "nightmare scenario" for us is to get code only with the last submission because if we fail to run it, we will not be able to let you make another code\r\n submission and validate your submission without using extra (different) validation data, which presents some chances that you would be wrongly eliminated because you did not pass the verification test.
", "rawMarkdown": "", "votes": null }, { "id": "8796", "postDate": "03/01/2012 22:09:56", "content": "[quote=Balazs Godeny;8752]
\r\nI'm using Java but relying on ffmpeg to grab the frames out of the avi files. So I can't provide a simple out of the box command line that does all the magic, the frame grabbing must happen separately before my Java code runs. I hope this is ok and can be\r\n addressed in the procedure description. As Rajstennaj Barrabas said this step is slow and requires lots of storage place but as there were no restrictions on these I'm assuming this is fine.
\r\n[/quote]
\r\nHopefully eveybody will be using the same version of ffmpeg. Please use up-to-date versions.
", "rawMarkdown": "", "votes": null }, { "id": "8815", "postDate": "03/02/2012 03:52:23", "content": "I have a perl script which finds all AVI files and renders them as a set of single-frame bitmaps. (It calls ffmpeg for the rendering).
\r\nIf everyone who needs rendered images could agree on the layout, I would be willing to submit this script to the contest organizers. They in turn could render all the files ahead of time, then use copies of this structure for validation purposes.
\r\nThe structure is a directory with the same name as the AVI file, containing numbered bitmaps.
\r\nSo for example, my devel/devel01 directory contains a directory K1, and within this directory are files K1.0001.bmp through K1.0165.bmp. Similarly for all the other directories and AVI files. (The RGB ones are named "M" as expected.)
\r\nAny interest?
", "rawMarkdown": "", "votes": null }, { "id": "8907", "postDate": "03/03/2012 00:25:41", "content": "\r\n\r\n"We will try to support standard libraries like ffmpeg. It is important that people post requests on this forum because we will give priority to the most popular ones."
\r\n
I'm using the OpenCV libraries and from the forum, it seems that some others are doing so as well.
", "rawMarkdown": "", "votes": null }, { "id": "8980", "postDate": "03/05/2012 03:57:43", "content": "We are using Matlab 2010. We were able to execute the sample codes without any errors. So I hope there will be no problem to run our system on newer versions of Matlab.
\r\nThanks
", "rawMarkdown": "", "votes": null }, { "id": "9106", "postDate": "03/06/2012 22:07:39", "content": "Hi -- We are using Fedora 16 with OpenCV + ffmpeg. Thanks!
", "rawMarkdown": "", "votes": null }, { "id": "9148", "postDate": "03/07/2012 20:52:50", "content": "You may now start uploading your models:
\r\nUpload your model as a zip archive together with your result file on the Kaggle website: at the bottom of the submission page, you will see Attach a model file and a button "Choose file".
The sooner you submit an example of your code the better. We will examine the code submissions in the order received and get beck to you if we encounter any problem.\r\n* It does not need to be your final version. * You can keep making new submissions and the code that will count towards the final evaluation will be the one submitted with your final entry.
", "rawMarkdown": "", "votes": null }, { "id": "9150", "postDate": "03/07/2012 21:20:30", "content": "I am running my code parallely using basic parfor loops. Can you also support this on your end ?
", "rawMarkdown": "", "votes": null }, { "id": "9151", "postDate": "03/07/2012 21:53:03", "content": "[quote=Manavender;9150]
\r\nI am running my code parallely using basic parfor loops. Can you also support this on your end ?
\r\n[/quote]
\r\nWe are not supporting parallelism, make sure your code can run on a single processor.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 8744, "author_name": "rajstennajbarrabas", "author_url": "", "post_date": "02/28/2012 23:03:46", "content": "Okay, I'll start.
\r\nQ1) I'm using perl for development. This runs on any windows system, but would need to be installed. By compiling the perl into bytecode, I can have the "executable" not require any additional perl libraries. Is this OK?
\r\nQ2) I'm using FFMPEG and ImageMagick for the initial digest. These are open source, but would need to be installed on the target system. The perl executable calls the DLL's installed by the FFMPEG/ImageMagick package.
\r\nQ3) In order to access the information, I run a digest pass which extracts the pixels into a more easily read form. This takes a looong time - the video has to be rendered, frames have to be examined, and data has to be written. (Note: This is only to deal\r\n with the .AVI file encoding. The recognition algorithm is much faster.) Are there any time constraints for the validation step?
\r\nQ4) The digest step I am using breaks out the videos into individual images. This takes a fair amount of space - about 200GB for the combined devel/valid set. Are there any space constraints?
\r\nIn the spirit of the requirement, I'm considering taking a different path to demonstrate that my system is automatic. This will only come up if I win, so it may be a non-issue. There are already several more likely contenders on the leaderboard.
", "votes": null, "replies": [] }, { "id": 8746, "author_name": "onemillionmonkeys", "author_url": "", "post_date": "02/29/2012 01:44:25", "content": "Hopefully you'll accept most platform (language/OS) requests (within reason) as some people have presumably invested a lot of time in this competition already and would be pretty disappointed to find out this late in the game that their platform is disallowed.\r\n To the best of my knowledge, there was no warning of this possibility up front.
\r\nIn my case, I've been developing in C++ on Linux (Ubuntu). I'm not positive if I personally will end up submitting anything, but I'm sure there are other Linux-based competitors out there.
", "votes": null, "replies": [] }, { "id": 8752, "author_name": "balazsgodeny", "author_url": "", "post_date": "02/29/2012 08:25:41", "content": "I'm using Java but relying on ffmpeg to grab the frames out of the avi files. So I can't provide a simple out of the box command line that does all the magic, the frame grabbing must happen separately before my Java code runs. I hope this is ok and can be\r\n addressed in the procedure description. As Rajstennaj Barrabas said this step is slow and requires lots of storage place but as there were no restrictions on these I'm assuming this is fine.
", "votes": null, "replies": [] }, { "id": 8793, "author_name": "iguyon", "author_url": "", "post_date": "03/01/2012 21:52:40", "content": "We will try to accommodate as many platforms as possible, but:
\r\n- We cannot promise that we will support every platform. Your best chances to have your platform supported will be to submit an early version of your system as soon as we open the platform for software submission. You will be allowed to keep submitting new\r\n versions with all your new submissions and only your final submission will count towards your ranking. However, this will give us a change to identify ahead of time possible difficulties with running your code on our computers.
\r\n- We will try to support standard libraries like ffmpeg. It is important that people post requests on this forum because we will give priority to the most popular ones.
\r\n- If you do not submit code with your final submission or if for any reason we fail to run the code you provided with your final submission, we will require that you bring your system (hardware and software) for validation to the site of the CVPR workshop\r\n where we will run tests. See http://www.kaggle.com/c/GestureChallenge/details/Evaluation for more details. This may result in the unfair elimination of honest candidates, so we hope that it can be avoided by letting people submit code a month before the deadline.
", "votes": null, "replies": [] }, { "id": 8794, "author_name": "iguyon", "author_url": "", "post_date": "03/01/2012 22:04:25", "content": "[quote=Rajstennaj Barrabas;8744]
\r\nOkay, I'll start.
\r\nQ1) I'm using perl for development. This runs on any windows system, but would need to be installed. By compiling the perl into bytecode, I can have the "executable" not require any additional perl libraries. Is this OK?
\r\nQ2) I'm using FFMPEG and ImageMagick for the initial digest. These are open source, but would need to be installed on the target system. The perl executable calls the DLL's installed by the FFMPEG/ImageMagick package.
\r\nQ3) In order to access the information, I run a digest pass which extracts the pixels into a more easily read form. This takes a looong time - the video has to be rendered, frames have to be examined, and data has to be written. (Note: This is only to deal\r\n with the .AVI file encoding. The recognition algorithm is much faster.) Are there any time constraints for the validation step?
\r\nQ4) The digest step I am using breaks out the videos into individual images. This takes a fair amount of space - about 200GB for the combined devel/valid set. Are there any space constraints?
\r\nIn the spirit of the requirement, I'm considering taking a different path to demonstrate that my system is automatic. This will only come up if I win, so it may be a non-issue. There are already several more likely contenders on the leaderboard.
\r\n[/quote]
\r\nQ1: Yes, we can install and run Perl. It is a good idea to provide self contained code not requiring library installations on our side.
\r\nQ2: For libraries: we prefer self-contained code, but will make some exceptions. We will need you to submit early versions of your code in advance and detailed instructions on how to install the libraries. We don't want last minute surprises. Make sure that\r\n there are no licensing problems in the way.
\r\nQ3: Be careful about the time it takes you to run your code on the final evaluation set. You will be under time pressure to produce the results on the final evaluation set. How does it take you to process the entire validation set?
\r\nQ4: 200 GB should be OK.
\r\nDon't take the leaderboard order as a good indication of who will win. Frequently people overfit the validation set. Keep doing good work!
", "votes": null, "replies": [] }, { "id": 8795, "author_name": "iguyon", "author_url": "", "post_date": "03/01/2012 22:08:46", "content": "[quote=Eric Jackson;8746]
\r\nHopefully you'll accept most platform (language/OS) requests (within reason) as some people have presumably invested a lot of time in this competition already and would be pretty disappointed to find out this late in the game that their platform is disallowed.\r\n To the best of my knowledge, there was no warning of this possibility up front.
\r\nIn my case, I've been developing in C++ on Linux (Ubuntu). I'm not positive if I personally will end up submitting anything, but I'm sure there are other Linux-based competitors out there.
\r\n[/quote]
\r\nThe best way to ensure that we support your platform will be to submit an early version of your code. The "nightmare scenario" for us is to get code only with the last submission because if we fail to run it, we will not be able to let you make another code\r\n submission and validate your submission without using extra (different) validation data, which presents some chances that you would be wrongly eliminated because you did not pass the verification test.
", "votes": null, "replies": [] }, { "id": 8796, "author_name": "iguyon", "author_url": "", "post_date": "03/01/2012 22:09:56", "content": "[quote=Balazs Godeny;8752]
\r\nI'm using Java but relying on ffmpeg to grab the frames out of the avi files. So I can't provide a simple out of the box command line that does all the magic, the frame grabbing must happen separately before my Java code runs. I hope this is ok and can be\r\n addressed in the procedure description. As Rajstennaj Barrabas said this step is slow and requires lots of storage place but as there were no restrictions on these I'm assuming this is fine.
\r\n[/quote]
\r\nHopefully eveybody will be using the same version of ffmpeg. Please use up-to-date versions.
", "votes": null, "replies": [] }, { "id": 8815, "author_name": "rajstennajbarrabas", "author_url": "", "post_date": "03/02/2012 03:52:23", "content": "I have a perl script which finds all AVI files and renders them as a set of single-frame bitmaps. (It calls ffmpeg for the rendering).
\r\nIf everyone who needs rendered images could agree on the layout, I would be willing to submit this script to the contest organizers. They in turn could render all the files ahead of time, then use copies of this structure for validation purposes.
\r\nThe structure is a directory with the same name as the AVI file, containing numbered bitmaps.
\r\nSo for example, my devel/devel01 directory contains a directory K1, and within this directory are files K1.0001.bmp through K1.0165.bmp. Similarly for all the other directories and AVI files. (The RGB ones are named "M" as expected.)
\r\nAny interest?
", "votes": null, "replies": [] }, { "id": 8907, "author_name": "onemillionmonkeys", "author_url": "", "post_date": "03/03/2012 00:25:41", "content": "\r\n\r\n"We will try to support standard libraries like ffmpeg. It is important that people post requests on this forum because we will give priority to the most popular ones."
\r\n
I'm using the OpenCV libraries and from the forum, it seems that some others are doing so as well.
", "votes": null, "replies": [] }, { "id": 8980, "author_name": "iftekharanam", "author_url": "", "post_date": "03/05/2012 03:57:43", "content": "We are using Matlab 2010. We were able to execute the sample codes without any errors. So I hope there will be no problem to run our system on newer versions of Matlab.
\r\nThanks
", "votes": null, "replies": [] }, { "id": 9106, "author_name": "danlief", "author_url": "", "post_date": "03/06/2012 22:07:39", "content": "Hi -- We are using Fedora 16 with OpenCV + ffmpeg. Thanks!
", "votes": null, "replies": [] }, { "id": 9148, "author_name": "iguyon", "author_url": "", "post_date": "03/07/2012 20:52:50", "content": "You may now start uploading your models:
\r\nUpload your model as a zip archive together with your result file on the Kaggle website: at the bottom of the submission page, you will see Attach a model file and a button "Choose file".
The sooner you submit an example of your code the better. We will examine the code submissions in the order received and get beck to you if we encounter any problem.\r\n* It does not need to be your final version. * You can keep making new submissions and the code that will count towards the final evaluation will be the one submitted with your final entry.
", "votes": null, "replies": [] }, { "id": 9150, "author_name": "mrkaggler", "author_url": "", "post_date": "03/07/2012 21:20:30", "content": "I am running my code parallely using basic parfor loops. Can you also support this on your end ?
", "votes": null, "replies": [] }, { "id": 9151, "author_name": "iguyon", "author_url": "", "post_date": "03/07/2012 21:53:03", "content": "[quote=Manavender;9150]
\r\nI am running my code parallely using basic parfor loops. Can you also support this on your end ?
\r\n[/quote]
\r\nWe are not supporting parallelism, make sure your code can run on a single processor.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "8743": "", "8744": "", "8746": "", "8752": "", "8793": "", "8794": "", "8795": "", "8796": "", "8815": "", "8907": "", "8980": "", "9106": "", "9148": "", "9150": "", "9151": "" }, "source": "meta" }