{ "id": 1591, "title": "Observations of the data", "url": "/competitions/GestureChallenge/discussion/1591", "author_name": "", "post_date": "2012-03-19T19:40:54.733Z", "votes": 2, "comment_count": 1, "views": 1439, "content": "

I have to withdraw from the competion, so I'm posting some observations and conclusions so that others can make use of them.

\r\n

 

\r\n

Observation 1: The distance measurement is per channel, but offset.

\r\n

 

\r\n

I took separate R, G, B histograms in the distance frames and compared the pixel counts. The data seem to indicate that the R, G, and B channels all measure the same distribution, but offset from each other. For example, the count of pixels which have a\r\n red = 10 is the same as the count of green = 12 and the same as blue = 8.

\r\n

 

\r\n

I concluded from this that the three channels are measuring the same distribution but offset from each other, and that calculating the distance at a pixel by averaging the channels (R+B+G)/3 would effectively fuzz the data.

\r\n

 

\r\n

To calculate the distance of a particular pixel, simply use the green channel value. This appears to have the best spread of values within the distribution.

\r\n

 

\r\n

Observation 2: The distance values are not continuous

\r\n

 

\r\n

The histograms also show gaps in the data counts - channel values which have a count of zero at periodic intervals.

\r\n

 

\r\n

For example, a Red channel histogram is shown below. The values at positions 5, 12, 19, 26 and so on are zero. This indicates, for example, that there are no pixels in the distance image which have a Red component of 26. This is true for all distance frames\r\n in all videos, and since the green histogram is at an offset from the red data, there are corresponding zeroes in the green channel, and also in the blue channel.

\r\n

 

\r\n

Calculating the *change* in value from one frame to the next without compensating for these zeroes will skew the results.

\r\n

 

\r\n

For example, consider the MSE of the same pixel taken from two succeeding frames. One frame might show a distance of 11 at that location, and the next might show a 13. The MSE will appear to be (13-11)^2 = 4 when in reality the distances differ by 1 so the\r\n MSE should be (12-11)^2 = 1. The extra distance is an anomaly, caused by the hole in the digitization technique. Since no pixel can have a distance of 13, the new pixel has be the next higher value.

\r\n

 

\r\n

When distances are remapped to avoid the zeroes, 212 separate distance values remain in a more-or-less continuous distribution.

\r\n

 

\r\n

Observation 3: Black (R=G=B=0) means "no information"

\r\n

 

\r\n

In the distance images, the value of "black" does not mean "really close", but instead means "no information". The Kinect apparently uses this value to report that the infrared signal has disappeared - hence, "no information".

\r\n

 

\r\n

For an example of this, look at devel05/K_26.avi and notice how the patterns change frame to frame. Any detection algorithm needs to account for this special value; for example, by relying on pixel area averages instead of absolute pixel counts.

\r\n

 

\r\n

Observation 4: The "no information" value (Black) is low-pass filtered

\r\n

 

\r\n

A histogram of the distance images plotted does not show a sharp vertical line at zero, but a smoothly sloping curve starting at around value 5 and rising up to meet the value at zero.

\r\n

 

\r\n

This indicates that the black value has been put through a low-pass filter, with the result that some of the black values appear as small-numbered values close to black. Practically speaking, this means that values close to black (around 5 and less) should\r\n also be considered the "no information" value. How close depends on the specific video - I think this has to do with the specifics of the AVI encoding.

\r\n

 

\r\n

Histogram of red channel in one of the distance frames:

\r\n

 

\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
1270
27
17
15
15
0
22
14
13
15
13
9
0
2
5
4
2
4
3
0
2
2
2
3
4
4
0
4
2
7
2
7
5
0
5
3
4
6
3
", "messages": [ { "id": "9544", "postDate": "03/19/2012 19:40:54", "content": "

I have to withdraw from the competion, so I'm posting some observations and conclusions so that others can make use of them.

\r\n

 

\r\n

Observation 1: The distance measurement is per channel, but offset.

\r\n

 

\r\n

I took separate R, G, B histograms in the distance frames and compared the pixel counts. The data seem to indicate that the R, G, and B channels all measure the same distribution, but offset from each other. For example, the count of pixels which have a\r\n red = 10 is the same as the count of green = 12 and the same as blue = 8.

\r\n

 

\r\n

I concluded from this that the three channels are measuring the same distribution but offset from each other, and that calculating the distance at a pixel by averaging the channels (R+B+G)/3 would effectively fuzz the data.

\r\n

 

\r\n

To calculate the distance of a particular pixel, simply use the green channel value. This appears to have the best spread of values within the distribution.

\r\n

 

\r\n

Observation 2: The distance values are not continuous

\r\n

 

\r\n

The histograms also show gaps in the data counts - channel values which have a count of zero at periodic intervals.

\r\n

 

\r\n

For example, a Red channel histogram is shown below. The values at positions 5, 12, 19, 26 and so on are zero. This indicates, for example, that there are no pixels in the distance image which have a Red component of 26. This is true for all distance frames\r\n in all videos, and since the green histogram is at an offset from the red data, there are corresponding zeroes in the green channel, and also in the blue channel.

\r\n

 

\r\n

Calculating the *change* in value from one frame to the next without compensating for these zeroes will skew the results.

\r\n

 

\r\n

For example, consider the MSE of the same pixel taken from two succeeding frames. One frame might show a distance of 11 at that location, and the next might show a 13. The MSE will appear to be (13-11)^2 = 4 when in reality the distances differ by 1 so the\r\n MSE should be (12-11)^2 = 1. The extra distance is an anomaly, caused by the hole in the digitization technique. Since no pixel can have a distance of 13, the new pixel has be the next higher value.

\r\n

 

\r\n

When distances are remapped to avoid the zeroes, 212 separate distance values remain in a more-or-less continuous distribution.

\r\n

 

\r\n

Observation 3: Black (R=G=B=0) means "no information"

\r\n

 

\r\n

In the distance images, the value of "black" does not mean "really close", but instead means "no information". The Kinect apparently uses this value to report that the infrared signal has disappeared - hence, "no information".

\r\n

 

\r\n

For an example of this, look at devel05/K_26.avi and notice how the patterns change frame to frame. Any detection algorithm needs to account for this special value; for example, by relying on pixel area averages instead of absolute pixel counts.

\r\n

 

\r\n

Observation 4: The "no information" value (Black) is low-pass filtered

\r\n

 

\r\n

A histogram of the distance images plotted does not show a sharp vertical line at zero, but a smoothly sloping curve starting at around value 5 and rising up to meet the value at zero.

\r\n

 

\r\n

This indicates that the black value has been put through a low-pass filter, with the result that some of the black values appear as small-numbered values close to black. Practically speaking, this means that values close to black (around 5 and less) should\r\n also be considered the "no information" value. How close depends on the specific video - I think this has to do with the specifics of the AVI encoding.

\r\n

 

\r\n

Histogram of red channel in one of the distance frames:

\r\n

 

\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
1270
27
17
15
15
0
22
14
13
15
13
9
0
2
5
4
2
4
3
0
2
2
2
3
4
4
0
4
2
7
2
7
5
0
5
3
4
6
3
", "rawMarkdown": "", "votes": null }, { "id": "9606", "postDate": "03/20/2012 17:18:32", "content": "

Observation: FFMPEG can turn the videos into images

\r\n

FFMPEG is free software and can be used to extract individual frames from the videos. The frames are usually easier to deal with than the raw video, and accessing the frames is generally much faster than rendering the video inline.

\r\n

The FFMPEG line to use is: "ffmpeg -i $VideoPath $ImageDir/Something.%04d.bmp"

\r\n

(Replace $VideoPath with the path to the video of interest, and $ImageDir with the destination directory on your system.)

\r\n

The -i argument identifies the input video file. The last part specifies the output image format (in this example, a BMP file) and the "printf" style feature at the end is recognized by FFMPEG as a file expression.

\r\n

This will generate image files "Something.0001.bmp Something.0002.bmp Something.0003.bmp" and so on in the specified $ImageDir. For example, the line: "ffmpeg -i K1.avi K1/K1.%04d.bmp" will grab K1.avi in the current directory, render it\r\n as frames, and store the frames as bitmap files K1.0001.bmp and so on. The frames will be stored in a subdirectory names "K1" in the current directory.

\r\n

Observation: Compressed data results in duplicated frames

\r\n

Using FFMPEG to render the frames, I noticed that frames are duplicated. For example, rendering devel01/K_1.avi results in 0001.bmp through 0004.bmp as the same image - identical in content. The first non-identical image is 0005.bmp. In similar manner, 0006.bmp\r\n is identical to 0005, and so on until 0009.bmp.

\r\n

I believe that this is an artifact of the compression used by the .AVI conversion. When compressing a video the system chooses a bit rate, for example 300K bits per second. If the video has a lot of change, then the compression will have to drop frames in\r\n order to achieve the specified compression.

\r\n

Since the amount of change is specific to the video, this means that the frame number of unique images will be different depending on which video is accessed. Devel01/K1 renders 1 unique image in 5 frames, while devel01/K10 renders one in two. There's\r\n apparently much less change frame by frame in video K_10, which allows the compression to render more unique frames.

\r\n

So if you render the video to individual frames, be on the lookout for identical frames. Or to put it more succinctly, the compression algorithm quantizes the actions in a temporal direction.

\r\n

Observation: The Depth image is offset from the color image, and a different size

\r\n

I've posted an image which overlays the Kinect and color images below. As can be seen from the image, the Kinect data and color data don't match - the Kinect image is a little to the left and above the color image.

\r\n

A system which attempts to correlate segments from one video stream to another needs to take this into account.

\r\n

Hmmm... Can't seem to get the image to load. I'll try to add it as an attachment

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 9606, "author_name": "rajstennajbarrabas", "author_url": "", "post_date": "03/20/2012 17:18:32", "content": "

Observation: FFMPEG can turn the videos into images

\r\n

FFMPEG is free software and can be used to extract individual frames from the videos. The frames are usually easier to deal with than the raw video, and accessing the frames is generally much faster than rendering the video inline.

\r\n

The FFMPEG line to use is: "ffmpeg -i $VideoPath $ImageDir/Something.%04d.bmp"

\r\n

(Replace $VideoPath with the path to the video of interest, and $ImageDir with the destination directory on your system.)

\r\n

The -i argument identifies the input video file. The last part specifies the output image format (in this example, a BMP file) and the "printf" style feature at the end is recognized by FFMPEG as a file expression.

\r\n

This will generate image files "Something.0001.bmp Something.0002.bmp Something.0003.bmp" and so on in the specified $ImageDir. For example, the line: "ffmpeg -i K1.avi K1/K1.%04d.bmp" will grab K1.avi in the current directory, render it\r\n as frames, and store the frames as bitmap files K1.0001.bmp and so on. The frames will be stored in a subdirectory names "K1" in the current directory.

\r\n

Observation: Compressed data results in duplicated frames

\r\n

Using FFMPEG to render the frames, I noticed that frames are duplicated. For example, rendering devel01/K_1.avi results in 0001.bmp through 0004.bmp as the same image - identical in content. The first non-identical image is 0005.bmp. In similar manner, 0006.bmp\r\n is identical to 0005, and so on until 0009.bmp.

\r\n

I believe that this is an artifact of the compression used by the .AVI conversion. When compressing a video the system chooses a bit rate, for example 300K bits per second. If the video has a lot of change, then the compression will have to drop frames in\r\n order to achieve the specified compression.

\r\n

Since the amount of change is specific to the video, this means that the frame number of unique images will be different depending on which video is accessed. Devel01/K1 renders 1 unique image in 5 frames, while devel01/K10 renders one in two. There's\r\n apparently much less change frame by frame in video K_10, which allows the compression to render more unique frames.

\r\n

So if you render the video to individual frames, be on the lookout for identical frames. Or to put it more succinctly, the compression algorithm quantizes the actions in a temporal direction.

\r\n

Observation: The Depth image is offset from the color image, and a different size

\r\n

I've posted an image which overlays the Kinect and color images below. As can be seen from the image, the Kinect data and color data don't match - the Kinect image is a little to the left and above the color image.

\r\n

A system which attempts to correlate segments from one video stream to another needs to take this into account.

\r\n

Hmmm... Can't seem to get the image to load. I'll try to add it as an attachment

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "9544": "", "9606": "" }, "source": "meta" }