Multimedia Labor

During my PhD (2002-2006) my work was closely related with the MIP lab, which is well equipped for the animation analysis of humans. My goal was to make use of the whole hardware and to integrate different subsystems, in a way that robust and efficient analysis of a person moving in front of the screen became possible. The integration included three cameras for observing the scene, 2 microphones for audio detection and issuing commands, 2 projectors for 3D-stereo visualization and 9 PCs for processing the data and rendering.
The movie (MPEG-1 9MB) on the right shows our person tracking and multidisplay rendering  system, where 4 linux PCs are evaluating the person's head position and 5 PCs are rendering the scene. The user's head position is computed and the scene view is adapted according to the head's height. By walking around on the floor, the person can navigate through the scene, standing in front means forward, left means rotate left and so on. For good visibility of the displays the scene has to be rather dim lighted, which makes the image processing very difficult. A Baysian approach using particle systems makes it possible in real-time though (see publications CAIP2005 ).

The system is also used for public demonstrations. In this movie (25MB MPEG-1) Prof. Koch first introduces the system. Later in the video, guests from the "Girl's Day" are enjoying the interaction themselves. The movie shows also very well the difficulties we are facing in this environemnt.


An  overview of the interaction space (click for 9MB MPEG-1 movie)

Real-Time Face Tracking

Two  pan-tilt cameras track the user's face during the interaction.  The tracking is especcially difficult, becuse the lighting is not only rather dim but changes dramatically during the rendering, as the main light in the scene is emitted from the screens. Therefore the color of light depends on the current visualized scene part.
We achieve robust tracking with a CONDENSATION approach using face color and a face detection/recognitin algorithm. The face color is stored as a dynamic histogram. While the face detection is a adapted version of a trained classifier cascade as available in  openCV.  Our adaption enables a faster use within the CONDENSATION. Results of the tracking can be seen in the right.
The image shows the sample distribution of the CONDENSATION particle filter. The state/position of particles is here the image position and the scale of the face.  Brighter rectangles represent higher probabilties. Visible in the movie is also the panning and tilting of the camera.  Please note that the camera does not move constantly, but as seldomly as possible, such that the  calculated projection matrix of the cam stays valid as long as possible.

Samples during Face Tracking
Sample distribution of the CONDENSATION particle filter
(click for  5MB MPEG-1 movie)

Gesture Tracking and Recognition

To enable real interaction with the virtual scene  pointing gestures are useful. Their detection and recogntion is realized in our system by tracking of one hand  again with a CONDENSATION approach. Both pan/tilt cameras are used additionally for tracking of the hand. Used features here are again skin color (taken the same as the face) and movement. The movement is calulated as the temporal image gradient over a few frames. The movement is a good feature for hand detection, as people tend to move their hands always, if they move at all (with respect to the global coordinate system).
In the movie the samples with a certain probability ae shown. The red cross marks the estimated hand position, which is calculated as the weighted mean of all sample positions. The state is here only 2D (the image position of the hand. The size of the red cross shows the uncertainty of the estimated position. This output of the particle filter is very useful for sensor fusion and makes the system more stable and robust.
The hand tracking integrated in the whole system can be seen in this movie(6MB).
The pointing ray of the person is calculated as the line extension between head and hand. The intersection point with the virtual scene can be calculated due to the fully calibrated system. However the position estimate is rather noisy, as visible  by observing the yellow ball. This is due to the small amount of cameras used for triangulation (only 2) and due to the blob like features taken (color and movement).

hand tracking
Hand and face tracking together (3MB movie)