Real-time Motion Capture

This reserach project was part of the Computing Research Association Distributed Mentor Project. The CRAW-DMP matches undergraduate female students with female mentors for a summer of research at the mentor's institution. The research was conducted at Carnegie Mellon University summer of 2002.

The Project
My project focused on capturing or recording real time human data and applying it to computer-generated characters. The characters are modeled in Maya and animated using incoming data from the motion capture system. One prospect of using real time capture is to enable the capture subjects to see their motions in real time. Up until this point, the person being captured and the researcher are not able to view the motion until after it has been captured. With real time, as the subject is moving, the generated motion is simultaneously displayed on screen and any improvements or changes to the motion can be made at that point rather than waiting to view the motion after the capture and any changes will require the motion to be recaptured.. Being able to view the motion in real time is a prospect for future projects and applications here at CMU.

What is Motion Capture?
Motion capture is a popular tool used for creating realistic motions. It is the process of capturing the movement of a real object in 3D space and mapping it onto a computer generated object. Motion capture has been used extensively in the entertainment business, a good example is the movie “Final Fantasy” in which most of the motions were created using motion capture. In general, motion capture is the process of capturing the motion of a live subject and applying it to a virtual (i.e. computer generated) subject. Motion capture eliminates many of the problems and difficulties found in traditional keyframing animation, however motion capture does have its limitations. For example, it is very difficult to modify a motion once its been captured and so if the motion wasn’t as good as the animator wanted it to be, then that motion will most likely be discarded and another motion will be captured. Overall, motion capture can be a reliable tool to create realistic looking motions fast and easy.

How does Motion Capture work?
For “optical” motion capture, a series of high resolution cameras with special strobe lights are set up around an area to be “captured.” Small spheres (markers) covered with a retro-reflective substance are placed in strategic locations on the person (or object). Specialized software will locate the markers seen through the cameras. The markers are then recorded as 3D coordinates (xyz). The collected marker data over time creates motion data.

What is real time Motion Capture?
There are two types of motion capture: offline and online (real time). In the case of the offline, the data is captured, stored, processed (cleaning data from noise is common) and then later applied to the computer character. However, in real time capture the motion data is captured and applied to the character simultaneously. Depending on the capture system, there should be no noticeable delay in the character’s movement compared to the movement of the real subject.

A common problem with real time motion capture is the presence of noise. If you plot the motion data you will notice that the curves aren’t always smooth, there are some unsettledness or jerkiness. There are many filtering systems and techniques to eliminate the “bad data”. During offline capture, one can go back to the data and “clean it up” before applying it to the computer character. But how about real time capture?

1. Marker placement: Using the marker placement guide, markers are placed on the subject(s). For humans there are either 40 or 41 markers total. Placing markers requires some practice, if some of the markers are not in their “expected” positions then that will result in generating bad data later on.

2. Calibrate the cameras: there are a total of 12 cameras in the lab. The calibration is necessary to ensure that all the cameras are able to view the area used for capture. Basically a number of markers are placed on the floor for “static calibration” and then a stick with markers is waved around the area of capture, this is for “dynamic calibration”.
   

Calibration Stick

Calibration Axis

3. Subject calibration: the subject of capture is calibrated. For human capture, the person would stand in a “T” position with the arms straight out for “static calibration” and then the person would do a simple motion such as walking across the room for “dynamic calibration”

4. Labeling: using the software “Workstation”, you can view the captured motion as a collection of markers moving over a period of time. Next step is to label the markers. It’s simply a matching game, you are given a list of default marker names and you are to match those names to the corresponding markers on the body. Workstation will then connect the markers to create segments.
   

(photo taken from www.vicon.com)

5. Ready to Capture: All the preprocessing is complete and you are ready to capture. Simply click on the start button and Vicon starts capturing the data.
   

Stage 2: Writing a plugin for Maya

The captured data from Vicon will be applied to a computer character generated in Maya. Maya is a 3D modeling and animation software. In order to apply the real time data to the character, there must be some type of a link or communication method between Maya and Vicon. This part of the project involves writing a plugin ( an additional program that enhances the features of an existing program) that will allow Maya to receive data from Vicon.

Writing the plugin requires a good understanding of networking concepts, an in-depth knowledge of how data is handled by Tarsus (a Vicon software used for motion capture) and how Maya maps incoming data to objects. This project seemed to be heavily involved since I didn’t know much about networking, Tarsus or Maya. Luckily, I was able to contact the people at Vicon and they told me that they had already written a plugin for Maya and that they’ll e-mail a copy to me.

Stage 3: Modifying the plugin

This is the general idea of how data is communicated between Tarsus and Maya:

1. Tarsus outputs data using named channels. Channels are strings of double values varying over time. Each channel corresponds to a specific source that could be either a marker or a body segment. In turn, the plugin creates channels (vectors) to store the incoming channel data. coming from the Tarsus channels.
2. The data stored in the vector channels is parsed, modified and sent out to Maya using routines in the Maya server library.

Tarsus data consists of the position(x,y,z) and orientation (qx, qy, qz ) of the body parts for the real time subject and the position (only) of the corresponding markers. However, Tarsus doesn’t provide joint angle data, which is what we need. Therefore, the code must be modified to extract joint angle information from the available position/orientation data.

• Calculating joint-angle information:

  Given the position and orientation of two body parts we can calculate the angle for the joint joining the two bodies using quaternions. A quaternion is a 4-tuple of real numbers [s,x,y,z] or [s,v], s is a scalar and v is a three-dimensional vector. First, represent the position/orientation data of both body parts as quaternions, find the difference of the two quaternions and the result is the angle for the joint connecting the two body parts.

Stage 4: Modifying the Vicon files

Part of the Vicon system are files called .MKR and .KM files, they contain lists of: 1) marker labels, 2) body segments and their corresponding markers. By default, the marker file (.MKR) for a human contains 40-41 markers and the kinematic model file (.KM) contains 17 body segments.

A typical human skeleton file contains around 30 body parts, which adds more degrees of freedom to the body and results in a better looking motion. To match with the current skeleton files being used, the marker and kinematic model files need to be modified to include more body segments. Both files can be viewed as text files and to change the number of segments requires typing in few additional text lines.