Measuring Fingertip Force by Imaging the Fingernail

The goal of this project is to measure fingertip forces by using a multi-camera system to image coloration changes in the fingernail and surrounding skin. Key advantages of this transduction method are that objects do not have to be instrumented with force sensors, and relatively unconstrained grasps can be measured.

Grasping studies to date have required the design of special test objects using key-like touch pads instrumented with force sensors, but the new technique now allows the characterization of grasp forces for ordinary objects of arbitrary shape. It also allows the measurement of fingertip forces when regrasping and manipulating objects, where the contact points change in an unpredetermined manner.

2D to 3D Image Registration [Haptics06] [CVPR07]

Fingernail locations will vary depending on the grasp and on the relative locations of the cameras. As a particular fingernail is imaged, it will be necessary to correspond points in the image to a reference image so that calibration results can be applied.

Fingernails are curved, so the analysis of the coloration changes has to be carried out in 3D. Currently there are many registration methods relying on naturally present features, such as edges, corners or texture. However, the fingernail is relative featureless and some features, related to color, change with force. Pairing of those features with neighborhood correlation gives a great number of outliers (wrong pairs). We currently use RANSAC to find the inlier pairs and use those pairs to calculate the tranform matrix. The 3D model of the fingernail is obtained with a Bumblebee Stereo camera from Point Grey Research.

Bunblebee (Point Grey)	Flea (Point Grey)	Lens (Tomron)

Elastic Warping [WHC07][CVPR07]

To compare fingernails from different subjects, finger images are registered to a finger atlas with simplified reference shapes. The fingernail is modeled as a disk with 70 pixel radius. The surrounding skin region is composed of a ring and an isosceles trapezoid. Ideally, the fingernails are mapped to the fingernail atlas, and the surrounding skin is mapped to the surrounding skin atlas. To do so, the fingernails in the reference images need to be segmented from the surrounding skin.

We use a Canny edge filter to automatically detect the boundary of the fingernail. However, because of the broken skin around the fingernail, the automatically detected boundary is noisy and can rarely form a smooth curve. We use cubic B-splines to fit the edges and achieve a close-loop contour. The region inside of the contour is the segmented nail.

A finger	Canny edge	Bspline	Boundary	Segmented nail

The nail and the surrounding skin are transformed to the atlas image with boundary-based elastic deformation transformation. We model both the fingernail and surrounding skin regions as elastic sheets that are warped by an external force field applied to the boundaries. Since elastic warping tends to preserve color pattern shapes and the relative position of the patterns, it suits color pattern comparison across subjects.

Elastic nail warping	Elastic skin warping	Combination

Weighted Least Squares Model [Haptics06]

Our research has identified that certain areas of the fingernail show a strong linear response of coloration to fingertip force, others do not. Not just the fingernail areas show this effect, certain areas of the surrounding skin particularly at the base of the nail show a strong linear response as well. The location of the good areas depends on the contact conditions.

Generally speaking, regions within the fingernail saturate at lower force levels than regions in the surrounding skin. The front of the fingernail saturates at higher force levels than the middle of the fingernail. The skin at the sides of the fingernail saturates at the highest force levels. The relatively low saturation level of the middle of the fingernail emphasizes the need to image the whole fingernail and surrounding skin.

Using the good mesh elements, a generalized least squares estimator is applied to predict fingertip force.

Automated Calibration

To improve our calibration procedure, we are developing a robot that will apply forces in three directions independently. The z-direction force is supplied by a single motor with a linear stage, on which a platform is mounted. A two-DOF pantograph is attached to the platform, on which is mounted a six-axis force sensor. The fingertip rests against the force sensor, while the first and second links of the finger are held in place by a small finger brace. A series of force controllers determine the input signals to each of the three motors. With this design, the test subject can hold the finger still while the machine provides the desired force, allowing for more methodical collection of calibration data.

Participants

Collaborators

This project is supported by NIH Grant 1R21EB004600-01A2.

Publications

Patents