The three reasons for difficulty in transfer function specification, which were outlined in the previous section, can be considered in the context of a conceptual gap between the spatial and transfer function domains. Having intuition and knowledge of both domains is important for creating good transfer functions, but these two domains have very different properties and characteristics. The spatial domain is the familiar 3D space for the geometry and volume data being rendered, but the transfer function domain is more abstract. Its dimensions (data value and two types of derivative) are not spatial, and the quantity at each location (opacity and three color channels) is not scalar. Thus, a principle of the direct manipulation widgets presented here is to link interaction in one domain with feedback in another, so as to build intuition for the connection between them. Also, the conceptual gap between the domains can be reduced by facilitating interaction in both domains simultaneously. We will provide a brief description of how a user might interact with this system and then describe the individual direct manipulation widgets in detail.
In a typical session with our system, a user might begin by moving and rotating a clipping plane through the volume, to inspect slices of data. The user can click on the clipping plane near a region of interest (for example, the boundary between two materials). The resulting visual feedback in the transfer function domain indicates the data value and derivatives at that point and its local neighborhood. By moving the mouse around, volume query locations are constrained to the slice, and the user is able to visualize, in the transfer function domain, how the values change around the feature of interest. Conversely, the system can track, with a small region of opacity in the transfer function domain, the data values at the user-selected locations, while continually updating the volume rendering. This visualizes, in the spatial domain, all other voxels with similar transfer function values. If the user decides that an important feature is captured by the current transfer function, he or she can effectively ``paint'' that region into the transfer function and continue querying and investigating the volume until all regions of interest have been made visible.
Another possible interaction scenario begins with a pre-determined transfer function that is likely to bring out some features of interest. This can originate with an automated transfer function generation tool , or it could be the ``default'' transfer function described in Section 7. The user would then begin investigating and exploring the dataset as described above. The widgets are used to adapt the transfer function to emphasize regions of interest or eliminate extraneous information.
The process outlined above for creating higher dimensional transfer functions is made possible by the collection of direct manipulation widgets described in the remainder of this section. Embedding the transfer function widget in the main rendering window (say, below the volume rendering, as has been done for all the figures in this paper) is a simple way to make interaction in either or both domains more convenient. More importantly, our system relies on event-based inter-widget communication. This allows information generated by a widget in the spatial domain to determine the state of a widget in the transfer function domain, and vice versa. All widgets are object oriented and derived from a master widget class which specifies a standard set of callbacks for sending and receiving events, allowing the widgets to query and direct each other. Also, widgets can contain embedded widgets, which send and receive events through the parent widget.
In a traditional volume rendering system, the process of setting the transfer function involves moving the control points (in a sequence of linear ramps defining color and opacity), and then observing the resulting rendered image. That is, interaction in the transfer function domain is guided by careful observation of changes in the spatial domain. We prefer a reversal of this process, in which the transfer function is set by direct interaction in the spatial domain, with observation of the transfer function domain. Furthermore, by allowing interaction to happen in both domains simultaneously, the conceptual gap between them is significantly lessened. We use the term ``dual-domain interaction'' to describe this approach to transfer function exploration and generation.
Figure 3 illustrates the specific steps of dual-domain interaction. When a position inside the volume is queried by the user with the data probe widget (Figure 3a), the values associated with that position (data value, first and second derivative) are graphically represented in the transfer function widget (3b). Then, a small region of high opacity (3c) is temporarily added to the transfer function at the data value and derivatives determined by the probe location. The user has now set a multi-dimensional transfer function simply by positioning a data probe within the volume. The resulting rendering (3d) depicts (in the spatial domain) all the other locations in the volume which share values (in the transfer function domain) with those at the data probe tip. If the features rendered are of interest, the user can copy the temporary transfer function to the permanent one (3e), by, for instance, tapping the keyboard space bar with the free hand. As features of interest are discovered, they can be added to the transfer function quickly and easily with this type of two-handed interaction. Alternately, the probe feedback can be used to manually set other types of classification widgets (3f), which are described later. The outcome of dual-domain interaction is an effective multi-dimensional transfer function built up over the course of data exploration. The widget components which participated in this process can be seen in Figure 4 (on colorplate), which shows how dual-domain interaction can help volume render the CT tooth dataset. The remainder of this section describes the individual widgets and provides additional details about dual-domain interaction.
The data probe widget is responsible for reporting its tip's position in volume space and its slider sub-widget's value. Its pencil-like shape is designed to give the user the ability to point at a feature in the volume being rendered. The other end of the widget orients the widget about its tip. When the volume widget's position or orientation is modified the data probe widget's tip tracks its point in volume space. The data probe widget can be seen in Figures 4 and 5 (on colorplate). A natural extension is to link the data probe widget to a haptic device, such as the SensAble PHANTOM, which can provide a direct 3D location and orientation .
The clipping plane widget is a basic frame type widget. It is responsible for reporting its orientation and position to the volume widget, which handles the actual clipping when it draws the volume. In addition to clipping, the volume widget will also map a slice of the data to the arbitrary plane defined by the clip widget, and blend it with the volume by a constant opacity value determined by the clip widget's slider. It is also responsible for reporting the spatial position of a mouse click on its clipping surface. This provides an additional means of querying positions within the volume, distinct from the 3D data probe. The balls at the corners of the clipping plane widget are used to modify its orientation, and the bars on the edges are used to modify its position. The clipping plane widget can also be seen in Figures 5 and 8 (on colorplate).
The main role of the transfer function widget is to present a graphical representation of the transfer function domain, in which feedback from querying the volume (with the data probe or clipping plane) is displayed, and in which the transfer function itself can be set and altered. The transfer function widget is shown at the bottom of all of our rendered figures. The backbone of the transfer function widget is a basic frame widget. Data value is represented by position along the horizontal axis, and gradient magnitude is represented in the vertical direction. The third transfer function axis, second derivative, is not explicitly represented in the widget, but quantities and parameters associated with this axis are represented and controlled by other sub-widgets. The balls at the corners of the transfer function widget are used to resize it, as with a desktop window, and the bars on the edges are used to translate its position. The inner plane of the frame is a polygon texture-mapped with a slice through the 3D lookup table containing the full 3D transfer function. The user is presented with a single slice of the 3D transfer function for a few reasons. Making a picture of the entire 3D transfer function would be a visualization in itself, and its image would visually compete with the main volume rendering. Also, since the goal in our work has primarily been the visualization of surfaces, the role of the second derivative axis is much simpler than the other two, so it needs fewer control points.
The data value and derivatives at the position queried in the volume (either via the data probe or clipping plane widgets) is represented with a small ball in the transfer function widget. In addition to the precise location queried, the eight data sample points at the corners of the voxel containing the query location are also represented by balls in the transfer function domain, and are connected together with edges that reflect the connectivity of the voxel corners in the spatial domain. By ``re-projecting'' a voxel from the spatial domain to a simple graphical representation in the transfer function domain, the user can learn how the transfer function variables (data value and derivatives) are changing near the probe location. The second derivative values are indicated by colormapping the balls: negative, zero, and positive second derivatives are represented by blue, white, and yellow balls, respectively. When the projected points form an arc, with the color varying through the colormap, the probe is at a boundary in the volume. These can be seen in Figures 4a, 4c, 5a, and 5c (on colorplate). When the re-projected data points are clustered together, the probe is in a homogeneous region, as seen in Figures 4b and 5b. An explanation of the inter-relationships between data values and derivatives which underly these configurations can be found in [12,13]. As the user gains experience with this representation, he or she can learn to ``read'' the re-projected voxel as an indicator of the volume characteristics at the probe location.
In addition to the process of dual-domain interaction described above, transfer functions can also be created in a more manual fashion by adding one or more classification widgets to the main transfer function window. The opacity and color contributions from each classification widget sum together to form the transfer function. We have developed two types of classification widget: triangular and rectangular.
The triangular classification widget, shown in Figures 2, 5, and 9, is based on Levoy's ``isovalue contour surface'' opacity function . The widget is an inverted triangle with a base point attached to the horizontal data value axis. The horizontal location of the widget is altered by dragging the ball at the base point, and the vertical extent is altered by dragging the bar on the top edge. As the height is modified, the angle subtended by the sides of the triangle is maintained, scaling the width of the top bar. The top bar can also be translated horizontally to shear the triangle. The width of the triangle is modified by moving a ball on the right endpoint of the triangle's top bar. The classification can avoid assigning opacity to low gradient magnitudes by raising a gradient threshold bar, controlled by a ball on the triangle's right edge. The trapezoidal region spanned by the widget (between the low gradient threshold and the top bar) defines the data values and gradient magnitudes which receive color and opacity. Color is constant; opacity is maximal along the center of the widget, and it linearly ramps down to zero at the left and right edges.
The triangular classification widgets are particularly effective for visualizing surfaces in scalar data. More general transfer functions, for visualizing data which may not have clear boundaries, can be created with the rectangular classification widget. This widget is seen in Figure 1. The rectangular widget has a top bar which translates the entire widget freely in the two visible dimensions of the transfer function domain. The balls located at the top right and the bottom left corners resize the widget. The rectangular region spanned by the widget defines the data values and gradient magnitudes which receive opacity and color. Like the triangular widget, color is constant, but the opacity is more flexible. It can be constant, or fall off in various ways: quadratically as an ellipsoid with axes corresponding to the rectangle's aspect ratio, or linearly as a ramp, tent, or pyramid.
For both types of classification widget (triangular and rectangular), additional controls are necessary to use the third dimension of the transfer function domain: the second derivative of the scalar data. Because our research has focused on visualizing boundaries between material regions, we have consistently used the second derivative to emphasize the regions where the second derivative magnitude is small or zero. Specifically, maximal opacity is always given to zero second derivative, and decreases linearly towards the second derivative extremal values. How much the opacity changes as a function of second derivative magnitude is controlled with a single slider, called the ``boundary emphasis slider.'' Because the individual classification widgets can have various sizes and locations, it is easiest to always locate the boundary emphasis slider on the top edge of the transfer function widget. The slider controls the boundary emphasis for whichever classification widget is currently selected. With the slider in its left-most position, zero opacity is given to extremal second derivatives; in the right-most position, opacity is constant with respect to the second derivative.
While the classification widgets are usually set by hand in the transfer function domain, based on feedback from probing and re-projected voxels, their placement can also be somewhat automated. This further reduces the difficulty of creating an effective higher dimensional transfer function. The classification widget's location and size in the transfer function domain can be tied to the distribution of the re-projected voxels determined by the data probe location. For instance, the rectangular classification widget can be centered at the transfer function values interpolated at the data probe's tip, with the size of the rectangle controlled by the data probe's slider. Or, the triangular classification widget can be located horizontally at the data value queried by the probe, with the width and height determined by the horizontal and vertical variance in the re-projected voxel locations.
The shading widget is a collection of spheres which can be rendered in the scene to indicate and control the light direction and color. Fixing a few lights in view space is generally effective for renderings, therefore changing the lighting is an infrequent operation.
The color picker is an embedded widget which is based on the hue-lightness-saturation (HLS) color space. Interacting with this widget can be thought of as manipulating a sphere with hues mapped around the equator, gradually becoming black at the top, and white at the bottom. To select a hue, the user moves the mouse horizontally, rotating the ball around its vertical axis. Vertical mouse motion tips the sphere toward or away from the user, shifting the color towards white or black. Saturation and opacity are selected independently using different mouse buttons with vertical motion. While this color picker can be thought of as manipulating this HLS sphere, it actually renders no geometry. Rather, it is attached to a sub-object of another widget. The triangular and rectangular classification widgets embed the color picker in the polygonal region which contributes opacity and color to the transfer function domain. The shading widget embeds a color picker in each sphere that represents a light. The user specifies a color simply by clicking on that object, then moving the mouse horizontally and vertically until the desired hue and lightness are visible. In most cases, the desired color can be selected with a single mouse click and gesture.