# Project 5: Snakes: Deformable Contour Segmentation

## 1. Overview

In this project we implement the deformable contour segmentation approach due to Trucco & Verry. This approach, also known as deformable snake segmentation optimizes a user-specified contour to segment an image. Such user interaction results in a robust algorithm for segmentation, but does not preclude complete automation of the process. The following sections discuss the theory of deformable contours and one possible greedy implementation. We further apply this algorithm to various contour images and examine the effects of various factors governing the optimization process. Finally, we conclude with a few remarks about the details of the implementation.

## 2. Deformable Contours: Snakes

The snake model is desribed as a controlled continuity model under the influence of image forces. Internal forces control the bending characteristics of the line, while image forces, such as the gradient, serve to push the snake toward image features. When the snake coordinate vector is defined parametrically as v(s) = (x(s), y(s)) , the total energy of the snake is defined as
E*snake = ∫ Esnake(v(s))ds = ∫ Eint(v(s))ds + ∫ Eext(v(s))ds
where Eint and Eext are the internal and external energy terms, respectively. Minimization of this combined energy functional implies that the internal energy works to ensure that a consistent shape representation for the snakse, while the external energy term attempts to change the shape properties of the snake (as it evolves) to capture relevant features.

The internal energy of a snake seeks to preserve the continuity and curvature through the evolution process. Thus, the internal energy can be defined as:
Eint = Econt + Ecurv
Each of the above energies can be computed in terms of the contour points defined by v(s). The continuity can be quantified as the rate of change of the contour points, whereas smoothness is the change in curvature, and can be measured as the rate of change of the tangent vectors along the contour. These energies are therefore given by the first and second derivatives of the contour v(s).

The external forces mostly depend on the application. For image segmentation, we want the deformable contour to find a particular feature within an image. Typically, features have well defined edges, and we wish to evolve the snake to match these edges. Thus, the external energy influencing the evolution of the snake must be related to this feature ! The most relevant and obvious measure of edges in an image is the gradient magnitude, and this is precisely what is used to denote the external energy. Mathematically,
Eext = -|| $\nabla$I ||
Since we wish to minimize both internal and external energies, we construct the external energy to equal the negative of the gradient magnitude.

Variational calculus can be used to formulate a solution to minimize E*snake and evolve the snake to match the desired contour in the image. However, this method is complex and it is tough to obtain a stable implementation.

## 3. Snakes: Discrete Implementation

Alternately, we can devise a discrete solution to the snakes evolution problem as follows.

We model the contour as c = {p1,p2,...,pN}, where pi represents the x- and y-coordinates of the ith point on the contour. Now, we can minimize the following equation in a

The paper we implement in this project describes a greedy algorithm to solve this minimization problem. This approach requires us to evaluate local neighborhoods of each pi to decide the best location for moving pi. We evaluate each candidate neighborhood point for the energies defined in the equation above, and move pi to the location with the minimum energy. The algorithm can run until a specified number of iterations or until point locations along the contour are stable.

Each of the energies related to the snake can be defined using discrete differences (as shown in class). The continuity energy can be defined in terms of the distance between points along the snake, as compared to the average distance between neighboring points. Thus, this term will seek to keep points along the snake as close together as possible.
The curvature energy is nothing but the second derivative at each point along the snake. Finally, each of these energies are normalized by division with the largest value in any given neighborhood.

As discussed earlier, the external energy is nothing but the negative gradient magnitude. However, a pre-processing step is required prior to computing the image gradient. Gaussian blurring is applied as a pre-processing step. This ensures a greater range for the deformable snake to latch onto image contours. Normalizing the gradient magnitude can lead to detection of spurious or weak edges and should be avoided. The influence of the external energy term can instead be controlled by the constant γ related to it.

## 4. Results: Synthetic Data

*hyperlink to image in actual size
This section describes the various experiments and provides a discussion on the corresponding observations. Most of the experiments are performed on a U-shaped image. This image is challenging and provides avenues to stretch the implementation and examine the contour evolution over a region involving concavities.

### 4.1 Contour Initialization and Number of Points

As a first step, we experiment with different initial contours. All other parameters remaining the same, the initial contour plays a very important part in the convergence of the snake evolution process. The images below describe the effect of generic contours versus user-defined contours. Figures in the left column indicate the initial contour in red, while the animation in the middle column shows the evolution of the deformable snake at regular intervals in the optimization process. The final contour is displayed in green in figures of the right column.

We try to automate the initialization process by using geometric primitives such as a circle and an ellipse as initial contours. As we can see in the first and second rows of the figure above, this approach does not converge very well ! In fact, the final contour due to the second initialization actually loops on itself !! The constants also play a major part in the eovlution, but this effect is not biased due to those values since they are the same for all three results displayed below.

One the other hand, user-defined contours result in much better segmentation. Thr third row displays the result of a meticulously initialized snake with a dense collection of points whereas the fourth row has a sparsely populated initialization and yet generates a decent segmentation. This does not imply that the results are invariant to the number of points on the initial contour. It is important to note that the granularity of the feature governs the required number of points on the initial snake. In the last row of the figure below, the user-initialized snake has lesser number of points on the outer curve of the U, but has a dense population of points near the concave section. This sensitivity is important to generate good results using deformable snakes.

From the above figure, we can conclude that the deformable snake algorithm is highly sensitive to initialization of the snake. One also needs to understand the granularity of the feature being segmented while deciding on the number of points and their distribution in the initialization process.

### 4.2 Effect of Gaussian Smoothing

The second experiment involves an examination of the external image forces governing the optimization process. We keep all parameters constant, and only vary the amount of Gaussian blurring applied to the image prior to computing the gradient magnitude. The figures below display the quiver plots of the gradient image after different amout of blurring. As we can see from these images, apart from the scaling factor γ the Gaussian blurring plays an important role in guiding the contour evolution.

Synthetic data: Gradient image after Gaussian blurring ( Note the increasing range for the snake to move)
(l) σ = 0.25 (c) σ = 0.5, (r) σ = 1.0

The amount of blurring determines the size of the region in which the snake can move under the influence of the external forces. As we can see from the images below, blurring with a wider Gaussian kernel enlarges the range of possible locations any point on the snake can move to. While this may be an advantage in most cases, it may turn out to be a flaw in cases where we have overlapping objects, or the discrimination at an edge is poor (e.g. MRI images of brain tissue).

### 4.2 Effect of Continuity constant α

This experiment is designed to look at the effect of the constants α, β, and γ on the segmentation results. The parameter α governs the continuity. For contour like the 'infinity' symbol in the figure below, it is important that the snake deform as much as possible to achieve the desired shape. α however acts to keep the points along the snake close together and reduces its 'elasticity'. Thus, a low value of α is desirable to segment shapes like the 'infinity' symbol. In the figure below, we see the user-initialized snake to the left, along the with evolution animation in the center and the final contour on the right. A high value of α will result in a much more 'circular' segmentation !

Synthetic data: (l) Initial Contour*, (c) Snake Evolution*, (r) Final Contour*

From the above figure, we can also conclude that the deformable snakes algorithm is fairly robust to noise added into this image using the GIMP software. Even though the noise results in spurious edges and holes in the contour, energy minimization only happens in the center and the snake evolves to fit this contour correctly.

### 4.4 Open (Illusory) Curves: the effect of Curvature constant β and Image constant γ

In the process of looking at shapes with missing information, we also discover the actions of the curvature constant β, and the constant γwhich governs the image force. In the figure below, we examine the results of snake evolution on a shape with missing edges which simply imply a boundary to the human visual system. In the figure below, the gray level contour has gaps in it, and we wish to segment out the entire contour as a single feature. As we can see from the second row, the deformable snake does a decent job with the right choice of parameters.
In the first row, the initial snake is roughly a user-initialized ellipse which is slightly biased toward the protrusion on the left as compared to the one on the right. The image in the center shows the animation of snake evolution with a relatively high value of β. The final contour on the right shows that the snake has not been able to recede into the right hand corner as well as it did on the left side. This behavior can be attributed to the value of β which governs the smoothness of the snake, and does not allow bending around corners unless the image forces are strong enough to govern the evolution. It must be noted that this experiment was run with a relatively low value of γ because of which the image force was not able to pull the snake into the right hand corner !

The images in the second row are results from a more balanced configuration of
β and γ. This balance allows the snake to follow the contour into both corners and achieve a much better segmentation. Of course, the user-initialization of the snake near the desired contour also helps ! In general, it seems preferable to initialize the snake just outside the feature to be segmented and let it work its way on to the contour. Perhaps, this is due to the nature of our discrete implementation as there is really nothing in the theory which implies such a preference.
This experiment also demonstrates the difficulty in segmenting corners. This problem can again be attributed to the inclusion of the curvature constraint, and its accompanying constant
β. The inherent trade-off between trying to maintain smoothness and squeezing into a corner results in the snake rounding off at corners as seen in the results above. The remedy would be to possibly make β a function of each point-location along the snake such that as this point approaches a corner the corresponding β becomes lower in value to allow this point to propagate into the corner.

## 5. Results: Real Data

*hyperlink to image in actual size

We briefly look at an example of a 2D CT image of the abdomen and try to use the deformable snake model to segment the lung. In the figure below, the left image shows the initial contour, the animation of the snake evolution process can be seen in the middle image and the final contour of the lung can be seen in the right image. For real data, it is tricky to get the balance of parameters right. Also, initializing the contour very close to the feature we wish to segment is the key.

Real data: (l) Initial Contour*, (c) Snake Evolution*, (r) Final Contour*

Unless there is an automated way to propagate the segmented contour onto the next slice (in a 3D scenario) it would be a tedious process to manually initialize the snake in each frame. One method might be to register each frame to the first one (to get rid of motion artifacts, etc.) and then propagate the segmented contour to the corresponding voxels in subsequent slices and optimize the snake further to capture the contour in that slice.

## 6. Conclusion

Active contours or deformable snakes are a handy tool for image segmentation. The method is robust to noise and illusory contours and has the potential to use prior knowledge (e.g. shape priors for initialization) about the object being segmented. In most cases however, user-interaction is crucial and the initialization process can affect the results hugely. The three parameters governing the various forces act in tandem to produce good results in general, but there is often a trade-off necessary between robustness of the approach and having a general method for all types of contours. Inspite of any drawbacks that we may have noticed, deformable snakes are widely used in a variety of image segmentation applications, and rightly so :)

## 7. Implementation Notes

My experimentation with Matlab continued with this project as it provided the opportunity to re-use some of the code from the Computer Vision class project, especially the modeules related to user-interation. My code is a collection of .m files and uses in-built Matlab functions for each of the steps in the analysis.