Project 6: Graph Cuts and Normalized Graph Cuts for Image Segmentation

Manasi Datar

---

1. Overview

In this project explore the graph cut and normalized graph cut methods with applications to image segmentation. These graph theoretic methods conceptualize the image as a connected, undirected graph with edge weights given by various affinity measures and find the 'cuts' which correspond to an optimal segmentation. We examine the use of various affinity measures by looking at the segmentation results. The study is restricted to small images due to computation limits. Finally, we conclude with a few remarks about the details of the implementation.

---

2. Images as Graphs

An image is conceptualized as a graph with a node at each pixel location. Edges represent relationships within pixels as defined by the image content. The edges can further be weighted based on a similarity criteria (discussed ahead in the report), also known as an affinity measure. Thus, the image is converted to a weighted undirected graph.

---

3. Affinity Measures

An affinity measure computes the similarity between two pixels. Of the many features that can be considered, we look at two of the most commonly used in this report. They are distance and intensity. Usually, a combination of both these measures is used to obtain an optimal segmentation. This section elaborates on the distance and intensity based affinity measures.

3.1 Distance measure

One way to define similarity is to look at the proximity of pixels. The assumption is that fundamentally, pixels close-by in an image will belog to one cohesive object. Thus, spatial distance between pixels is a good measure of affinity. This distance is usually weighted using a Gaussian kernel to ensure that the influence of far away pixels falls off exponentially. Formally, such an affinity measure is defined as
We can measure the similarity between pixels based on the spatial distance between them. We desire a measurement that falls off quickly with distance, so we define the affinity as an exponential function of distance as:
Here, σ_d controls how far away pixels influence the current location. A high value of σ_d implies that we may be looking for a large object so that far away pixels may be considered similar, whereas a low value of σ_d imples that only close-by pixels may be termed similar.

3.2 Intensity measure

A similar argument can be made in regards to the intensity of pixels. We may want to consider pixels "near-by" in the intensity space to be similar. Again, we would like to have an exponential drop in the manner in which far away pixels interact, and hence we define the intensity based affinity measure as follows:

---

4. Graph Cuts for Segmentation

Intuitively, sorting the pixel neighborhoods based on the affinity measures described above should give us a good segmentation of the image ! This is exactly what the graph cuts method aims to achieve. By formulating the problem as one of optimization of affinity measures, and solving using Lagrange multipliers, we have the following linear system to solve:
where w is the solution given by the eigen vector of the affinity matrix A. As discussed in the reading material, the values of w signify the probability of a pixel belonging to a cluster. Thus, we can look at eigenvectors corresponding to decreasing eigenvalues and obtain a segmentation by labelling pixels having a sufficiently high probability.

The distribution of the eigenvalues depends on the size of the kernel selected for the affinity measure, and may provide an insight into the number of clusters present in the given image. However, the free parameters are not easy to tune, and it is best to know the number of clusters apriori.

---

5. Normalized Graph Cuts for Image Segmentation

The graph cut approach is known to have problems dealing with outliers these pixels end up having the minimum 'cut' in the partitioning of the graph. An improvement is the normalized graph cut method, where we partition the graph using a cut such that its cost is small compared to the affinity of the individual components.

The normalized cost of cutting a weighted undirected graph V into two components A and B is given as:
where cut(A,B) is the sum of edge weights in V connecting elements in A and B, assoc(A,V) and assoc(B,V) are the sum of all edge weights in A and B respectively, to all other edges in V. Intuitively, we wish to find a cut such that the individual components have high affinity while the cost of the cut (given by the affinity between clusters) is low.

Formulating this problem as an optimization problem, and following the solution discussed in the reading material, we compute real values for the solution thus obtaining the following system:
Here we choose the eigenvector associated with the second smallest eigenvalue. Further thresholding is necessary to determine the cluster to which each pixel belongs. For multiple clusters, we look at successive eigenvectors corresponding to increasing eigenvalues.

---

6. Results

The above methods were implemented in Matlab and tested on the two images shown below. Each image was created to be 25x25 pixels.

6.1 Graph Cut Method

We begin with the graph cut method. The first experiment involves the 3-segment image, with added Gaussian noise having zero-mean and a small variance. Even though intensity affinity measure would have been sufficient to segment the three regions present in this image, we use a combination of distance+intensity affinity measures to obtain the best result. The table below displays the various components of the analysis along with the final segmentation result at the bottom right.
The affinity matrix hints at the structure within the image via its blockiness, but does not prove useful in its native form. We cannot decipher the number of components from the affinity matrix. In this case, the eigenvalue distribution isn't particularly useful either and does not help us infer that there are indeed three components in the image.
From the top two eigenvectors visualized in the bottom row, we can see the components represented as non-zero weights of the corresponding pixels. The third component is extracted as the set of unlabelled pixels. The image at the bottom right displays the final segmentation obtained.


Next, we move on to the second synthetic image. In this image, an intensity based affinity measure would again be sufficient to separate the foreground from the background. However, we use a combination of distance+intensity to seek each of the 3 shapes and the background as separate segments. However, graph cut method fails to extract more than a foreground/background segmentation. The reason is clear from the eigenvector distribution. All the pixels are accounted for by the first eigenvector itself. Thus, even if there exist non-zero weights in the other projections, they are not useful since all pixels have been assigned to the foreground or background cluster. This segmentaion is displayed in the bottom-right image of the table below.

6.1 Normalized Graph Cut Method

We look at the 3-segment image again and use the normalized graph cut method for this experiment. The results are tabulated in Figure. 4 below. In this experiment also, we choose a combination of distance+intensity affinity measure for better results. The thresholding scheme to decide the labels is slightly different in this case. Only positive weights are used for labelling. It appears that this is important in obtaining a 'sane' result with the normalized graph cut method. The eigenvectors used in the segmentation are shown in the bottom row, along with the final segmentation result.


It is not apparent from this experiment that the normalized graph cut performs better than the original graph cut method.
However, an experiment with a noisy synthetic image-2 proves the improvement normalized graph cuts provide over the original method. Figure. 5 below displays the results of the normalized graph cut method to segment the 4-component synthetic image.


We again use the distance+intensity based affinity matrix for this segmentation, and use only positive weights to threshold the labels. The middle row of Fig. 5 above shows the weights along the various eigenvectors used for the segmentation. The discrimination is much more clear in this result than the original graph cut formulation.

---

7. Conclusion

The graph cut and normalized graph cut methods lead to a different formulation of the segmentation problem and provide an intuitive solution. Normalized graph cut method provides an improvement over the graph cut method which is unable to distinguish between components similar in intensity even if they are spatially well separated. Apriori knowledge of the number of components is an advantage, since it is not always clear from the eigenvalue distribution. Hierarchical schemes could be devised to keep including eigenvectors till the segmentation accounts for a certain percentage of the variation present in the image, but this would still be a heuristic solution. Affinity measures such as texture will definitely boost the quality of segmentation for real images. In fact, edge inforrmation may be modulated into an affinity matrix and provide another good affinity measure.

However, graph cut methods provide an interesting solution to the segmentation problem, and were definitely worth exploring.

---

8. Implementation Notes

For the final project of this class, I chose to continue working with Matlab, since it provided in-built linear solvers appropriate for this problem. It was easy to integrate the in-built functions to create a working prototype of the graph cut and normalized graph cut methods.