Gordon Kindlmann

CS 6964
Project 1
Image Mosaicing

Results + Discussion



These are results of using moss for image mosaicing. Note that below, the small original images which were stithced together are links to full-resolution images, but these are in PPM format, so the browser may need to be configured to view these (this way you can recreate the results here). The full-resolution mosaic images are saved as JPGs.

We start with a single image of the Federal Reserve Chairman, split into two pieces with xv. This was my first test of the mosaicing program because translations are the only required transformations between images, and thus are easy to debug. Here are the two source images:

Here is the correspondence data:
NRRD00.01
dimension: 3
type: float
encoding: ascii
sizes: 2 2 4

 130  79    12   8
 179  88    60  17
 127 136     8  65
 176 151    57  79
Here is the command-line which joined them:
moss gs.ppm gs.nrrd 0 gsA.ppm gsB.ppm
Here is the result:
If the points had been chosen perfectly, the needed transformation would have been a pure translation, and the smaller image would not have been warped slightly, but it was close enough.

An example with three images. At the end of my stay in the Program of Computer Graphics at Cornell University, I packed up all belongings in boxes which were shipped to Utah via UPS. I had to document this. Three of the images (dimensions 387x256) were taken from the same viewpoint:

Here is the point data to stitch them:
NRRD00.01
dimension: 3
type: float
encoding: ascii
sizes: 2 3 13
# box09      box08     box07

 172 204    59  57    -1 -1
 215 154   105   7    -1 -1
 236 223   123  77    -1 -1
 262 216   149  71    -1 -1
 362 178   250  37    -1 -1
 313 232   198  89    -1 -1
  -1 -1    109 129    51  34
  -1 -1    154 121    98  28
  -1 -1    182 160   125  66
  -1 -1    176 244   120 151
  -1 -1    268 130   212  35
  -1 -1    377 121   323  27
  -1 -1    352 226   294 131
The command line:
moss box.ppm box.nrrd 1 box09.ppm box08.ppm box07.ppm
The result:
Notice that because the middle image (#1) was chosen as the reference image, its edges are straight, while the projective frames of the other two images are slightly distorted. The soft blending between images seems to be doing a good job.

Here is an example of how the mosaicing works when the target is planar, and the camera viewpoint is varying. These images will be recongnizable to anyone who works in the Merrill Engineering Building:


It should also be clear that these were taken from very different viewpoints. Here is a shortened version of the point data used for mosaicing; the full set of correspondence points is also available.
NRRD00.01
dimension: 3
type: float
encoding: ascii
sizes: 2 4 36
# w3        w0       w1       w2

 264 105  134  10   -1 -1    -1 -1 
 369  84  231   8   -1 -1    -1 -1 
                 ...
 530 197  353 109   -1 -1    -1 -1 
 509 382  340 230   -1 -1    -1 -1 
  -1 -1   434   6   81  72   -1 -1 
  -1 -1   451 136  106 227   -1 -1 
                 ...
  -1 -1   565 282  246 399   -1 -1 
  -1 -1   566 306  248 429   -1 -1 
  -1 -1   256 164  -1 -1     428 107
  -1 -1   313 165  -1 -1     489 109
                 ...
  -1 -1     3 249  -1 -1     140 192
  -1 -1     2 340  -1 -1     139 293
The command line is just like all the others:
moss wall.ppm wall.nrrd 1 w3c.ppm w0.ppm w1.ppm w2c.ppm
From the correspondence data above we can see that image #1 ("w0.ppm") is the one to which all the others are connected; this image also happens to be the reference image:
Any image can be chosen as the reference image, but if there is one which has the least perspective distortion, then it makes a good choice for the reference. Choosing a different reference image makes it look like same mosaic is being viewed from a different location. A slightly different command line:
moss wall2.ppm wall.nrrd 2 w3c.ppm w0.ppm w1.ppm w2c.ppm
And a slightly different viewpoint; the image in the upper right is now the reference one.
These also demonstrate the blending that happens between images. The portrait in the middle of the top row didn't get quite registered between the two overlapping images, so the top of its frame is broken into two segments. Although they are not aligned, the mosaic blends smoothly between the two images.

Finally, a more complex mosaic. This is shows the advantage of being able to set up the mosaic with any kind of adjacency graph topology; not just restricted to linear tilings (suitable for landscape panaramas). The eight source images:


Setting up the correspondence data for so many images was a bit of a pain. One sensible strategy was to build up the mosaic one image at a time; with each new image I had to determine which existing image had the most overlap, and then determine the matching point coordinates. The command line:
moss clutter.ppm clutter.nrrd 2 clutter02.ppm clutter03.ppm \
clutter04.ppm clutter05.ppm clutter06.ppm clutter08.ppm \
clutter09.ppm clutter10.ppm
This makes the third image (#2, "clutter04.ppm") the reference one, so the slide-rule is not distorted very much, however in the lower left area, on the desk, there is obvious distortion.
If we try to fix that distortion by making the first image (#0, "clutter02.ppm") be the reference, then everything else gets stretched out.
As long as we are reprojecting images to a plane and not some other surface such as a cylinder, this problem is unavoidable. Projecting onto something other than a plane would mean that straight lines would not be reprojected straight, so the slide rule would look wrong.

This mosaic also serves as a good demonstration of the benefit of fading gradually between images. One portion of the mosaic above was selected for this. If the weighting given to each image is reversed, so that the edges weigh more heavily than the interior, then we can see just how dissimilar in color and brightness adjacent images are:

With the weighting back to normal, the transitions between the images are smooth enough to be unseen. Notice also that besides the differences in brightness and color, some slight position mis-calibrations are also being hidden by this blending scheme, especially where my ties and belts are cut by the edge of one image:

Finally, a note about how many correspondence points are best for the mosaicing. The mathematics of the situation dictate that at least four points be required. Here are two images do demonstrate the effect of varying the number of correspondence points. There is not a huge amount of overlap between images, so it is important to get the correspondence right:

If only four points are used, they must be chosen very precisely, and even then, the result may be pretty awful. For example, one set of correspondence data with four points:
NRRD00.01
dimension: 3
type: float
encoding: ascii
sizes: 2 2 4

 888 585 153 588
 760 536  26 546
 897 422 164 426
 743 326   9 332
And the resulting mosiac:
Small errors in determining point coordinate will show up if there are only four points, and the error will be worse if the camera's optical center was not exactly fixed between the images. Also, if one is using only four points, then no one point can be collinear with any other two. If this happens, then moss detects that it does not have sufficient information to compute the mosaic, and it complains (demonstrating what my biff library is good for):
moss: trouble computing the mosaic:
[moss] mossDoit: trouble learning correspondence info
[moss] mossSetMatrix: trouble calculating corresp. projective transforms
[moss] _mossCalcPTs: trouble with pseudo inverse for i,j=0,1
[moss] _mossPseudoInverse: no pseudo-inverse due to small singular values
[moss] _mossPseudoInverse: abs(W[7]) = 0.00226571 < 0.005 = tiny
The "tiny" value used is defined in moss.h. Even if the projective transform can be calculated, it had better be invertible, since its inverse is needed in the final step of creating the mosaic image. Here is the error message for this condition:
moss: trouble computing the mosaic:
[moss] mossDoit: trouble learning correspondence info
[moss] mossSetMatrix: trouble calculating corresp. projective transforms
[moss] _mossCalcPTs: can't invert projective transform for i,j=0,1
[moss] _moss3x3Invert: matrix has tiny determinant (0.00315269)
Adding correspondence points generally does a great deal to make the mosaic more reasonable. A new correspondence data file, in which the first four points are copied from above:
NRRD00.01
dimension: 3
type: float
encoding: ascii
sizes: 2 2 8

 888 585 153 588
 760 536  26 546
 897 422 164 426
 743 326   9 332

 813 474  81 481
 813 514  80 521
 763 410  31 416
 900 219 168 226
And the resulting mosaic:
Note that there is still a slight misalignment in the mountain tops. Even with the addition of four more points, the alignment is not going to get much better. There is only so much distortion that can be expressed with projective transforms, and if some of the first correspondence points were poorly chosen, then they could be corrupting the results using many more points. Using four more points which try to correctly nail down the top edge of the mountains, now the yellow parking line below the silver car (in the center) is visibly misaligned.
NRRD00.01
dimension: 3
type: float
encoding: ascii
sizes: 2 2 12

 888 585 153 588
 760 536  26 546
 897 422 164 426
 743 326   9 332

 813 474  81 481
 813 514  80 521
 763 410  31 416
 900 219 168 226

 832 202 101 208
 792 265  61 271
 893 243 161 249
 776 206  46 211