Project 1
Image Mosaicing
Results + Discussion
These are results of using moss for image mosaicing. Note that below, the small original images which were stithced together are links to full-resolution images, but these are in PPM format, so the browser may need to be configured to view these (this way you can recreate the results here). The full-resolution mosaic images are saved as JPGs.We start with a single image of the Federal Reserve Chairman, split into two pieces with xv. This was my first test of the mosaicing program because translations are the only required transformations between images, and thus are easy to debug. Here are the two source images:
Here is the correspondence data: 
NRRD00.01 dimension: 3 type: float encoding: ascii sizes: 2 2 4 130 79 12 8 179 88 60 17 127 136 8 65 176 151 57 79Here is the command-line which joined them:moss gs.ppm gs.nrrd 0 gsA.ppm gsB.ppmHere is the result:If the points had been chosen perfectly, the needed transformation would have been a pure translation, and the smaller image would not have been warped slightly, but it was close enough. 
An example with three images. At the end of my stay in the Program of Computer Graphics at Cornell University, I packed up all belongings in boxes which were shipped to Utah via UPS. I had to document this. Three of the images (dimensions 387x256) were taken from the same viewpoint:
Here is the point data to stitch them: 
The command line:NRRD00.01 dimension: 3 type: float encoding: ascii sizes: 2 3 13 # box09 box08 box07 172 204 59 57 -1 -1 215 154 105 7 -1 -1 236 223 123 77 -1 -1 262 216 149 71 -1 -1 362 178 250 37 -1 -1 313 232 198 89 -1 -1 -1 -1 109 129 51 34 -1 -1 154 121 98 28 -1 -1 182 160 125 66 -1 -1 176 244 120 151 -1 -1 268 130 212 35 -1 -1 377 121 323 27 -1 -1 352 226 294 131The result:moss box.ppm box.nrrd 1 box09.ppm box08.ppm box07.ppmNotice that because the middle image (#1) was chosen as the reference image, its edges are straight, while the projective frames of the other two images are slightly distorted. The soft blending between images seems to be doing a good job. 
Here is an example of how the mosaicing works when the target is planar, and the camera viewpoint is varying. These images will be recongnizable to anyone who works in the Merrill Engineering Building:
It should also be clear that these were taken from very different viewpoints. Here is a shortened version of the point data used for mosaicing; the full set of correspondence points is also available. 
The command line is just like all the others:NRRD00.01 dimension: 3 type: float encoding: ascii sizes: 2 4 36 # w3 w0 w1 w2 264 105 134 10 -1 -1 -1 -1 369 84 231 8 -1 -1 -1 -1 ... 530 197 353 109 -1 -1 -1 -1 509 382 340 230 -1 -1 -1 -1 -1 -1 434 6 81 72 -1 -1 -1 -1 451 136 106 227 -1 -1 ... -1 -1 565 282 246 399 -1 -1 -1 -1 566 306 248 429 -1 -1 -1 -1 256 164 -1 -1 428 107 -1 -1 313 165 -1 -1 489 109 ... -1 -1 3 249 -1 -1 140 192 -1 -1 2 340 -1 -1 139 293From the correspondence data above we can see that image #1 ("w0.ppm") is the one to which all the others are connected; this image also happens to be the reference image:moss wall.ppm wall.nrrd 1 w3c.ppm w0.ppm w1.ppm w2c.ppmAny image can be chosen as the reference image, but if there is one which has the least perspective distortion, then it makes a good choice for the reference. Choosing a different reference image makes it look like same mosaic is being viewed from a different location. A slightly different command line: 
And a slightly different viewpoint; the image in the upper right is now the reference one.moss wall2.ppm wall.nrrd 2 w3c.ppm w0.ppm w1.ppm w2c.ppmThese also demonstrate the blending that happens between images. The portrait in the middle of the top row didn't get quite registered between the two overlapping images, so the top of its frame is broken into two segments. Although they are not aligned, the mosaic blends smoothly between the two images. 
Finally, a more complex mosaic. This is shows the advantage of being able to set up the mosaic with any kind of adjacency graph topology; not just restricted to linear tilings (suitable for landscape panaramas). The eight source images:
Setting up the correspondence data for so many images was a bit of a pain. One sensible strategy was to build up the mosaic one image at a time; with each new image I had to determine which existing image had the most overlap, and then determine the matching point coordinates. The command line: 
This makes the third image (#2, "clutter04.ppm") the reference one, so the slide-rule is not distorted very much, however in the lower left area, on the desk, there is obvious distortion.moss clutter.ppm clutter.nrrd 2 clutter02.ppm clutter03.ppm \ clutter04.ppm clutter05.ppm clutter06.ppm clutter08.ppm \ clutter09.ppm clutter10.ppmIf we try to fix that distortion by making the first image (#0, "clutter02.ppm") be the reference, then everything else gets stretched out. 
As long as we are reprojecting images to a plane and not some other surface such as a cylinder, this problem is unavoidable. Projecting onto something other than a plane would mean that straight lines would not be reprojected straight, so the slide rule would look wrong. 
This mosaic also serves as a good demonstration of the benefit of fading gradually between images. One portion of the mosaic above was selected for this. If the weighting given to each image is reversed, so that the edges weigh more heavily than the interior, then we can see just how dissimilar in color and brightness adjacent images are:
With the weighting back to normal, the transitions between the images are smooth enough to be unseen. Notice also that besides the differences in brightness and color, some slight position mis-calibrations are also being hidden by this blending scheme, especially where my ties and belts are cut by the edge of one image: 
Finally, a note about how many correspondence points are best for the mosaicing. The mathematics of the situation dictate that at least four points be required. Here are two images do demonstrate the effect of varying the number of correspondence points. There is not a huge amount of overlap between images, so it is important to get the correspondence right:
If only four points are used, they must be chosen very precisely, and even then, the result may be pretty awful. For example, one set of correspondence data with four points: 
And the resulting mosiac:NRRD00.01 dimension: 3 type: float encoding: ascii sizes: 2 2 4 888 585 153 588 760 536 26 546 897 422 164 426 743 326 9 332Small errors in determining point coordinate will show up if there are only four points, and the error will be worse if the camera's optical center was not exactly fixed between the images. Also, if one is using only four points, then no one point can be collinear with any other two. If this happens, then moss detects that it does not have sufficient information to compute the mosaic, and it complains (demonstrating what my biff library is good for): 
The "tiny" value used is defined in moss.h. Even if the projective transform can be calculated, it had better be invertible, since its inverse is needed in the final step of creating the mosaic image. Here is the error message for this condition:moss: trouble computing the mosaic: [moss] mossDoit: trouble learning correspondence info [moss] mossSetMatrix: trouble calculating corresp. projective transforms [moss] _mossCalcPTs: trouble with pseudo inverse for i,j=0,1 [moss] _mossPseudoInverse: no pseudo-inverse due to small singular values [moss] _mossPseudoInverse: abs(W[7]) = 0.00226571 < 0.005 = tinyAdding correspondence points generally does a great deal to make the mosaic more reasonable. A new correspondence data file, in which the first four points are copied from above:moss: trouble computing the mosaic: [moss] mossDoit: trouble learning correspondence info [moss] mossSetMatrix: trouble calculating corresp. projective transforms [moss] _mossCalcPTs: can't invert projective transform for i,j=0,1 [moss] _moss3x3Invert: matrix has tiny determinant (0.00315269)And the resulting mosaic:NRRD00.01 dimension: 3 type: float encoding: ascii sizes: 2 2 8 888 585 153 588 760 536 26 546 897 422 164 426 743 326 9 332 813 474 81 481 813 514 80 521 763 410 31 416 900 219 168 226Note that there is still a slight misalignment in the mountain tops. Even with the addition of four more points, the alignment is not going to get much better. There is only so much distortion that can be expressed with projective transforms, and if some of the first correspondence points were poorly chosen, then they could be corrupting the results using many more points. Using four more points which try to correctly nail down the top edge of the mountains, now the yellow parking line below the silver car (in the center) is visibly misaligned. 
NRRD00.01 dimension: 3 type: float encoding: ascii sizes: 2 2 12 888 585 153 588 760 536 26 546 897 422 164 426 743 326 9 332 813 474 81 481 813 514 80 521 763 410 31 416 900 219 168 226 832 202 101 208 792 265 61 271 893 243 161 249 776 206 46 211
