CS498: Computer Vision

CS498

Lab 4

Prelab

Before lab, take two pictures, with the camera lense at the same position, but pointing in slightly different directions. Try get at least a 50-80% overlap between the images. It is best if you take the pictures in a large area, since this will reduce the difficulties coming from taking a picture from multiple positions. Here is an example, although do be sure to take some pictures with more overlap than this. (They will be useful in later labs.)

Many cameras today have multi-megapixel images. In their uncompressed form, these images are very large. Since we usually maintain several copies of an image in matlab, this can easily fill your computer's memory. I recommend reducing the size of these images until Matlab does not warn you that they are too large to display on your screen when you run imshow

Deliverables

The lab checklist will be provided in hard-copy.

Please follow the provided templates transforms.m and stitch.m for your top-level scripts ((Updated after lab based on student feedback. Indeed, most of the highlights throughout this page were updated after lab.)

In all your source files, be sure to include your name and the quarter (e.g. Winter 2012-2013.) And, in the templates you copy from below, be sure to edit the author block that follows the help block so that it is accurate for you.

Useful Matlab commands for this lab

ndgrid: Create a set of n-dimensional grids with the values of the index for each point in the grid. e.g. [ii,jj,kk] = ndgrid(1:2,1:3,1:4);
hold on: Keep the currents contents of a graph. The next plot will be plotted on top. e.g. hold on
hold off: Opposite of hold on.
plot: Plot 2-d data. e.g. plot(jj,ii,'ro-'). If you use doc for help, be sure to check out the LineSpec link.
imfuse: Overlay two images with 0.5 alpha compositing. e.g. IFused = imfuse(I1,I2,'blend');
size: Find the size of all or the given dimension of a matrix. e.g. size(stack) could return [400,300,5] and size(stack,2) would return 300
length: Return size of first non-singleton dimension. e.g., if size(A) is [1 7 5], length(A) returns 7. This is meant to be used with column and row vectors which have only one dimension greater than 1
x(:): Flatten a matrix to a column vector. e.g., col = H(:) flattens the matrix H into a 9x1 matrix
reshape: Restore the shape of a flattened matrix. e.g., x = reshape(x(:),size(x)) flattens and then restores x to its original shape.
*: Matrix multiplication can be used to duplicate rows or columns. For example, x * ones(1,5) produces five duplicates of the column vector x
repmat: An alternative to the matrix multiplication approach on the previous line. I've found it to be slower in practice.
[a,b;c,d]: Concatenate the matrices a,b, c, and d into a larger matrix.

Homographic point transforms

Start in the template transforms.m

TODO: Instruct students to create a matrix of the form

  [i₁ i₂ i₃ ... i_n*m]
  [j₁ j₂ j₃ ... j_n*m]
  [1 1 1 ... 1]

where every point in the grid has a column in the matrix. Also, perhaps add a trasformPoints matlab script? ... or not

As discussed in class, you can compute many useful warpings using matrix multiplication.

For example, you can rotate a point counter-clockise by multiplying it by the matrix

  [i_new] = [cos(theta), -sin(theta)] [i_old]
  [j_new]   [sin(theta),  cos(theta)] [j_old]

Which we can represent in short form as

 i_new = R i_old

Create a grid of points using ndgrid. Plot this grid on top of your first image using hold on and plot commands. Space the points in the grid 10-50 pixels apart so they don't overlap when you plot theml

Now transform this grid once for each bullet using

a rotation 45 degrees clockwise
a translation 30 pixels right and 50 pixels down
a scaling to increase the size of the image by a factor of 2.5
All three of the above in the order they are above

Plot the points on a blank image -- that is, a white image the size of one of your images.

Transforming the image

Once you are able to transform points, you can move on to transforming an image. You can start by transforming an image in your top-level script, then cut (not copy) your code and paste it into an m-file using the template below.

The most natural way to transform an image is, for each pixel in the original image, to compute the indices of the corresponding pixel in the new image, and then color that pixel the same color as the original image -- in other words, to project the original image onto the new transformed image. However, in practice, this can result in some pixels in transformed image not getting any color at all, even though they are inside the transformed area. If you are enlarging an image, this approach will produce scattered individual dots of color wherever a pixel lands.

Instead, we transform the pixel indices from the destination image back into the original space, and copy the color from there into the new image.

You can do this using a for-loop, or, if you prefer, using image indexing and resizing. If you choose the indexing route, you might be able to obtain faster results (though this approach can be memory-bound if you use large images). This technique requires a variety of methods:

Flattening an array with x(:)
reshape
sub2ind
ind2sub
multiplying by ones(x,k), e.g. h'*ones(1,5) will repeat the column h' five times.
repmat, although I have found the ones trick above to be faster.
Concatenating matrices, e.g. [a,b;c,d] or equivelently, [a b;c d]

There are lots of fun tricks you can play with these methods. In particular, you can assign to a square region from a column vector (or vise-versa) if they both have the same size:

x = zeros(4,5);
x(:) = 1:prod(size(x));

When you map the pixel locations, they most often will not fall exactly on a pixel in the source image. Just round to the nearest pixel, and use the value of that pixel.

Once you are able to successfully apply a transformation to an image, copy-paste the function header below (using Chrome or Firefox) into a new .m file. Save the .m file into the same directory as your top-level. This should also be your Matlab working directory. (Matlab should automatically name the m-file transformImage.m when you save it.) Then cut-paste your transformation code into the file and edit it to fit the description in the comment. After that, you can use transformImage like any other Matlab function. You can even use help transformImage. Matlab will look up the filename transformImage.m automatically when you use transformImage in the command line or your code.

Note that in the provided template, it says H transforms from the source image to the destination image, that is, from the image being transformed to the image it is transformed on to.

% INew = transformImage(szDest,Isource, H)
% 
% Transform image Isource an image of size szDest.
%
%  szDest -- size of the image to create (size of destination,  that is,
%            szDest == size(INew) when done).
%  Isource -- the image to transform
%  H -- The homography supplied is from source to dest.
%    [iDest]   [         ] [iSource]
%    [jDest] = [    H    ] [jSource]
%    [1]       [         ] [1]
%
%  INew -- The transformed image, which has size szDest, and is 
%  in the coordinate frame of the destionation image, but contains the transformed source image.

% Author: Phileas Fogg
% Term: Winter 2012-2013
% Course: CS498 - Computer Vision
% Assignment: Lab 4 - Homographic image warping

function INew = transformImage(szDest, Isource, H)

Repeat the transformations from the previous part, now moving the image instead of the points. You should not see gaps in the image when it is complete.

Marking correspondences in an image

Now move over to the top-level script stitch.m

Using ginput, mark four to ten points on the first image. Mark the corresponding points in the second image. That is, mark four distinct points on objects in the first image, and mark the same points on the objects in the second image. Choose points that you can identify well, such as corners. Choose points that are widely spaced in the image so that your program does not have to extrapolate beyond them too much.

Save the points you use somewhere in your source code, so you don't have to remark the points every time you run the program. This should not be hard-coded in one of your matlab functions, but should be hard-coded into your top-level script.

Compute the means of the i and j coordinates in both images. Subtract the mean of one image from the mean of the other image to find the translation between the two images.

Using a similar strategy to the previous part, create a transformation that maps the points in the right image to the points in the left image, just translating the points by the distance between the means.

Once you have code that works, move it into the method findTranslation, which conforms to the following heading:

% H = findTranslation(matches)
%
% matches -- list of matches in rows of [iSrc jSrc iDest jDest]
%
% H -- translation transformation [iSrc,jSrc,1]' = H*[iSrc,jSrc,1]'

% Author: Phileas Fogg
% Term: Winter 2012-2013
% Course: CS498 - Computer Vision
% Assignment: Lab 4 - Homographic image warping

function H = findTranslation(matches)

Creating a Panography

Now you can use your image transformation techniques from the first section to map one image onto another!

After fusing your two images, you should have something like this:

One of the fun ideas below is to modify things a bit so you get a black matted margin around the destination image:

To see some artistic works created by manually arranging images in a similar style (but also with rotations), visit the flickr panography group

Fun Ideas

Try creating a black mapped area around the image, as shown in the second image above. You can either modify the transform to go from the larger image, or modify your transforming function to accept an "offset" for the origin of the image, which is taken into account before transforming.
Use bilinear interpolation instead of rounding to the nearest pixel. This gives a smoother image when you enlarge.
Try stitching several images in a row to create a fuller panography (6.1.2, p. 277/)
The imfuse approach we take here causes the images to be darker where they don't overlap. Can you make the images have full brightness like Figure 6.5. (p. 278/) by creating your own alpha-mapping strategy? You can create a mask for the transformed image(s) by transforming an all-ones image using the same mapping function
Can you build an entire panorama, going full-circle around a point? You may want to project the images on the the surface of a cube. Could you find the homography to go from one cube surface to the next? This would probably require some full 3-D camera geometry. Our textbook has a chapter on this subject.
Google Street-view is built as a sequence of panoramas, centered at distinct points on a street. Can you build a simple "street view" of a pretty spot on campus?
Do you have your own fun ideas that you would like to try?

Readings

Szeliski (2011) Section 6.1 - 6.1.4 (pp. 275-280/)

Lab 4: Homographic image warping