Mosaics

VR Seattle: http://www.vrseattle.com/
Full screen panoramas (cubic): http://www.panoramas.dk/
Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html

Today’s Readings
• Szeliski and Shum paper (sections 1 and 2, skim the rest)
– http://www.cs.washington.edu/education/courses/455/08wi/readings/szeliskiShum97.pdf
Image Mosaics

+ + … + =

Goal
• Stitch together several images into a seamless composite
How to do it?
Basic Procedure
• Take a sequence of images from the same position
– Rotate the camera about its optical center
• Compute transformation between second image and first
• Shift the second image to overlap with the first
• Blend the two together to create a mosaic
• If there are more images, repeat
Project 2
1. Take pictures on a tripod (or handheld)
2. Warp to spherical coordinates
3. Extract features
4. Align neighboring pairs using RANSAC
5. Write out list of neighboring translations
6. Correct for drift
7. Read in warped images and blend them
8. Crop the result and import into a viewer

Roughly based on Autostitch

• By Matthew Brown and David Lowe
• http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
Aligning images

How to account for warping?

• Translations are not enough to align the images
• Photoshop demo
Image reprojection

mosaic PP

The mosaic has a natural interpretation in 3D

• The images are reprojected onto a common plane
• The mosaic is formed on this plane
Image reprojection
Basic question
• How to relate two images from the same camera center?
– how to map a pixel from PP1 to PP2

PP2
Answer
• Cast a ray through each pixel in PP1
• Draw the pixel where that ray intersects PP2

PP1

Don’t need to know what’s in the scene!

Image reprojection

Observation
• Rather than thinking of this as a 3D reprojection, think of it
as a 2D image warp from one image to another
Homographies
Perspective projection of a plane
• Lots of names for this:
– homography, texture-map, colineation, planar projective map
• Modeled as a 2D warp using homogeneous coordinates

 wx'  * * *   x 
 wy'   * * *   y 
 w * * *   1 
 
p’ H p
To apply a homography H
• Compute p’ = Hp (regular matrix multiply)
• Convert p’ from homogeneous to image coordinates
– divide by w (third) coordinate
Image warping with homographies

image plane in front image plane below

black area
where no pixel
maps to
Panoramas
What if you want a 360 field of view?

mosaic Projection Sphere

Spherical projection

• Map 3D point (X,Y,Z) onto sphere

Y
Z X • Convert to spherical coordinates

• Convert to spherical image coordinates

unit sphere

– s defines size of the final image

» often convenient to set s = camera focal length

unwrapped sphere

Spherical image
Spherical reprojection
How to map sphere onto a flat image?
• to

Y
Z X

side view

top-down view
Spherical reprojection
How to map sphere onto a flat image?
• to
– Use image projection matrix!
– or use the version of projection that properly
Y
accounts for radial distortion, as discussed in
Z X
projection slides. This is what you’ll do for
project 2.

side view

top-down view
Correcting radial distortion

from Helmut Dersch

Modeling distortion

Project
to “normalized”
image coordinates

Apply radial distortion

Apply focal length

translate image center

To model lens distortion

• Use above projection operation instead of standard
projection matrix multiplication
Spherical reprojection

input f = 200 (pixels) f = 400 f = 800

Map image to spherical coordinates

• need to know the focal length
Aligning spherical images

Suppose we rotate the camera by  about the vertical axis

• How does this change the spherical image?
Aligning spherical images

Suppose we rotate the camera by  about the vertical axis

• How does this change the spherical image?
• Translation by 
• This means that we can align spherical images by translation
Spherical image stitching

What if you don’t know the camera rotation?

• Solve for the camera rotations
– Note that a pan (rotation) of the camera is a translation of the sphere!
– Use feature matching to solve for translations of spherical-warped images
Project 2
1. Take pictures on a tripod (or handheld)
2. Warp to spherical coordinates
3. Extract features
4. Align neighboring pairs using RANSAC
5. Write out list of neighboring translations
6. Correct for drift
7. Read in warped images and blend them
8. Crop the result and import into a viewer

Roughly based on Autostitch

• By Matthew Brown and David Lowe
• http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
Feature detection summary
Here’s what you do
• Compute the gradient at each point in the image
• Create the H matrix from the entries in the gradient
• Compute the eigenvalues.
• Find points with large response (- > threshold)
• Choose those points where - is a local maximum as features
The Harris operator
- is a variant of the “Harris operator” for feature detection

• The trace is the sum of the diagonals, i.e., trace(H) = h11 + h22
• Very similar to - but less expensive (no square root)
• Called the “Harris Corner Detector” or “Harris Operator”
• Lots of other detectors, this is one of the most popular
The Harris operator

Harris
operator
Multiscale Oriented PatcheS descriptor
Take 40x40 square window around detected feature
• Scale to 1/5 size (using prefiltering)
• Rotate to horizontal
• Sample 8x8 square window centered at feature
• Intensity normalize the window by subtracting the mean, dividing by
the standard deviation in the window

40 p
ixels 8 pixels

CSE 576: Computer Vision

Adapted from slide by Matthew Brown

Feature matching
Given a feature in I1, how to find the best match in I2?
1. Define distance function that compares two descriptors
2. Test all the features in I2, find the one with min distance
Feature distance
How to define the difference between two features f1, f2?
• Simple approach is SSD(f1, f2)
– sum of square differences between entries of the two descriptors
– can give good scores to very ambiguous (bad) matches

f1 f2

I1 I2
Feature distance
How to define the difference between two features f1, f2?
• Better approach: ratio distance = SSD(f1, f2) / SSD(f1, f2’)
– f2 is best SSD match to f1 in I2
– f2 ’ is 2nd best SSD match to f1 in I2
– gives small values for ambiguous matches

f1 f2' f2

I1 I2
Evaluating the results
How can we measure the performance of a feature matcher?

50
75
200

feature distance
Computing image translations

What do we do about the “bad” matches?

Richard Szeliski CSE 576 (Spring 2005): Computer 30

Vision
Project 2
1. Take pictures on a tripod (or handheld)
2. Warp to spherical coordinates
3. Extract features
4. Align neighboring pairs using RANSAC
5. Write out list of neighboring translations
6. Correct for drift
7. Read in warped images and blend them
8. Crop the result and import into a viewer

Roughly based on Autostitch

• By Matthew Brown and David Lowe
• http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
RAndom SAmple Consensus

Select one match, count inliers

(in this case, only one)
Richard Szeliski CSE 576 (Spring 2005): Computer 32
Vision
RAndom SAmple Consensus

Select one match, count inliers

(4 inliers)
Richard Szeliski CSE 576 (Spring 2005): Computer 33
Vision
Least squares fit

Find “average” translation vector

for largest set of inliers
Richard Szeliski CSE 576 (Spring 2005): Computer 34
Vision
RANSAC
Same basic approach works for any transformation
• Translation, rotation, homographies, etc.
• Very useful tool

General version
• Randomly choose a set of K correspondences
– Typically K is the minimum size that lets you fit a model
• Fit a model (e.g., homography) to those correspondences
• Count the number of inliers that “approximately” fit the model
– Need a threshold on the error
• Repeat as many times as you can
• Choose the model that has the largest set of inliers
• Refine the model by doing a least squares fit using ALL of
the inliers
Project 2
1. Take pictures on a tripod (or handheld)
2. Warp to spherical coordinates
3. Extract features
4. Align neighboring pairs using RANSAC
5. Write out list of neighboring translations
6. Correct for drift
7. Read in warped images and blend them
8. Crop the result and import into a viewer

Roughly based on Autostitch

• By Matthew Brown and David Lowe
• http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
Assembling the panorama

Stitch pairs together, blend, then crop

Problem: Drift

Error accumulation
• small errors accumulate over time
Project 2
1. Take pictures on a tripod (or handheld)
2. Warp to spherical coordinates
3. Extract features
4. Align neighboring pairs using RANSAC
5. Write out list of neighboring translations
6. Correct for drift
7. Read in warped images and blend them
8. Crop the result and import into a viewer

Roughly based on Autostitch

• By Matthew Brown and David Lowe
• http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
 Problem: Drift

(x1,y1)

(xn,yn)

copy of first image

Solution
• add another copy of first image at the end
• this gives a constraint: yn = y1
• there are a bunch of ways to solve this problem
– add displacement of (y1 – yn)/(n -1) to each image after the first
– compute a global warp: y’ = y + ax
– run a big optimization problem, incorporating this constraint
» best solution, but more complicated
» known as “bundle adjustment”
Full-view Panorama

+ +
+ +
Different projections are possible
Image Blending
Project 2
1. Take pictures on a tripod (or handheld)
2. Warp to spherical coordinates
3. Extract features
4. Align neighboring pairs using RANSAC
5. Write out list of neighboring translations
6. Correct for drift
7. Read in warped images and blend them
8. Crop the result and import into a viewer

Roughly based on Autostitch

• By Matthew Brown and David Lowe
• http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
Feathering

+
1 1
0 0

=
Effect of window size

1 left 1

right
0 0
Effect of window size

1 1

0 0
Good window size

“Optimal” window: smooth but not ghosted

• Doesn’t always work...
Pyramid blending

Create a Laplacian pyramid, blend each level

• Burt, P. J. and Adelson, E. H., A multiresolution spline with applications to image mosaics, ACM
Transactions on Graphics, 42(4), October 1983, 217-236.
 Alpha Blending
I3

I1
Optional: see Blinn (CGA, 1994) for details:
http://ieeexplore.ieee.org/iel1/38/7531/00310740.pdf?isNumb
er=7531&prod=JNL&arnumber=310740&arSt=83&ared=87&a

I2 rAuthor=Blinn%2C+J.F.

Encoding blend weights: I(x,y) = (R, G, B, )

color at p =
Implement this in two steps:
1. accumulate: add up the ( premultiplied) RGB values at each pixel
2. normalize: divide each pixel’s accumulated RGB by its  value
Q: what if  = 0?
Poisson Image Editing

For more info: Perez et al, SIGGRAPH 2003

• http://research.microsoft.com/vision/cambridge/papers/perez_siggraph03.pdf
Image warping

h(x,y)
y y’
x x’
f(x,y) g(x’,y’)

Given a coordinate transform (x’,y’) = h(x,y) and a

source image f(x,y), how do we compute a
transformed image g(x’,y’) = f(h(x,y))?
Forward warping

h(x,y)
y y’
x x’
f(x,y) g(x’,y’)

Send each pixel f(x,y) to its corresponding location

(x’,y’) = h(x,y) in the second image
Q: what if pixel lands “between” two pixels?
Forward warping

h(x,y)
y y’
x x’
f(x,y) g(x’,y’)

Send each pixel f(x,y) to its corresponding location

(x’,y’) = h(x,y) in the second image
Q: what if pixel lands “between” two pixels?
A: distribute color among neighboring pixels (x’,y’)
– Known as “splatting”
Inverse warping

h-1(x,y)
y y’
x x’
f(x,y) g(x’,y’)

Get each pixel g(x’,y’) from its corresponding location

(x,y) = h-1(x’,y’) in the first image
Q: what if pixel comes from “between” two pixels?
Inverse warping

h-1(x,y)
y y’
x x’
f(x,y) g(x’,y’)

Get each pixel g(x’,y’) from its corresponding location

(x,y) = h-1(x’,y’) in the first image
Q: what if pixel comes from “between” two pixels?
A: resample color value
– We discussed resampling techniques before
• nearest neighbor, bilinear, Gaussian, bicubic
Forward vs. inverse warping
Q: which is better?

A: usually inverse—eliminates holes

• however, it requires an invertible warp function—not always possible...
Project 2
1. Take pictures on a tripod (or handheld)
2. Warp to spherical coordinates
3. Extract features
4. Align neighboring pairs using RANSAC
5. Write out list of neighboring translations
6. Correct for drift
7. Read in warped images and blend them
8. Crop the result and import into a viewer

Roughly based on Autostitch

• By Matthew Brown and David Lowe
• http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
Other types of mosaics

Can mosaic onto any surface if you know the geometry

• See NASA’s Visible Earth project for some stunning earth mosaics
– http://earthobservatory.nasa.gov/Newsroom/BlueMarble/
– Click for images…