In this work we seek to automatically align historical photographs
and nonphotographic renderings, such as paintings and line drawings,
to a 3D model of an architectural site. Specifically, we wish to
establish a set of point correspondences between local structures on
the 3D model and their respective 2D depictions. The established
correspondences will in turn allow us to find an approximate viewpoint
of the 2D depiction with respect to the 3D model.We focus on
depictions that are, at least approximately, perspective renderings
of the 3D scene.We consider complex textured 3D models obtained
by recent multiview stereo reconstruction systems [Furukawa et al.
2010] as well as simplified models obtained from 3D modeling
tools such as Trimble 3DWarehouse. Example results are shown in
Figure 1.
Why is this task important? First, nonphotographic depictions
are plentiful and comprise a large portion of our visual record.
We wish to reason about them, and aligning such depictions to
reference imagery (via a 3D model in this case) is an important
step towards this goal. Second, such technology would open up a
number of exciting computer graphics applications that currently
require expensive manual alignment of 3D models to various forms
of 2D imagery. Examples include interactive visualization of a 3D
site across time and different rendering styles [Debevec et al. 1996;
Levin and Debevec 1999], model-based image enhancement [Kopf
et al. 2008], annotation transfer for augmented reality [Snavely
et al. 2006], inverse procedural 3D modeling [Aliaga et al. 2007;
Musialski et al. 2012], or computational rephotography [Rapp 2008;
Bae et al. 2010]. Finally, reliable automatic image-to-3D model
matching is important in domains where reference 3D models are
often available, but may contain errors or unexpected changes (e.g.,
something built/destroyed) [Bosch´e 2010], such as urban planning,
civil engineering, or archaeology.
The task of aligning 3D models to 2D nonphotographic depictions
is extremely challenging. As discussed in prior work [Russell