Most recent digital camera and imaging news items - RSSDigital camera and imaging news items - HTML

How to reconstruct your digital picture flops

Every once in a while, we all have shot a beautiful digital picture that turned out to be spoiled by some unwanted objects, when checking the harvest of a shooting day at the computer monitor in the quiet of our home. James Hays and Alexei Efros of Carnegie Mellon University have come up with a technique that can recreate these images to the way we intended them to be. They have developed an algorithm that patches up areas in these images by searching for a replacement area in similar images in a large image database...
Other interesting articles and the Carnegie Mellon Graphics website
"Scene Completion Using Millions of Photographs"


What can you do with a million images? In this paper we present a new image completion algorithm powered by a huge database of photographs gathered from the Web. The algorithm patches up holes in images by finding similar image regions in the database that are not only seamless but also semantically valid. Our chief insight is that while the space of images is effectively infinite, the space of semantically differentiable scenes is actually not that large. For many image completion tasks we are able to find similar scenes which contain image fragments that will convincingly complete the image. Our algorithm is entirely data-driven, requiring no annotations or labelling by the user. Unlike existing image completion methods, our algorithm can generate a diverse set of results for each input image and we allow users to select among them. We demonstrate
the superiority of our algorithm over existing image completion approaches.

1 Introduction

Every once in a while, we all wish we could erase something from our old photographs. A garbage truck right in the middle of a charming Italian piazza, an ex-boyfriend in a family photo, a political ally in a group portrait who has fallen out of favor [King 1997]. Other times, there is simply missing data in some areas of the image. An aged corner of an old photograph, a hole in an image-based 3D reconstruction due to occlusion, a dead bug on the camera lens. Image completion (also called inpainting or hole-filling) is the task of filling in or replacing an image region with new image data such that the modification can not be detected.

There are two fundamentally different strategies for image completion. The first aims to reconstruct, as accurately as possible, the data that should have been there, but somehow got occluded or corrupted. Methods attempting an accurate reconstruction have to use some other source of data in addition to the input image, such as video (using various background stabilization techniques, e.g. [Irani et al. 1995]) or multiple photographs of the same physical scene [Agarwala et al. 2004; Snavely et al. 2006].

The alternative is to try finding a plausible way to fill in the missing pixels, hallucinating data that could have been there. This is a much less easily quantifiable endeavor, relying instead on the studies of human visual perception. The most successful existing methods [Criminisi et al. 2003; Drori et al. 2003; Wexler et al. 2004; Wilczkowiak et al. 2005; Komodakis 2006] operate by extending adjacent textures and contours into the unknown region. This idea is derived from example-based texture synthesis [Efros and Leung 1999; Efros and Freeman 2001; Kwatra et al. 2003; Kwatra et al. 2005], sometimes with additional constraints to explicitly preserve Gestalt cues such as good continuation [Wertheimer 1938], either automatically [Criminisi et al. 2003] or by hand [Sun et al. 2005].

Importantly, all of the existing image completion methods operate by filling in the unknown region with content from the known parts of the input source image. Searching the source image for usable texture makes a lot of sense. The source image often has textures at just the right scale, orientation, and illumination as needed to seamlessly fill in the unknown region. Some methods [Drori et al. 2003;Wilczkowiak et al. 2005] search additional scales and orientations to gain additional source texture samples. However, viewing image completion as constrained texture synthesis limits the type of completion tasks that can be tackled. The assumption present in all of these methods is that all the necessary image data to fill in an unknown region is located somewhere else in that same image. We believe this assumption is flawed and that the source image simply doesn't provide enough data except for trivial image completion tasks.

Continue to read the complete article (11MB pdf file) with some very explicatory sample images…

Additional information: Other interesting articles and the Carnegie Mellon Graphics website
August 8, 2007
go to top of page