MPI-I-2011-4-001
How not to be seen – inpainting dynamic objects in crowded scenes
Granados, Miguel and Tompkin, James and In Kim, Kwang and Grau, Oliver and Kautz, Jan and Theobalt, Christian
February 2011, 35 pages.
.
Status: available - back from printing
Removal of dynamic scene elements from video is an extremely challenging prob-
lem that even movie professionals often solve through days of manual frame-by-
frame editing. The disoccluded regions in the video have to be inpainted in a
coherent way, even if originally occluded objects or background are dynamic. To
make this problem easier, we propose a new approach for video
inpainting that can
deal with complex scenes with dynamic backgrounds and many non-periodically
moving occluding scene elements. It is built on the idea that a spatio-temporal
hole created by a removed scene element can be filled by copying information
from other space-time locations in the video, where objects and background are
unoccluded. Inpainting is performed by solving a combinatorial optimization
problem that searches for the optimal pattern of pixel shifts. Solving
this problem
naively, even on short videos, quickly becomes infeasible. The primary contribu-
tions of this work are a new energy functional with desirable convergence prop-
erties, an efficient hierarchical solution strategy, and an effective
search space
reduction strategy that restricts potential pixel shifts to regions
around tracked ob-
jects in the scene. A simple interface enables the user to optionally
support the
algorithm in marking and tracking dynamic objects. Our approach can efficiently
inpaint holes even in HD videos with many occlusions, and requires only little
user input.
URL to this document: https://domino.mpi-inf.mpg.de/internet/reports.nsf/NumberView/2011-4-001
BibTeX
@TECHREPORT{GranadosTompkinInKimGrauKautzTheobalt2011,
AUTHOR = {Granados, Miguel and Tompkin, James and In Kim, Kwang and Grau, Oliver and Kautz, Jan and Theobalt, Christian},
TITLE = {How not to be seen – inpainting dynamic objects in crowded scenes},
TYPE = {Research Report},
INSTITUTION = {Max-Planck-Institut f{\"u}r Informatik},
ADDRESS = {Stuhlsatzenhausweg 85, 66123 Saarbr{\"u}cken, Germany},
NUMBER = {MPI-I-2011-4-001},
MONTH = {February},
YEAR = {2011},
ISSN = {0946-011X},
}