Live Inverse Rendering

light transport components of a scene - geometry, reflectance and illumination. While the theory
and practices for estimating environmental geometry and self-localization on mobile devices has
progressed rapidly, the task of estimating scene reflectance and illumination from monocular
images or videos in real-time (termed live inverse rendering) is still at a nascent stage. The
challenge is that of designing efficient representations and models for these appearance parameters
and solving the resulting high-dimensional, non-linear and under-constrained system of equations at
frame rate.
This thesis talk will comprehensively explore various representations, formulations, algorithms and
systems for addressing these challenges in monocular inverse rendering. Starting with simple
assumptions on the light transport model –of Lambertian surface reflectance and single light bounce scenario –
the talk will expand in several directions by including 3D geometry, multiple light
bounces, non-Lambertian isotropic surface reflectance and data-driven reflectance representation to
address various facets of this problem. In the first part, the talk will explore the design of fast
parallel non-linear GPU optimization schemes for solving both sparse and dense set of equations
underlying the inverse rendering problem. In the next part, the current advances in machine learning
methods will be applied to design novel formulations and loss-energies to give a significant push to
the state-of-the-art of reflectance and illumination estimation. Several real-time applications of
illumination-aware scene editing, including relighting and material-cloning, will also be shown
to be possible for first time by the new models proposed in this thesis.