Reconstructing Reflection Maps using a Stacked-CNN for Mixed Reality Rendering

ieee-tvcg-2020-light-prediction
Figure: Left to right: an input photograph taken by a conventional camera, the trained SCNN, the predicted 360° reflection maps (RMs) from three roughness levels in the SCNN, rendering and composition using the predicted RMs, and two more examples of an indoor and outdoor scene. Top-right virtual objects: teapot, bunny, and armadillo. Bottom-right virtual objects: park bench and traffic cone.

Abstract

Corresponding lighting and reflectance between real and virtual objects is important for spatial presence in augmented and mixed reality (AR and MR) applications. We present a method to reconstruct real-world environmental lighting, encoded as a reflection map (RM), from a conventional photograph. To achieve this, we propose a stacked convolutional neural network (SCNN) that predicts high dynamic range (HDR) 360° RMs with varying roughness from a limited field of view, low dynamic range photograph. The SCNN is progressively trained from high to low roughness to predict RMs at varying roughness levels, where each roughness level corresponds to a virtual object’s roughness (from diffuse to glossy) for rendering. The predicted RM provides high-fidelity rendering of virtual objects to match with the background photograph. We illustrate the use of our method with indoor and outdoor scenes trained on separate indoor/outdoor SCNNs showing plausible rendering and composition of virtual objects in AR/MR. We show that our method has improved quality over previous methods with a comparative user study and error metrics.

Publication and authors

IEEE Transactions on Visualization and Computer Graphics, "Reconstructing Reflection Maps using a Stacked-CNN for Mixed Reality Rendering"
Andrew Chalmers, Junhong Zhao, Daniel Medeiros, Taehyun Rhee (Computational Media Innovation Centre, Victoria University of Wellington, NZ)