Real-Time Mixed Reality Rendering for Underwater 360° Videos
Learn about the Underwater Toolkit, a mixed reality toolkit that enables seamless blending of virtual objects into underwater 360° videos in real time.
![ismar19-paper-img](https://www.wgtn.ac.nz/__data/assets/image/0003/1771860/ismar19-paper-img.png)
Abstract
We present the Underwater Toolkit, a mixed reality (MR) toolkit that enables seamless blending of virtual objects into underwater 360° videos (360-video) in real time. It is fully integrated into commercial game engines such as Unity3D and Unreal Engine 4 (UE4), providing a complete pipeline for underwater 360° videos. The toolkit provides real-time underwater lighting (caustics, god rays, fog, and particulates) to ensure that the virtual objects are lit and blend similarly to each frame of the underwater video semiautomatically and in real time. Our toolkit’s user-friendly interface enables users to fine-tune the underwater lighting parameters so they can match the lighting observed in the 360-video for improved visual quality and seamless blending. In our demonstration, users will be able to immerse themselves into underwater 360-videos using a HMD. Using motion controllers, the users will be able to interact with fish by feeding or catching them. An additional user will be able to interact with our toolkit, changing underwater lighting parameters to seamlessly blend fish into the 360-video in real time.
![ieeevr19-poster-img](https://www.wgtn.ac.nz/__data/assets/image/0011/1771859/ieeevr19-poster-img.png)
![ismar19-demo-img](https://www.wgtn.ac.nz/__data/assets/image/0010/1771858/ismar19-demo-img.png)
Publication and authors
ISMAR 2019 paper "Real-Time Mixed Reality Rendering for Underwater 360° Videos"
Stephen Thompson, Andrew Chalmers and Taehyun Rhee (Computational Media Innovation Centre, Victoria University of Wellington, NZ)
ISMAR 2019 demo "Underwater Toolkit: Mixed Reality Object Blending for 360° Videos"
Stephen Thompson, Andrew Chalmers, Daniel Medeiros and Taehyun Rhee (Computational Media Innovation Centre, Victoria University of Wellington, NZ)
IEEEVR 2019 poster "Real-time Underwater Caustics for Mixed Reality 360° Videos"
Stephen Thompson, Andrew Chalmers and Taehyun Rhee (Computational Media Innovation Centre, Victoria University of Wellington, NZ)
Acknowledgments
This project was supported by the Entrepreneurial University Programme funded by TEC and in part by the Smart Ideas project funded by MBIE in New Zealand. We thank Boxfish Research for providing 360-videos captured with their underwater 360° camera.