The big picture: researching coherent video generation
Modern camera technology means that anyone can capture high-quality videos of events as they occur, from a street performance to the sun setting over the horizon or a colourful splash of flowers in the park. Sharing videos of such everyday events on social media has evolved into an integral part of many people’s lives. However, the photographer may not have the skills required to capture the event in its full range.
ECS lecturer Dr Fanglue Zhang’s research focuses on coherent video generation – where videos with dynamic foreground objects can be combined, so that the captured videos are spatially and temporally complementary to each other. “Imagine you’re walking down the street and stop to watch a group of dancers. By being on one side of the street, you’re only able to capture one perspective. But that’s true of 20 other people who are spread all around the street. They each get a different perspective. How can we seamlessly combine all the videos, so that everyone gets to see the performance in its entirety? That’s the focus of my research,” he explains.
Keen on identifying how to offer better software and methods for novice users of digital media software, Fanglue chose the area of image-based computer graphics because technologies related to image and video editing, enhancement and rendering have emerged as a key research field in modern computer graphics. In fact, the idea for this direction of research came from Fanglue’s own experience. “I was walking along the waterfront and I saw a performer – he was on skates and playing the violin at the same time. People all around him were trying to capture it with their cell phones. But he was moving really fast – naturally, some people kept losing him from their frames. That’s when I began thinking – what if there was a tool to generate a coherent video focusing on the object in the foreground, based on inputs from the various cameras?”
As part of the research, Fanglue has been collaborating with researchers from other universities and institutions, including Tsinghua University and Adobe Research, which has added significant depth to the nature of the research. “The research includes combining many techniques from computer vision and computer graphics, in order to manage the smaller problems that may arise. Maintaining a system large enough to manage and process input data like this is a huge challenge. But I see the results we’re getting from continuous effort and it keeps me going because I believe I’m on the right track.”
Originally from China, Fanglue has lived in Wellington for the last couple of years. “It’s a wonderful place to live. People are very friendly and the work-life balance is really good.”
“Wellington being such a key hub for IT, I’m happy to be doing research in the computer graphics space here. It’s exciting to constantly use new methods to create visual content. Right now, the coherent video generation tool is designed for novices who don’t have professional skills in image and video-editing. There’s definitely a possibility that it can be commercialised.
“There are so many moments from everyday lives that people capture as videos – I just want to help everyday people edit their photos and videos to a professional standard”.