First-person videos become smooth hyperlapses thanks to Microsoft

News

by Nino LeitnerAugust 13^th, 2014

Microsoft is hardly known as a player in the video and film industry, but their Research division turned out some pretty spectacular image processing innovation before (incorporated into Bing Maps).

Now they have found a way to make point-of-view / first-person video more watchable. An inherent problem of the typical GoPro strapped to your head is that humans do not move steadily – it’s our brains that make us think that we move smoothly, in fact there is a lot of shake (hence the need for handheld gimbals as pioneered by Freefly Systems with the MoVi). That means that our POV-GoPro shots almost NEVER look anything like the GoPro marketing department makes us think it would look like.

In comes Microsoft with the demo of a yet-unpublished software that can not only de-shake these first person videos, but actually make them extremely smooth. In their demo video below, they demonstrate the different looks of the input material, the sped-up timelapse version of that (unbelievably shaky) and the Microsoft hyperlapse version after processing.

The downside is that it only seems to work for sped-up (i.e. timelapse) versions of those point-of-view videos, not the realtime recordings, which would certainly be more useful in day-to-day use (I am actually currently personally deeply involved in a large first-person project that would greatly benefit just from that!). However, the results of Microsoft’s hyperlapses are nothing short of amazing: Using some kind of 3D camera path mapping, the route becomes much smoother and actually seems to reconstruct footage at the edges. They also show what normal stabilization looks like on that same footage, and it doesn’t come even remotely close.

They are working on putting all of this goodness into a Windows app (yeah, I know … come on, it’s 2014, please give us a Mac app too!).

Until then, head over to their Microsoft Research page where you can download the technical paper, supplemental material and a high-res video demo.

From their site:

We present a method for converting first-person videos, for example, captured with a helmet camera during activities such as rock climbing or bicycling, into hyper-lapse videos, i.e., time-lapse videos with a smoothly moving camera. 

At high speed-up rates, simple frame sub-sampling coupled with existing video stabilization methods does not work, because the erratic camera shake present in first-person videos is amplified by the speed-up. 

Scene Reconstruction

Our algorithm first reconstructs the 3D input camera path as well as dense, per-frame proxy geometries. We then optimize a novel camera path for the output video (shown in red) that is smooth and passes near the input cameras while ensuring that the virtual camera looks in directions that can be rendered well from the input.

Next, we compute geometric proxies for each input frame. These allow us to render the frames from the novel viewpoints on the optimized path.

Proxy Geometry

Stitched & Blended

Finally, we generate the novel smoothed, time-lapse video by rendering, stitching, and blending appropriately selected source frames for each output frame. We present a number of results for challenging videos that cannot be processed using traditional techniques.