Massively Enhanced Timelapses with NVIDIA’s New AI Technology

Artificial Intelligence

by Mascha DeikovaFebruary 27^th, 2023

Whether for music videos, features or documentaries, timelapses are a well-known and widely-used tool for capturing change in motion. Though with random flickering, objects popping up out of nowhere and other so-called artifacts, they are anything but flawless – well, at least, they were, according to researchers from Aalto University in Finland. Together with NVIDIA, they developed a new AI technology for massively enhanced timelapses.

There are three key elements to a visually compelling timelapse: camera, tripod and a whole lot of patience. Seems easy at first sight, doesn’t it? However, if you have ever captured a dynamic process yourself, you know how immensely tricky it can be. In one of our previous articles, we explained setting the right time intervals during the shoot. But what if the goal is to observe something over days, months, or even years? The new generative model from NVIDIA can become an irreplaceable assistant. Why and how? Let’s take a closer look.

Solving the flickering problem through a new deep-learning model

When it comes to creating timelapses, NVIDIA wanted to kill several birds with one stone. First, reduce flickering which can happen easily if you capture long timelapses, e.g. multi-year video observations. Second, control changes and clear up the most annoying ones from the timelapse, such as drastic weather development, a random person walking through the frame, or a spider web that’s suddenly covering the camera lens – all the unexpected things a filmmaker faces.

And indeed, they did it. In a thorough video presentation, one of the researchers, Erik Härkönen, demonstrates how rough and messy input data from a webcam magically transforms into a smooth and stunning timelapse. A chosen landscape explores different seasons in all its beauty, and we are able to witness it without flickering. In the research documentation, you may read all the technical details on how NVIDIA’s AI was trained and tested.

Adjusting your timelapse footage afterward

It goes even further. The deep-learning model also allows us to alter the footage afterward. You know, an oldie but goodie “fix it in post”. Depending on your artistic goal, you control when the changes take place in the video – weather, time of day, and even the time of year.

Another example: let’s say over a couple of years you watch a tree grow in the foreground of your timelapse. Using AI, you can now decide whether you want to see this process in detail, or if you want to fix the growth in time. The latter option will prevent the tree from hiding the beautiful landscape behind it.

Achieving quality even with missing data

The NVIDIA team’s approach is based on Generative Adversarial Networks (GAN), which basically means that their AI is good at generating plausible data – imagine long timelapses. How often are there days or weeks of missing footage? The video material is usable regardless, as it usually shows a wide timespan very quickly. But now, NVIDIA’s AI is capable of synthesizing everything that’s in between, further enhancing the final video.

Creating timelapses with NVIDIA – results

So here it is – the new way to edit your timelapse. Making videos smooth while also showing the essence of the changes. Compared to previous research attempts, this model from NVIDIA can control weather and flickering, and NOT wash out and of the interesting details. If you want to try it, the source code and pre-trained models are available to download on GitHub. You may need some experience and expertise in AI training though.

The featured image is from the paper “Disentangling random and cyclic effects in time-lapse sequences”. Credit: Aalto University, Finland and NVIDIA

Another source used: Two Minute Papers

It’s incredible how far AI-based tools have already come to improve the quality of videos. Or is it? As a commentator on YouTube pointed out, when we make everything perfect on a whim, it starts losing its meaning. That comment made me think about the beauty of imperfection. What are your thoughts on the topic? In the future will you use this technology from NVIDIA to create your timelapses? Let us know in the comments below.