Must-Have AI Tools for Filmmakers in 2023

Artificial Intelligence

by Mascha DeikovaDecember 28^th, 2023

The Collins Dictionary named AI the word of 2023. Rightly so! The development of artificial intelligence and its ever-growing popularity burst to a new level this past year. Personally, I don’t know anyone who hasn’t heard about ChatGPT or the various text-to-image generators. The discussion around the ethics of using this technology is also extensive. Yet, deep-learning models have much more to offer than just funny fake pictures of movie stars or generated texts for social media. Here, you will find some filmmaker’s go-to AI tools from 2023 that can enhance your workflow and make your job easier.

A short heads-up first. Some other dictionaries (like Cambridge Dictionary) picked “Hallucinate” as the Word of the Year 2023. It’s also related to AI and recently acquired a new meaning, in addition to the traditional one.

When an artificial intelligence (= a computer system that has some of the qualities that the human brain has, such as the ability to produce language in a way that seems human) hallucinates, it produces false information.

A quote from Cambridge Dictionary

So, use AI tools with caution and don’t believe everything its artificial brain generates. In the shortlist below, you will find the AI tools I personally apply in my filmmaking routine and find quite useful. However, it doesn’t mean they are flawless.

AI tools from 2023 for simplifying video production

While 2022 was all about AI image generators that could suddenly turn simple words into beautiful artwork, 2023 finally provided an answer to the question: why can’t we do the same with videos? In spring, the New York-based company Runway rolled out a deep-learning model Gen-1 (video-to-video approach), followed a couple of months later by Gen-2 (text-to-video and image-to-video).

Runway’s software works directly in the browser and is very user-friendly. They offer a free plan with a limited amount of generations, in case you want to test it first. After signing up, you can choose between one of their so-called AI magic tools and start creating. Gen-1 allows you to change the style of your source video (for instance, into a fully cartoonish look), whereas Gen-2 is capable of generating clips from scratch. To try it out, click “Start with Text” and provide a detailed description of the scene you aim to create. The “Preview” button will show you four still images based on your text prompt. Once you make a selection and proceed, it will serve as the initial frame for the generated video.

As you see in my example above, the clip only goes for 4 seconds, which is the current limitation for these AI models. The default resolution of the generated video is 768×448. You can upscale it later with Runway’s paid subscription or with other AI upscalers (like the well-known Topaz Labs).

At the moment, I can’t imagine using the Gen-2 video results in a commercial edit alongside other shots. If you try it in different scenarios, you will often get quite a sloppy output, to be honest. Yet, this model will undoubtedly continue to develop. Over time, its results just might replace stock footage, if only because you can get more precise visualizations of whatever shot you need, and much faster.

Runway also offers other AI tools for filmmakers worth looking at, like Motion Brush (which adds naturalistic and consistent motion to a still image), or artificial slow-motion. Head over here to get an overview of all their products.

Luma AI for location scout and shot previsualization

My favorite AI tool from 2023 is hands down Luma AI. It’s something I recommend that every filmmaker installs on their phone. Why? Because it’s super useful for location scouting and building previsualizations.

Based on the so-called NeRF technology, Luma AI creates photorealistic, high-quality 3D models out of simple videos in an incredibly short time. And that, directly on your smartphone! Picture stumbling onto a perfect location or having to scout it alone. Make a quick video (with any phone or camera you have on hand), upload it to Luma AI, and let its artificial powers turn it into a detailed 3D environment (which can then be exported into the 3D software of your choice). If you have an iPhone 11 or newer, you can even capture video directly in Luma’s app, following pop-up tips for better results.

Not only does this scan offer a comprehensive 360-degree look at a possible location for your DP, or production designer, but it allows you to animate the camera directly in the app. Using simple keyframes, we can previsualize any shots in a given location and export them with no third-party applications needed. As Luma AI’s technology also flawlessly recreates the lighting conditions, it can be an ideal tool for quickly previsualizing scenes in a selected space.

You can use Luma AI at no cost, with a small exception for the “video to API” feature, which comes at a nominal fee of $1 per capture. Although I doubt that you will need to integrate Luma’s photorealistic 3D NeRF models into a different application, even if that situation arises, the cost is relatively low.

Face morphing with Picsi.Ai for storyboards and deep fakes

Another AI tool that emerged in 2023, and that I found particularly impressive, is called Picsi.Ai. No doubt you have already heard of deepfakes. If not, this is a technology that allows filmmakers to convincingly replace faces in movies (for example, to make an actor appear younger, or completely change their appearance).

Normally, to make a decent deepfake, you have to train a deep-learning model using hundreds of photos (at least for Hollywood standard results). Yet, in 2023, another technology emerged – the so-called “face morphing” – and I was introduced to it with the help of Picsi.Ai.

This tool is still in beta (as most such state-of-the-art techniques are) and operates through a Discord bot. What it does is seamlessly superimpose a selected face onto a target picture or video gif, akin to a deepfake. The key distinction is that only a single source image is required. The rest happens thanks to the AI’s astonishing guessing skills. Just look at my result below:

It looks like fun, but at the same time, this tool can assist you with several tasks. For example, you can create shots for storyboards in one of the image generators (such as Adobe Firefly) and then employ face morphing to ensure a consistent appearance for your main character. Or, as Picsi.AI lets us apply their technology to short gifs, we can also use it for previsualizations (or story reels, as some call them).

Anyway, you can try it out for free. Every day you receive 50 credits, which equals 50 face morph generations. To start using this AI tool, create a new server on Discord and invite the “InsightFaceSwap” bot to your server by clicking here. Now, you will need to remember only two commands:  /saveid Name to register a source photo (free plan enables up to 20), and /swapid Name plus uploaded image or gif to morph faces with one of your registered identities.

Other AI tools from 2023 worth looking into

2023 was full of AI launches, so it was for sure not easy to keep up. We always try our best to keep you on track, so here are some other handy tools you might have missed (and definitely shouldn’t):

• Audio Enhancer from Adobe, which is already integrated into Premiere (Beta), or can be accessed in the browser. It’s an AI tool that can turn an audio of poor quality into one with studio sound, no kidding. We wrote about it here, along with other AI tools for audio postproduction. By the way, DaVinci Resolve has a similar AI feature called “Voice Isolation”;
• Automated captions and text-based editing are integrated into the post-production software of your choice. Read how it works here;
• AI-based stock music search for those who are tired of browsing through hundreds of melodies to find a perfect tune;
• Colourlab’s Freelab plugin, a zero-cost addition for Adobe Premiere Pro, After Effects, or DaVinci Resolve, can convert any shot (even of a completely unknown origin) into REC709 with a single click.

And what are your favorites? Which new AI tools from 2023 became a normal part of your workflow? Are there any applications or deep-learning models that you’re waiting to be released in 2024? Let’s talk about technology in the comments below!

Feature image source: generated with Midjourney V6 for CineD