AI Tools for Filmmakers – Recap and Trends of 2024

Artificial Intelligence

by Mascha DeikovaToday

In 2023, we had just begun touching the surface of generative AI’s capabilities, baffled and excited that suddenly robots could draw pictures and converse in humanlike language. This past year, on the other hand, was a wild ride at breakneck speed (in every way) – a myriad of emerging AI video generators for creating photorealistic moving images (with weird artifacts, but still), new enhanced post-production capabilities, full AI-based production applications, and so on. Let’s look at what 2024 brought us regarding AI tools for filmmakers and some current trends in this area.

Over the past years, we have tried our best to keep you updated on AI-enhanced tools and developments relevant to our industry. And boy, did we receive backlash and get criticized for it! However, we still believe that filmmakers should keep up with technology updates and at least become familiar with what it is capable of nowadays. So, here it is, your short recap “Previously on AI!”

The era of AI video generators

The beginning of the year was marked with one announcement after another. Following Runway’s lead, companies (from small startups to tech giants) started to present their own text-to-video generators. The first “big news” came from Midjourney, an already well-known AI for image creation (however, their promised video generation tool still hasn’t been released). Then, there was Google with Lumiere (which became Google’s Veo later in the year), OpenAI with Sora, which caused a lot of buzz in February (and was released to the general public only a few weeks ago), Luma AI’s Dream Machine, and the list can go on and on.

It seems no company with its own AI research can stand aside and restrain from participating in this generative AI race. Thus, even Amazon recently presented the so-called Nova, a variety of generation models, including Nova Canvas for images and Nova Reels for videos.

With the ability to process text, image, and video as prompts, customers can use Amazon Nova-powered generative AI applications to understand videos, charts, and documents, or generate videos and other multimedia content.

A quote from the anouncement

According to Amazon’s announcement, you can see a “Pasta City” above. This is a short video advertisement created by Amazon Ads using the freshly launched text-to-video generator. With this example, the company aims to showcase innovative approaches for advertisers. To be honest, it seems scary. I can vividly imagine how product pages on Amazon will soon be drowned in artificial content (we will talk about it below).

AI-enhanced software for film production

However controversial the topic of generative AI may be, its development does bring some useful AI tools for filmmakers. Last year, we highly praised AI language understanding for bringing automated transcripts and captions into the editing software. This year, the developers have taken it a step further. They started to augment applications, tailored specifically for filmmakers, with AI-enhanced functions.

One such highlight of 2024 has been PRODUCER – Maker Machina, an all-in-one software that tackles shoot production topics with automated solutions. For instance, once you have your storyboard filled in with reference pictures and important notes on shots, the application will automatically create a shot list for each day based on the information that you provided. After you connect the actors and add your crew members for this project, it will generate a decent call sheet. The main idea behind such tools is the following: if we use AI to take over mundane, repetitive tasks, we gain more time for creative processes, which is not a bad idea per se.

Another example for digging deeper into the AI nerve is the shift by companies previously focused on a single niche to now incorporating more automated tools and features for filmmakers. For instance, Artlist. We all know it as a stock library with music and video clips. In 2024, they announced an AI-powered search (which was somewhat to be expected). But then, Artlist also presented an AI text-to-speech generator, expanding their offer. My assumption is that stock companies try to keep up and rapidly think of new tools because they are also afraid of generative AI’s capabilities and quick development. After all, this past year literally buzzed with new music generators that can create coherent songs (even with lyrics!) that won’t replace original composers (in my opinion) but can surely replace stock melodies over time.

Multi-modal emerging AI tools for filmmakers

A further trend we’ve observed developing in the generative AI tools for filmmakers in 2024 is multi-modality. Nowadays, it’s not only about which software can generate a better image, a more realistic voiceover, or a less troubled video clip. It’s also about combining those for the filmmaker’s convenience to achieve a faster result. The most prominent example from the past year is LTX Studio. This generative AI software is designed to handle everything from storyboarding and visualizing your concept to generating music and voice-overs. Our Johnnie met with the Lightricks company’s CEO and Co-Founder, Zeev Farbman. They discussed not only the software’s potential but also the negative impact it may have on video creators:

Announced play’s Holly is also an example of multi-modality but aimed at video editors. Their software, which is still in the testing phase, can create a rough cut based on your text prompt. For that, the tool uses machine learning to analyze delivered video clips. Afterward, the neural network provides them with detailed metadata and transcribes speech (understanding almost 100 languages). Sure, many editors don’t want anyone (or anything, for that matter) to take over their rough cut. Yet these functions alone in one software are pretty useful, aren’t they?

Photorealism and abundance

A trend that we can’t look past is that companies developing and training generative AI try to achieve results that are as close to reality as possible. Meaning: images that look like photos, video clips that could have been shot on a smartphone, and voices that can trick you into thinking they are real people. All that brings along a troublesome tendency that, more often than not, makes it hard to distinguish real content from generated. That’s another reason why we should educate ourselves and the people around us on AI. Knowing their capabilities can amplify critical thinking and remind us to surf more cautiously. Don’t believe everything you see, read, or hear. This has always been an important tip for Internet users, but now more than ever.

At the same time, we notice (as you probably do as well) that social media is flooded with AI-generated content. There is so much of it that it no longer inspires or excites. People are already fed up with neural network creations, and this trend will worsen. Thus, human-made art with unique ideas and insights starts to become scarce. That’s why I believe it will stay valuable, even though it is often more expensive and time-consuming.

Massive backlash towards AI tools for filmmakers

Last but not least, this year, we’ve seen how filmmakers react to AI and its development. People are scared of losing their jobs to the machines. They are furious that deep-learning models scrap the Internet and use their art for training without any residuals or attributions. On top of that, they are angry at huge corporations and companies that prefer neural networks to humans and at governments for the lack of restrictions and regulations. There are a lot of unresolved ethical questions and, above all – a massive fear behind all the criticism and hate that we get from our articles explaining AI tools.

We don’t say that this fear is unreasonable. Neither do we say that there are no consequences of generative AI development for filmmakers (Johnnie published a comprehensive guide on the options that we have nowadays and which may help to overcome the general panic –please read it). What we are saying is that the point of no return has already been reached. We can’t stick our heads in the sand, pretending that the generative AI revolution hasn’t happened. Instead, we should gain knowledge of it and implement the tools that will help us augment and speed up our processes but not replace our creativity. And develop our craft even more. Now, anyone can tip a few words into a text field and generate a video clip. But how many of these people understand visual storytelling, camera language, the power of rhythm in editing, the added level of narration in the score, and the importance of the smallest nuances in the voice and acting? A good question, indeed.

What about you? What trends in AI tools for filmmakers have you noticed over the past year? How do you react to this development? Share your thoughts with us in the comments below!

Feature image: generated with Midjourney for CineD.