Adobe Firefly Video Model Introduced – Generative Extend, Image-to-Video, AI-Powered Dubbing

Artificial Intelligence

by Mascha Deikova6 days ago

Earlier this year, Adobe shared a sneak peek into their research on generative AI for video. They showed us the first tests, which included removing disturbing elements from shots, filling in gaps in the timeline with generated footage, and other “magical” tricks. Now it is finally official: the new Adobe Firefly video model will join the company’s family of generative AI later in 2024. What are we to expect? Read below!

AI image generator Adobe Firefly is quite popular among creators for several reasons. First, developers use training data to which they have legal rights, which makes their software commercially safe and more ethical than similar AI tools from competitors. Second, lots of Firefly features are directly integrated in Adobe’s software: For instance, AI vector generation in Illustrator or Generative Fill in Photoshop.

What Adobe hasn’t managed so far is to release Firefly for video, although we know the company’s developers have been working on it for the past couple of years. Will the news from IBC 2024 finally break the long waiting phase?

Adobe Firefly Video Model: a reminder

In the sneak peek during this year’s NAB, Adobe showed many AI tools for filmmakers and video creators. Among them were text-to-video and image-to-video generators, as well as some useful editing features, like adding objects, masking out disturbing elements in one click, or extending shots directly in the timeline. Head over here to read our coverage of the entire demo session.

Fast-forward to now, and the company announces that the Firefly Video Model is ready for beta and coming to the market. According to Adobe’s press release, the first AI-powered features for video will at least include the so-called Generative Extend and creating videos from text and image prompts.

For instance, potential uses could include leveraging fine-grained controls like camera angle and camera motion to produce compelling animations, effects, and more; generating an insert into a key video shot with camera motion; pairing video with professionally captured footage to tell stories faster.

A quote from the Adobe’s press-release 

Where will we see these tools powered by Adobe Firefly?

However, there is still no official launch date. As Adobe’s developers mentioned in the press presentation, it’s more important for them to get the AI-powered tools right than to get them out quickly.

According to the announcement, the new Firefly Video Model will be available “later this year.” After the launch, text-to-video and image-to-video generators can be tested on Firefly.Adobe.com, and Generative Extend will be embedded directly into Premiere Pro. When other previewed functions (like object removal) will go live – remains unknown.

Join the waitlist here to stay updated and be notified when the Firefly Video Model is available in beta. 

AI dubbing and lip sync

Apart from the features already demonstrated, Adobe also announced some new Firefly-powered tools for their enterprise services. The most impressive among them is the AI dubbing and lip sync. This feature allows users to translate spoken dialogue in video into different languages while preserving the original speaker’s voice and ensuring lip sync.

The service is already available for enterprise customers as an API in beta for seamless integration into existing workflows and as a web application. We haven’t tried it yet, but the demo presentation during Adobe’s live event was shockingly persuasive.

What do you think about the Adobe Firefly video model? Are you waiting for it to be released? What functions seem the most interesting and useful to you? Share your thoughts with us in the comments below!

Feature image source: Adobe.