Artificial Intelligence is not just an overhyped set of keywords buzzing around the internet. It’s also a rapidly evolving new system of tools that are already changing the industry immensely. Putting aside the worrying question of whether neural networks will take over our jobs, we’d rather talk about how filmmakers can use this technology to their advantage. Let’s start with image generators like Midjourney and Stable Diffusion for creating mood boards with AI tools!
I’m sure every one of you has tried to generate an image from text using AI at least once. There are dozens of them out there: from the famous DALLE-2 to Deep Dream Generator and Nightcafe, just to name a few. It’s easy to get lost in the ocean of opportunities, so in this brief overview, I’ll show you a couple of selected tools that don’t require coding skills and that I myself use at the beginning of a new video project.
Midjourney – AI-based tool for the first creative ideas and mood boards
Midjourney is a small independent research lab with a huge community. Their AI creates images from text descriptions via so-called “prompts” and operates online through the simple Discord bot. They are still in the beta phase which means Midjourney constantly brings new features to the table. Probably, even while I’m writing this text.
After you’ve created a Discord account and joined their beta, you will be redirected to the Midjourney server and can get started. Just go over to one of the newbies’ channels and type in “/imagine” followed by a text description of what you want to have visualized.
Adventure Filmmaking with Russ Malkin
How do you combine filmmaking with traveling? How do you make sure you can come home in one piece? And how do you turn it into your job? Join Russ Malkin on Adventure Filmmaking.
In my opinion, Midjourney is perfect for boosting creativity at the very start of the project and for turning your ideas into mood boards. For example, I have a story in mind about a small girl who has been lost in the forest. Suddenly, she comes across an empty mystically-looking camper van. I don’t know what “mystically” precisely means though. So, I go over to Midjourney for some sparks of inspiration and type in my rough ideas. By the way, you can also set up the aspect ratio for generated images by adding “–ar 16:9 (2:1, 9:16, etc.)” at the end of your prompt. In addition to this, their machine learning system is aware of different artists and filmmakers, so you can use the styles of your favorite ones.
What you get after only mere seconds of waiting is four different image variations based on your wording. Now it’s possible to develop and upscale each of them several times until you get the desired results. Or leave it as is and adjust your prompt to get a completely new one. In my opinion, it’s worth a try out if you’re at a loss for concept ideas.
In my case, I already had some visuals in my head. So, it took me only 10 minutes of playing around with Midjourney to put together a decent mood board for a possible short film project. Impressive, right?
A lesser-known feature: combining different images and altering them
But what about something more down to Earth? Let’s say, we’re about to shoot a commercial and currently looking for a creative setup to present our product. For example, we want to sell an idea of putting the new Panasonic LUMIX S5 II we recently reviewed here into a wildlife habitat for the product shot. Just to underline its powerful autofocus function.
A lesser-known feature of Midjourney provides us with the possibility to feed it one or more images and then combine them as we like. To try it, upload your references directly into the chat. Use the same “/imagine“ prompt field and paste the links to your pictures adding some text if needed. There is also a new “/blend” command now which is even simpler to use, but it doesn’t allow adding text notes yet. 50 seconds, and voilà! The first pitch image for the client is ready to be sent (don’t forget to mention it’s only a rough concept though).
To learn more about other features and commands, please head over to the Midjourney user guide.
Downsides of Midjourney
- The free Midjourney package has a restricted amount of AI jobs (0,4 hr of GPU time or around 20 prompts). Also, in the trial version, you have to look for your generated images in endless Discord chats alongside other users. In paid subscription plans you can chat directly with Midjourney Bot.
- Midjourney creates beautiful but not very precise images (say hello to the weird tiger from the image above!). You cannot influence the results directly, except by changing the wording which could get annoying.
- It’s addictive. Once you start, you’re in danger of falling into the rabbit hole and spending the whole night generating new visuals.
Stable Diffusion for precise visualizations of your concept
Another deep learning, text-to-image system interesting for filmmakers is called Stable Diffusion. Compared to Midjourney, it’s available as an open-source project, it’s completely free of charge, and it can also work on the GPU of your computer. There are different models based on Stable Diffusion that can be downloaded and used offline: from the ones trained to generate dog images to the ones which allow you to create comic strips. For the sake of this experiment, we will be using one similar to Midjourney called Dreamlike Diffusion 1.0. (Mac OS users can try Diffusionbee).
It can do everything that Midjourney does, even if I find its generated visuals somehow less artistic. So, let’s skip that part and dive into what Stable Diffusion is much better at – precision and flexibility.
Their image generator offers a great variety of parameters and custom settings to play with. That way you gain much more control over the final result. For example, it’s possible to alter only a selected area of your image with the “Inpaint” function. Let’s say, I didn’t like how the girl’s character from my first test turned out so I uploaded that image to Stable Diffusion and asked it for help.
Turning pictures from your head into concept art
Another great way to use Stable Diffusion is to transform a quick & dirty collage into a piece of concept art. When you have a concrete idea in your mind and just need it to be visualized precisely, try the following: put together some images in any graphic design program (even in Paint). Feed it to the neural network, add a text description of what you want to change, and play with different settings. Enjoy the results.
Advanced users can also train the AI on photo portraits of themselves or their actors and use the created character models to make realistic storyboards. Still, even mentioning all that I’ve just scratched the surface of what Stable Diffusion is capable of. The best way to explore its possibilities is to try it out.
Downsides of Stable Diffusion
- The biggest disadvantage of Stable Diffusion is its complexity. You will need a hell of a time to get into the process and to understand how everything works. But once you’ve gone through it, this AI can increase your productivity and become a powerful tool to communicate your vision.
- Stable Diffusion is trained on a dataset that consists of 512×512 resolution images (768×768 in updated version 2.0). The quality of generated pictures noticeably degrades when you change image dimensions from the “expected”. This limitation can be overcome by AI-based upscaling tools.
- If you run Stable Diffusion on your hardware, it’s recommended to have a GPU with at least 8 GB VRAM and enough space on your hard drive (2 to 4 GB pro downloaded model). Diverse services also offer Diffusion models online but you either have to pay for them or they are much more complicated to use than the offline interface.
Conclusion
Visual AI-powered applications can become very useful tools in the filmmaker’s kit. Midjourney is wonderful for enhancing the creative process at the beginning of the concept phase. Stable Diffusion helps to visualize the precise image from your head for a pitch or a storyboard.
There are also neural networks like KIVE AI which structures libraries of images & moods. Using deep learning it divides pictures into categories and labels each of them with detailed metadata. That way, you no longer need to organize your creative folders yourself.
Of course, there is much more to it. I hope that this article gives you a taste of helpful applications for creating visual mood boards with AI tools quickly and easily. In the following overviews, we’ll look at other neural networks that can support you in scriptwriting, previs, and postproduction processes. Stay tuned!
Featured Image: created with Midjourney by CineD
What about you? Have you already used these or other visual AI tools for your video projects? What other great applications do you find useful in your workflow? Let us know in the comments below!
