Midjourney’s Vary Region Feature Challenges Adobe’s Generative Fill – Review

Artificial IntelligenceReview

by Mascha DeikovaAugust 31^st, 2023

A long-awaited news for all AI art lovers! Midjourney has recently rolled out the new inpainting function, which basically allows users to alter selected parts of their image. It is still in the testing phase, but the results are already quite impressive. Some call the update “an answer to Adobe’s Generative Fill”. Others react with the excited, “Finally!” We also tried out the Midjourney’s Vary Region feature and think it has the potential to support us in different filmmaking tasks. How so? Let’s explore together!

Midjourney is considered one of the best image generators on the market. As the developers belong to an independent research lab, they manage to release new updates and features at breakneck speed. (Just a couple of weeks ago, we were experimenting with the latest Zoom Out function, for example). Users also appreciate the precise language understanding and incredible photorealistic results of this deep-learning model.

Yet, one of the central things Midjourney lacked was the possibility to change selected areas of your image. Compared to Stable Diffusion, which had an Inpaint function from the beginning on, or Adobe’s Generative Fill, Midjourney users couldn’t adjust the details of their generated visuals. That was frustrating, but finally, this issue won’t be a problem anymore. Well, at least to some extent.

Before we dive into the tests, tips, and tricks for Midjourney’s Vary Region Feature, a heads-up. If you have never used this AI image generator before, please read our article “Creating Artistic Mood Boards for Videos Using AI Tools” first. There you will learn the basics of working with Midjourney’s neural network.

Two ways to use Midjourney’s Vary Region feature

Users can access the new feature through Midjourney’s Discord Bot, as usual. After you generate and upscale an image, the button “Vary (Region)” will appear underneath it.

When you click on the button, a new window with an editor will pop up directly from your chat. There, you can choose between a rectangular selection tool or the freehand lasso. Use one or both to select the area of your image that you want to refine.

Now you have two possibilities. The first one is to click “submit” and let Midjourney regenerate the defined part of the visual. In this case, it will try to correct mistakes within this area and get a better result according to your original text input. In my example, the neural network created new visualizations of the medieval warrior and matched it to the background.

An alternative approach to making use of Midjourney’s new Vary Region feature involves implementing the Remix mode. This way, you can change the contents of the selected area completely by writing a new prompt. You might need to enable it by typing “/settings“ and clicking on “remix mode”.

Changing the text prompt to refine your image

Once you’ve enabled the Remix mode, an additional text box will appear in the editor, which will allow you to modify the prompt for the selected region. Describe precisely what you want to see in that area. Be specific about the details you’d like to introduce or exclude (a few tips on wording follow below). Don’t worry, the AI will preserve the original aspect ratio of the root image.

As you see in the screenshot above, I decided to change the entire environment around my warrior, teleporting him from a foggy forest into an abandoned village. The results were unexpectedly good. 3 out of 4 image variations matched my description precisely and didn’t contain any weird artifacts. Check it out yourself:

Of course, a single successful test does not set a precedent, and my other experiments turned out less encouraging. However, for the tool, which came out only recently and is still in the beta test, the results seem amazing.

What’s especially great about the new Midjourney’s Vary Region feature is that it introduces flexibility. By upscaling regenerated images in between, you can improve parts of your image as many times as you need to get the desired result. Let’s say you have a specific shot in mind and you want to convey it to your cinematographer, producer, or production designer. Now, it seems possible to really get it from your head onto the paper without any drawing skills. While it may involve some trial and error, the potential is there.

Tips for the best image result

As with other neural networks, Midjourney is still learning. So, don’t expect wonders from it straight away. In order to get the best result out of the Vary Region feature, here are some tips you may follow (which combine suggestions from the AI developers and myself):

• This feature works best on large regions of the image. According to creators, if you select 20% to 50% of the picture’s area, you will get the most precise and consistent results.
• In cases where you decide to alter the prompt, the neural network will provide a better outcome if your new text matches that specific image. For example, Midjourney has no problems adding a hat to a character. Yet, if you ask it to draw an extremely unusual scenario (like an elephant in the room – pardon the pun!), the system might not give you the result you intended.
• The function also respects some of the commands and parameters working in Midjourney. So, don’t forget about the power of the “—no” command, a parameter utilized for negative prompts. This prompts the AI to eliminate the specified elements from the image.

Possible ways to use Midjourney’s Vary Region feature in filmmaking

As you probably know, I love using Midjourney for visual research, creating artistic mood boards, and preparing pitch papers for upcoming projects. The latest update will definitely simplify this conceptual task and be useful in situations when I want to communicate my specific vision. As they say, a picture is worth a thousand words.

Apart from that, you might use Midjourney’s Vary Region function to create a fast previz. The system remembers your initial selection when you return to the image after altering specific parts of it. This allows you to utilize the tool multiple times for generating diverse scenarios. Accordingly, I was able to put my warrior into different scenarios and then animate them in the form of match-cuts for a short preview of his hero’s journey. It didn’t take much time, and the video result speaks for itself:

I’m not suggesting that it will always suit the purpose, but for some scenes or sequences such a previz is enough.

Comparing inpainting in Midjourney to Adobe’s Generative Fill

What Midjourney’s Vary Region definitely lacks at this point in time is the possibility to upload your own image (or film still) and then refine parts of it with the help of artificial intelligence. This would allow us to prepare filmed scenes for VFX, or even quickly mask out some disturbing elements in the shot.

Sounds cool, right? This is already within the capabilities of Adobe’s Generative Fill. Based on their own AI called Adobe Firefly, this function is available in Photoshop (Beta). You can install the software and try it out if you have a Creative Cloud subscription. In the following example, I took a film still from my latest short and changed a part of the image, just like that. Now, the protagonist can enjoy a slightly more appealing dinner:

Generative Fill can also work as an eraser for the designated area. If you don’t type anything in the prompt, it will make an effort to eliminate the chosen elements by employing content-aware fill techniques. Midjourney, on the other hand, always tries to put something new into the defined area.

So no, in my opinion, Midjourney is by no means the new Generative Fill. However, it’s developing in this direction, and hopefully, similar functions will be introduced soon. Why “hopefully”? The quality of the pictures created by this artistic image generator is still hard to beat, even with Adobe’s innovations.

Other problems and limitations

We already touched on the issue concerning the selected area’s size. In one of the tests, I tried to replace only the warrior in the wide shot. Changing the prompt accordingly, I hoped to get a beautiful elven woman in a long green dress, and the results were not promising.

The only decent picture I managed to generate after a couple of trials was the last one, where the woman stands with her back to the viewers. Others seem not only quite disturbing but also weird. All that, despite the fact that usually, Midjourney can draw absolutely stunning humans and human-like creatures. If we ask it to make variations of the troubled elf from the pictures above, using the same prompt, it instantly comes up with an amazing outcome:

So, hopefully, in the next updates, the model will continue to learn and eventually apply its new skills to the smaller selected parts of the image as well.

Some other limitations and problems I noticed while playing with Midjourney’s Vary Region feature are:

• The interface of the built-in editor is bulky and awkward to use. Compared to the variety of flexible tools in Photoshop, Midjourney’s lasso will take some getting used to. Yet, it’s remarkable that developers could embed a functioning editor directly into Discord.
• Additionally, there is no eraser. As a result, no fast changes can be made to your selection. At the moment, you can only “undo” the steps in which you marked various areas of the image.
• Midjourney’s Vary Region tool is compatible with the following model versions: V5.0, V5.1, V5.2, and Niji 5.
• Midjourney users in the Discord community also noticed that when you do many generations of regional variations, the whole image gradually becomes darker and darker.

Conclusion

In every article on AI tools, I mention the question of ethics, and I won’t stop doing so. Of course, we don’t want artificial intelligence to take over our jobs, or big production companies to use generated art without proper attribution to the initial artists whose works the models were trained on. Yet, such tools as Midjourney can also become helping hands, which support us in mundane tasks, or help enhance our work and unleash new ideas for film, art, and music. A careful and ethical approach is key here. Therefore, it’s important to learn how to use neural networks and keep up with the updates.

So, what do you think about Midjourney’s Vary Region feature? Have you already tried it out? What are some other ways of using it in your film and video projects?

Feature image credit: created with Midjourney for CineD