Simple Motion Capture With AI – No Suit Needed

Artificial Intelligence

by Mascha DeikovaMarch 22^nd, 2023

“I need your clothes, your boots, and your motorcycle… Wait, and also your motion capture suit!” It seems spooky yet it’s undeniable: artificial intelligence tools have been growing and evolving at breakneck speed. We can either despise new technology or see it as a helping hand. Let’s stick with the latter, shall we? Apart from creating artistic mood boards within minutes and enhancing concept ideas, neural networks can also massively simplify your previsualization and animation processes. How so, you ask? Let’s talk about motion capture with AI – a zero-cost way to start animating from scratch.

If you have ever observed behind-the-scenes of any film using a CGI character then you are familiar with the concept of a tight black suit covering every inch of an actor’s body while they impersonate an imaginary Gollum, say. A motion capture suit helps to record human acting and to track body motion for further animation processes. Such hardware is incredibly expensive going up to hundreds of thousands of dollars. Well, seems that there is another way now, which should put a fat check on this industry injustice.

Quick and simple motion capture with AI

What if you could capture motion at a fraction of the cost of traditional studios – no suits, using mobile phones and off-the-shelf cameras, and still get the same high-fidelity results? Move.ai claims to offer exactly that, once they fully release their AI-based markerless technology. According to the company website, this tool is able to track pose, gait, root motion, and even the mannerisms of any age or body type. The feedback from beta testers has been positive so far. Starting this month, everyone can try out this technology for couple of minutes after signing up.

Move.ai is not a breakthrough invention, even if they claim their patented machine-learning software is unique. Today, the market is already packed with different varieties of AI mocap tools (Deepmotion, Plask, and Kinetix, just to name a few). To take a closer look at how this technology works, I picked the Rokoko Video because it’s free of charge, and their slogan promises anyone will be able to use it. (I’m definitely “anyone” here, considering that I’ve never done any motion capture in my life). And secondly, Rokoko is a well-established developer of conventional motion capture suits which are widely used for high-budget productions (like the ones you watch on Netflix). So, I considered it an interesting and bold move toward new technology, which might ruin their initial business in the future.

How to use AI motion capture

After the first attempt, my answer is – the AI mocap is quite simple, user-friendly, and intuitive. As an example, here is a short step-by-step instruction for the Rokoko Video tool:

• Capture your movement with a webcam directly in the browser or upload a phone video. (The recorded dance of your actor will also suffice). Tips: your full body should be visible at all times, avoid baggy clothes, and make sure to use a tripod. The maximum file size allowed is 300 MB.

• Trim the clip as you like, click “Animate”, and let AI make its magic: using a powerful cloud engine, Rokoko extracts the motion of the body and limbs. In my case, it took around 10 minutes to process 50 seconds of the initial video.

• Afterward, you can access and clean up your data in Rokoko Studio software – for example, to adjust foot locking – and then take it to the final 3D application of your choice, like Blender, Cinema 4D, or Unity.

You can watch the detailed tutorial from the developers here:

Possible application fields for motion capture with AI

For this test, I captured a short video on the old iPhone X with no artificial lighting. To keep the process as effortless as possible, the clip only consists of various yoga poses and dance moves shot directly in my living room against the wall. These are some results that came out of Rokoko Studio:

Not a flawless body-tracking example, to be honest. But it’s working! Just a few clicks, several minutes – and that’s it. What excited me most was that I didn’t need any special animation skills at this stage. Of course, you may still require them when you want to transform a moving skeleton into a plausible character, but that’s another story.

If you’re a beginner animator looking for an easy way to get started, or a professional who wants to experiment with their CGI character, AI-based mocaps may be just your cup of tea. To me, the most suitable use of this technology is a quick, good-quality, animated previs for your short movie or a less demanding animation for a VFX shot.

Downsides of AI motion capture

• According to the CEO of Rokoko, Jakob Balslev, the AI-generated data isn’t as accurate as that produced by a traditional motion capture suit. For example, it extracts full-body motion but doesn’t include the precise movements of hands or facial expressions. (You can see how my hands were suddenly behind my back in the example above).
• Rokoko Video cannot offer high fidelity and real-time streaming, which are very important features of mocap suits. Will the launch of move.ai prove to become a solution? Time will tell.

Photorealistic 3D landscapes for your previs with NVIDIA AI

Speaking of so-called previses. You may have heard about NVIDIA Canvas – a free AI tool for Windows that allows its users to quickly turn simple brushstrokes into realistic landscapes using neural networks. It’s been around for a couple of years and is still in the beta phase, but recently the developers announced a mind-blowing update for this app, namely – the ability to create 360-degree environment maps.

The update is not here yet, but when it comes out, it might become big. Imagine how a simple sketch in Canvas magically transforms into a 3D visualization of your idea. Open the export in your animation application and apply it as a 360-degree background. Add a quick AI motion capture from the previous step, play with the camera moves, and the first previs for a scene is ready. As simple as that.

Conclusion

Will artificial intelligence software fully replace expensive mocap suits? To be discussed. The ongoing conflict between sentient beings versus machines also won’t be resolved in the near future. But as long as new AI tools make our life simpler and work processes more efficient, why not use them?

What do you think? Would you use AI motion capture for your projects? Why or why not? Let’s talk in the comments below.

Feature image: a showcase, created with help of AI motion capture. Image credit: Mascha Deikova, CineD