No Camera High-Res Enough for Apple Vision Pro, According to Canon

Virtual Reality

by Omri Keren LapidotMarch 13^th, 2024

With 23 million pixels spread across two 4K displays (one per eye), the Apple Vision Pro features an incredibly detailed display. Combine that with the ability to look around the spatial virtual reality environment, and now you are facing an exorbitant amount of pixels required to fill this insatiable hunger for resolution and details. Such high demands lead to a problem in finding a camera that may opt to fulfill them. Canon representatives claim that no camera available today can do it.

In a recently published interview, PetaPixel’s Jaron Schneider discussed these challenges with a contingent of Canon executives. They’ve all emphasized the importance of augmented and virtual reality (AR, VR), and they’ve also claimed no camera today can provide such a demanding video feed. While this claim is probably mathematically true, I’d like to dissect it, because in my eyes things aren’t so simple or decisive.

The Apple Vision Pro displays are only one small component

While the dual displays are impressive in their own right, it’s the spatial nature of the Apple Vision Pro that really raises the bar here. Once the user moves their head, new visual information streams across into the twin displays. As long as this information comes from the device’s camera array there’s no problem, but if it originates from an external imagery or video then the source material must cover an incredible amount of detail.

Let’s not forget – augmented or virtual reality has more than two dimensions, meaning that getting closer or farther away from the image should also be considered and accounted for when creating footage for the Apple Vision Pro, as well as the refresh rate of the system. These technical specifications pose a great challenge for aspiring creators looking to move into this niche.

What tools do we have today?

Canon itself produces a dedicated AR/VR set with its RF 5.2mm f/2.8 L Dual Fisheye 3D VR Lens and R5 or R5C cameras. This unique lens projects two image circles over the 8K sensor. Working on a single sensor bypasses any synchronization issues associated with multi-camera solutions, but compromises raw resolution. This system is probably the most compact, outdoor-ready system capable of adequate results.

Other options are also available. Canon’s executives claim the required benchmark for the Apple Vision Pro revolves around 14K (or about 100 Megapixels) While such cameras are hard to find, there is the Insta360 TITAN that comes pretty close. With 8 Micro Four Thirds sensors overlapping the spherical camera body, this one captures an impressive 11K 360 VR footage.

These are a lot of pixels, generating a lot of data. The TITAN uses 9 memory cards. Eight are assigned to the eight different cameras while the ninth collects additional sync information, gyroscopic motion stats, etc. Here we encounter another challenge – data throughput.

Extreme alternatives

If there’s a will there’s a way, and everybody wants a piece of this new tech. One rather extreme solution that comes to mind is Sphere’s Big Sky Camera in all its 18K glory. As perfect as this solution may sound, we’re talking about one of the rarest cameras in the world with very demanding operational requirements. Oh, and no depth perception… Other options are camera assemblies. These may provide adequate quality depending on the cameras used, but they are even more operationally demanding and require a high level of expertise both in terms of videography and in terms of the electro-mechanical rigging required for such contraptions.

Big data

The higher the resolution, the “bigger” the data. Though Apple surely compresses the footage coming to the Vision Pro most efficiently, there’s no way out of the raw resolution requirements dictated by such a device. Producing, editing, and sharing this kind of digital information is a daunting task, at least at this point in time (and technology)

AI. Must mention AI.

It seems like almost every tech-related article must mention AI in some way, but it’s actually very relevant here. AI-based tools possess some unique abilities. I’m not talking about the hyped video-generating tools such as SORA and LTX Studio. A day may come (could be tomorrow…) when these generators are up to the task, but as of today, they will probably generate more problems than solutions. The relevant tools here are resolution enhancers, which have been pretty efficient in recent years and have been here long enough to provide professional-level results with adequate consistency and reliability.

Peak performance is rarely required

Don’t we all just love 8K? Don’t you take photos and videos of your kids (or groceries) with the highest resolution possible? Of course not. Peak performance is very nice. Sometimes required, but most of us do most stuff at lower settings, and this applies to much more than filmmaking, to be fair. The same will probably go for the Apple Vision Pro. As nice as immersing ourselves in the Grand Canyon while editing or emailing may be, we don’t need every image to cover entire virtual surroundings. As I see it, one of the Apple Vision Pro’s biggest features is its multitasking ability. So no, I probably won’t be able to fully immerse myself in my recent music video, documentary, or feature, but I believe it’s good enough for now, and beyond. Immersive AR/VR content still has some major non-technical hurdles to leap over before it is fully immersed into the mainstream (pun intended).

Creating spatial content professionally

For those who wish to create specific high-res spatial content, I sadly bear no significant news. Gear-wise, alternatives are scarce, quite expensive, and require a considerable amount of expertise. Alas – early adopters pay a premium, but those who brave these challenges may place themselves in a unique position for the future.

Do you see yourself creating dedicated content for the Apple Vision Pro? Do you think AR/VR will become mainstream content consumption devices in the near future? Let us know in the comments.