Vision Pro isn’t dead yet. Here’s How Apple Can Give It New Life with VisionOS

Sometimes I sit down and draw strange doodles in the air that hang and expand and rotate when I touch them. I draw them in the air with a thick, black paint made by Logitech called Muse, while I’m dressed. Apple Vision Pro the headset.

The process sounds like magic, and yet Apple often seems surprisingly indifferent to this opportunity. Professional-oriented 3D creation tools that make 3D art like this don’t really come to life in Vision Pro. And Apple didn’t bother to make a local version of its Pencil. This sense of half-finished frustration is exactly what makes the Vision Pro, now more than two years old and in its second hardware iteration, sometimes feel dead.

But that’s not the case. In fact, Apple plans a lot for them. Small, simple, but not at least for a few years, according to the latest reports of Mark Gurman. Meanwhile, Apple’s AI-powered smart glasses may not arrive until at least late next year… and later still for those who are enabled for display they can get features like Vision.

But as we look at Apple WWDC software conference Next week, it’s a good time for Apple to finally release the Vision Pro and its many hidden possibilities. Not just in that device itself, but in everything that comes behind it. AI is included.

More power under the hood than it touches

I love the idea of ​​virtual computing — specifically, floating screens and apps around me where they’re needed. But what Apple has delivered so far is only a limited portion of what’s possible. Apple’s according to the real person avatars are just one aspect of that hidden potential.

I know the possibilities because I look at and test products like this all the time and talk to people who are testing solutions that don’t exist yet. The Vision Pro was seen as Apple’s biggest product failure under Tim Cook — who knows who owns it? — but it’s also widely accepted as the most advanced VR/AR device out there. The M5 processor it uses, the eye-tracking finesse it packs, the quality of the far- and near-range motion sensor, the cameras that combine the view of the world around you into pass-through video, are the best.

What’s negative is how Vision Pro fails to use all of this to deliver truly useful tools, and how it fails to explore the ideas Apple has to address in the expected range of AI-powered wearables that don’t exist yet.

I expect Apple to make glasses, AirPods with a camera, and maybe a universally recognized type of pendant or pin. But for now, the Vision Pro is a real product that already packs a lot of this potential, if only Apple would let it go.

And no, an endless supply of immersive 3D video movies and sporting events is not the answer for a $3,500 device. But what about realistic 3D scans, which can be made with Gaussian splatting and displayed in Vision Pro? Or, AI that can see the world worn on the head and guide you in the middle of multiple projects?

Visual intelligence can and should happen in Vision Pro first

Camera-aware AI, or multimodal AI, is a growing area that most major AI platforms have already entered. Meta already covers it smart glassesand Google and Samsung they mix it up true integrated headphones already exist, and in glasses coming at the end of this year.

Google and Samsung released Vision Pro-like Galaxy XR headset last fall to test ideas, like Gemini’s always-on mode that can see your location and the apps you’re using. Now that Apple has a partnership with Google installing Siri with Gemini, the road seems open to similar experiments on Apple hardware, too.

I am curious and concerned about how AI integrates with our sensors in wearables, and how it can invasion of privacy both the wearer and anyone around the wearer. But in Vision Pro, Apple has a great way to test this, using all the headphones’ sensors and processors to test these visions beforehand. smart glasses, smart pins or smart AirPods be released.

There’s another small area in AI that doesn’t concern agents, known as Gaussian splatting, that Vision Pro’s OS should explore more closely. Unlike Apple stereoscopic, immersive videos shot with multiple camera lenses, Gaussian splats can create complete holographic-style photos and videos using AI to stitch them together. Apple has been introducing more 3D Spatial layers to Vision, Personas being a big part of it. But Vision Pro should be part of an overall 3D scanning system with apps that Apple is creating for iPhones and headsets, as extensions of the camera app on existing phones.

Connect all Apple devices to Vision Pro

The Logitech Muse is like Apple’s Vision Pro pencil that Apple never bothered to make. Likewise, Sony’s PlayStation VR 2 administrators can connect with Vision Pro to play games in ways that are not possible otherwise, since Apple does not make their own local controllers.

Some of this is understandable, since the Vision Pro is an experimental product, and Apple often relies on companies like Logitech to test peripheral ideas that have not yet been made (this happens with iPad keyboard cases).

And yet the Vision Pro has yet to connect seamlessly with other Apple products that have been around for years. While AirPods connect with it, and Macs can extend monitors or stream Mac apps to the headset, iPhones, iPads and watches are oddly left out, except for streaming iPhone/iPad screens via AirPlay.

I want to share apps and extend monitors from iPhones and iPads around me, letting the Vision Pro act as a kind of shared computer, like it does with a Mac. There’s no reason why they can’t. iPhone-level chips can run macOS now, like MacBook Neo proved. Sure, they can share screens and extend apps, or let me magically transform all these devices into one headset that sees them all.

The Apple Watch is perhaps the worst piece left out: A wrist-mounted control panel with motion tracking, and it would be a great interface with Vision Pro if Apple allows it.

Open the road to professional applications, one way or another

The Vision Pro’s biggest flaw, for me, isn’t its price or size. It’s how the headset lags behind Macs and iPads in being a true computer to fit the word “pro”, even with an M5 processor under its belt now.

What do I mean by pro? I mean video editing suites, music creation tools, 3D graphics programs — anything that can and should be in VisionOS to enable the kind of creative work that Apple prides itself on.

It’s strange to me because Vision Pros are apparently used by filmmakers, usually as monitors set up to see videos or 3D models. They should be all doors to do everything. The massive visual monitoring and 3D interface should enable simulation and creativity that I’ve seen even on less powerful headsets like the Meta. Question 3.

If getting developers to create powerful apps is too difficult to promote and Apple doesn’t want to do them (which seems to be the case after the new version of App Creators Studio doesn’t include any Vision Pro settings), extending and streaming from Macs and iPads can help. VisionOS can render 3D content from Macs and make the most of it by using foveated streaming, a trick that delivers high resolution only where your eyes are looking directly.

Paving the way for developers to extend Mac apps ready for VisionOS, or to do this for iPads and iPhones, is a necessary step. If not, when it comes to doing serious work, I’d probably ditch the Vision Pro.