Here's how spatial user interfaces work in visionOS

How does Apple’s new “spatial computing” platform visionOS work exactly?

At WWDC23 this week, Apple detailed a bunch of interesting tidbits about how the new Vision Pro headset works. Apple detailed how buttons look and behave in the spatial computer, how they are pressed without any physical controls, and how apps work in 3D.

Here’s how Apple’s spatial interface works.

visionOS: Principles of spatial design

Keynote dimming the environment background, with the cursor hovering over the Close button — Every window has a grabbable bar to move it around, along with a button to close it.
Photo: Apple

In the developer session “Principles of spatial design,” Apple laid out the basics of window behavior in visionOS so coders can begin to wrap their heads around creating apps for the headset. In the image above, notice the thin bar floating underneath the window. This is the Window bar, much like the Home bar on the iPhone. Users grab it to move windows around in space. When moved, windows reorient to stay pointed at the user. A dot next to the window bar is the close button. To resize the window, the user focuses their eyes on the corner of a window to reveal a control.

Windows can grow and shrink freely, unconstrained by any physical limitations, but they should remain small so as not to block too much of the user’s field of view, Apple said. Apps can have many windows for floating palettes and elements.

Pressing and holding the headset’s Digital Crown automatically centers a window to the user’s field of view, so they don’t lose anything in the 3D space.

How to present windows in visionOS

Apple encouraged developers to lay out windows in landscape, as it’s easier for the user to turn their head left and right, rather than up and down. Apple said the most important information in an app should be placed in the center. Dynamic Scale will make sure windows stay the same visual size as the user pushes them backward in space.

While developers have the freedom to make windows appear anywhere, Apple has tips for how they should work. Developers should spawn windows out of arm’s reach, so users interact with them remotely using their eyes and hands rather than with direct “touch.” Too many windows can make the user feel cramped — even if they’re virtual. And apps shouldn’t require a lot of movement, as the user might be using the device sitting down with the Vision Pro plugged in.

Full Spaces for virtual presentations

The Full Space experience in Keynote — Practice your presentation in a Full Space that simulates a theater or lecture hall.
Photo: Apple

Apps that need the user’s full attention, like videos or Keynote presentations, can dim the room around them. Full Spaces replace the user’s background with a completely virtual 3D space.

Windows can project shadows or glow onto the surface beneath them to make them appear real.

How the apps on Vision Pro look and feel

The app grid in visionOS is inspired by the honeycomb pattern on the Apple Watch. However, labels under app icons will make them easier to find, according to the presenters in the “Design for spatial user interfaces” session. Icons are displayed in 3D, with shadows that appear when you look at them. They can have layers of depth, as in the Apple TV app.

The frosted, blurry background that the user interface is built on looks very similar to the blur effect on iOS and macOS. But with the specular highlights around the edges, it looks more like a real pane of frosted glass. Apple encouraged developers not to use completely opaque windows, as they make the user feel claustrophobic or hide important parts of the real environment.

Text defaults to white on a dark gray background. Fonts are slightly heavier than they appear on iOS — Regular paragraph text and Semibold titles on iOS are weighted Medium and Bold on visionOS, respectively. Tracking (the space between letters) has been slightly increased.

Buttons and layouts

Close-up of Music app showing a button with 60 points of space — Buttons need 60 points of space to be easily tappable.
Photo: Apple

Whereas on iOS the Steve Jobsian rule-of-thumb was for buttons to be at least 44 points tall, in visionOS, elements must be at least 60 points, Apple said. Buttons that appear small may still have 60 points of tappable area around them — so long as they’re the only tappable item in the area.

When the user looks at an interactive element, like a button, it’ll subtly glow and shine, so they know it’s something they can press.

How to convert a 2D interface to a 3D one

Expanded tab bar floating in front of an app with a sidebar — Stare at a tab bar and it’ll appear with labels.
Photo: Apple

When importing from iOS or iPadOS, all the app’s windows, tab bars and sidebars are re-created. An app window becomes a floating piece of glass in visionOS. And the tab bar at the bottom of the iPhone becomes a vertical strip of icons that floats to the left after conversion for the headset. If the user looks at the icon bar for a few seconds, labels will temporarily appear. Sidebars for secondary navigation take up the full height of the window on the left side, like on macOS.

Toolbars, menus, popovers and sheets are no longer bound by two-dimensional constraints in visionOS — they become “ornaments,” which float slightly below and in front of the background window. When a popover or sheet requires the user’s immediate attention, the background window is pushed back and the sheet appears at the same distance from their eyes, so they don’t have to refocus.

How to interact using your eyes and hands

Diagram showing gestures on visionOS — Here’s how gestures work without a screen to touch or a cursor to click.
Photo: Apple

Vision Pro supports a wide variety of input methods, said the presenters in the “Design for spatial input” talk. Users can look with their eyes and press their fingers together to click; or they can reach out and “touch” elements directly. Alternatively, they can hook up a Bluetooth keyboard and mouse to use an iPad-like pointer.

Pinching thumb and pointer finger together is equivalent to tapping the screen or clicking the mouse; users scroll by pinching their fingers and dragging their hand up or down; pinching with both hands will let them zoom and rotate elements on the screen. This is how users can control a window from far away.

Hand touching a floating keyboard — Reach out and touch parts of the screen when you need to.
Photo: Apple

Bring a window closer, and the user can touch it as if it’s a giant floating iPad in front of them. The user can reach their hands out to scroll or tap, although Apple cautions that this can cause fatigue after a long period of time. Because the user can’t feel this happening under their fingertips, immediate visual and audio feedback — such as glowing buttons and clicking noises — helps give a sense of physicality to the virtual elements.

Users can turn on dictation inside a text field just by looking at the microphone icon inside, so they can enter text fluidly and seamlessly without making a tap gesture or typing on a virtual keyboard.

Further information

This is only a summary of three session videos. Apple published 14 more videos that go into great detail about how Safari and games work in visionOS, how to make apps accessible and how these user interfaces are implemented for developers.

Here’s the list of visionOS sessions:

Throughout the week, Apple plans to publish 23 more videos all about Vision Pro and visionOS. Although the visionOS software development kit will not become available until later this month, and access to hardware for testing is obviously extremely limited, Apple wants developers to start planning how their apps will work for the new device.

Here’s how spatial user interfaces work in visionOS