Platforms State of the Union explains Apple Intelligence and new OS features

After this morning’s WWDC24 keynote, the Platforms State of the Union explained the nitty gritty details on Apple Intelligence, code completion in Xcode, Swift 6 and the latest software updates.

Susan Prescott, Apple vice president of Worldwide Developer Relations, opened the event by thanking the hard work of all Apple developers. “Thank you for making this such an incredible and exciting ecosystem.”

While the morning keynote has become more of a general interest event for the public to see what’s coming later this year, the State of the Union is a show for developers to see how they can adapt their apps to take advantage of the new APIs and software tools. There was a lot to talk about this year.

All the details on Apple Intelligence from the Platforms State of the Union 2024

Apple Intelligence takes advantage of the power of Apple silicon on-device for low latency and a better experience. It can run fast and privately.

Apple’s Foundation model that powers Apple Intelligence is a balance between power, performance and specialization. Adapters can be plugged in on top of the Foundation model to give this singular model all kinds of different superpowers. It also means that your device doesn’t need to simultaneously keep an LLM, image model and action model loaded in memory all the time.

Private Cloud Compute lets software take advantage of additional compute power off-device. While it operates remotely in Apple servers, it has no persistent storage, so there isn’t any information to be stolen, sold or hacked into. According to Sebastien Marineau-Mes, Apple’s vice president of Intelligent System Experience Engineering, it is an “unprecedented” level of secure cloud computing. He claims it is “the most advanced security architecture” for a cloud service of its kind.

Developers can easily incorporate Apple Intelligence into their own apps. Any standard text view can tap into the system text generation features; any standard image input can accept Genmoji with a single line of code. The Image Playground API gives third-party apps the same experience available in Messages and Keynote.

With a major update to CoreML, developers can also easily import third-party ML models directly in their apps, like Whisper, Stable Diffusion, Mistral, Llama and more. Compared to macOS Sonoma, macOS Sequoia can run third-party ML models many times faster with a fraction of the resources.

Smarter Siri in your apps

App Intents get a big boost in capabilities for integrating Siri inside your app, with hundreds of new actions inside apps. Siri can now respond properly if you stumble over your words or refer back to earlier context in the conversation.

Siri will be able to control any feature inside your app, like tap buttons inside the toolbar, based on the description in code — with no additional work. It’s context-aware as well, so if a string of text on the screen mentions a contact in your phone, you can ask Siri to call them, and Siri can look them up. This will be available in a future software update.

A new Spotlight API will enable Siri to search through and access data stored inside your app, like podcasts in your library or notes you’ve written. What makes this feature special is that it’s available across all your apps, so Siri can draw from all your information whether you’re writing an email in Gmail or making a presentation in Excel.

Code completion in Xcode

Xcode 16 infuses the power of generative models to help you code faster, using the best practices and standards. It uses a unique model specifically trained on Swift, and can recognize the symbols and objects inside your project. Write comments in your code as prompts for functions that it will write for you. You’ll see the suggested code inline; just hit Tab to fill it in. Code completion will receive automatic updates over time to keep it up-to-date with the latest APIs.

Code completion requires a Mac with Apple silicon and at least 16 GB of unified memory.

A larger and more powerful model that runs in the cloud, called Swift Assist, can “answer your coding questions” and experiment with new APIs inside your projects. Ken Orr, Apple senior manager for Xcode and Swift Playgrounds, described it as “great for prototyping,” giving you quick code and even sample data to fill out a prototype UI. Give it a suggestion, and Swift Assist will continue to personalize and modify your code.

Like on-device code completion, Swift Assist will be continually updated on the latest Apple APIs and features. It will be available later this year.

Your code is never stored on Apple servers and is never used for training data — a sharp contrast to GitHub Copilot, which has drawn controversy and even legal challenges for being trained on other people’s open source projects.

Swift 6 brings powerful concurrency and testing features on its tenth birthday

Swift was created to be a simple, modern language for app development to replace C, C++ and Objective-C. Apple is dogfood testing Swift to replace C code across all levels of the company’s own software.

To make Swift accessible to everyone, Swift is now coming to Fedora and Debian Linux, and even Windows, through Visual Studio Code.

The latest changes in Swift 6 makes concurrent programming easier by eliminating data-race conflicts. Analyzed at compile time, data-race safety will point out potential race conditions in your app. Because moving over to Swift 6 and adopting concurrency requires many changes to your code, you can opt-in to Swift 6 module-by-module, rather than spending weeks moving the entire codebase over at once.

Now that Apple silicon adds more CPU cores to Apple products than ever before, adding concurrency to your app can significantly speed up performance.

Swift Testing, introduced nine months ago, “makes it easy to write complex checks” to your code. It is available in both Xcode and Visual Studio Code.

SwiftUI and other software tools continue to get easier to use

SwiftUI is an incredibly easy way to build out user interfaces across all of Apple’s platforms. Because of its declarative syntax, complex user interface code remains very readable. Apple used SwiftUI to build the new Passwords app, Image Playground app, Photos app and more. SwiftUI is also used extensively on watchOS and visionOS.

• SwiftUI previews are now easier to set up. Previews use the same build artifacts as full compiles, so you don’t have to worry about discrepancies between the two.
• On Vision Pro, SwiftUI can now customize or expand buttons when you look at them.
• On the Mac, SwiftUI can now customize the appearance of an app toolbar.
• Gesture recognition and animations have been implemented in SwiftUI, so you don’t need to switch over to AppKit for these features.

SwiftData, an API for managing data inside your app, can now be plugged into any kind of data store, like JSON — not just Apple’s own CoreData.

RealityKit 4, and its tools like Reality Composer Pro, are now unified across all of Apple’s platforms. That means the code you write for your visionOS app can be reused to make AR apps for iPad or 3D apps for Mac.

New APIs in iOS power customizable controls and app icons

The new Controls API will let third-party apps assign controls to the Lock Screen, Control Center or the Action button. Through the Lock Screen button, developers can even make a section of their app available to the user without unlocking the device.

iOS apps can now have three different modes: Light, Dark, and Tinted. Developers can add new alternate versions inside the Xcode App Icon Catalog.

A new registration API makes passkeys easier for the user. Passkeys can now automatically create an account or sign a user into their app without any intermediate step — and the user is notified that their passkey has been saved to the keychain.

iPadOS gets a new, redesigned tab bar for app navigation. It floats at the top of the screen, and automatically expands to fill the side of the screen with more detail when necessary using a simple API.

watchOS 11’s new Smart Stack and interactive complications

The new Smart Stack in watchOS 11 builds on the same code already used to create Live Activities, and interactive complications reuse the same code that’s used to make widgets from iOS. They can be context-aware based on the time of day, connected AirPods and more.

A new hand gesture API lets third-party apps use the Double Tap feature on Apple Watch Series 9 and Ultra 2 inside their apps.

Porting PC games to Mac gets even easier

Every year, Apple makes it easier to port games to the Mac. Apple showed off how recent releases like Cyberpunk 2077 are easily run on the Mac without any work done on the code.

The new Game Porting Toolkit 2 makes it even easier to adapt a PC game for the Mac. It also lets developers then bring Mac games to iPhone and iPad. It now supports advanced graphics features like ray tracing, and API improvements to shader memory models and residency management. Once you bring the code over to Xcode, you can debug the HLSL shader code to make sure it runs smoothly.

Game Porting Toolkit 2 also comes with additional documentation and sample code to help developers take full advantage of Apple hardware.

visionOS 2

visionOS is built on the decades of history of macOS, iOS and iPadOS. SwiftUI, ARKit and RealityKit are core to making great apps for visionOS. Here are some of the new features in visionOS 2:

• New APIs make it easy to tie custom 3D objects, called Volumes, with an app’s user interface. Volumes are now resizable by the user in 3D space, and can now have app ornaments that dynamically stay within the view of the user no matter where they are in the room.
• TabletopKit does the heavy lifting to make sure a 3D volume stays put on top of a real-world surface.
• New APIs let developers choose whether the user’s hands should appear in front of, through, or behind a user interface element.
• Advanced APIs let apps be aware of all the surfaces in a room, like walls, tables, easels, paintings, etc.
• Apps can now tap into a user’s movement around a room or a series of connected rooms.
• Object tracking APIs let app elements stay tied to a physical object in the user’s space.

These new features should allow for exciting new apps in visionOS 2.

Watch Apple’s session videos to learn how to implement these new features

Apple’s SDKs make it easy to build new apps across all Apple platforms. The best way to make an app for iPhone, iPad, Mac, Apple Watch, Apple TV and Vision Pro is with the native SDKs, because you can reuse all the hard work Apple puts into making sure its software is secure, accessible, feature-rich, beautiful and easy to use.

Over 100 session videos will be published throughout the week for the nitty gritty details on implementing all of the features announced in this morning’s keynote. You can find these videos in the Developer app or on the Apple Developer YouTube channel.

D. Griffin Jones

D. Griffin Jones is a writer, podcaster and video producer for Cult of Mac. Griffin has been a passionate computer enthusiast since 2002, when he got his first PC — but since getting a Mac in 2008, he hasn’t turned back. His skills in graphic and web design, along with video and podcast editing, are self-taught over 20+ years. Griffin has a bachelor’s degree in computer science and has written several (unpublished) apps for Mac and iOS. His collection of old computers is made up of 40+ desktops, laptops, PDAs and devices, dating back to the early ’80s. He brings all of these creative and technical skills, along with a deep knowledge of Apple history, into his work for Cult of Mac.