Of all the things Apple revealed Tuesday, to me the most impressive was the iPhone 13’s new Cinematic mode. The depth-of-field effect creates amazing focus transitions between subjects.
Apple basically built an AI cinematographer into the iPhone 13.
Whodunnit
Cinematic mode marks a new era in computational photography, which leans on digital processing to produce amazing photographic effects that rival or exceed what can be accomplished with traditional cameras and lenses.
The video camera in the iPhone 13 continuously analyzes the scene, and can recognize when subjects move into frame — and focus on them.
If the subject then turns their head away to look at something, the camera will automatically focus on whatever they look at. It is, as the name implies, very cinematic — and it’s done automatically in software.
Apple showed off Cinematic mode in a short movie shot entirely on a new iPhone 13. Check it out below:
Look at the shot at the 1:02 mark, where the dog turns to look at the butler standing behind it. At first, the shot focuses on the dog. Then it focuses on the butler. This is the kind of shot you can easily imagine normal users like us attempting. And if the results turn out like Apple’s, it’ll make for some great home movies.
No small feat
Apple says bringing Cinematic mode to iPhone was “no small feat.”
First, the system generates a ton of depth data so it knows the precise distance to all the people, pets and places in a scene. It does this continually at 30fps. Then the Neural Engine, which has been trained to act like a Hollywood cinematographer and focus puller in one, makes continuous decisions about what should be in focus, and applies a smooth focus transition if that changes.
“The sheer computational power needed to run the machine-learning algorithms, render autofocus changes, support manual focus changes, and grade each frame in Dolby Vision — all in real time — is astounding,” the company said on its website.
But it’s not all strictly automatic: You can take control of the system as you shoot, or adjust it afterward while editing.
New era in computational photography
Photo: Apple
Cinematic mode is impressive because the iPhone 13 camera knows what it’s looking at. It’s a big advance for computational photography, which has been around for a while. Early examples include digital panoramas, which are stitched together in software, or high dynamic range photos, which combine multiple images shot at different exposures into one image.
We are now entering an era where the camera can perform far more interesting and intricate computations. For example, light field effects — adding 3D info to the scene — allows for impressive Portrait effects or adjusting bokeh in post-production.
Apple’s new Live Text feature, due to be rolled out soon in iOS 15, is a similarly impressive feature that recognizes text in a scene, and makes it editable for cutting or passing into another app. I’ve been playing around with it in the iOS beta, and it’s kinda miraculous. It even recognizes (some) of my awful, spidery handwriting.
And take Visual Lookup, also in iOS 15, which can identify plants, animals, landmarks, artwork and other objects in your Photos library.
In all these examples, the camera isn’t just a dumb piece of glass and metal. The built-in AI can recognize things and objects in the scene. And not just in photos. It’s now working in video in real time.
It remains to be seen if Cinematic mode works as well in real life as it does in Apple’s demos. But wow, what a brilliant idea. It makes you wonder just how far Cupertino can push its amazing camera, which it cleverly disguised as a slender smartphone.
