Apple tests AI software to transform images into video


Apple tests Keyframer, AI software to transform images into video
Keyframer is only a research project, but it confirms that Apple is hard at work on LLM-based AI software.
Screenshot: Apple

The latest demonstration of Apple’s AI expertise is Keyframer, software that turns an image into a video based on natural language instructions.

It offers an early glimpse at the type of AI tools that might come bundled into future versions of macOS or iOS.

Keyframer: An Apple animation app built on LLM-based AI

In recent months, Apple faced increasing criticism for not doing more with artificial intelligence, arguably the hottest sector in tech. But CEO Tim Cook promises that Apple continues to put an enormous amount of effort into AI. The resulting software simply hasn’t been released yet. Cook said recently that he remains “incredibly excited” about announcements in this area coming “later this year.”

But we got a fresh taste of what’s ahead with a new research paper from three Apple researchers: “Keyframer: Empowering Animation Design using Large Language Models” (.pdf). It’s the brainchild of Tiffany Tseng, Ruijia Cheng and Jeffrey Nichols.

They describe their creation like this: “With Keyframer, users can create animated illustrations from static 2D images via natural language prompting.” Users just input an image, then describe how an animated version should look.

But, unlike so many current AI image generators, Apple’s animation software is intended for professional designers. It “supports exploration and refinement of animations through the combination of prompting and direct editing of generated output.”

Users can do much more than put in an image and a description and then accept the AI’s results. They can use tools built into Keyframer to make changes — say, pick different colors. It’s more of a collaboration between designers and AI.

Simplifying animation with AI

The researchers are exploring the possibility of using natural language to make otherwise complex tasks simpler. “The use of descriptive natural language prompts has the potential to reduce the technical expertise needed to create design artifacts and provide novices with educational opportunities to improve their design skills,” they say in the paper.

They did not develop their own large-language model. Nor did they use the one Apple itself reportedly created. They used OpenAI’s ChatGPT-4 instead.

They tested Keyframer with 13 participants with a range of animation and programming experience. “Participants were happily surprised by the efficacy of the system to turn natural language instruction into animations,” said the Apple researchers.

Apple researchers also recently unveiled an open-source AI model that edits images based on users’ natural language commands.


Daily round-ups or a weekly refresher, straight from Cult of Mac to your inbox.

  • The Weekender

    The week's best Apple news, reviews and how-tos from Cult of Mac, every Saturday morning. Our readers say: "Thank you guys for always posting cool stuff" -- Vaughn Nevins. "Very informative" -- Kenly Xavier.