Siri doesn’t have all the answers, but Apple’s digital assistant continues to get smarter.
Loup Ventures fires 800 questions at the leading digital assistants for its twice-a-year IQ Test to measure the practical use of voice as a means of computing input on smartphones. They are all improving, but not at the same rate.
Google Assistant continues to outperform the big three. It understood every question and correctly answered 92.9 percent. Siri fell two questions short on perfect understanding and answered 83.1 percent correctly. Alexa made the most dramatic improvement. It understood 799 of the questions and answered 79.8 percent.
Cortana was dropped from the competition because Microsoft has pivoted towards integrating the digital assistant with the operating systems of its former competitors.
IQ test measures ‘within reason’
“With scores on our test quickly approaching 100 percent, it may seem like digital assistants will soon be able to answer any question you ask them, but we caution this is not the case,” Loup Ventures‘ Gene Munster and Will Thompson wrote. “Today, they are able to understand, within reason, everything you say to them, and the primary use cases are well built out, but they are not generally intelligent.”
Digital assistants are graded on five categories of questions: local, commerce navigation, information and command.
Siri bested its competitors on command questions, flexing its muscles on calling, texting, emailing, calendar and music requests. Siri has always topped this category, this time correctly answering 93 percent, to Google’s 86 percent and Alexa’s 69 percent.
As the Loup Ventures chart shows, Google Assistant topped the other four categories, even commerce (92 percent), a category that should be owned by Amazon’s Alexa (71 percent).
Smartphone users who routinely use digital assistant seem to use voice to purchase items at lower rates. Google Assistant was better than Alexa at answering more research-based commerce questions, such as product specs and locations to purchase.
Since it is a competition, Alexa would win the Most Improved Player award. Its correct answer percentage went up 18 points, to Google’s seven and Sir’s five.
Loup Ventures does a separate test for the voice assistants on smart speakers because of variations in products and applications.