¿Qué? Siri destroys Cortana and Google Now on language accuracy

Three-and-a-half years after the debut of Siri, virtual assistants haven’t yet become a user interface element on par with, say, the mouse cursor — but that’s not through any lack of trying.

According to a new study carried out for Venture Beat, Siri not only defeats Microsoft rival Cortana and Google’s Google Now automated assistants in understanding English; it absolutely slays them when it comes to other languages.

¡Viva Siri!

As Venture Beat’s Sophie Salonga explains, to test the multilingual abilities of all three virtual assistants, the team posed 18 questions in English, Italian, French, German and Mandarin.

“We gave each question a possible score of 1.0, 0.5 and 0.0. Answers that scored a 1.0 gave our testers a quick and easy correct response. 0.5 answers weren’t quite as straightforward and came back with a list of search results, where somewhere within the clutter, we found the correct answer. 0.0 responses did not understand our questions, gave irrelevant search results, or by their own admission, stated that they could not complete the task.”

In a close-fought battle, Siri scored top in English, with a score of 14.5/18, followed by 11.5 for Google Now and 12.5 for Cortana.

It’s a strong defense against accusations that Apple has lost its edge when it comes to Siri.

It was the other languages where Siri really left her mark, however. In Italian, Siri scored an identical 14.5/18, while Google Now sank to 8.5 and Cortana landed with just 6/18. French and German results were reportedly similar to Italian — although Siri got marginally worse, while her two rivals got better — although not enough overtake Siri. Finally, in Mandarin, Siri scored a dazzling 13.5/18, while Google Now crashed and burned with 1.5 and Cortana struggled to a 5.

As the study concludes:

“Siri won this battle, Cortana was voted ‘most likely to go to web search’ and Google Now needs to go back to college and pick up a few Mandarin classes.”

While there is certainly an element of luck involved with the test, based on which questions happened to be asked, it’s also a strong defense on Siri’s part against accusations that Apple has lost its edge when it comes to virtual assistants.

Siri’s multilingual abilities are also particularly good at a time when Apple is pushing into new markets like China. With 955 million speakers worldwide (roughly three times the number of English speakers), Mandarin is by far the world’s most-spoken language. Tim Cook has talked about how China will become Apple’s primary market in the near future, and there’s no doubt that Siri can be a powerful tool in helping with adoption.

As Apple continues to grow in developing markets with lower rates of literacy, I can see virtual assistants which allow people to speak with their smartphones having a level of use they don’t in the U.S., where they essentially offer a substitute for text-based queries.

Apple’s growing its Siri team all the time, and while this test shows there’s still a way to go (using virtual assistants to check the stock market was something all three companies struggled with), this is definitely a net positive for Cupertino’s automated PA.

Source: Venture Beat

Via: iMore

Luke Dormehl

Luke Dormehl is a U.K.-based journalist and author, with a background working in documentary film for Channel 4 and the BBC. He is the author of The Apple Revolution and The Formula: How Algorithms Solve All Our Problems … and Create More, both published by Penguin/Random House. His tech writing has also appeared in Wired, Fast Company, Techmeme and other publications.