Google Home has come out on the top for the second consecutive year in the Smart Speaker IQ test conducted by Loup Ventures, a US-based venture capital firm. The Google Assistant, which powers the search giant’s speaker, was able to answer most of the 800 real-world queries posed to the four major smart speakers in the market, including Apple HomePod (Siri), Amazon Echo (Alexa), Harmon Kardon Invoke (Microsoft Cortana). Even though Google’s Assistant was able to retain its position, the competitors like Alexa and Cortana have shown significant improvements. Apple Homepod, which reached the consumers earlier this year, was not included in the company’s last Smart Speaker IQ Test.
According to Loup Ventures, Google Assistant understood all the questions that it was asked and answered 88 percent of them correctly. It was followed by Apple Siri, which misunderstood 3 questions and achieved a 75 percent accuracy in answering.
Amazon’s Echo landed the third spot by not being able to grasp 8 queries and correctly answering 73 percent of them. The Invoke smart speaker with Cortana rightly answered 63 percent and misunderstood just 5 questions.
In 2017, Google Home was able to answer 81 percent of the questions accurately vs. Alexa at 64 percent and Cortana at 56 percent. In a separate test done by the firm in February this year, Siri had answered 52 percent of the questions correctly.
This data clearly shows Apple has massively improved Siri’s performance on the HomePod with Cortana and Alexa also getting a decent bump.
Among other learnings, Siri was found to be outperforming Google Assistant in responding to commands, whereas the Apple assistant underperformed in information and commerce related queries.
Proper nouns seem to be a major pain point for most smart speakers, which are now able to at least understand virtually all queries. Additionally, Loup noticed improvements in specific productivity-related questions, which all assistants had earlier problems answering.
It is important to note that with Loup’s testing done in the US, a market for which these smart speakers are the most optimised for, and the noted performance is unlikely to be observed by buyers in India and other markets.
Detailing the methodology of the testing, Loup Ventures revealed that it asked the same set of questions to all smart speakers. The questions ranged a wide variety of topics in five categories — local, commerce, navigation, information, and command. The firm used the second generation of the Amazon Echo and Google Home speakers.
Smart speakers continue to be one of the biggest battlegrounds for the tech giants. While Google Home maybe able to beat Amazon Echo in terms of performance, the Amazon speaker leads the speaker market.
According to a report published by Canalys recently, the worldwide smart speaker shipments grew 137 percent year-over-year to reach 19.7 million units in the previous quarter (Q3 2018), of which 6.3 million units were Amazon Echo speakers. Google grabbed the second place by shipping 5.9 million Home units. The China-only Alibaba and Xiaomi smart speakers were present on third and fourth spots respectively.