Video: Benchmarking Voice Assistants, Pt. 3: Comparison Questions
Learn more about intelligent assistants at the next SpeechTEK conference.
Read the complete transcript of this clip:
Ronald Schmelzer: Comparison questions, alright.
Kathleen Walch: Alright so with this we wanted to ask comparison questions: What's bigger, an ant or a tiger? And we tried to make these fairly obvious so that, you know, to humans, we know, you know, we're not asking "Oh, this bird or this bird, what's bigger?" No, we made it clear, an ant or a tiger, and we wanted to see if these voice assistants understood these basic concepts.
Ronald Schmelzer: Right, and sort of like the, there's always an enterprise reason, a business reason for all these questions, right? The first one is to make sure that the system you're building a skill for has an understanding of the basic concepts that you're trying to build your skill for. This one is basically if you're trying to build, let's say, a scheduling app, and you're like. "Oh, you know, will this train take longer to get there than to drive," you know, for example, or, like, "Is this meeting longer than the amount of time I have this conference room set up for?" These are very basic questions and it's interesting that you're asking to do the second order thing, which is, "Figure out the measurement that you want, and then compare it to another measurement and see if it can give you an answer for that. So the failure rates are kind of interesting here.
- [Machine Voice] Alexa, what is the nearest star?
Ronald Schmelzer: Oh I'm gonna pause this, stop.
Kathleen Walch: Yeah, I was gonna say. We ask a lot of humans this question as well so let's ask the audience. What is the nearest star to the Earth?
Ronald Schmelzer: The sun.
Kathleen Walch: Okay, great.
Ronald Schmelzer: Very good.
Kathleen Walch: A lot of people say the moon.
Ronald Schmelzer: I don't know why. Our Chinese audience didn't quite get that, right? Of course, some people might be tempted to say "Alpha Centauri," because they might think you know Proxima--
Kathleen Walch: Less people said that than the moon.
Ronalde Schmelzer: The scientists among us, okay.
- [Alexa] The nearest star is the sun.
Ronald Schmelzer: Okay, Alexa got it right.
- [Machine Voice] Hey Google, what is the nearest star?
- [Google Assistant] Alpha Centauri, here's a summary from the website space.com. The two main stars are Alpha Centauri A, and Alpha Centauri B. They are an average of 4.3 light years from Earth. The third star is Proxima Centauri.
Ronald Schmelzer: All right, so, you get the idea. And there's a lot of them that are the same thing like comparing sizes of animals none of them really got them right and things like that, so, okay.
Related Articles
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer present conclusions from their benchmarking tests on intelligent assistants in this clip from their presentation at SpeechTEK 2019.
27 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the seventh test in their benchmarking project, testing the intelligent assistants' understandings of slang and colloquialisms in this clip from their presentation at SpeechTEK 2019.
20 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the fifth and sixth tests in their benchmarking project, testing the intelligent assistants' emotional IQ and common sense in this clip from their presentation at SpeechTEK 2019.
13 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the fourth test in their benchmarking project, testing the intelligent assistants' handling of reasoning and logic in this clip from their presentation at SpeechTEK 2019.
06 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the first test in their benchmarking project, testing the intelligent assistants' grasp of basic concepts in this clip from their presentation at SpeechTEK 2019.
15 Nov 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer introduce the Cognilytica Voice Assistant Benchmark for testing the intelligence of devices such as Alexa, Siri, Google Home, and Cortana in this clip from their presentation at SpeechTEK 2019.
08 Nov 2019