Video: Benchmarking Voice Assistants, Pt. 2: Basic Concepts
Learn more about intelligent assistants at the next SpeechTEK conference.
Read the complete transcript of this clip:
Ronald Schmelzer: The next category question is, we basically try to figure out, can these devices answer very basic concepts? And we have a few of them, for example, "Is a ton a unit of measurement? How much does a ton of peas weigh?" Right, a standard kids' question, right?
Kathleen Walch: And so you may say, "Well, that's a silly question. I'm never gonna be asking that, but if you're ordering mulch, for example, or you work with very large volumes of weight, it's nice to know that these systems are able to handle that. Also, if you're building a skill on top of one of these devices, and it can't handle what a ton is, or what a pound is, or what a kilo is, you need to know that before you build your skill on top of that.
Ronald Schmelzer: And also, we're asking sort of a little bit of an interesting question where we're sort of giving the answer in the question, right? So, "How much does a pound of peas weigh?" is, well, we just said it was a pound. So, the question is, does the system understand that we're trying to get-- we're providing an answer in the question. And, well, we got some interesting responses. You see a lot of those red bars. Each one of those bars is the graph of the various different assistants and there are categories of response-- green meaning getting it right, red meaning getting it really wrong, yellow meaning irrelevant, and orange--
Kathleen Walch: Yeah the yellow and orange both, did not provide the right answer. There were just different categories of not providing a right, so one isn't better than two, we just decided to use numerical instead of letters or characters, whatever else.
Ronald Schmelzer: But there's an awful lot of red here where there shouldn't be. So let's play Alexa, and I think it's Google Home answering this question.
[Computerized Voice] Alexa, how much does a ton of peas weigh? Alexa, how much does a ton of peas weigh?
[Computerized Voice] Alexa, how much does a ton of peas weigh? Just refused to answer that.
[Computerized Voice] Alexa, how much does a pound of peas weigh?
[Alexa] One pound of any material equals one pound.
Ronald Schmelzer: Huh, interesting.
[Computerized Voice] Alexa, is a pound a unit of measurement?
[Alexa] Sorry, I don't know that one.
[Computerized Voice] Alexa, is a pound a unit of measurement?
[Alexa] No, one pound is not a unit of measurement.
Ronald Schmelzer: Hm, all right. Something is going on there, it doesn't understand tons, it understands pounds, it understands kilos, it doesn't... There's a lot of.. you have to kind of.. When you're building a skill, be aware of its, obvious data bias, right?
Kathleen Walch: Right. Be aware of its data bias. Be aware of its limitations as well.
Related Articles
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer present conclusions from their benchmarking tests on intelligent assistants in this clip from their presentation at SpeechTEK 2019.
27 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the seventh test in their benchmarking project, testing the intelligent assistants' understandings of slang and colloquialisms in this clip from their presentation at SpeechTEK 2019.
20 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the fifth and sixth tests in their benchmarking project, testing the intelligent assistants' emotional IQ and common sense in this clip from their presentation at SpeechTEK 2019.
13 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the fourth test in their benchmarking project, testing the intelligent assistants' handling of reasoning and logic in this clip from their presentation at SpeechTEK 2019.
06 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the third test in their benchmarking project, testing the intelligent assistants' grasp of cause and effect in this clip from their presentation at SpeechTEK 2019.
29 Nov 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the first test in their benchmarking project, testing the intelligent assistants' handling of comparison questions in this clip from their presentation at SpeechTEK 2019.
22 Nov 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer introduce the Cognilytica Voice Assistant Benchmark for testing the intelligence of devices such as Alexa, Siri, Google Home, and Cortana in this clip from their presentation at SpeechTEK 2019.
08 Nov 2019