Video: Benchmarking Voice Assistants, Pt. 7: Idiomatic Speech
Learn more about intelligent assistants at the next SpeechTEK conference.
Read the complete transcript of this clip:
Kathleen Walch: We also thought it was important to test on slang and colloquialism. Not everybody uses the same word to describe the same object, so we wanted to see how good they were with that. So some of the questions-- it varies by culture and location, even age-based. Some people use different words for things. So a good example is soda. Some people call it pop. Some people call it soda. You can also call it Coke. You're kind of all talking about the same thing, but you're using different words to describe that. So were these able to get it? We also, when we did this, since we did have interns that were not from the United States, they didn't--
Ronald Schmelzer: Get the slang.
Kathleen Walch: Yeah, understand all of this as well.
Ronald Schmelzer: They're like, "Hit the sack"? I'm like, "Yeah, tired, uh, yeah ." "Down to earth? "Are you a space person?" Like, oh my goodness. I mean, I'm sure Chinese has all sorts of expressions that we'd be like, "I don't understand why."
Kathleen Walch: That just goes to show, if you speak English, you don't always understand some of those slangs.
Ronald Schmelzer: But the important thing is, we're like, "Well, have these devices been trained in slang "in addition to understanding accents?" So we gave them a test. Let's see how these devices did.
- [Tester] Alexa, I'd like to shoot the breeze with you.
Ronald Schmelzer: Nope.
- [Tester] Siri, I'd like to shoot the breeze with you.
- [Siri] Who, me?
Ronald Schmelzer: Oh, not a bad one. So if you see the chart here for this one, Siri actually did really well here. Actually, there's a lot of orange responses and, of course, a bunch of red, but a whole bunch of green. So it's like, it's clear that when they designed Siri, it was meant to be much more conversational and have this sort of personality. I don't know how to explain it, but like it was trained to understand more of these colloquialisms, and you can have a random banter. So of course, if you gave your three-year-old access to Alexa, Google Home, Siri, and Cortana, they would probably have the most enjoyable experience, not the most productive one, but the most enjoyable experience with Siri, is sort of like our assessment.
Related Articles
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer present conclusions from their benchmarking tests on intelligent assistants in this clip from their presentation at SpeechTEK 2019.
27 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the fifth and sixth tests in their benchmarking project, testing the intelligent assistants' emotional IQ and common sense in this clip from their presentation at SpeechTEK 2019.
13 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the fourth test in their benchmarking project, testing the intelligent assistants' handling of reasoning and logic in this clip from their presentation at SpeechTEK 2019.
06 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the third test in their benchmarking project, testing the intelligent assistants' grasp of cause and effect in this clip from their presentation at SpeechTEK 2019.
29 Nov 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the first test in their benchmarking project, testing the intelligent assistants' handling of comparison questions in this clip from their presentation at SpeechTEK 2019.
22 Nov 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the first test in their benchmarking project, testing the intelligent assistants' grasp of basic concepts in this clip from their presentation at SpeechTEK 2019.
15 Nov 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer introduce the Cognilytica Voice Assistant Benchmark for testing the intelligence of devices such as Alexa, Siri, Google Home, and Cortana in this clip from their presentation at SpeechTEK 2019.
08 Nov 2019