Video: Benchmarking Voice Assistants, Pt. 4: Understanding Cause & Effect
Learn more about intelligent assistants at the next SpeechTEK conference.
Read the complete transcript of this clip:
Kathleen Walch: Then we decided to test cause and effect. So what happens when ice melts or if you break something in to two parts, how many parts are there? And we thought it was important to ask this series of questions because we wanted to see if it understood the next step and what would happen if you did something.
Ronald Schmelzer: Right and so you can think, for example, we're always thinking about cooking because, first of all, I love to cook and Kathleen and I like to cook. But it's also like a very easy explanation.
Kathleen Walch: Yeah, for everybody to understand.
Ronald Schmelzer: Okay, well, if you gave me the recipe for this, what if I put this in the refrigerator? You know, what if I do this? You'd want the systems to provide a little bit of help there. Otherwise, if you're building an app, let's say if you're building an Alexa skill or a Google app, you have to encode all this knowledge in because the platform doesn't have any of this cooked in to it. Not to play on words there.
- [Woman On Video] Alexa, what happens when you melt ice?
- [Alexa] I don't know that one.
- [Woman On Video] Hey Cortana, what happens when you melt ice?
- [Cortana] According to answers.com, as ice melts, it's framework collapses and it becomes more dense as it turns to liquid water.
Ronald Schmelzer: That's a long-about way of getting there. I figured out what you were trying to ask, which is great. So obviously Cortana, as you start looking at our benchmark and if you look at our data, you'll see that Cortana has been trained much better on scientific and medical and that sort of stuff.
Kathleen Walch: Right, and this also brings up a good point because we've started to see voice assistance in more situational settings. Such as hospital rooms, for example. And if a patient is using it to ask specific questions, like, "What will happen if I take Tylenol in my current condition?" And it doesn't know that or it provides a wrong answer, that can be potentially deadly.
Ronald Schmelzer: Right, and actually it is true, there was an article recently about how many hospitals now are putting voice assistance in to patient care.
Kathleen Walch: Into patient rooms.
Ronald Schmelzer: Some of it's really simple, like you know, change the television channel and you know, don't call the assistant to do those really...
Kathleen Walch: Also with bed settings, it can lift the bed up or down, it can turn the lights on and off, depending on the state that the patient is in. They may not be able to move around, so this helps them feel a little bit more in control of everything.
Ronald Schmelzer: Yeah, but just case in point...
Kathleen Walch: Yeah, so doing basic voice commands with your device, okay, but when you start asking it questions that require a little bit more intelligence, then this is where you need to know its limitations.
Related Articles
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer present conclusions from their benchmarking tests on intelligent assistants in this clip from their presentation at SpeechTEK 2019.
27 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the seventh test in their benchmarking project, testing the intelligent assistants' understandings of slang and colloquialisms in this clip from their presentation at SpeechTEK 2019.
20 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the fifth and sixth tests in their benchmarking project, testing the intelligent assistants' emotional IQ and common sense in this clip from their presentation at SpeechTEK 2019.
13 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the fourth test in their benchmarking project, testing the intelligent assistants' handling of reasoning and logic in this clip from their presentation at SpeechTEK 2019.
06 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the first test in their benchmarking project, testing the intelligent assistants' grasp of basic concepts in this clip from their presentation at SpeechTEK 2019.
15 Nov 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer introduce the Cognilytica Voice Assistant Benchmark for testing the intelligence of devices such as Alexa, Siri, Google Home, and Cortana in this clip from their presentation at SpeechTEK 2019.
08 Nov 2019