Notre Dame Researchers Developing AI Technology to Interpret Nonverbal Communication

Haley Hilt might not speak, but her grandfather, Ed Fennell, will tell you she says plenty.

After nearly two decades caring for Haley, who has a rare neurodevelopmental disorder, Fennell recognizes the subtle gestures, expressions, and sounds that signal when she is happy, uncomfortable, tired, or simply wants something. But he also worries about the future: Who will understand those signals when the people who know her best are no longer there?

"The vast majority of nonverbal or minimally verbal people do have something to say. They're people! They have joy. They have sorrow. They have pain. They have great feelings. They just don't have the language to do it," Fennell says.

Researchers at the University of Notre Dame are exploring whether artificial intelligence can help bridge that gap.

Computer engineering scientist Jarek Nabrzyski and his team are developing Voice4PIMD, a personalized AI system for people with profound intellectual and multiple disabilities.

Using cameras and sensors, the technology can capture facial expressions, eye gaze, body movements, heart rate, vocalizations, and other signals. Caregivers help train the system to recognize what those cues mean for a particular person, with the eventual goal of translating them into text or audio notifications.

The research is still in its early stages. Nabrzyski's team is beginning additional testing with families through LOGAN Community Resources in South Bend, Ind., while researchers are also exploring applications for people with autism and dementia.

People with profound disabilities, he says, have historically received too little attention from research funding agencies, leaving significant questions about communication, caregiving, and quality of life unexplored.