Notre Dame researchers are developing AI technology to interpret nonverbal communication
Voice4PIMD analyzes an individual's expressions, gestures, eye gaze, and other nonverbal cues—and translates them into information caregivers can understand.
NOTRE DAME, Ind., Oct. 6, 2026 /PRNewswire/ -- Haley Hilt may not speak, but her grandfather, Ed Fennell, will tell you she says plenty.
After nearly two decades caring for Haley, who has a rare neurodevelopmental disorder, Fennell recognizes the subtle gestures, expressions, and sounds that signal when she is happy, uncomfortable, tired, or simply wants something. But he also worries about the future: Who will understand those signals when the people who know her best are no longer there?
"The vast majority of nonverbal or minimally verbal people do have something to say. They're people! They have joy. They have sorrow. They have pain. They have great feelings. They just don't have the language to do it," Fennell says.
Researchers at the University of Notre Dame are exploring whether artificial intelligence can help bridge that gap.
Making nonverbal communication visible
Computer engineering scientist Jarek Nabrzyski and his team are developing Voice4PIMD, a personalized AI system designed for people with profound intellectual and multiple disabilities.
Using cameras and sensors, the technology can capture facial expressions, eye gaze, body movements, heart rate, vocalizations, and other signals. Caregivers help train the system to recognize what those cues mean for a particular person, with the eventual goal of translating them into text or audio notifications.
The technology is intentionally individualized. A smile, movement, or sound may mean something entirely different from one person to another.
That personalization could eventually offer families something difficult to replicate: continuity of communication as caregivers change over a person's lifetime.
Advancing AI for an underserved population
The research is still in its early stages. Nabrzyski's team is beginning additional testing with families through LOGAN Community Resources in South Bend, Indiana, while researchers are also exploring applications for people with autism and dementia.
People with profound disabilities, he says, have historically received too little attention from research funding agencies—leaving significant questions about communication, caregiving, and quality of life unexplored.
For Notre Dame researchers, Voice4PIMD represents one possibility for what human-centered AI can accomplish when technological advances begin with the dignity and needs of the person.
Learn more about how Notre Dame is fighting for the future of nonverbal communication.
Contact: [email protected].
About the University of Notre Dame
The University of Notre Dame is a leading Catholic research university and a member of the Association of American Universities (AAU), a consortium of the nation's leading public and private research universities. The only explicitly religious institution in the AAU, Notre Dame advances research and scholarship in service to humanity, pursuing discoveries that address complex challenges, improve lives, and contribute to the common good.
SOURCE University of Notre Dame
Share this article