Reducing viseme confusion in speech reading

Abstract

Although our sense of hearing, smell, and vision allow us to perceive things at a distance, the detection of many day-to-day events relies exclusively on our hearing. For example, finding a ringing phone lost in a sofa, hearing a child cry in another room, and use of a car alarm to locate a vehicle in a car park. However, individuals with total or partial hearing loss have difficulty detecting the audible signals in these situations. We have developed VisAural, a system that converts audible signals into visual cues. Using an array of head-mounted microphones, VisAural detects the direction of a sound, and places LEDs at the periphery of the user's visual field to guide them to the source of the sound. We tested VisAural with nine people with hearing impairments and found that this approach holds great promise but needs to be made more responsive before it can be truly helpful.