Visual perturbation experiment
To better understand audiovisual speech processing, we investigated the effects of viewing time-synchronized videos of a 3D tongue avatar on vowel production by healthy individuals. A group of 15 American English-speaking subjects heard pink noise over headphones and produced the word head under four viewing conditions: First, while viewing repetitions of the same vowel, /ɛ/ (baseline phase), then during a series of “morphed” videos shifting gradually from /ɛ/ to /æ/ (ramp phase), followed by repetitions of /æ/ (maximum hold phase), and finally repetitions of /ɛ/ (after effects phase). Results of a formant frequency (F1) analysis indicated that the visual mismatch phases (ramp and maximum hold) caused all subjects to align their productions to the visually-presented vowel, /æ/. No subjects reported being aware that their vowel quality had changed. We conclude that the visual moving tongue stimuli produced entrainment to the viewed vowel category, rather than adaptation in the opposite direction of the perturbation. Further experimentation is needed to determine whether these effects are due to inherent imitation behaviors or subjects’ lack of agency with the tongue avatar.
We are now investigating these same effects in adults with Parkinson’s disease, individuals who typically exhibit hypoarticulation, presumably from overestimating their motor gestures.
For more information:
Katz, W.F., and Prabhakaran, D. (2016). Sensorimotor response to visual imagery of tongue displacement. Interspeech, San Francisco, 2090 – 2094.
Katz, W.F., and Prabhakaran, D. (in progress). Sensorimotor response to visual tongue movement during vowel production by individuals with Parkinson’s disease.