Audio and Visual Cues in a Two-Talker Divided Attention Speech-Monitoring Task

Abstract

Although audiovisual (AV) cues are known to improve speech intelligibility in difficult listening environments, little is known about their role in divided attention tasks that require listeners to monitor multiple talkers at the same time. In this experiment, a call-sign-based multitalker listening test was used to evaluate performance in two-talker AV configurations that combined zero, one, or two channels of visual information (neither, one, or both talkers visible) with zero, one, or two channels of audio information (no audio, both talkers played from the same loudspeaker, and both talkers played through different, spatially separated loudspeakers). The results were analyzed to determine the relative performance levels that would occur with each AV configuration with target information that was equally likely to originate from either of the two talkers in the stimulus. The results indicate that spatial separation of the audio signals has the greatest impact on performance in multi-channel AV speech displays and that caution should be used when presenting a visual representation of only a single talker unless that talker is known to be the highest priority talker in the combined AV stimulus. Potential applications of this research include the design of improved audiovisual speech displays for multi-channel communications systems.

References

Bertelson, P. , & Radeau, M. (1976). Ventriloquism, sensory interaction, and response bias: Remarks on the paper by Choe, Welch, Gilford, and Juola. Perception and Psychophysics, 29, 578–585.

Bronkhorst, A. (2000). The cocktail party phenomenon: A review of research on speech intelligibility in multiple-talker conditions. Acustica, 86, 117–128.

Brungart, D. , Simpson, B. , Ericson, M. , & Scott, K. (2001). Informational and energetic masking effects in the perception of multiple simultaneous talkers. Journal of the Acoustical Society of America, 110, 2527–2538.

Crispien, K. , & Ehrenberg, T. (1995). Evaluation of the “cocktail party effect” for multiple speech stimuli within a spatial audio display. Journal of the Audio Engineering Society, 43, 932–940.

Driver, J. (1996). Enhancement of selective attention by illusory mislocation of speech sounds due to lip-reading. Nature, 384, 66–68.

Driver, J. , & Spence, C. (1994). Spatial synergies between auditory and visual attention. In C. Umilta & M. Moscovitch (Eds.), Attention and performance XV (pp. 311–331). Cambridge, MA: MIT Press.

Ericson, M. , & McKinley, R. (1997). The intelligibility of multiple talkers spatially separated in noise. In R. H. Gilkey & T. R. Anderson (Eds.), Binaural and spatial hearing in real and virtual environments (pp. 701–724). Hillsdale, NJ: Erlbaum.

McGurk, H. , & McDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748.

Nelson, W. T. , Bolia, R. S. , Ericson, M. A. , & McKinley, R. L. (1999). Spatial audio displays for speech communication: A comparison of free-field and virtual sources. In Proceedings of the Human Factors and Ergonomics Society 43rd Annual Meeting (pp. 1202–1205). Santa Monica, CA: Human Factors and Ergonomics Society.

10.

Reisberg, D. (1978). Looking where you listen: Visual cues and auditory attention. Acta Psychologica, 42, 331–341.

11.

Rudmann, D. , McCarley, J. , & Kramer, A. (2003). Bimodal displays improve speech comprehension in environments with multiple speakers. Human Factors, 45, 329–336.

12.

Spence, C. , Ranson, J. , & Driver, J. (2000). Cross-modal selective attention: On the difficulty of ignoring sounds at the locus of visual attention. Perception and Psychophysics, 62, 410–424.

13.

Sumby, W. , & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215.

14.

Summerfield, Q. (1987). Some preliminaries to a comprehensive account of audiovisual speech perception. In B. Dodd & R. Campbell (Eds.), Hearing by eye: The psychology of lipreading (pp. 3–51). New York: Erlbaum.