0 + 1 > 1

Experiment 1

In Experiment 1, we examined auditory-visual interactions in a coherent-motion-detection task using a two-interval forced-choice paradigm. The task was to determine which of two stimulus intervals contained visual coherent motion. It is critical to note that, in addition to including classic unisensory and auditory-visual conditions, we investigated the influence of auditory motion on performance during this visual detection task when identical auditory motion was presented during the two intervals and the auditory stimuli therefore did not provide information useful for the detection task.

Method

Stimuli and design

Visual stimuli were dynamic dot patterns of low motion coherence (i.e., only a small minority of the dots moved in the same direction). Each pattern was displayed for an interval of 600 ms, and dot speed was 60°/s. We used only one coherent-motion direction: leftward. Colocalized auditory motion (with the same duration and speed as the visual motion) was created by varying the relative amplitude between speakers placed to the left and right of the monitor on which the visual stimuli were displayed. This process produced a strong percept of lateral motion.

For each subject, we determined the visual and auditory coherence levels required to achieve each of three levels of accuracy in the task. These coherence levels were determined by obtaining psychometric functions for each modality from separate calibration blocks that were run 1 to 2 days before the experimental session. For each subject, visual coherence levels that were chosen produced approximately 55% to 60%, 70% to 75%, and 80% to 85% correct detection in the calibration session; these levels corresponded to difficult, intermediate, and easy levels of difficulty, respectively. Auditory coherence levels that were selected produced approximately 75% and 90% correct detection.

Procedure

The direction, speed, location, and timing (within each stimulus interval) parameters of the coherent visual motion were fixed across trials, and therefore there was little uncertainty about the motion signal. Each trial started with a fixation point, on which observers were instructed to fixate throughout the trial. Subjects performed a two-interval, forced-choice visual-coherent-motion-detection task, in which they were shown two displays in separate intervals: One interval (either the first or second) contained coherent visual motion, and the other contained only random motion (see Fig. 1). At the end of each trial, observers were prompted to press one of two keys to indicate in which interval they perceived coherent motion. Feedback (“correct” or “incorrect”) was provided after each response to keep the participants engaged in the task over the course of the experiment. Subjects were told that although the task was visual, they should do their best to pay attention to both auditory and visual stimuli.

OPEN IN VIEWER

Fig. 1.

Trial types used in Experiment 1 (a–d) and Experiment 2 (b–d). In all trials, observers were presented with two sequential stimulus intervals in which they were shown patterns of moving dots (represented here with small arrows for purposes of illustration). In one interval, some of the dots moved in the same direction (highlighted here with shading), and in the other interval, the dots moved randomly. Speakers were placed to the left and right of the monitor on which the visual stimuli were presented, and observers heard a sound during each interval. The direction of the sound varied with the trial type. After the second interval, a response prompt asked observers to identify whether the first or the second interval contained coherent motion. In informative-congruent trials (a), the interval with visual coherent motion was accompanied by sound moving in the same direction, and the interval with random motion was accompanied by nonmoving sound. In noninformative-congruent trials (b), both intervals were accompanied by sound moving in the same direction as the visual coherent motion. In noninformative-incongruent trials (c), both intervals were accompanied by sound moving in the direction opposite that of the coherent visual motion. In visual-only trials (d), no sound accompanied the images.

Sixty-three subjects were divided into three groups. All three groups completed the same visual-only trials (in which there was no auditory stimulus; Fig. 1d), but each group received a different type of audiovisual (AV) trial. Observers in the informative-congruent group received AV trials with sound that was informative for the task (i.e., sound was present in both stimulus intervals, but the sound moved only in the interval with coherent visual motion and in the direction of that motion; Fig. 1a). Observers in the noninformative- congruent group received AV trials in which identical sound was presented in both intervals (i.e., the sound moved in the same direction as the coherent visual stimulus; Fig. 1b). Because the sound moved identically in both intervals, it provided no indication as to which interval contained the visual coherent motion and thus provided no useful information for the detection task. If the observers in this group closed their eyes, they could not do the task. Observers in the noninformative-incongruent group received AV trials in which sound moved in both intervals but in the direction opposite that of the coherent visual motion (Fig. 1c).

Each participant completed a total of 960 trials. Twenty-five percent of these trials were visual only (80 trials at each visual coherence level: easy, medium, and difficult), and 75% were AV trials (120 trials of each of two types at each of the three visual coherence levels). Trials were presented in pseudorandom order. See the Supplemental Material available online for additional details on the methodology and participants.

Results

Because the results from the two auditory-coherence levels were similar, we have collapsed these data. Although the three groups performed similarly on visual-only trials, the trials featuring AV stimuli yielded different results; the Group × Trial Type × Visual Difficulty interaction was significant, F(4, 120) = 8.4862, p < .01. This effect was not influenced by the interval in which the visual target was presented (the four-way interaction was not significant). As expected, the Trial Type × Visual Difficulty interaction was significant in the group that completed informative-congruent trials, F(4, 40) = 15.15, p < .01. Specifically, subjects performed better when the visual stimulus was accompanied by congruent informative sound than when it was not accompanied by sound (Fig. 2a), both at the difficult level, t(20) = 5.69, p = .000, and at the medium level of difficulty, t(20) = 4.27, p = .0004 (two-tailed, paired t test using Bonferroni-corrected α = .05/9 = .0055).

OPEN IN VIEWER

Fig. 2.

Results of Experiments 1 and 2: mean percentage of correct responses as a function of visual difficulty level and trial type. Results of Experiment 1 are shown separately for the three participant groups. Each group completed visual-only trials in addition to one type of audiovisual trials: (a) informative congruent, (b) noninformative congruent, or (c) noninformative incongruent. Results of Experiment 2 (d) are shown for the single participant group, which completed visual-only, noninformative-congruent, and noninformative-incongruent trials. Error bars show withingroups standard errors. Asterisks denote significant differences between trial types (p < .05, Bonferroni-corrected for multiple comparisons).

Surprisingly, visual motion detection was also significantly enhanced by the noninformative congruent sound, though to a lesser degree (Fig. 2b). The Trial Type × Visual Difficulty interaction was significant, F(2, 40) = 3.60, p < .05. Sound had an effect at the medium difficulty level, t(20) = 3.11, p = .005 (two-tailed, paired Bonferroni-corrected t test of visual-only vs. noninformative-congruent trials), but not at the other difficulty levels. In contrast, noninformative sound moving in the direction opposite that of visual motion did not yield any benefit (Fig. 2c), nor did it cause a significant decrease in performance (p > .05 for all difficulty levels) relative to visual-only trials. The size of the effect of noninformative congruent sound on the medium-difficulty visual task was moderate (Cohen’s d = 0.42).

Experiment 2

To scrutinize the relationship between the visual-only, noninformative-congruent, and noninformative-incongruent trials further, we conducted Experiment 2. In this experiment, the three trial types were presented in pseudorandom order to each observer, thus allowing comparison of all three trial types within the same individual.

General Discussion

The patterns of results found in the two experiments reported here are highly consistent despite significant changes in experimental design, such as within-subjects versus between-groups design and feedback versus no feedback. This consistency of results provides evidence for the robustness of the observed effects. Taken together, these data demonstrate that identical moving sound presented in two stimulus intervals can improve the detection of visual coherent motion in one of the intervals, even though it does not provide information about which interval contains the visual coherent motion. Further, this enhancement occurs only when auditory stimuli move in the same direction as visual stimuli.

How can noninformative sound improve performance? In theory, facilitation could be caused by auxiliary factors, but the pattern of results makes this possibility unlikely. For example, the absence of enhancement in the noninformative-incongruent trials rules out the possibility that the effect results from general attentional modulation by sound. It is also unlikely that the enhancement stems from uncertainty reduction by sound, for two reasons. First, the direction, speed, timing, and position of the visual coherent-motion signal were fixed, and observers had been familiarized with the signal during calibration; therefore, there was little uncertainty—if any—about the signal. Second, any putative reduction in uncertainty should have also occurred in the noninformative-incongruent trials.

Similarly, the enhancement cannot be due to any influence of auditory motion on eye movements. First, subjects were instructed to fixate. Second, the lifetime of each dot was no longer than 20 ms, thus not allowing for any benefit from smooth-pursuit eye movement. Third, the opposite-moving sound in the incongruent trials would have deteriorated performance, which was not the case. Therefore, it appears that the enhanced performance in the noninformative-congruent trials reflects sensory interactions between visual and auditory modalities. Indeed, the sensory interactions would be expected to be specific to concordant directions of visual and auditory motion for which integration would occur, and these interactions should not occur for strongly conflicting auditory and visual directions that would not get integrated (i.e., in the noninformative-incongruent trials; Shams & Beierholm, 2010). It is also interesting to note that this pattern of results is consistent with previously reported patterns of auditory modulation of visual-motion perceptual learning, in which congruent sound facilitates learning (Kim, Seitz, & Shams, 2008; Seitz, Kim, & Shams, 2006), and incongruent sound does not affect learning (Kim et al., 2008).

It is important to note that the pattern of AV enhancement differed between the informative- and noninformative-congruent groups in Experiment 1; this suggests that different mechanisms underlie these two types of enhancement. The informative-congruent group’s pattern demonstrates inverse effectiveness (Meredith & Stein, 1983): that is, maximal benefit from sound for the weakest visual stimulus and little or no benefit for the easiest visual condition, which already produces high performance. In contrast, the noninformative-congruent group shows more benefit for the medium visual-difficulty level and no benefit for the weakest visual stimulus.

This pattern of effects can be qualitatively explained by a combination of two factors. The differential effect for the difficult (low-coherence) visual stimulus compared with the medium-coherence visual stimulus in the noninformative- congruent trials could potentially be due to a multiplicative interaction between the two modalities at a sensory level. In this type of interaction, if either modality is weak, the interaction itself is also weak and thus difficult to detect, which could explain why no benefit was observed at the difficult-visual-stimulus level. In contrast, the absence of an enhancement for the easy (high-coherence) visual stimulus may simply be due to a ceiling effect, as similarly no enhancement occurred at this level of difficulty in the informative-congruent trials. Therefore, the obtained pattern of results could conceivably be due to a subset of directionally tuned neurons in a visual motion-processing area (e.g., middle temporal, or MT) that are multiplicatively modulated by auditory neurons with the same directional preferences. Indeed, some functional neuroimaging studies have suggested that sound can modulate MT+ activation (Poirier et al., 2005; Scheef et al., 2009), and multiplicative modulation of neurons in MT has been reported to occur in the case of attention (Treue & Martínez Trujillo, 1999). Moreover, in monkeys, ventral intraparietal area receives input from both MT and auditory regions and contains auditory-visual neurons with directional motion selectivity (Graziano, 2001; Lewis & Van Essen, 2000). However, further research is necessary to uncover the neural mechanisms underlying these interactions.

Because a two-interval forced-choice paradigm was used in this study, the improved accuracy in visual motion detection found in the presence of congruent but noninformative auditory motion cannot be attributed to a change in response criterion, and therefore it indicates improved sensitivity in visual detection. These results provide evidence for sensory interaction between auditory and visual motion-processing pathways. These findings may appear to conflict with results from previous studies that examined auditory-visual interactions in motion perception and did not find evidence of sensory integration. We suspect that the absence of evidence for sensory interactions in these previous studies is due to the fact that in those studies, both modalities are informative (i.e., the task could be performed on the basis of each modality alone). Therefore, the sensory interaction effects—which appear to be significantly smaller in size—are masked by the comparatively larger perceptual and decision-level interactions. Comparing Figures 2a and 2b, it can be seen that the auditory-visual interactions as observed in the informative-congruent trials (Fig. 2a) are much larger than the facilitatory effect seen in the noninformative-congruent trials (Fig. 2b). The reason we were able to detect the sensory interactions between the two modalities is likely because the higher-level interactions are disabled by rendering sound noninformative for the task, in effect unmasking the smaller sensory effects.

The observed enhancement of visual detection accuracy in the presence of noninformative congruent auditory motion cannot be explained by the standard models of multisensory integration (e.g., Alais & Burr, 2004b; Angelaki, Gu, & DeAngelis, 2009; Ernst & Banks, 2002; Ernst & Bülthoff, 2004; Witten & Knudsen, 2005) depicted in Figure 3a. These models are based on the assumption that sensory cues (e.g., auditory signal A and visual signal V) are caused by a source s in the environment, and given this source, that the sensory signals are conditionally independent of each other. This assumption of conditional independence is justified as follows: Up to the stage of processing in which the interaction between modalities occurs (which can be as low level as primary cortical or subcortical brain areas), it is assumed that the two pathways are separate and therefore the noise processes that corrupt the sensory signals are independent of each other. The absence of an arrow between A and V in Figure 3a represents this conditional independence. The cross-modal interactions in these models occur during the process of inferring the value of s (e.g., direction of motion, distance, size) from sensory cues A and V (i.e., after an estimate of s is generated by each sensory modality).

OPEN IN VIEWER

Fig. 3.

Graphical models (Pearl, 1988) typifying models of multisensory integration. The black and gray arrows represent the direction of generative processes and inference processes, respectively. In the standard model of multisensory integration (a), sensory cues, for example, auditory signal A and visual signal V, are caused by a source s in the environment, and given this source, the sensory signals are conditionally independent of each other. In an alternative model (b), there can be an interaction between the two modalities at the level of sensory representations, as indicated by the arrows between A and V.

In contrast, the results of the current study can most easily be explained by auditory-visual sensory interactions (Fig. 3b), and for the sake of parsimony, that is the explanation we favor. In such a model, the sensory representations are not conditionally independent of each other, and interaction between the two modalities occurs at both the perceptual and sensory levels of processing. However, an alternative explanation based on perceptual interactions (i.e., without the link between A and V) cannot be ruled out entirely. For example, it is conceivable that a causal-inference model (Körding et al., 2007; Shams, Ma, & Beierholm, 2005; Wozny, Beierholm, & Shams, 2008) operating on complex features or a combination of features (e.g., making an inference about the coherence level of motion as well as the direction of motion) could explain the observed interactions. Further research is required to explore these possibilities and to gain further insight into the computational as well as neural mechanisms of auditory-visual motion processing.

Conclusion

Our results indicate that auditory motion can facilitate perception of visual motion, even when it does not provide information relevant for the task. This finding suggests that interaction between the two modalities in motion processing can occur at a sensory level in addition to higher computational levels.