Abstract
This article aims to investigate whether there is a difference in audiovisual integration in school-aged children (aged 6 to 13 years; mean age = 9.9 years) between the selective attention condition and divided attention condition. We designed a visual and/or auditory detection task that included three blocks (divided attention, visual-selective attention, and auditory-selective attention). The results showed that the response to bimodal audiovisual stimuli was faster than to unimodal auditory or visual stimuli under both divided attention and auditory-selective attention conditions. However, in the visual-selective attention condition, no significant difference was found between the unimodal visual and bimodal audiovisual stimuli in response speed. Moreover, audiovisual behavioral facilitation effects were compared between divided attention and selective attention (auditory or visual attention). In doing so, we found that audiovisual behavioral facilitation was significantly difference between divided attention and selective attention. The results indicated that audiovisual integration was stronger in the divided attention condition than that in the selective attention condition in children. Our findings objectively support the notion that attention can modulate audiovisual integration in school-aged children. Our study might offer a new perspective for identifying children with conditions that are associated with sustained attention deficit, such as attention-deficit hyperactivity disorder.
Introduction
In real life, events are multisensory experiences and are rarely unimodal. For example, in a general communicative exchange with our friends, we hear the words, see the friend’s lips moving, and notice their expressions simultaneously. The brain can integrate this multisensory information to provide a complete and coherent cognition and can consequently allow for appropriate behavioral responses (Simon, 2008). This means that in the multisensory world, we form our perceptions and exhibit our behaviors based upon the interaction of information from various modalities. In adults, bimodal audiovisual stimuli can be discriminated or detected more accurately and faster than unimodal auditory or visual stimuli presented alone (Giard & Peronnet, 1999; Stein, London, Wilkinson, & Price, 1996; Wu, Yang, Gao, & Kimura, 2012; Yang et al., 2014; Yang et al., 2013). This phenomenon is called audiovisual integration.
Audiovisual integration abilities have also been investigated in children (Barutchu, Crewther, & Crewther, 2009; Brandwein et al., 2011; Dionne-Dostie, Paquette, Lassonde, & Gallagher, 2015). In a simple visual and auditory stimuli detection task, school-aged children exhibited faster reaction times to audiovisual stimuli when compared with the unimodal stimuli (Barutchu et al., 2009). Furthermore, the neural processing mechanism has been researched in children using high-density electrical mapping. The results showed that audiovisual integration occurred over frontal-central scalp regions from approximately 100 to 120 ms and approximately 200 ms (Brandwein et al., 2011). These results indicated that children have the ability to integrate bimodal stimuli even though their brains are immature. Moreover, the developmental progression of multisensory temporal function was examined by analyzing responses on an audiovisual simultaneity judgment task in 6- to 23-year-old participants (Neil, Chee-Ruiter, Scheier, Lewkowicz, & Shimojo, 2006). The results demonstrated that sensitivity to audiovisual temporal asynchrony increases with age, with adults being less likely to bind more temporally disparate multisensory stimuli than are younger participants. Furthermore, development of multisensory spatial integration was also investigated, which found age-dependent eccentricity and modality effects on response latencies and that audiovisual integration emerges late in the first year of life (Hillock-Dunn & Wallace, 2012). These results are consistent with neurophysiological findings from multisensory sites in the superior colliculus of infant monkeys, which show that multisensory enhancement of responsiveness is not present at birth but emerges later in life. In addition, some studies have demonstrated that, in contrast to human adults, school-aged children are not yet able to optimally combine this type of sensory information. For example, Nardini, Jones, Bedford, and Braddick (2008) showed that young children did not integrate visual and movement-related information during spatial navigation (Nardini et al., 2008). Barutchu et al. (2009) also observed reduced audiovisual facilitation of motor response times in 8- and 10-year-old children relative to adults; however, Corinne et al. (2007) reported no effect of age in 5- to 19-year-olds on perception of the sound-induced flash illusion (Corinne et al., 2007). Therefore, differences in the rate of maturation across sensory systems or the degree of task complexity may influence multisensory processing.
Some researchers have investigated audiovisual integration that is elicited by a pair of visual and auditory stimuli that were presented over auditory background noise in school-aged children and adults (Barutchu et al., 2010). During the audiovisual detection task, the auditory signal-to-noise ratios were manipulated using white noise (0.1–24 kHz), and the results showed that the facilitating effect of audiovisual integration diminished with reduced auditory signal-to-noise ratio in both adults and children. These results suggested that changes in audiovisual integration with increased auditory noise might be due to changes in attention bias. Attention is an important factor in the integration processing of visual and auditory stimuli. Attention can modulate audiovisual integration processes in adults (Fairhall & Macaluso, 2009; Giray & Ulrich, 1993; Koelewijn, Bronkhorst, & Theeuwes, 2010; Miller, 1982; Talsma, Senkowski, Soto-Faraco, & Woldorff, 2010). The effects of selective attention to a single sensory modality (visual or auditory stimulus) and divided attention (paying attention to both the visual and auditory stimuli) on audiovisual integration have been investigated with verbal stimuli in adults (Mozolic, Hugenschmidt, Peiffer, & Laurienti, 2008). The results indicated that audiovisual integration was attenuated under the selective attention condition. Talsma, Doty, and Woldorff (2007) used tones and simple stimuli to demonstrate that integration effects were absent in the early stage during selective attention using event-related potential measurements in adults (Talsma et al., 2007); however, whether similar audiovisual integration occurs in school-aged children under divided and selective attention condition remains unclear.
To explore the relationship between attention and audiovisual integration in school-aged children, we designed a visual and/or auditory detection task that included three blocks. In Block 1, the participants were instructed to respond to all the target stimuli, including a visual target, auditory target, and audiovisual target. Thus, attention was divided into visual and auditory modalities. This is called divided attention. In Block 2, the participants were told to attend to the visual target stimuli containing the unimodal visual stimulus and the visual segment of the audiovisual stimuli while ignoring all auditory stimuli. Therefore, this block tested selective attention to visual stimuli. In Block 3, the participants were required to respond to the auditory target stimuli, neglecting visual signals. Therefore, this block tested selective attention to auditory stimuli. By comparing the audiovisual integration between divided attention and selective attention, we examined whether the audiovisual integration would be weaker under the selective attention condition than under the divided attention condition in school-aged children.
Methods
Subjects
Thirty healthy volunteers (aged 6–13 years; mean age = 9.9 years) participated in the present study. All of the participants had normal or corrected-to-normal vision (none of the participants were color blind) and normal hearing capabilities. The participants provided written informed consent to participate in this study, which was previously approved by the ethics committee of Hubei University.
Stimuli
Stimulus presentation and response collection were accomplished using Presentation software (Neurobehavioral Systems Inc., Albany, California, USA). Unimodal visual stimuli, unimodal auditory stimuli, and bimodal audiovisual stimuli were presented randomly to the left or the right hemispace. Each stimulus type had two subtypes in total: the target stimulus and the task-irrelevant stimulus.
The visual target stimulus was a red and white block (5.2 × 5.2 cm with a subtending visual angle of 5°) that was presented on a black background on a ^21-inch computer monitor positioned 60 cm in the front of the participant’s eyes (Figure 1(a)). These visual stimuli were randomly presented to the lower left or lower right quadrant of the screen (at a 12° visual angle to the left or right of the center and a 5° angle below the central fixation). The auditory target stimulus consisted of white noise at 60 dB. The auditory stimuli were presented to the left or the right ear through earphones. The audiovisual target stimulus consisted of the simultaneous presentation of both visual and auditory target stimuli. In addition, task-irrelevant stimuli were presented at a frequency of 20% of the total stimuli and were similar to the visual and auditory target stimuli in order to sustain participant’s attention. The task-irrelevant visual stimulus was a black and white block and the task-irrelevant stimulus was a 1000 Hz sinusoidal tone with a linear rise and fall time of 5 ms and amplitude of 60 dB.
The experimental design. (a) Examples of the stimulus sequences in the experiment. Subjects sat approximately 60 cm from the screen. All of the stimuli were presented randomly, and each stimulus had an equal probability of appearing to the left or to the right of the central fixation point. (b) The three blocks. Block 1: the participants were instructed to respond to all target stimuli, including the visual target, auditory target, and audiovisual target. Block 2: the participants were told to attend to the visual stimuli that contained the unimodal visual stimulus and the visual segment of the audiovisual stimuli, while ignoring all auditory stimuli. Block 3: the participants were instructed to respond to the auditory stimuli that contained the unimodal auditory stimulus and the auditory segment of the audiovisual stimuli.
Procedure and Tasks
The experiment was performed in a dimly lit, sound-attenuated, and electrically shielded room (laboratory room; Hubei University, China). Participants sat on a comfortable chair with their head fixed by a chin-rest. Each participant completed all three blocks with different attention to modality conditions (Figure 1(b)). The three blocks were measured in random order as generated a by randomizing function.
First block (divided attention condition): The participants were instructed to attend to all the target stimuli, including the visual target, auditory target, and audiovisual target. Attention was assigned to visual and auditory modalities. Thus, the participants’ task was to indicate whether a target stimulus appeared in the left or the right hemispace by pressing the left or the right key with the index or the middle finger of their right hand as quickly and accurately as possible.
Second block (selective attention to only visual stimulus condition): The participants were told to attend to the visual target stimuli that contained the unimodal visual stimulus and the visual segment of the audiovisual stimuli, while ignoring all auditory stimuli. Thus, the participants’ task was to press the left button of a computer mouse when visual target stimuli were presented on the left side and to press the right button of a computer mouse when visual target stimuli were presented on the right side as quickly and accurately as possible.
Third block (selective attention to auditory stimulus condition): The participants were instructed to respond to the auditory stimuli containing the unimodal auditory stimuli and the auditory segment of the audiovisual stimuli. Thus, the participants’ task was to press the left button of a computer mouse when an auditory target stimulus on the left side and the right button when an auditory target stimulus on the right side as quickly and accurately as possible.
Each block consisted of 100 visual stimuli, 100 auditory stimuli, and 100 audiovisual stimuli. All of the stimuli were presented randomly, and each stimulus had an equal probability of appearing to the left or to the right of the central fixation point. The duration of each type of stimulus was 150 ms, and the inter-stimulus interval ranged between 1300 and 1800 ms. Throughout the experiment, the subjects were required to fix their eyes on a centrally presented fixation point on a screen. The subjects were instructed to not press the button when a task-irrelevant stimulus appeared. At the beginning of the experiment, the participants performed a few practice trials to ensure that they understood the paradigm and became familiar with the stimuli.
Data Analysis
Hit rates and response times to target stimuli were computed separately for each stimulus type. Hit rates were the number of correct responses to target stimuli divided by the total number of target stimuli. Response time data were analyzed for correct responses. The response times were first analyzed to remove outliers, which were defined as responses that occurred faster or slower than three standard deviations from the mean response time for each subject. The data for responses to task-irrelevant stimuli were submitted using false alarm rates. To confirm the level of audiovisual behavioral facilitation, the difference in response times, hit rate, and false alarm rate between unimodal visual and bimodal audiovisual stimuli were calculated (Li, Yang, Sun, & Wu, 2015).
Response times, hit rates, and false alarm rates were submitted to one-way analysis of variance (ANOVA; three levels: visual, auditory, and audiovisual levels) in the divided attention condition. With regard to selective attention, the paired-sample t test was used to analyze the response times, hit rates, and false alarm rates (unimodal visual or auditory stimuli and bimodal audiovisual stimuli). To compare divided attention and visual-selective attention, a repeated-measures ANOVA was used with the factors attention condition (divided and visual-selective attention) and stimulus type (unimodal visual and bimodal audiovisual stimuli), and age as a covariate was analyzed. Similarly, to compare divided attention and auditory-selective attention, an ANOVA was used with the factors attention condition (divided and auditory-selective attention) and stimulus type (unimodal auditory and bimodal audiovisual stimuli), while age was analyzed as a covariate. The Bonferroni correction was applied to adjust the multiple comparisons. All statistical analyses were conducted using SPSS version 17.0 (SPSS, Tokyo, Japan) and the overall alpha level was set at p < .05.
Results
Divided Attention
Mean Response Times, Hit Rates and False Alarm Rates for Divided and Selective Attention Conditions.
Standard deviations are given in parentheses.
These effects for hit rates were statistically expressed as a main effect of the factor stimulus type (F2,87 = 7.82, p = .001). The post-hoc comparisons showed that hit rates differed significantly (p < .001) between unimodal auditory and bimodal audiovisual stimuli; however, no significant difference (p = .474) was found between unimodal visual and bimodal audiovisual stimuli. No main effect of false alarm rates was found for the factor stimulus type (F2,87 = 1.82, p = .168). These results indicated that audiovisual integration occurred in the divided attention condition (the responses to the audiovisual stimuli were faster).
Selective Attention
The paired-sample t test was used to analyze the response times, hit rates, and false alarm rates in response to unimodal visual and bimodal audiovisual stimuli under the visual-selective attention condition. The results showed that no significant difference was found between unimodal visual and bimodal audiovisual stimuli for response times to target stimuli (t = 1.32, p = .196), hit rates to target stimuli (t = 1.13, p = .265), or false alarm rates to standard stimuli (t = 0.38, p = .710). The results indicated that audiovisual integration was absent when only visual stimuli received the attention of the participants.
In the auditory-selective attention condition, the response times to the audiovisual stimuli were faster than unimodal auditory stimuli (t = 5.54, p < .001). Moreover, the paired-sample t test showed that bimodal audiovisual stimuli could be detected more accurately by an individual than can unimodal auditory stimuli being presented alone (t = 2.58, p = .015). Yet, regarding the false alarm rates, there was no significant difference (t = 0.47, p = .645) between unimodal auditory stimuli and bimodal audiovisual stimuli.
Comparison of Divided Attention and Selective Attention
Selective attention includes visual-selective attention and auditory-selective attention; therefore, we compared these components individually with divided attention by submitting the data to an ANOVA. The 2 attention (divided and selective) × 2 stimulus type (unimodal and bimodal) ANOVA analyzed the response times, hit rates, and false alarm rates. And the sphericity was not violated for response times, hit rates, and false alarm rates (p > .05).
Divided Attention and Visual-Selective Attention
A 2 attention (divided and visual selective) × 2 stimulus type (unimodal visual and bimodal audiovisual) ANOVA that analyzed the response times showed a significant interaction (F1,28 = 8.944, p = .006) between attention condition and stimulus type. The post-hoc comparisons found that the response times to the audiovisual stimuli were significantly faster than unimodal visual in the divided (p < .001) but not the visual selective (p = .196) attention condition. Similarly, a 2 attention × 2 stimulus type ANOVA that compared the false alarm rates found no significant interaction between attention condition and stimulus type (F1,28 = 0.307, p = .584). A 2 attention × 2 stimulus type ANOVA that also compared the hit rates showed no significant interaction (F1,28 = 2.361, p = .136) between attention condition and stimulus type.
Divided Attention and Auditory-Selective Attention
Similar comparisons were performed for the response times, hit rates, and false alarm rates in the divided attention and auditory-selective attention conditions. No significant interaction was found between attention condition and stimulus type for the false alarm rate (F1,28 = 0.131, p = .720) and hit rate (F1,28 = 2.224, p = .147). However, a significant interaction was found for the response time (F1,28 = 9.930, p = .004) between attention condition and stimulus type. The interaction between attention condition and stimulus type for the auditory conditions was co-driven by the response time differences between unimodal and bimodal stimuli in both the divided (p < .001) and the auditory-selective (p < .001) attention conditions. These results indicate that the response times were significantly different between divided attention and auditory-selective attention.
The above results indicate that audiovisual integration effect is influenced by attention. To confirm the level of audiovisual behavioral facilitation between the divided attention condition and the selective attention condition, difference in response times, hit rate, false alarm rate between unimodal visual and bimodal audiovisual stimuli was calculated in Figure 2. Additionally, the 2 modality (unimodal auditory–bimodal, unimodal visual–bimodal) × 2 attention (selective and divided) ANOVA that was used to analyze the response times showed a significant interaction (F1,28 = 11.094, p < .005) between modality and attention condition. In addition, post-hoc comparisons indicated that audiovisual behavioral facilitation in the divided attention condition was larger than in the visual-selective attention condition (p < .001; Figure 2(a)) or auditory-selective attention condition (p < .001; Figure 2(b)). No significant interaction was found between modality and attention condition for hit rate (F1,28 = 3.414, p = .075; Figure 2(c) and (d)) and false alarm rate (F1,28 = 0.121, p = .730; Figure 2(e) and (f)). These results demonstrate that attention influences the level of audiovisual behavioral facilitation.
Difference of response time, hit rate, false alarm rate. The differences were obtained by subtracting response for the bimodal target stimuli from response for the unimodal target stimuli. (a) Difference of response time under selective attention to visual stimuli and divided attention conditions. (b) Difference of response time under selective attention to auditory stimuli and divided attention conditions. (c) Difference of hit rate under selective attention to visual stimuli and divided attention conditions. (d) Difference of hit rate under selective attention to auditory stimuli and divided attention conditions. (e) Difference of false alarm rate under selective attention to visual stimuli and divided attention conditions. (f) Difference of false alarm rate under selective attention to auditory stimuli and divided attention conditions.
Discussion
The results of this study show that, overall, directing attention to the auditory and/or visual information in an audiovisual detection task can have an influence on the integration of audiovisual information. In the divided attention condition, significant audiovisual integration occurred in school-aged children. This result is in agreement with findings from previous studies, which showed that the response time to bimodal audiovisual stimuli is faster than to unimodal auditory or visual stimuli in children (Barutchu et al., 2009; Brandwein et al., 2011) and adults (Molholm et al., 2002; Teder-Sälejärvi, Russo, McDonald, & Hillyard, 2005).
Nevertheless, although participants are fully informed to focus their attention on a certain modality, they were still unable to completely ignore the stimuli from other modality, particularly in auditory-selective condition. Thus, audiovisual integration was found under auditory-selective attention while ignoring visual stimuli. This result is consistent with previous studies in adults (Li, Wu, & Touge, 2010; Lovelace, Stein, & Wallace, 2003), which showed that task-irrelevant visual information enhanced auditory detection. This phenomenon indicated that ignoring visual information was also difficult in children. However, in the present study, task-irrelevant auditory stimuli failed to promote the visual judgment. These results are not in agreement with those of previous studies in adults (Lippert, Logothetis, & Kayser, 2007; Stein et al., 1996; Wu, Li, Bai, & Touge, 2009), which showed that visual detection was improved by simultaneous sound. Some possible reasons for these inconsistent results may be because the sound processing mechanism of children is different than that of adults (Fox, Anderson, Reid, Smith, & Bishop, 2010; Orekhova et al., 2013; Ponton, Eggermont, Khosla, Kwong, & Don, 2002). Sound processing has been investigated in an auditory Go/No-go task, which found that P2 and N2 components were significantly different between children and adults even though some early and late Event-Related potential (ERP) components showed similarities (Barry, De Blasio, & Borchard, 2014). Additionally, processing of novel auditory information involves the more frontally distributed P3 components and larger late frontal negativities in children than in adults (Maatta et al., 2005). Therefore, it is perhaps because of the child’s sound processing features at the immature stage that task-irrelevant auditory stimuli could not facilitate visual detection. However, further electrophysiological studies are needed to confirm and elucidate these neural mechanisms to improve our understanding of the developmental aspects of cognitive processing.
In this study, the audiovisual integration facilitation effect in the divided attention condition was stronger than that in the selective attention condition (Figure 2). The relationship between the development of sustained attention and the ability to inhibit responses to irrelevant stimuli has been discussed in children (Paus, 1989). Paus found a positive correlation between the number of signals that were detected and the number of successfully suppressed external stimuli, and this ability might be related to the maturation of frontal cortical functions. Klimkeit et al. reported that the development of attention and executive functions in normal children was parallel (Klimkeit, Mattingley, Sheppard, Farrow, & Bradshaw, 2004). These results suggest that school-aged children already have the ability to divide their attention between different signals (distribute attention to visual and auditory modalities) and select a specific task to perform (attend a single sensory modality). In adults, previous studies have shown that less sensory information was available for integration when attention was assigned to one sensory modality while ignoring another modality (Johnson & Zatorre 2005, 2006). Audiovisual integration in selective and divided attention conditions has been investigated using cumulative distribution functions, showing that modality-specific selective attention attenuates audiovisual integration (Mozolic et al., 2008). Furthermore, it was reported that audiovisual integration was larger in the attended conditions than in the unattended conditions (Talsma & Woldorff 2005). Therefore, our results in children were consistent with these findings in adults, which showed that audiovisual integration facilitation effects are larger in the divided attention condition than in the selective attention condition. However, the brain processing mechanism may differ between children and adults, although the behavioral results in children are consistent with adults. Therefore, further electrophysiological studies are needed to confirm and elucidate neural mechanisms in children. In addition, the development of audiovisual integration in children with autism spectrum disorder (ASD) has been investigated. In doing so, it was observed that children with ASD were delayed in audiovisual integration compared to the control group (Taylor, Isaac, & Milne, 2010). The results indicated that the children with ASD had deficit in audiovisual integration. Thus, our data might provide a new perspective for identifying children with conditions that are associated with sustained attention deficit, such as attention-deficit hyperactivity disorder.
In summary, the main focus of this study was to determine whether selective attention and divided attention affects audiovisual integration in school-aged children. Our results showed that significant audiovisual integration was elicited by divided attention. However, audiovisual integration was absent in the visual-selective attention condition, but audiovisual integration was present in the auditory-selective attention condition. Audiovisual behavioral facilitation in the divided attention condition was larger than in the selective attention condition. The results support the idea that attention can modulate audiovisual integration.
Footnotes
Acknowledgements
The authors would like to thank all individuals who participated in the study.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by JAPAN SOCIETY FOR THE PROMOTION OF SCIENCE (JSPS) KAKENHI grant numbers 25249026 and 25303013, a Grant-in-ASid for Strategic Research Promotion from Okayama University, and the National Science Foundation of Hubei University (098379, WY).
