Abstract
The present eye-tracking study aimed to investigate the impact of auditory speech information on 12-month-olds’ gaze behavior to silently-talking faces. We examined German infants’ face-scanning behavior to side-by-side presentation of a bilingual speaker’s face silently speaking German utterances on one side and French on the other side, before and after auditory familiarization with one of the two languages. The results showed that 12-month-old infants showed no general visual preference for either of the visual speeches, neither before nor after auditory input. But, infants who heard native speech decreased their looking time to the mouth area and focused longer on the eyes compared to their scanning behavior without auditory language input, whereas infants who heard non-native speech increased their visual attention on the mouth region and focused less on the eyes. Thus, it can be assumed that 12-month-olds quickly identified their native language based on auditory speech and guided their visual attention more to the eye region than infants who have listened to non-native speech.
From birth onwards, infants encounter a socially-rich environment, hearing and seeing talking faces on a daily basis, especially when communicating with their caregivers. Such early face-to-face communication should lead to a close binding between face and speech. It provides infants with redundant or complementary auditory-visual speech information, which could enhance their perception, understanding and learning of spoken language.
Although there is a substantial body of literature on infants’ auditory processing of speech (e.g., Werker & Tees, 1999), far fewer researchers have focused on the visual information that infants extract from talking faces. Weikum and colleagues (2007) addressed the question whether visual speech cues alone are sufficient for infants’ language discrimination. The authors used silent video clips of English-French bilingual speakers who were reciting different sentences in either French or English in a habituation paradigm. Monolingual 4- and 6-month-old infants were able to visually discriminate between the two languages, whereas infants at 8 months of age were not. The decline in the ability to visually discriminate their native language from an unfamiliar language observed in 8-month-olds is concordant with the perceptual narrowing view (Scott, Pascalis, & Nelson, 2007), that is, a tendency for infants to still develop perceptual abilities for native attributes, while declining for non-native attributes that infants have little experience with. This view has been described, for example, in studies of infants’ phonetic development (Kuhl, Tsao, & Liu, 2003; Werker & Tees, 1984), face discrimination (Kelly et al., 2007; Pascalis, de Haan, & Nelson, 2002), and scanning of own- and other-race faces (Liu et al., 2011; Wheeler et al., 2011).
A recent study by Pons, Lewkowicz, Soto-Faraco and Sebastián-Gallés (2009) investigated infants’ intersensory speech perception. The authors compared 6- and 11-month-old English and Spanish infants’ ability to orient toward silent videos of a Spanish woman pronouncing the syllables “ba” or “va” before and after auditory-only familiarization with one of the two syllables. The /ba/ vs. /va/ phonological contrast is known to be perceived by adult English speakers but not by Spanish ones. As expected, no visual preference for either of the silent videos was observed prior to auditory familiarization for any of the four groups of children. After the auditory-only familiarization with either the /ba/ or the /va/ syllable, 6-month-old English and Spanish infants looked longer at the matching visible syllables, but at 11 months of age only the English infants did so. The fact that 6-month-olds looked at the corresponding visual stimulus more than at the non-corresponding stimulus suggests that these infants have perceived the relationship between the auditory and the visual stimulus, and thus that they performed intersensory matching. The fact that older infants did not match audible and visible attributes of non-native speech is interpreted by Pons and colleagues as suggesting that infants’ sensitivity to intersensory speech narrows during the first year of life and that perceptual narrowing is a domain general developmental process.
By tracking eye movements, it was demonstrated that the way in which infants process audible-talking faces in their native language changes over the course of the first year: Lewkowicz and Hansen-Tift (2012) investigated infants’ visual attention to facial regions in 4- to 12-month-old English-learners while they watched and listened to a female speaking their native (English) or a non-native (Spanish) language. The authors were particularly interested in potential differences regarding the face-scanning behavior at the end of the first year. Independently of language familiarity, 4-month-olds looked longer at the eye region, 6-month-olds gazed equally long at the eyes and the mouth, and 8-month-old infants looked longer at the mouth area. Interestingly, 12-month-olds gazed equally long at the eyes and the mouth when native speech was spoken, whereas they maintained their looking toward the mouth area for the non-native language. The authors concluded that perceptual narrowing and growing native language expertise explained this behavior. Whereas the processing of non-native speech required infants to access complementary audio-visual cues from the mouth region, the processing of native speech was less depending on such cues.
To sum up, on the one hand, monolingual 8-month-old infants seem to no longer be able to visually discriminate their native language from an unfamiliar language, an ability infants have at 4 and 6 months of age (Weikum et al., 2007). On the other hand, when auditory speech information is presented, 12-month-old infants show a differential scanning pattern depending on whether the faces talked in their native or non-native language (Lewkowicz & Hansen-Tift, 2012), and 11-month-olds seem to remain sensitive to visual cues to native speech contrasts after audible-only speech input (Pons et al., 2009). These findings lead to the general question of the extent to which auditory speech has an impact on the visual processing of silently-talking faces at the end of the first year.
To address this question, we used a variant of the intersensory matching procedure (Pons et al., 2009) and compared 12-month-old German infants’ gaze behavior to facial regions of faces silently-speaking German (native) and French (non-native) fluent speech before (baseline trials) and after (post-test trials) auditory-only familiarization with one of the two languages. In accordance with the study of Weikum and colleagues (2007), who did not observe any visual language discrimination abilities in 8-month-olds, 12-month-old infants also should not show a visual preference for either of the languages. Moreover, we predicted no difference in scanning patterns between the German- or the French-speaking face in the baseline trials. By contrast, in the post-test trials, corresponding with the results of Lewkowicz and Hansen-Tift (2012), we expected infants’ looking times to the mouth region to increase when French auditory language is presented, in comparison to baseline, and infants’ looking times to the eyes to increase when German auditory language is presented.
Method
Participants
Forty monolingual German-learning 12-month-old infants (M = 377 days, SD = 8 days; age range: 365 days – 394 days, 21 males) were tested. All infants were full-term with no visual or auditory deficits, as reported by parents. The data from seven additional infants were discarded from the final sample due to equipment failure (two), failure to complete the calibration procedure (four) or extreme fussiness (one).
Stimuli
Visual stimuli were silent video clips of four female bilingual German-French speakers. Recording took place in Germany (two speakers) and in France (two speakers). The speakers were recorded against a blue background looking directly into the camera while they were reciting a German and a French version of a children’s short story in a child-directed manner. Each of the 30-second video clips showed a full-face image of the speaker and measured 20.6 cm × 18 cm when displayed side-by-side on the monitor, separated by an 11-cm gap. Both the French and German videos were edited to make sure that they started on a closed mouth and that the first mouth-opening was synchronized.
Auditory stimuli consisted of the same story and were presented at conversational sound pressure level (65dB ± 5dB).
Apparatus and procedure
Caregivers were informed of the purpose of the study and gave written consent for their infant to participate. Each infant was tested individually in a baby lab, the caregiver sitting on a chair with the infant on her lap. To prevent parents from influencing the looking behavior of their infants, they were told to keep their eyes closed and to refrain from talking for the duration of the experiment. The infants were seated on the caregiver’s lap at a distance of 60 cm in front of a 22-inch monitor (resolution: 1280 × 1024 pixels). Stimuli were presented by using E-Prime 2.0 software (Psychology Software Tools, Sharpsburg, PA).
Eye-tracking data were captured by a Tobii X120 eye tracker with a sampling rate of 60 Hz. Infants completed an infant-adapted 5-point calibration sequence, which was run using Tobii Studio software (Tobii Technology, Sweden). Calibration was checked for accuracy and repeated up to three times if necessary.
After showing an attention-getter, we used a procedure adapted from Pons et al. (2009), which is a variant of the intersensory matching procedure. Importantly, in this procedure, the sound is not presented at the same time as the visual stimuli to ensure that audiovisual synchrony is not mediating intersensory matching.
There were six 30-second trials (see Figure 1): in the first and second trials (baseline condition), infants were presented with two side-by-side silent video clips, displaying one bilingual speaker uttering the same story in French on one side and in German on the other side. The left-right position of French and German videos was counterbalanced across infants in the first trial and reversed in the second one. In the third trial (auditory familiarization trial), infants were presented with a voice speaking either French or German while they were watching an attention-getter. Infants were randomly assigned to one of two auditory groups (German or French) and the specific speaker the infants listened to was different from the speaker they had seen in the baseline trials. In the fourth trial (post-test trial), we presented the two initial silent videos again. The fifth and sixth trials were a repetition of the auditory familiarization and post-test trial but the left-right presentation of the silent videos was reversed in the sixth trial. The voices and silent videos of the four female bilingual German-French speakers were counterbalanced across infants.
Schematic representation of the procedure used in the current study. Only the German auditory condition is shown.
Data analysis
Data were analysed for the total duration of fixations within an area of interest. Fixations were defined as having a minimum radius of 35 pixels and a minimum duration of 100 ms. Three Areas of Interest (AOIs) were created: whole face, mouth and eyes. The AOI of the whole face was defined by outlining the face with the hair excluded. A small buffer zone of approximately 1 cm allowed for head movements during the recording (see Liu et al., 2011; Wheeler et al., 2011). The mouth AOI and the eyes AOI were created in the same way as in the study of Lewkowicz and Hansen-Tift (2012) by defining rectangular AOIs that covered both eyes and the mouth, respectively, for each stimulus face (see Figure 2).
Example of eyes and mouth AOI plots, 81 × 95mm (72 × 72 dpi).
For the three AOIs (face, mouth and eyes) of each French- or German-speaking face, gaze data were aggregated across the baseline and post-test trials, respectively. Thus, data analysis was conducted with the AOIs (face, mouth and eyes) of the bilingual person speaking French (resp. German) during the baseline trials (first and second trials), and the AOIs of the bilingual person speaking French (resp. German) during the post-test trials (fourth and sixth trials).
Results
Preliminary analyses revealed no significant effects of infants’ gender or of the bilingual speakers’ identity on infants’ gaze behavior to AOIs. Thus, the data for these two factors were collapsed for subsequent analyses.
Face AOI
As a dependent variable, we computed a preferential looking-score by dividing the fixation time on one face (e.g., face AOI of the face speaking German) by the amount of total looking time (sum of looking times at the face AOI of the German and French faces) across the baseline and post-test trials, respectively. To determine whether the infants showed a preference for one of the visual speeches during the baseline condition, and whether such a preference could be affected by the auditory speech the infants were familiarized with, we submitted the preferential looking scores to an ANOVA with “visual speech” (German, French) and “test” (baseline, post-test) as within-subjects factors, and “auditory group” (French, German) as a between-subjects factor.
This analysis did not show any significant effect (all ps > .6), indicating that infants did not show a preference for either of the faces during baseline (MGerman = 50%, SDGerman = 11%; MFrench = 50%, SDFrench = 11%) and that this pattern was not influenced by auditory familiarization, neither after infants were exposed to German auditory speech (MGerman = 48%, SDGerman = 10%; MFrench = 52%, SDFrench = 11%) nor after the infants heard French (MGerman = 50%, SDGerman = 11%; MFrench = 50%, SDFrench = 11%).
Proportional fixation on mouth and eyes AOIs
The fixation data were converted into proportional fixation times within the mouth and eyes AOI relative to the total on-face fixation times. To analyse whether the infants showed different looking patterns within the faces after auditory familiarization, we computed a four-way ANOVA with “AOI” (mouth, eyes), “visual speech” (German, French) and “test” (baseline, post-test) as within-subjects factors and “auditory group” (French, German) as a between-subjects factor.
This analysis failed to show any effect of visual speech, F(1, 38) = .04, p = .83, η2 = .00, nor any interaction with this factor. This seems to indicate that infants scanned the German- and French-speaking faces in a similar way, even after auditory familiarization. However, the ANOVA revealed a significant main effect of AOI, F(1, 38) = 126.23, p < .001, η2 = .77. Infants looked longer at the mouth AOI (Mmouth = 48%, SDmouth = 20%) than at the eyes AOI (Meyes = 14%, SDeyes = 12%). We also found a significant AOI × auditory group interaction, F(1, 38) = 9.78, p < .01, η2 = .21 and a significant AOI × test × auditory group interaction, F(1, 38) = 24.87, p < .001, η2 = .40, indicating that during the post-test trials, the looking patterns toward the mouth and the eye region of both the German- and French-speaking faces were influenced by the auditory input. However, the looking time was affected similarly for German and French faces, as there was no difference in looking times between both faces (see above). As illustrated in Figure 3, this latter interaction is explained by the fact that the auditory familiarization modulated looking behavior in an opposite manner for French and German.
Mean and standard errors of proportional looking times (%) to mouth and eyes AOI within baseline and post-test trials of both auditory groups.
Whereas infants auditorily-familiarized with French looked longer at the mouth region of both faces during the post-test trials (M = 57%, SD = 15%) than during baseline (M = 48%, SD = 11%; t(19) = −2.18, p < .05), infants who had heard German looked shorter at the mouth region of both faces during the post-test trials (M = 35%, SD = 14%) than during baseline (M = 48%, SD = 17%; t(19) = 5.09, p < .001). Conversely, whereas infants auditorily-familiarized with French looked shorter at the eyes region of both faces during the post-test trials (M = 9%, SD = 5%) than during baseline (M = 14%, SD = 8%; t(19) = 2.23, p < .05), infants who heard German looked longer at the eyes region of both faces during the post-test trials (M = 23%, SD = 9%) than during baseline (M = 14%, SD = 10%; t(19) = −3.42, p < .01).
Discussion
The main goal of the present study was to investigate whether 12-month-olds’ visual exploration of faces speaking in their native language (German) or in a non-native (French) language was influenced by auditory speech. The results showed that listening to auditory speech did not lead to different visual preferences for the German- and French-speaking faces. Auditory speech rather affected infants’ visual scanning of the faces but independently of whether they were silently speaking French or German. After auditory exposure to their native language, infants looked less at the mouth of both faces, whereas looking times at the eyes increased. Conversely, after auditory exposure to their non-native language, infants looked more at the mouth of both faces, whereas looking times at the eyes decreased.
These results are consistent with the study of Lewkowicz and Hansen-Tift (2012) who showed that 12-month-old infants looked longer at the mouth when they heard and saw a person speaking in a foreign language (Spanish), in contrast to infants who attended equally-long on the mouth and the eyes when the talker spoke their native language (English). However, Lewkowicz and Hansen-Tift (2012) presented the infants with visual and auditory speech simultaneously, whereas our design allowed us to isolate the impact of separately presented auditory speech on infants’ looking patterns to silently-talking faces.
According to Lewkowicz and Hansen-Tift (2012), our results can be considered in the context of multisensory perceptual narrowing. As infants at the end of the first year have specialized speech perception skills of their native language (Werker & Tees, 1999), it can be assumed that the processing of their native speech does not necessarily rely on language-specific visual inputs like the movements of the mouth. On the contrary, the processing of unfamiliar speech relies more on the processing on the mouth. In line with this interpretation, our results showed that after hearing their native language, infants focused less on the mouth and increased looking times toward the eyes of the silently-talking faces, probably to gain access to socially relevant cues (Langton, Watt, & Bruce, 2000). In contrast, infants who were presented with French auditory speech (their non-native language) focused less on the eyes and showed increased visual attention to the mouth, possibly to profit from complementary audio-visual information that is available in this area.
With regard to infants’ visual processing of silently-talking faces without previous listening to auditory speech (baseline trials), our results have demonstrated that the infants did not show any preference for one of the two visual speeches, suggesting that they were not able to discriminate their native language from a non-native language on the basis of visual cues alone. These results confirm and extend the findings of Weikum and colleagues (2007) by showing similar results among infants older than 8 months. Unexpectedly, our findings showed that infants focused longer on the mouth than on the eyes on both the German- and the French-speaking faces in the baseline (see Figure 1).Thus, without the opportunity to listen to speech, infants at 12 months of age looked mainly at the mouth compared to the eyes of silently-talking faces. Our results are in line with studies indicating that older infants focus longer on the mouth of in-sound presented talking faces compared to smiling faces (Tenenbaum, Shah, Sobel, Malle, & Morgan, 2012), or when they are exposed to silently-talking faces of young children (Frank, Vul, & Saxe, 2011) or of their mother (Hunnius & Geuze, 2004). It is reasonable to assume that in our study, infants’ attention was mainly attracted by the movement of the mouth when no auditory speech was presented before.
Importantly, the auditory input did not enable the 12-month-old infants to discriminate the visually presented French and German languages neither in terms of a general visual preference nor in terms of differential scanning patterns between the faces. Thus, even for their native language, infants did not perform cross-modal matching of heard and seen speech. This result seems to be in contrast to Pons and colleagues’ result (2009) indicating that 11-month-old English infants successfully discriminated between the visually presented syllables “ba” and “va” after they had heard one of these syllables acoustically. Thus, it seems that, at the syllable level, discrimination of visual speech or cross-modal matching of visual-auditory information can be enabled by prior auditory speech in infants at the end of the first year. At the utterance level, however, auditory speech does not enable visual language discrimination. It can be assumed that cross-modal processing of fluent speech, especially when amodal cues (e.g., synchrony) are eliminated due to sequential presentation of auditory and visual speech, is too difficult for infants at this age. Another interpretation could be that infants at the end of the first year do not notice the differences between the visual cues carried by two different languages at the utterance level.
A limitation of the present study is that only 12-month-olds were tested and future research will have to be conducted using our design to examine younger infants. This will allow us to determine the extent to which higher interest in visual speech combined with less auditory language understanding, which can be assumed in younger infants, affect their visual processing of silently-talking faces.
Footnotes
Funding
This study was supported through a grant from the German Research Foundation (Deutsche Forschungsgemeinschaft) for GS (SCHW 665/11-1) and ANR--10-FRAL-017 for HL & JG.