Abstract
Emotions have been shown to modulate low-level visual processing of simple stimuli. In this study, we investigate whether emotions only modulate processing of visual representations created from direct visual inputs or whether they also modulate representations that underlie visual mental images. Our results demonstrate that when participants visualize or look at the global shape of written words (low-spatial-frequency visual information), the prior brief presentation of fearful faces enhances processing, whereas when participants visualize or look at details of written words (high-spatial-frequency visual information), the prior brief presentation of fearful faces impairs processing. This study demonstrates that emotions have similar effects on low-level processing of visual percepts and of internal representations created on the basis of information stored in long-term memory.
Ample evidence now indicates that emotions affect human cognition (i.e., Phelps, 2006). In the case of visual perception, for example, fearful faces are identified more efficiently than neutral faces (e.g., Fox et al., 2000). Moreover, the effect of emotion on perception is not restricted to higher levels of visual processing. In fact, neuroimaging studies have demonstrated that emotional pictures elicit more activation in the early visual areas (such as V1) than do neutral pictures (e.g., Lang et al., 1998; Morris et al., 1998; Vuilleumier, Richardson, Armony, Driver, & Dolan, 2004). In addition, two studies converge in demonstrating that fearful faces enhance contrast sensitivity (Phelps, Ling, & Carrasco, 2006) and orientation sensitivity of low-spatial-frequency Gabor patches (Bocanegra & Zeelenberg, 2009). The amygdala is presumably at the root of this effect: at an early stage, the emotional valence of the stimulus is processed by the amygdala which—through backward projections to early visual areas—modulates subsequent low-level perceptual processing (Vuilleumier, 2005). This hypothesis is supported by studies demonstrating that the effect of emotion on perception can be eliminated in patients with amygdala lesions (Anderson & Phelps, 2001; Vuilleumier et al., 2004).
Although evidence suggests that the emotional state of participants (evoked by the short presentation of faces primes) modulates early visual processing during perception, no study to date has documented similar effects of emotion during processing of visual representations created on the basis of information stored in long-term memory (i.e., on visual mental images). Such a finding would not only illuminate our knowledge of effects of emotion on information processing, but also would provide another source of evidence that visual mental images rely in part on representations that exist relatively early in the visual processing stream (see Kosslyn, Thompson, & Ganis, 2006).
In this study, we investigated the effect of fearful faces on participants’ performance in a mental imagery task and in a visual perception task. In both tasks, we asked participants to judge characteristics of written lowercase words following a brief presentation of faces. On half of the trials, participants decided whether the overall shape of a word increases from left to right—such as occurs in the word “sell”—either by visualizing the letters of the words as they had appeared (during an initial study phase) after low-pass filtering (mental imagery task, see Figure 1a) or by seeing the low-pass-filtered word on the screen (visual perception task, see Figure 1b). Thus both the stimuli and the task itself led participants to process only the coarse contours of the word; these were the low-spatial-frequency trials (LSF). On the remaining trials, participants visualized or saw high-pass-filtered stimuli and judged whether the first and the last letters of each word both contain an enclosed space—such as in the word “gold”. On these trials, both the stimuli and the task led participants to process high-resolution visual information; these were the high-spatial-frequency trials (HSF).
Procedures (a) in the mental imagery task, and (b) in the visual perception task.
Given that the amygdala sends efferent magnocellular projections to portions of early visual areas (Amaral, Behniea & Kelly, 2003; Freese & Amaral, 2005) that are involved in processing coarse LSF visual information, we predicted facilitation—shorter response times (RTs) and fewer errors (ERs)—following presentation of fearful faces for LSF trials. In contrast, we predicted that having just seen fearful faces should impair performance when fine-grained HSF visual information needed to be processed; this prediction stems from previous findings (Bocanegra & Zeelenberg, 2009) with low-level perceptual tasks that documented inhibitory interactions between magno- and parvocellular pathways (see also Yeshurun & Carrasco, 2000).
Method
Participants
We recruited 32 right-handed volunteers with normal or corrected-to-normal vision from Harvard University and the local community (19 females and 13 males with an average age of 21.1 years). Data from 2 additional participants were not analysed because they performed at chance levels, and hence we had no reason to believe that they actually performed the task. Participants received either pay or course credit. Participants were randomly assigned to the mental imagery task or the visual perception task. All participants provided written consent and were tested in accordance with national and international norms governing the use of human research participants. The research was approved by the Harvard University Institutional Review Board.
Materials
Stimuli were presented on a 17-in IBM monitor (1,280 × 1,024 pixels, resolution and refresh rate of 75 Hz). A light grey fixation point (0.4° × 0.4°) was displayed on a 13.7° × 13.7° dark grey background throughout the entire trial. To induce emotion, we selected 24 monochrome pictures of faces (6 males and 6 females of different ethnicities), displaying neutral and fearful expressions, from the Extended Yale Database B. All pictures were taken with the identical lighting source and camera angle. A copy of each picture (7.8° × 6.1°) was positioned at 3° to the right and to the left of the fixation point. We presented two faces along with a fixation point in order to limit participants’ eye movements. In the mental imagery task, words were presented auditorily along with the fixation display and one black bracket (4.9° × 1.7°) positioned at 6.1° to the right and one black bracket positioned at 6.1° to the left of the fixation point.
All 672 words were selected from the Clark and Paivio (2004) norms of concrete words. Words were rated low on the imagery scale and high on the familiarity scale (respectively M = 2.3 and M = 6.2 on a seven-point scale). Word length varied from 3 to 8 letters (M = 5.03). Finally, the stimulus words had a high frequency of occurrence as defined by the Kucera-Francis norms (M = .92 in a log base 10). Words were selected so that half had an ascending shape envelope and half did not (for the LSF task), and half had both the first and last letters enclosed and half had either the first or the last letter enclosed (for the HSF task). Words used for the LSF and HSF task were equated in length, frequency of occurrence, imagery and familiarity ratings, ts < 1. To design the visual stimuli, words were written in a 98-pt lowercase Arial black font within two black brackets that were 4.9° high on a 13.7° × 13.7° dark grey background. In Photoshop, we applied a Gaussian blur (radius of 11.5 pixels) or a high pass filter (radius of 11.5 pixels) to the written words to create, respectively, LSF and HSF stimuli (see Figure 2). In addition, we created two sets of 26 pictures of each letter of the alphabet (in the same font as the words) and submitted them to the same transformations as the words. Finally, all words were recorded aloud on a computer, read by a male native speaker of English. The average duration of the audio files was 612 ms.
Examples of a low-spatial-frequency (LSF) and a high-spatial-frequency (HSF) stimulus.
Procedure
The participants were tested individually, sitting approximately 75 cm from a computer screen.
In the mental imagery task, participants first learned to create LSF or HSF visual mental images of each letter. Each version (high or low pass filtered) of each letter was presented twice. On each trial, participants first studied the letter (3 s); then, after the letter had disappeared, they created a mental image of it. Finally, the letter was displayed again (3 s) to allow participants to correct their mental image.
In both tasks, on each test trial, first a fixation point was displayed for 500 ms and then a pair of the same face (75 ms), one to either side of the fixation point, appeared. On half the trials (fully counterbalanced with all other variables), the faces were presented in their upright orientation, and on half the trials they were presented in an inverted orientation. In the mental imagery task, after a 40-ms inter-stimulus interval (ISI), one of the words was presented auditorily along with the fixation point and the brackets. Participants were asked to form a mental image of the word as if it was printed within the brackets. In the LSF trials, participants visualized the written words using the LSF versions of the letters. In the HSF trials, they generated the image using the HSF versions of the letters. In the visual perception task, after the 40-ms ISI, one of the versions of the words was presented for 40 ms within the brackets and was replaced by the fixation point. A new trial started 500 ms after participants provided an answer. If participants did not provide an answer within 2 s in the visual perception task (or 3 s in the mental imagery task), a new trial started. Whenever the fixation point was present on the screen, participants were to keep their gaze focused on it.
In the LSF trials, participants decided whether or not the overall shape of the word was ascending from left to right, such as in the word “sell”. In the HSF trials, participants judged whether the first and the last letters of the word both had an enclosed space, such as in the word “gold”. Participants used their dominant hand to respond, pressing the “b” key to indicate that the word was ascending or that the word had two enclosed letters and the “n” key if the word did not possess the queried property. By pressing one of the keys, a clock was interrupted, which started when the audio file stopped (mental imagery task) or when the word disappeared (visual perception task). The nature of the response was also recorded.
In each task, participants first performed two practice blocks of 48 trials, one for each type of trial, in which we provided feedback on their responses. The order in which the practice blocks were performed was counterbalanced across participants. Then participants performed 3 blocks of 96 LSF trials and 3 blocks of 96 HSF trials. In each task, a given word was used only once. LSF and HSF blocks were alternated: half of the participants started with a LSF block, half with a HSF block. At the beginning of each block, written instructions were provided that indicated the type of trials the participants were to perform. In each block, 48 of the 96 words were ascending from left to right (LSF) or had enclosed spaces in their first and last letters (HSF). Each word was associated with a pair of the same face. Each pair of faces was either fearful or neutral and was presented either upright or inverted. In each block of 96 trials, 48 words were associated with pairs of fearful faces, half of which were presented inverted, and 48 words with neutral faces, half of which were inverted. In each task, across participants, the assignment of a set of words to a specific face cueing condition (i.e., fear/upright, fear/inverted, neutral/upright, neutral/inverted) was fully counterbalanced. Trials were randomized within each block, except that on no more than 3 trials could occur in a row of instances where the same key needed to be pressed or where the same emotion of the face was displayed in the same orientation.
Results
For each participant, we averaged the RTs and the number of errors separately for each type of expression of the faces (fearful vs. neutral), their orientation (upright vs. inverted), and the type of trial (LSF vs. HSF).
As a preliminary analysis, we performed a 2 (mental imagery task vs. visual perception task) × 2 (fearful vs. neutral faces) × 2 (LSF vs. HSF trials) analysis of variance (ANOVA). No significant interactions were observed for the RTs, F(1, 30) = 2.51, p = .12) or for the ERs (F < 1), which suggests that the emotional faces had comparable effects in the two tasks. In the subsequent analyses of the results, all factors were within-participant (so we computed repeated-measure ANOVAs), and when we compared two means, we computed one-tailed t tests in accordance with our clear hypotheses; all alpha levels were adjusted with a Bonferroni correction.
In the mental imagery task, as predicted, the emotional value of the faces (fearful vs. neutral) affected RTs and ERs differently in the two types of trials (LSF vs HSF), F(1, 16) = 12.42, p < .005, ηp2 = .44, for RTs; and F(1, 16) = 16.20, p < .001, ηp2 = .50, for ERs (see Figures 3a and 3b). On the LSF trials, having just seen fearful faces led participants to evaluate imagined shape contours faster (relative to having just seen neutral faces) (1,461 ms vs. 1,538 ms), t(16) = –2.96, p < .0005, d = –.22, and to make fewer errors (8.9% vs. 11.5%), t(16) = –2.30, p < .015, d = –.26. On the HSF trials, this pattern was reversed: the participants required more time and made more errors in the imagery task following fearful faces than following neutral faces: respectively (1,558 ms vs. 1,514 ms), t(16) = 2.21, p < .025, d = .15, and (10.2% vs. 8%), t(16) = 2.13, p < .025, d = .28.
RTs and ERs in each task for (a, b) upright and (c, d) inverted faces for judgements that require using high or low spatial frequencies. No error bars are displayed, given that all variables are repeated-measure (see Cumming & Fish, 2005).
In the visual perception task, effects of the emotional value of the faces and of the spatial frequencies of the pictures interacted as predicted, both for the RTs, F(1, 14) = 50.02, p < .0001, ηp2 = .79 (Figure 3a), and ERs, F(1, 14) = 25.29, p < .0001, ηp2 = .64 (Figure 3b). Having just seen fearful faces facilitated processing in the LSF condition compared to having just seen neutral faces (581 ms vs 619 ms), t(14) = –5.99, p < .0001, d = –.41, and led to fewer errors (9.2% vs. 11.9%), t(14) = –2.65, p < .01, d = –.37, whereas the opposite effect occurred in the HSF condition (524 ms vs. 509 ms), t(14) = 3.61, p < .005, d = .21, and, for ERs (6.2% vs. 4%), t(14) = 4.26, p < .0005, d = .75.
In addition, in both the mental imagery and visual perception tasks, half the faces were inverted, in order to control for any possible effect of visual differences between fearful and neutral faces. For these trials, the emotional value of the faces had no effect on either the RTs or the ERs on both LSF and HSF trials, ts < 1, nor did emotional value interact with the type of trials (HSF vs. LSF, Fs < 1) in both tasks (see Figure 3c and 3d).
Discussion
The results clearly demonstrate that emotions modulate low-level processing of complex stimuli even in the absence of visual input. In fact, when a fearful face was presented to a participant, subsequent low-level processing of complex LSF visual representations was enhanced, whereas processing of complex HSF visual representations was impaired. The facilitation that resulted from seeing fearful stimuli on LSF stimuli could stem from an evolutionary adaptation in response to danger: in fearful situations, one needs to process preferentially the coarse (i.e., low-spatial-frequency) visual information about an object, given that such information is sufficient to determine whether this object represents a danger (öhman, 1986). The impairment of fine-grained (i.e., high-spatial-frequency) visual information processing following fearful faces may reflect inhibitory interactions between the magnocellular and the parvocellular pathways (i.e., an increase in the sensitivity of magnocellular neurons, which in turn inhibits parvocellular neurons) across spatial frequencies (Bocanegra & Zeelenberg, 2009; Burr, Concetta Morronne, & Ross, 2002; Yeshurun & Carrasco, 2000).
Moreover, we demonstrated that emotions modulate processing of visual mental images comparably to how they modulate processing of visual percepts. These results are not only further evidence that visual mental imagery shares neural mechanisms with visual perception, but also support the hypothesis that both functions draw on early visual cortex (Kosslyn & Thompson, 2003).