Abstract
P.M. is a synesthete who experiences colors when viewing alphanumeric symbols. Her search for a target differing from distractors by a synesthetic color feature takes the form of a pop-out search. Thus, it would seem that synesthesia can occur preattentively. However, discrepancies between the regression functions of response times observed in target-present trials and target-absent trials, and the fact that fast response times occur only when the target is within a few degrees of visual angle from fixation, indicate that P.M.'s synesthesia does not occur preattentively, but rather is within the focus of attention. We conclude that synesthesia is a genuine perceptual phenomenon that can have substantial influence on visual processing.
In an autobiographical book titled What Do You Care What Other People Think? physicist Richard Feynman (1988) confessed: “When I see equations, I see letters in colors—I don't know why…. And I wonder what the hell it must look like to the students” (p. 59). Notice Feynman's choice of words: “seeing” the letters in color. The intrusion of color sensations systematically triggered by stimuli that are not colored per se (e.g., alphanumeric symbols printed in black) has been labeled color synesthesia. Recent research on synesthesia has proved it to be a genuine phenomenon. Several studies have shown interference effects in naming the “objective” color of an object (e.g., a letter A printed in blue) when at the same time the object evokes another “subjective,” or synesthetic, color (e.g., Dixon, Smilek, Cudahy, & Merikle, 2000; Mattingley, Rich, Yelland, & Bradshaw, 2001; Odgaard, Flowers, & Bradman, 1999).
A specific focus of current research on synesthesia is demonstrating its perceptual nature. The crucial evidence for such an account of synesthesia must be based on replicating well-known perceptual effects. For example, when an object has a color similar to its background, it becomes difficult to visually segregate the object. Such a synesthetic mimetism, due only to the virtual colors, has been demonstrated by the slowed identifications of target letters when their synesthetic color is congruent, rather than incongruent, with the color of the background (Smilek, Dixon, Cudahy, & Merikle, 2001). Most important, synesthetes can identify and localize a target object (e.g., the digit 2) among distracting objects (e.g., several digits 5) more efficiently than nonsynesthetes if the target and distractors differ by a synesthetic color feature (Palmeri, Blake, Marois, Flanery, & Whetsell, 2002). Some synesthetes might also introspect that the synesthetic colors make the targets pop out.
However, several cognitive models view pop-out effects as the result of early visual processing prior to attention (e.g., a preattentive analysis of features; Treisman & Gelade, 1980). Moreover, a result truly comparable with pop-out search would be a parallel search, that is, search characterized by a flat slope in the graph of response time (RT) as a function of the number of distractors (cf. Friedman-Hill & Wolfe, 1995; Joseph, Chun, & Nakayama, 1997). Nevertheless, reductions in both the slope coefficient and the intercept of the search functions (Palmeri et al., 2002) or in interference from flankers (Ramachandran & Hubbard,2001) carry the implication that selective attention might not be crucial for evoking synesthetic colors. Indeed, it seems that distinguishing a 5 and a 2 would require attention in order for the observer to conjoin the similar features of the symbols in different higher-level shape descriptions. Thus, the finding that synesthesia can change a “difficult” search into a less difficult one appears to contradict the assumption that focal attention must precede symbol identification and that the synesthetic colors depend on such prior identification. Palmeri et al. (2002) surmised that attentional selection may not be crucial to the synesthetic binding of form with color. Some accounts propose a direct link or unique neural connectivity between early visual areas and color areas in the brains of synesthetes (cf. Rich & Mattingley, 2002).
Our main goal in the present study was to address the following question: Are synesthetes' advantages in visual search tasks really irreconcilable with early-selection theories? We present here evidence that synesthetic pop out can in fact be compatible with early-selection theories.
P.M., A SYNESTHETE
P.M. is a 60-year-old female musician who reports “strong” grapheme-color synesthesia (cf. Martino & Marks, 2001), such that most letters of the alphabet and single digits have a unique position in color space (see Fig. 1, top panel). She reports no synesthesia for any other stimuli (including music and its notation). P.M. has normal vision and color sensitivity, as assessed with the Farnsworth-Munsell 100 Hues Test (score=26). As has been found for other synesthetes (e.g., Svartdal & Iversen, 1989), P.M. shows high consistency in pairing colors to symbols, as revealed by her selection of colors from the pantone palette (of Microsoft Word 2000) in two sessions 4 months apart. Specifically, correlation analyses between the colors chosen for uppercase letters during the first session and lowercase letters during the second session showed highly significant relations for hue, t(27)=37, p<.0001,R=.99; saturation, t(27)=8, p<.0001, R=.83; and luminance, t(27)=12, p<.0001, R=.92. Color matches to digits alsoshowed significant results for hue, t(9)=11, p<.0001, R=.97; saturation, t(9)=8, p<.0001, R=.94; and luminance, t(9)=11, p<.0001,R=.97.
Alphanumeric symbols shown in the colors selected by P.M. to match her synesthetic experience (top panel) and illustration of the stimuli used in the search tasks (bottom panel). In the bottom panel, each target (first column) is shown alongside the distractor (second column) with which it was paired. The third and fourth columns show the synesthetic colors of the stimuli.
IS THERE REALLY SYNESTHETIC POP OUT?
P.M. was asked to search on the computer screen for predefined targets among predefined distractors of varying set sizes. Searches could be easy or difficult, depending on whether target and distractors differed by a basic feature of shape (i.e., curved vs. straight alphanumeric symbols) and—supposedly—by their synesthetic color (see Fig. 1, bottom panel). In each task, the distractors' set size varied from 1 to 20, the number of trials was 640, and there was a 50% probability that the target was present (in a random position). A group of control subjects (n=8), matched to P.M. by age (M=56, SD=5), sex, and educational level, was also tested. Their performance (see Table 1, top row) confirmed that searching for an A among Os or for a V among 8s was an easy search, resulting in a flat function typical of pop-out searches. In contrast, searching for an A among 4s or for a 2 among 5s was a difficult search; that is, slope coefficients for these search functions were significantly different from zero.
If the virtual colors of synesthesia have no effect on such search tasks, then P.M.'s searches would be undistinguishable from those of a group of control subjects. In contrast, if virtual colors have a perceptual reality that can affect the salience of a target among its distractors, then we might expect the performance of our synesthete to differ from normal performance. Indeed, P.M. showed search slopes similar to those of the control subjects in three of the four tasks. Specifically, her search slopes were undistinguishable from those of the control subjects in the two easy searches, in which targets differed in curvature from distractors (in the case of search for an A among Os, target and distractor differed in synesthetic color as well). Moreover, as did the control subjects, P.M. found search difficult when the target differed from the distractors neither in simple shape features nor in synesthetic color (i.e., search for an A among 4s). In contrast, although search for a 2 among 5s was difficult for control subjects, for P.M. this search showed no significant effect of distractors' set size on RTs (see Fig. 2, top panel, and Table 1). Such absence of slope is a hallmark of pop-out searches. This result suggests that added synesthetic color difference may act as a basic feature for visual search, and that it can effectively eliminate the need for a serial search (or item-by-item reallocation of attention). Thus, pop out is observable in synesthetically based search tasks and does not exist only in phenomenological reports.
P.M.'s performance searching for a 2 among 5s. Each scatter plot shows response time on target-present trials as a function of number of distractors. The top panel presents results for targets at all eccentricities. In the bottom panel, response times for trials with targets presented at eccentricities of 3° and 6° (circles) are graphed separately from response times for trials with targets presented at eccentricities of 10°, 11°, and 12° (triangles).
DOES SYNESTHETIC POP OUT POSE A PARADOX FOR EARLY-SELECTION THEORIES OF VISUAL ATTENTION?
When confronted with the fact that a synesthetic color difference seemed to help her in finding targets (by eliminating the interfering effect of increasing numbers of distractors), P.M. was surprised, because she thought she would have to identify a letter before it could evoke its subjective color. P.M.'s insight highlights a paradox that arises if one tries to account for the results within the framework of early-selection theories of attention (e.g., Treisman & Gelade, 1980). In such theories, attention is the “glue” that conjoins basic features, so that when a target and a distractor differ by a conjunction of features, the attentional focus must shift serially among items to find a target. In contrast, when items differ by basic features (e.g., curvature; Wolfe, Yee, & Friedman-Hill, 1995), a narrowing of attention onto the salient item occurs instantaneously (bypassing a serial exam of other items). Because a 2 and a 5 differ only by arrangement of features and not by the presence versus absence of a single feature (Palmeri et al., 2002), it would seem impossible for search for a 2 among 5s to yield a pop-out search function based exclusively on perceiving a virtual color—prior to this experience of color it would be necessary to perform a serial search in order to first identify the form.
Results for Search Tasks With Black Stimuli
However, there was another interesting anomaly in P.M.'s search: Whereas the control subjects displayed positive slopes for both target-present and target-absent trials in difficult searches and flat slopes forboth target-present and target-absent trials in easy searches (see Table 1), P.M.'s search for a 2 among 5s showed inconsistency between the target-present and target-absent slopes. That is, despite the flat slope for target-present trials, the target-absent trials showed a positive slope (see Table 1). However, many past studies with normal subjects have shown that RTs on target-absent trials are independent—or nearly independent—of set size in parallel, or pop-out, searches (cf. Friedman-Hill & Wolfe, 1995; Joseph et al., 1997).
One way to account for the anomaly is to surmise that when P.M. searches for an item defined by a conjunction of shape features, her attention shifts serially in both target-present trials and target-absent trials. However, in target-present trials, an early attentional fixation might—by chance—include the target. If we assume that the attention window shifts serially but can sample a cluster of symbols in parallel (Treisman, 1982; Treisman & Gelade, 1980), then when the target symbol is within the attentional focus, partial evidence from “early returns” of feature processing will activate the symbol's node. That is, the presence of a feature such as a horizontally oriented curve could imply a 2, whereas a vertically oriented curve could imply a 5. After a symbol's node is activated, the activation will gradually spread to its synesthetic color. If the initial activation of the synesthetic color increases the salience of the target, this in turn can influence the scope of the attentional focus. Hence, parallel processing within the focus of attention could result in a quick narrowing of attention to the location of the target and to its fast identification (the synesthetic pop out).
One prediction from such an attentional account of synesthesia is that P.M.'s fastest target identifications should tend to occur for targets in locations close to central fixation (i.e., at the initial location of the attentional focus). Such an advantage may also be expected for the control subjects (cf. Saarinen, 1993; Wolfe, O'Neill, & Bennett, 1998), but we would expect such an effect to be abnormally strong for P.M. An analysis of variance with degree of eccentricity as the fixed factor and RT to target-present trials as the random variable confirmed that targets were identified progressively faster the closer to the zone of fixation for both the control subjects, F(4, 311)=25, p<.0001, and P.M., F(4, 299)=167, p<.0001. Targets 3° and 6° from center were identified faster by P.M. than by the control subjects (see Fig. 3); in both cases, P.M.'s RTs were 2.5 SDs faster than the mean RTs of the control subjects.
Comparison of control subjects' and P.M.'s performance searching for a 2 among 5s. Mean response times of control subjects and P.M. are graphed as a function of the target's distance (in degrees of visual angle) from central fixation. Error bars show standard deviations.
That these effects reflect processing within P.M.'s attentional focus is supported by an independent measurement of her scope of attention (i.e., functional field of vision, or FFOV; cf. Pringle, Irwin, Kramer, & Atchley, 2001; Sanders & Houtmans, 1985). In a separate task, we asked P.M. to localize a tilted line among 11 vertical lines presented together for 150 ms. She performed the task most accurately (i.e., with greater than 50% probability) when the target line appeared precisely within 3° and 6° of visual angle from fixation (mean accuracy=59%, SD=12, and 54%, SD=13, respectively); her accuracy dropped dramatically for targets 9° from fixation (mean accuracy=11%, SD=10), F(2, 45)=3.3, p=.04. This shows that an estimate of PM's “default” scope of attention encompasses a zone no greater than 6° of visual angle.
The findings we have described suggest an explanation not only of how synesthesia could transform a difficult search into a less difficult one (cf. Palmeri et al., 2002), but also of how the search function could, in some circumstances, resemble that of a pop-out search. Given that the targets fell within P.M.'s estimated FFOV 40% of the time, there was a significant proportion of trials in which her estimated scope of attention included synesthetically salient targets. These trials could have diluted the overall effects of the distractors' set size. Indeed, as illustrated in the bottom panel of Figure 2, P.M.'s search slope for targets presented within 6° of fixation (i.e., her estimated FFOV) were flat, y=831−4.5x, p=.17, R=.12, but there was a significant effect of set size on search slope for targets presented at eccentricities greater than 6°, y=1,299+7.2x, p=.001, R=.24. Thus, it seems that although distractors' set size had effects when targets were at eccentricities greater than P.M.'s FFOV, the effects of set size were greatly diluted at smaller eccentricities and failed to affect search slopes significantly in those conditions. Note, however, that even for targets at 10° to 12° of eccentricity, the search slope was rather shallow (slope=7.2) compared with that of the control subjects for all eccentricities combined (slope=16.7). In our interpretation, a decline in slope would be expected also for trials in which the synesthete reallocates attention over a new subset of stimuli, because there would be a proportion of trials in which the target would be included by chance within the new focus of attention.
SYNESTHETICALLY INDUCED COLOR SALIENCY IS NOT TRIGGERED PREATTENTIVELY
P.M. and the control subjects performed two additional tasks in which they searched for a 2 among 5s and two additional tasks in which they searched for an A among Os. The only difference from the earlier searches was that in one search each, the 2s and 5s, and the As and the Os, were colored congruently to the synesthetic colors indicated by P.M., whereas in the other search task, the colors were switched so that the objective and subjective colors of target and distractors were incongruent. Remarkably, P.M. showed no difference between searches in which the digits were congruently and incongruently colored, and her RTs were neither faster nor slower than the control subjects' (see Table 2). Moreover, P.M.'s searches with colored stimuli were much more efficient than her previous searches (see Table 1) when the same target and distractors differed only by their synesthetic color. Clearly, synesthetic colors are not like the basic color features of a preattentive stage of processing. In fact, interference or facilitatory effects of synesthetic colors might occur only after attention has focused on the target. In a study consistent with this view, Mattingley et al. (2001) showed that the interference of synesthetic colors in Stroop-like tasks vanishes in conditions that interfere with attention (i.e., very rapid stimulus presentations plus masking, such that synesthetes are able to report the color features but unable to report the letters' shapes explicitly). Thus, synesthesia may not be exceptional with regard to the general claim that “almost all of what is considered to be conscious vision cannot occur without attention” (Nakayama & Joseph, 1998, p.286; see also Mack & Rock, 1998).
Results for Search Tasks With Colored Stimuli
Footnotes
Acknowledgments
We thank Kristen L. Geiger and Michele Miozzo for comments and advice on early drafts of this article. Constructive criticisms by Eric Odgaard and Jeremy Wolfe have also greatly helped to improve the study. We are very grateful to P.M. for her cooperation and interest in the study. Bruno Laeng was partially supported by a grant from the Norges Forskningsråd (Nr. 149439).