Abstract
While visual saliency may sometimes capture attention, the guidance of eye movements in search is often dominated by knowledge of the target. How is the search for an object influenced by the saliency of an adjacent distractor? Participants searched for a target amongst an array of objects, with distractor saliency having an effect on response time and on the speed at which targets were found. Saliency did not predict the order in which objects in target-absent trials were fixated. The within-target landing position was distributed around a modal position close to the centre of the object. Saliency did not affect this position, the latency of the initial saccade, or the likelihood of the distractor being fixated, suggesting that saliency affects the allocation of covert attention and not just eye movements.
There is conflicting evidence regarding the extent to which irrelevant distractors can be ignored in a search task. In simple visual search, distractors can affect the speed and efficiency of search, and they can also affect the execution of saccadic eye movements in a systematic way (Theeuwes, De Vries, & Godjin, 2003). In naturalistic stimuli, search tasks have been key in distinguishing between top-down and bottom-up guidance of eye movements. For example, whilst contrast within low-level feature channels (or visual saliency) may predict where people fixate in some conditions (Itti & Koch, 2000; Parkhurst, Law, & Niebur, 2002), when people are given an item to search for saliency is a poor predictor of fixation location (Foulsham & Underwood, 2007; Henderson, Brockmole, Castelhano, & Mack, 2007; Underwood & Foulsham, 2006). In these cases participants were often able to saccade to the target after only a few fixations. They were presumably guided by their knowledge of what the target looked like (as in models by Navalpakkam & Itti, 2005; Rao, Zelinsky, Hayhoe, & Ballard, 2002), or by expectations of where in a scene it might occur (Torralba, Oliva, Castelhano, & Henderson, 2006).
If guidance processes are able to completely override the attraction of salient regions then the selection of nontargets should be determined only by the degree to which these regions resemble the target, and not by saliency. For example, if you are searching for a bottle of milk on a shelf you should be distracted by other bottles or white objects and not by a very salient item such as an orange, which looks nothing like a bottle of milk. Chen and Zelinsky (2006) reported a similar result, showing that people almost never looked at a highly salient object (the only coloured item in a greyscale display) when it was not the target. If, however, saliency remains to be important in a natural visual search then we would expect the saliency of distractor objects to influence their fixation, independently of their similarity to the target.
The questions addressed here are whether a distractor affects search and whether this is moderated by distractor saliency. This would be the case if salient items are fixated in preference of targets, but an alternative is that distractors might prolong search even when they are not fixated. Related to this question is the “remote distractor effect” (Walker, Deubel, Schneider, & Findlay, 1997), in which the presence of another object in the display affected the latency of saccades to a target, even when the two were positioned far apart. This suggests that the distractor increases competition within the saliency map that determines where the eyes will move (Findlay & Walker, 1999). The potency of a remote distractor depends on its proximity to the current fixation, suggesting an “extended fixation zone”, in which objects delay the disengagement of the eyes (Walker et al., 1997). In the present experiments, one might expect a distractor, if it is fixated, to prolong search depending on whether it is near or far from the target. What are the effects if it is not fixated?
Several experiments with realistic objects have looked at when or how often a target is fixated but few have looked at where within a target the eyes first land. If the saliency of objects around a target makes a difference, their effect might be seen in a shifting of covert attention or an alteration in the landing site. There is a large literature on saccade landing positions within words in reading that have identified the “preferred viewing location” (PVL) as being located slightly to the left of the centre of the word (Rayner, 1979). The landing position has consequences on subsequent processing. For example, the “optimal viewing position” can describe the position in the word where identification is easiest, and refixation is least likely, and this tends to be close to the PVL (O'Regan & Jacobs, 1992). If the first fixation lands at the edge of a word an additional fixation on this word is more likely than if it lands in the centre, and conversely the fixation duration is likely to be shorter (O'Regan, Levy-Schoen, Pynte, & Brugaillere, 1984). Few studies have looked at landing positions in objects. Henderson (1993) extended the findings from reading to a regular grid of four line drawings of objects that were inspected sequentially. Initial fixation position was distributed around the centre, with more variability in the direction of the saccade than in the direction perpendicular to it, and this was correlated with fixation duration and refixation probability. It was optimal to land near to the centre. Is there a systematic within-object landing site, and if so is its position influenced by the saliency of the adjacent object?
Method
Participants
A total of 16 student volunteers with normal or corrected-to-normal vision took part. The participants contributed to all conditions and were paid for taking part. Following repeated calibration and drift errors and a high proportion of data loss, the eye movement data for one participant were excluded, although their response data are retained in the analysis.
Stimuli, design, and apparatus
This experiment used arrays of photo-realistic objects. Figure 1 shows some examples of the stimuli. The pictures were constructed by arranging eight objects on two imaginary concentric circles around the image centre. The whole display subtended 34° by 27°. The inner and outer circles had radii of 6° and 11°, respectively. The objects were taken from a set of 30, and all were coloured photographs of items of food and drink, resized and centred inside a region that was 3.5° square, although exact object size varied slightly (means: width = 2.2°, height = 3.0°). Two configurations were used in which objects appeared on the horizontal and vertical axes or on the oblique axes, making object locations less predictable. The background was one of 10 colours. As colour contrast is believed to contribute to a saliency map of visual attention, altering the background colour changed the relative saliency of different objects. For example, a green apple was less salient on a green background than on a red background.
Two example stimuli, demonstrating the two different configurations used. Some objects were more salient than others due to the contrast between their colour and that of the background.
A set of potential stimuli was created with different objects and backgrounds. No objects were repeated in the same array. To estimate the relative saliency of different objects, the arrays were screened using the Itti and Koch (2000) saliency map model (source code available from http://ilab.usc), which can indicate the combined feature contrast at different locations.
The target was a single object that varied from trial to trial and was not highly salient—it was not selected within the first five shifts of attention made by the saliency model. Each object was featured as the target at least once. The object nearest to the target, and on the same vector from centre, was labelled the distractor. The two stimulus factors were target eccentricity (inner or outer) and distractor saliency (salient or nonsalient). Distractor saliency was manipulated so that half the distractors were salient (the most salient object in the array according to the model), and half were not salient (not featuring in the top five saliency rankings and therefore no more conspicuous than the target). As the model measures relative saliency there is always a most salient object, but in the nonsalient condition this object was not adjacent to the target. Each target position was used equally often, and each object appeared equally often as target, as distractor, and elsewhere in the display. This ensured that no spatial locations or particular objects were any more important for the task than the others. The 2 × 2 design gave four conditions: inner target salient distractor (IS), inner nonsalient (IN), outer salient (OS), and outer nonsalient (ON). The 80 target-present trials were divided equally into the four conditions and were interleaved with 80 trials that did not contain the target.
Eye movements were recorded using the head-mounted Eyelink II system (SR Research), and participants were supported on a chin rest. The system recorded eye position from the pupil image at 500 Hz, with an instrumental resolution of 0.01° and an average gaze position accuracy of less than 0.5°. Fixations and saccades were determined using a built-in online parser (with velocity and acceleration thresholds of 30°/s and 8,000°/s2, respectively).
Procedure
After practice with 10 trials the eye tracker was calibrated. Each trial started with an image of the target object, presented in the centre of a white background for 2,000 ms. Following this, a drift correct marker appeared in the centre of the screen. Participants pressed a key whilst fixating the marker, and the experiment continued when the tracker confirmed this and corrected for any drift. The search array was then displayed, and participants pressed one of two buttons to indicate the presence or absence of the target. If no response was made within 2,000 ms, the trial was classed as incorrect. All participants saw the same 160 trials in a random order. The eye tracker was recalibrated after every 40 trials.
Results
Performance was first assessed in terms of accuracy and manual reaction time (RT). If salient objects capture attention more than nonsalient objects then they might distract more and thereby decrease speed and accuracy performance. This is explored further by looking at the eye movement data, specifically the time to first fixate the target, and the probability of fixating the distractor. The sequence of objects selected by the saliency model, and its correlation with the objects fixated when there is no target present, is also investigated. Finally, the within-target landing position is measured to see whether people show a preferred viewing location and whether the distractor affects this.
Task performance
Trials were scored as errors when the response was incorrect, or in rare cases when the trial timed out without a response. The proportion of correct responses was close to ceiling (hits = 93%, rejections = 96%), and no further analysis was performed on response accuracy. Eccentricity had a reliable effect on RT, F(1, 15) = 27.1, MSE = 3,620, p < .001, as did saliency, F(1, 15) = 8.6, MSE = 2,942, p = .01. Participants were faster to respond to the target when it was closer to the centre (means, inner = 761 ms, outer = 839 ms), and they were slowed when it was next to a salient distractor (salient = 820 ms, nonsalient = 780 ms). The interaction was not significant, F(1, 15) = 1.76, MSE = 3,426, p = .205; distractor saliency had a similar effect at both eccentricities. RT to target-absent trials was slower (968 ms compared to all target-present trials, 800 ms); t(15) = 9.5, p < .001.
Eye movements toward target and distractor
A salient distractor might slow RT by interfering with target processing at fixation. The time to fixate the target supplements RT by specifying search time, excluding any time spent processing the target and deciding and executing the response. Incorrect trials were excluded from this measure, as were the minority of trials where the target was not fixated (18%). The results were similar to those in RT. There was a substantial effect of eccentricity, F(1, 14) = 171.9, MSE = 2,089, p < .001, with less time taken to fixate an inner target (mean = 380 ms) than an outer target (534 ms). Distractor saliency also had an effect, F(1, 14) = 16.5, MSE = 912, p = .001, with a salient distractor prolonging the time taken to fixate the target (473 ms/441 ms for salient/nonsalient distractors). These factors did not interact, F(1, 14) = 2.3, MSE = 1,381, p = .153. An additional question was whether the distractors influenced the latency of the very first saccade, which might indicate that programming saccades to a target whilst overriding a nearby salient distractor took longer. Participants took an average of 195 ms from stimulus onset to initiate their first saccade, and this did not differ between conditions, all effects, F(1, 14) < 1.
The saliency of an unrelated, adjacent object had an effect on search time before the target was fixated. Was this because salient distractors were more likely to be fixated prior to the target, thus requiring additional eye movements? The proportion of trials where the distractor was fixated was computed for each participant. The means showed that fixation of the distractor in the inner target conditions was very rare (mean percentage of trials, IS = 5%, IN = 1%). Fixation occurred much more often when it intervened between the centre and the target (OS = 38%, ON = 42%). Repeated measures analysis of variance (ANOVA) confirmed that there was a reliable effect of eccentricity, F(1, 14) = 108.5, MSE = 0.019, p < .001, but no effect of distractor saliency, F(1, 14) < 1. Salient distractors were not fixated more often, and so this cannot explain their influence on search.
The probability of fixating objects was also analysed to probe the influence of relative saliency of an object when no target was present. An average of 3.8 fixations (following the first) were made in target-absent trials, and only around half of the objects could be fixated in the 2,000-ms time limit. How were these objects selected? The saliency model can predict the relative saliency of each object and therefore the order in which they should be fixated. To see how well this predicted the target-absent search behaviour the probability of fixating an object at least once on a trial was plotted as a function of the object's saliency rank (see Figure 2).
The probability of fixating an object in target-absent trials, according to its eccentricity and its model-predicted saliency rank. A probability of 1 would indicate that an object with that rank and position was fixated every time it appeared in an array. Error bars show the standard error of the mean. Note that being selected later by the model did not lead to a lower probability of fixation.
Eccentricity had a large effect on the probability of inspection, F(1, 14) = 149.2, MSE = 0.018, p < .001. As was the case with the distractor in target-present trials, objects were much more likely to be fixated when they were on the inner circle. Saliency rank also had an effect, F(4, 56) = 6.4, MSE = 0.004, p < .001, but as can be seen from Figure 2 this was in the direction opposite to that expected. The most salient object in the array, ranked 1, was fixated somewhat less often than objects that were selected later by the model. A reliable interaction, F(4, 46) = 4.1, MSE = 0.0052, p = .031, indicated that saliency rank only affected inspection probability in objects on the inner circle. This is probably due to a floor effect as objects in the outer positions were rarely fixated at all.
Within-object landing positions
The distractors might have had their effects by shifting the site of fixation within the target to one that was less than optimal. We therefore looked at the end point of the first saccade to enter the target region. The saccade data were transformed in the following ways. The saccades were filtered according to amplitude: Only saccades entering the target that were greater than 1° were included. This excluded small corrective saccades that came from just outside the target region. The landing position of the saccade was calculated relative to the centre of the target, so that a saccade that landed exactly on the centre of the target was considered to have an offset of zero. To render all saccades and target positions comparable the offset from the target centre was separated into horizontal and vertical components, and the direction was coded relative to the centre of the search array. Negative offsets indicate positions inside the target circle, whereas positive offsets are closer to the edge of the display.
Figure 3 shows the distribution of landing positions, for horizontal and vertical offsets, plotted with bins of width 0.5°. The distributions peak on the inside of the target circle, between 0.25° and 0.75° from the object's centre. Beyond these regions the proportion of saccades decreases in both directions, with only around 10% of all landing sites over- or undershooting the target by more than 1.25°, which would be beyond the edge of the target object. The mean offsets (–0.23° and –0.19° for horizontal and vertical, respectively) confirm that fixations landed short of target centre.
The distribution of landing positions within the target region for horizontal (top) and vertical (bottom) offsets. Data points show the mean (with standard error bars) proportion of all the first saccades that landed at this position. The position axes show the midpoint of 0.5° bins. Negative offsets indicate undershoots that landed between the centre of the object and the centre of the whole array. The mean landing position is shown by a dashed line. The shaded area in each case represents the average dimensions of the target object within this region.
The arrangement of the stimuli allow for some clear predictions as to how landing positions change with the adjacent object. If the distractor makes a difference to the accuracy of the saccade then we might expect the average landing position of inner targets to shift outwards (towards the distractor). Conversely, outer targets, which have a distractor intervening between the centre and the target, might lead to a landing position shifted inwards. If saliency affects the extent to which an adjacent object attracts attention, this effect might be stronger for salient distractors. The distribution of first-saccade landing positions is shown for each of the four conditions in Figures 4 (horizontal offsets) and 5 (vertical offsets). The distribution in each case is similar. To quantify any differences a repeated measures ANOVA compared the mean landing position (dotted lines in Figures 4 and 5) as a function of eccentricity and saliency. For horizontal offsets, there were no reliable main effects and no interaction, all Fs(1, 14) < 1. Vertical landing positions showed a main effect of eccentricity, F(1, 14) = 10.3, MSE = 0.036, p < .01, with outer targets leading to a more negative mean landing position (i.e., one landing closer to the centre of the array) than inner targets. Distractor saliency had no effect, and there was no interaction, both F(1, 14) < 1.
Horizontal landing position distributions for each of the four conditions. IS = inner target salient distractor, IN = inner target nonsalient distractor, OS = outer target salient distractor, ON = outer target nonsalient distractor.
Vertical landing position distributions for each of the four conditions. IS = inner target salient distractor, IN = inner target nonsalient distractor, OS = outer target salient distractor, ON = outer target nonsalient distractor.
These data provide novel evidence for a preferred viewing position within objects. Is this position functional? The optimal viewing position observed in reading single words is the fixation site that leads to the fastest processing. In objects, a central landing site is associated with the lowest refixation probability and the longest subsequent fixation duration (Henderson, 1993). To search for this effect of landing position on subsequent behaviour the Euclidian distance between target centre and first-saccade landing position was compared with the likelihood of making at least one other fixation on the same object and the mean duration of the fixation following the saccade. If peripheral landing sites are nonoptimal the target might need to be fixated again to be fully processed, and thus the current fixation might be terminated early to allow for a better position. Figure 6 gives the mean refixation probability and the mean fixation duration for each position.
The optimal landing position within the target. The first entry saccades were binned according to their Euclidian distance from the target. Data points show the mean probability of refixation (top panel) and the mean duration of the subsequent fixation (bottom panel) for saccades in each bin. Error bars show the standard error of the mean.
Consistent with predictions, these measures were not constant across landing positions. Refixations were relatively infrequent when the saccade landed within about 1.5 degrees of the target centre, occurring about 30% of the time. When the saccade landed further away, a refixation became much more likely, so that a landing site more than 2 degrees away (and therefore beyond the edge of the object) led to a refixation in 65% of cases. The mean fixation duration was less variable for all but the most distant position. Subsequent fixations at the very edge of the target region were much shorter than those that landed on the target. The effect of landing position on refixation probability was reliable, F(3, 42) = 5.6, MSE = 0.022, p < .005. This analysis did not include the most distant positions as there were too few data points, and for this reason the same analysis on mean fixation duration was not reliable, F(3, 42) < 1.
Discussion
Performance of the search task was affected by both object eccentricity and, more interestingly, distractor saliency. Targets were found more slowly when they were on the outer circle, outer distractors were less likely to be fixated, and eccentricity was a good predictor of whether objects would be fixated in target-absent trials. Objects at the edge of the display were only rarely fixated, indicating that it was relatively easy to reject objects as nontargets without fixating them.
The role of bottom-up saliency on search has been refuted (Chen & Zelinsky, 2006; Foulsham & Underwood, 2007), but here more salient distractors had more of an impact on search. Distractors that were highly salient were associated with more time before fixating the target and longer RTs than nonsalient objects. This supports the saliency model and shows that bottom-up information can have an effect on search. Even when there were no systematic differences in their similarity to the target, salient distractors were more likely to slow search. Top-down guidance did not completely override bottom-up saliency in this experiment.
A problem for a bottom-up approach in this task is the absence of a relationship between model-predicted object saliency and fixation probability in target-absent trials. Saliency might be expected to have a noticeable effect in these trials, as there was no target object to be inspected. Despite this, objects that were ranked highly salient by the model were no more likely to be fixated than those that were not. Whilst the search array was relatively less complex and varied than the natural scenes used in other experiments, Itti and Koch (2000) argue that their model is a useful predictor even in a simple feature/conjunction search for oriented lines. Assuming that the model is a reliable estimate of bottom-up saliency then a top-down strategy must predict the fixation patterns. How were objects selected in these trials? Unlike search within a real scene, the present results cannot be explained by a bias towards locations deemed more relevant for search: Targets were equally likely to be positioned anywhere in the array so there was no contextual guidance in this way. It seems likely that the objects were selected based on their similarity to the target. Several models have been proposed that could potentially simulate this with the complex objects here (Rao et al., 2002; Zelinsky, Zhang, Yu, Chen, & Samaras, 2005).
The results concerning within-object landing positions are concordant with research in reading and simple objects that shows that viewers prefer to saccade close to the centre of the region of interest. The first saccades made to the target region here tended to land on the inside of the target, rather than exactly on the target centre. This can be seen as a tendency to undershoot a saccade target, which is also an explanation posited for the left-of-centre PVL observed in reading (McConkie, Kerr, Reddix, & Zola, 1988).
The presence of another object might be expected to skew the modal landing position, and, for vertical offsets, saccades to targets on the outer circle landed closer to the distractor. However, this was not the case for horizontal offsets. The landing site in this study also affected subsequent eye movements, with saccades failing to land near the centre of the target being more likely to lead to a short fixation duration and a refixation. The implication of this is that the modal landing site is optimal for processing the target and that when the saccade does not land here it is more efficient to terminate the current fixation and refixate than to continue trying to extract the required information.
Why were salient objects more distracting? We can discount several possibilities. Saliency prolonged the time to first fixate the target, and so distraction did not just occur when targets were being processed at fixation. The initial saccadic latency was not affected by saliency, and neither was the within-target landing position. Most important, salient distractors were not fixated any more often, meaning that they must have had their effects in peripheral vision, presumably by drawing covert attention. This may have led to slower subsequent saccades or more nontargets having to be fixated before reaching the target. Saliency gave an indication of the potency of a distractor that was remote from fixation. Although there is evidence for the effect of saliency on fixation placement, this is the first suggestion that model-predicted saliency makes a difference even when it does not necessarily draw fixation.
The experiment suggests that there are regularities in the landing position of saccades within natural objects, that top-down guidance cannot completely dominate, and that the saliency of a distractor makes a difference in search, even when it is not fixated.