The sensory nature of knowledge: Sensory priming effects in semantic categorization

Despite several decades of research, the question of the contents of long-term memory is one of the key unresolved issues relating to the field of human cognition. Increasing numbers of both behavioural and neuroscientific studies have demonstrated the existence of conceptual representations that possess a perceptual nature since they correspond to the patterns of activation of the neuronal systems associated with sensorimotor experiences (see, e.g., Barsalou, 1999, 2005; Glenberg, 1997; Martin & Chao, 2001; Martin, Ungerleider, & Haxby, 2000; Pulvermüller, 1999; Versace, Labeye, Badard, & Rose, 2009; Weinberger, 2004). The memory encoding of the perceptual components of our experiences is therefore assumed to involve the systems that are associated with the perceptual mechanisms rather than to take the form of specific and differentiated memory systems. Thus, all forms of knowledge emerge from the activation, integration, and synchronization of these multiple systems.

These mechanisms involved in the activation and integration of the perceptual components of memory traces are therefore assumed to form the basis for categorization. For example, Barsalou (e.g., 1999, 2005) has taken a particular interest in the perceptual nature of concepts and has developed the idea of concept simulators. According to Barsalou, to categorize the picture of a hammer as a tool, the neural systems for vision, action, audition, touch, and so on, re-enact the experience of a hammer. This re-enactment is thought to involve the activation of sensorimotor areas, but also of associative areas that register the configurations of sensory features at both the intramodal and the intermodal levels. Any given simulator can produce an unlimited number of simulations, with these different simulations corresponding to multimodal representations of the different instantiations of a concept. The exemplar that is actually generated depends on a number of different factors (context, participant's state, goals, etc.).

Many studies designed to demonstrate the perceptual nature of conceptual knowledge have employed various paradigms involving the interference between or the facilitation of items of information that are available either simultaneously (Stroop-type paradigms) or successively (long-term or short-term priming paradigms). These studies (for a review, see Versace et al., 2009) are based on the idea that all the perceptual components of an experience that are encoded in memory can be reactivated when the participant is confronted with an object, irrespective of the input modality associated with the object in question (object seen, heard, felt, etc.). For example, Zwaan, Stanfield, and Yaxley (2002) asked their participants to read sentences that evoked an object or animal in a certain position, with the shape of the animal or object being different depending on the position (e.g., an eagle in flight or perched on its nest). After each sentence, a drawing of an object or an animal was presented, and the participants either had to indicate whether or not this object/animal had been mentioned in the sentence (Experiment 1) or were simply asked to name the object/animal (Experiment 2). In both experiments, responses were faster when the shape of the object or animal was similar in the drawing and the sentence (see also Graf, Kaping, & Bülthoff, 2005; Richardson, Spivey, Barsalou, & McRae, 2003; Stanfield & Zwaan, 2001; Zwaan & Yaxley, 2003, 2004).

In the motor domain, Ellis and Tucker (2000) investigated the influence of the activation of motor representations resulting from a visual stimulus on the processing of an auditory stimulus. The participants had to identify sound volumes using a response mechanism that required them either to use their thumbs and index fingers or to grasp with the whole hand. Each sound was preceded by a picture of an object that implied either delicacy (using the thumb and index finger) or more global grasping. The pictures were presented for 700 ms, and the participants were told to memorize them for subsequent recognition. The authors observed a congruence effect between the type of grasping induced by the object and the type of response to the sound (see also Castiello, Lusher, Mari, Edwards, & Humphreys, 2002; Craighero, Fadiga, Rizzolatti, & Umiltà, 1998; Edwards, Humphreys, & Castiello, 2003; Glenberg & Kaschak, 2002; Tucker & Ellis, 1998).

However, an interpretation of the results of these studies in terms of amodal representations cannot be excluded. For example, since there is usually a large interval between the first stimulus (the prime) and the second stimulus (the target), one interpretation would be that the prime activates an amodal representation, which, in turn, activates the sensory or motor areas involved in the processing of the target.

The aim of the present study was therefore to show the perceptual nature of conceptual knowledge by using a priming paradigm that does not permit an interpretation in terms of amodal representation. The paradigm employed represented an adaptation of one used in an earlier study (Brunel, Labeye, Lesourd, & Versace, 2009). The Brunel et al. study demonstrated that the presentation, in a test phase, of a shape that was associated with a sound in the learning phase reactivates the auditory component (see also Meyer, Baumann, Marchina, & Jancke, 2007) and may influence the processing of a target sound (high or low pitched). In our new procedure we examined whether such priming effects could generalize to the perception of visual objects that were associated with sounds. This new paradigm was divided into two phases. The first phase consisted in learning a systematic association between a geometrical shape and a white noise. The second phase consisted of a short-term priming paradigm in which a shape (either associated or not with a sound in the first phase) preceded a picture of an object, which the participants had to categorize as representing either a large or a small object (more or less than 50 cm high). The objects were chosen in such a way that their principal function either was typically associated with noise (e.g., a blender) or was not associated with noise (e.g., a screwdriver).

We consequently hypothesized that if categorization involves the activation of sensory areas that encode the perceptual components of memory traces, then the categorization of a target picture that is typically associated with noise (referred to as “noisy” targets in contrast to “silent” targets) should be influenced by the prior presentation of a prime that was accompanied by a white noise in the encoding phase. The physical characteristic of white noise is that it contains all the auditory frequencies, from low pitched to high pitched. Therefore, the prime shape that has previously been associated with a white noise is able to prime the “noisy” picture targets irrespective of the physical characteristics of the sounds (reactivated by the “noisy” targets).

Moreover, to demonstrate that the activated memory components are perceptual and not amodal in nature, we used two different levels of stimulus onset asynchrony (SOA) between the prime and the target (see Brunel et al., 2009). If we consider that the activated auditory memory component retains all of its encoded characteristics (duration and frequency) when the SOA is shorter than the duration of the sound that was associated with the shape during the learning phase, then the activation of the auditory component should interfere with the processing of the noisy target, since the white noise that is thought to be reactivated contains auditory frequencies that are different from that of the target sound. In this context, the interference effect is due to the temporal overlap between the reactivation of the previous associated sound and the sound reactivated by the noisy target picture.

In contrast, if the SOA is the same as or longer than the duration of the sound that was associated with the shape during the learning phase, then the activation of the auditory component should facilitate the processing of the noisy target picture. In this context, there is no longer any temporal overlap between the reactivation of the previous associated sound and the sound reactivated by the noisy target picture. The facilitation effect could therefore be explained by the fact that the preactivation of a sensory modality facilitates subsequent processing in this modality (see, e.g., Pecher, Zeelenberg, & Barsalou, 2004) and does so to a greater extent as the particular frequency of the sound reactivated by the noisy target becomes more prevalent in the reactivated white noise.

Similar interference and facilitation effects have been found by Brunel et al. (2009). However, these authors used high-pitched and low-pitched sounds as targets. The influence of the prime on the target sounds demonstrated the perceptual nature (auditory in the authors’ study) of the memory component reactivated by the shape of the prime. Therefore the aim of the present experiment was to demonstrate that the categorization of a target picture that is typically associated with noise involves the activation of sensory areas that encode the perceptual components of memory traces. Such a finding would confirm that the processing of semantic knowledge systematically involves the reactivation of relevant sensory properties, just as if these sensory properties were actually present. Our assumption is consistent with the idea that access to a concept is intimately linked with an implicit mental simulation (Barsalou, 2008; Cassanto, Willems, & Hagoort, 2009; Zwaan, 2004).

EXPERIMENT

Method

Participants

A total of 32 right-handed voluntary participants were recruited for the experiment. All of them were students at the University Lumière Lyon 2, France, and had normal or corrected-to-normal vision.

Stimuli and materials

Very simple stimuli were used in the learning phase to permit the precise control of the visual and auditory stimulus components. The visual stimulus was a geometric shape, in the form of either a 7-cm square or a circle with a radius of 3.66 cm. The auditory stimulus in the learning phase consisted of a white noise. The auditory stimulus was presented monophonically for a duration of 500 ms. To make the experiment less monotonous, the square and circle could be displayed in four different levels of grey.

The test phase consisted of a large/small categorization task: A total of 64 coloured target pictures were selected, 32 of which depicted a small object (below 50 cm real size) and 32 of which portrayed a large object (greater than 50 cm). To test our hypothesis, it was necessary to differentiate between objects whose principal function is typically associated with noise (which we refer to as “noisy” targets, e.g., a blender) and objects that are not typically associated with noise (“silent” targets, e.g., a screwdriver). In line with this approach, we selected objects as a function of their sound properties: For example, a “nail” is mostly encoded as a silent object except when it is used together with a hammer, whereas a “blender” is always associated with sound when it is used. Therefore, in our study “nail” was considered to be a silent target while “blender” was considered to be a noisy one. However, the large objects were more heterogeneous than the small objects in terms of size and were also more difficult to control in terms of sound emission, in particular because of the high level of correlation between size and sound intensity. Consequently, large objects were used as distractors only. We therefore selected 16 pictures of small noisy objects, 16 pictures of small silent objects, and 32 pictures of large objects, about half of which were predominantly noisy, while the other half were predominantly silent (see the complete list of stimuli in the Appendix). The size of the pictures was fixed at 397 pixels in width by 285 pixels in height, with a resolution of 75 pixels per inch.

The experiment was carried out on a Macintosh microcomputer (eMac G4). Psyscope software X B41 LDEC8 (J. D. Cohen, MacWhinney, Flatt, & Provost, 1993) was used to create and manage the experiment. In addition, a chin rest was used to keep the distance between the participants and the monitor constant (60 cm). So the range of visual angles of the stimuli was 12° to 14° width and 10° to 13° height.

Procedure and design

After filling in a written consent form, each participant was tested individually during a session that lasted approximately 10 minutes. The experiment consisted of two phases. The first phase (the learning phase) was based on the hypothesis that the incidental repetition of a sound–shape association should lead to the integration of these two different components in memory (see the procedure in Figure 1a). Consequently, each trial consisted of the presentation of a shape (a square or a circle) for 500 ms, with one of these shapes being presented simultaneously with the white noise. The participants were told that their task was to judge, as quickly and accurately as possible, whether the shape was a square or a circle. They indicated their response by pressing the appropriate key on the keyboard. All the visual stimuli were presented in the centre of the screen, and the intertrial interval was 1,500 ms. For half of the participants, the squares were presented with the white noise, and the circles were presented without sound, while the opposite configuration was used for the other half of the participants. Each shape was presented 32 times (8 times for each of the four grey-scale levels) in a random order. Half of the participants used their right index finger for the square response and their right middle finger for the circle response, with the response fingers being inverted for the other half of the participants.

OPEN IN VIEWER

Figure 1.

Organization of the trials in (a) the first and (b) the second phase.

The second phase consisted of a short-term priming paradigm (see the procedure in Figure 1b). The prime was one of the two shapes presented during the learning phase (a square or a circle). In half of the trials, the prime was the shape that had been associated with the white noise during the encoding phase (sound prime), while for the other half, the prime was the shape that had not been associated with the white noise (silent prime). Half of the participants saw the prime presented for a period of 100 ms, while the other half saw it presented for 500 ms. The participants had to judge as quickly and accurately as possible whether the target picture represented a large or a small object (larger or smaller than 50 cm) by pressing the appropriate key on the keyboard. It is important to stress here that all the participants were instructed to keep their eyes open during the entirety of this phase. As the target appeared immediately on the disappearance of the prime, the SOA between the prime and the target was also 100 ms and 500 ms in the two groups of participants, respectively. All the visual stimuli were presented in the centre of the screen, and the intertrial interval was 1,500 ms. Each participant saw a total of 64 trials—that is, 8 trials in each of the experimental conditions resulting from the crossing of target size (large vs. small), target sonority (noisy target vs. silent target), and prime sonority (noisy prime vs. silent prime). The targets that were preceded by a sound prime for half of the participants were preceded by a silent prime for the other half of the participants. The order of the different experimental conditions was randomized.

Results and discussion

The mean correct response latencies and the mean percentages of correct responses were calculated across subjects and items for each experimental condition. Latencies below 250 ms and above 1,250 ms were removed (less than 3% of the data). Separate analyses of variance were performed on the latencies and percentages of correct responses, with subjects as random variables (F _s ), prime type (prime associated or not associated with a sound in the learning phase) and small target type (noisy target or silent target) as within-subjects factors, and SOA (100 ms or 500 ms) as the between-subjects factor. Analyses of variance were also performed with small items as random variables (F _i ), prime type and SOA as within-items factors, and target type as between-items factor.

The analyses performed on the proportion of correct responses revealed neither a significant main effect nor any interaction. This result could be explained in terms of ceiling effects since the overall level of correct responses was 93.98% (see Table 1). As far as the latencies are concerned, the analyses revealed only a significant interaction between SOA, prime type, and target type, F _s (1, 30) = 14.755; p < .001, and F _i (1, 30) = 20.743; p < .001. As Figures 2a and 2b clearly show, priming effects were, as we expected, observed only for the noisy targets, and these priming effects were positive for the 500-ms SOA and negative for the 100-ms SOA.

OPEN IN VIEWER

Figure 2.

(a) Mean response times (RTs), with 100-ms stimulus onset asynchrony (SOA), as a function of target type, for each prime type. Error bars represent standard error. (b) Mean RTs, with 500-ms SOA, as a function of target type, for each prime type. Error bars represent standard error.

OPEN IN VIEWER

Table 1.

Mean percentages of correct responses in each experimental condition for small targets

Target	Prime type	SOA
		100 ms	500 ms
Noisy	Silent	92 (2.7)	96 (1.1)
	Sound	90 (4.0)	94 (1.7)
Silent	Silent	94 (3.9)	96 (1.6)
	Sound	91 (3.9)	97 (1.3)

Note: SOA = stimulus onset asynchrony. Standard errors in parentheses.

Separate analyses of variance were performed for each SOA in order to illustrate these results. For the 100-ms SOA, the analysis revealed a significant interaction between prime type and target type, F _s (1, 15) = 10.6, p < .01, and F _i (1,30) = 7.00, p < .05 (see Figure 2a). As expected, planned comparisons showed that noisy targets were categorized less rapidly when they were preceded by a sound prime than when they were preceded by a silent prime, F _s (1, 15) = 18.59, p < .001, and F _i (1, 30) = 28.24, p < .001. In contrast, no priming effect was observed for silent targets, F _s < 1 and F _i < 1.

In the case of the 500-ms SOA, the analysis also revealed a significant interaction between prime type and target type, F _s (1, 15) = 6.24, p < .05, and F _i (1, 30) = 5.84, p < .05 (see Figure 2b). Planned comparisons showed that the noisy targets were categorized more rapidly when they were preceded by a sound prime than when they were preceded by a silent prime, F _s (1, 15) = 9.09, p < .01, and F _i (1, 30) = 24.21, p < .001. In the case of the 100-ms SOA, no priming effect was observed for the silent targets, F _s < 1 and F _i < 1.

While large items were excluded from the a priori analyses, we conducted separate a posteriori analyses of variance on the latencies and percentages of correct responses, with subjects as random variables, prime type (prime associated or not associated with a sound in the learning phase) and large target type (noisy target or silent target) as within-subjects factors, and SOA (100 ms or 500 ms) as the between-subjects factor (see Table 2). The analyses performed on the latencies and correct responses revealed only a main effect of the target type, with large noisy targets being processed more slowly, F(1, 30) = 10.67, p < .01, and less accurately, F(1, 30) = 8.74, p < .01, than large silent targets. These results underlined the fact that the large targets were more heterogeneous in size than the small targets and might explain the lack of significant interaction between prime type, target type, and SOA. Nevertheless it is perhaps worth noting that the trends in the data clearly support the findings with the small objects.

OPEN IN VIEWER

Table 2.

Mean response times and mean percentages of correct responses in each experimental condition for large targets

Target	Prime type	SOA
		100 ms		500 ms
		RT (ms)	CR (%)	RT (ms)	CR (%)
Noisy	Silent	765 (19)	81.0 (4.7)	812 (27)	84.7 (2.7)
	Sound	800 (25)	79.3(5.8)	799 (25)	78.9 (5.1)
Silent	Silent	730 (28)	91.3 (2.2)	770 (24)	87.4 (4.9)
	Sound	735 (19)	89.8(2.8)	778 (24)	89.7 (3)

Note: RT = response time. CR = percentage of correct responses. SOA = stimulus onset asynchrony. Standard errors in parentheses.

General Discussion

The aim of the present study was to show the perceptual nature of conceptual knowledge by means of a priming paradigm that excluded an interpretation exclusively in terms of amodal representations. Our hypothesis was that if categorization involves the activation of sensory areas that encode the perceptual components of memory traces, the categorization of a target picture that is typically associated with a sound should be affected by the prior presentation of a prime that was associated with a white noise during the learning phase. Moreover, if the SOA between the prime and the target is shorter than the duration of the sound that was associated with the shape during the learning phase, then the preactivation of the auditory component should interfere with the processing of the “noisy” target. In contrast, if the SOA is the same as or longer than the duration of the sound that was associated with the shape during the learning phase, then the preactivation of the auditory component should facilitate the processing of the noisy target. The results clearly confirmed our predictions.

The perceptual nature of the auditory component reactivated by both the sound prime and the noisy target is demonstrated, in particular, by the fact that we obtained an interference effect with an SOA of 100 ms and a facilitatory effect with an SOA of 500 ms. We interpreted the interference effect obtained at the 100-ms SOA as reflecting an overlap between the components reactivated by the sound prime and those activated by the processing of the noisy target. When an SOA of 500 ms was used, there was no temporal overlap. The observed facilitatory effect could be explained in terms of the preactivation, by the sound prime, of auditory areas that facilitate the categorization of noisy targets only. If the memory components that code the sounds associated with the prime and the target objects are amodal, then there is no reason why the short SOA should lead to an interference effect. Moreover, since the white noise that is thought to be reactivated by the prime is different from the auditory representation of the noisy target, it is difficult to predict any priming effect whatever the SOA.

The present paper questions the idea that semantic knowledge is amodal in nature. Indeed, this idea is associated with a multisystem, abstractive description of memory, which foregrounds the concept of memory systems, with each system being associated with different representational forms and thus with specific contents (e.g., Anderson, 1983; N. J. Cohen & Squire, 1980; Squire, 1987; Tulving, 1995). One of these systems, semantic memory, is thought to contain abstract, decontextualized knowledge that individuals possess concerning the world around them (conceptual knowledge). Our results are very difficult to interpret within this structural approach as they demonstrate the existence of auditory sensory properties encoded with visual properties into memory traces. These results provide a strong argument in favour of the idea that access to conceptual knowledge is linked to the simulation of the sensory dimension integrated with the concept (see Barsalou, 2008).