i-Perception

A sensory substitution device for blind persons aims to provide the missing visual input by converting images into a form that another modality can perceive, such as sound. Here I will discuss the perceptual learning and attentional mechanisms necessary for interpreting sounds produced by a device (The vOICe) in a visuospatial manner. Although some aspects of the conversion, such as relating vertical location to pitch, rely on natural crossmodal mappings, the extensive training required suggests that synthetic mappings are required to generalize perceptual learning to new objects and environments, and ultimately to experience visual qualia. Here I will discuss the effects of the conversion and training on perception and attention that demonstrate the synthetic nature of learning the crossmodal mapping. Sensorimotor experience may be required to facilitate learning, develop expertise, and to develop a form of synthetic synaesthesia.

Visual-to-auditory sensory substitution devices have various benefits over tactile counterparts (eg, less hardware limitations), but they also suffer from several drawbacks (eg, learning time, potentially unpleasant sounds). An ‘ideal’ device would be intuitive to learn, pleasant to listen to, and capture relevant visual information in sufficient detail. In this presentation, we outline the general problem of how to convert an image into sound, and we give an overview of some possible approaches to the problem. We then go on to describe our own recent explorations using Interactive Genetic Algorithms (IGAs). IGAs enable a highly dimensional problem space to be explored rapidly. Initially, a set of orthogonally varying settings need to be identified (eg, different levels of maximum and minimum pitch, different ways of mapping lightness-loudness, musical vs non-musical intervals), and a set of random permutations of these settings are chosen. Participants then evaluate the ‘fitness’ of these different algorithms (eg, by selecting what the correct image is for a given sound). The fittest algorithms are then ‘bred’ together over successive generations. Using this approach, we compare the performance of evolved devices against one of the main existing devices (the vOICe) in three tasks: audio-visual matching, aesthetic preference, and auditory discrimination.

About one-quarter of our brain “real estate” is devoted to the processing of vision. So what happens to this vast “vision” part of the brain when no visual input is received? We are working with novel high-tech multisensory ‘glasses’ that convert visual information from a tiny video camera into sensory signals that the blind can interpret. In this talk I will mainly highlight work done using The vOICe algorithm (Meijer et al 1992). We have devised a training program which teaches blind individuals to use such a device. Following approximately 30 hours of training, congenitally blind individuals can use this device to recognize what and where various objects are, for instance, within a room (like a chair, glass, and even people and their body posture; eg, see http://brain.huji.ac.il/press.asp). Additional training is given specifically for encouraging free “visual” orientation enabling blind individuals to walk in corridors while avoiding obstacles and applying hand-“eye” coordination (eg, playing bowling). A main focus of the project is using this unique “set-up” to study brain organization and brain flexibility. For example, we are elucidating how the subjects' brains use preserved functions on one hand and on the other hand, reorganize to enable to process this new sensory language (eg, See Amedi et al Nature Neurosience 2007; Stiem-Amit et al 2011; Reich et al 2011). I will also focus on novel spectral analysis approaches to study large-scale brain dynamics and to look into the binding problem: how we integrate information into a coherent percept, an old question in neuroscience which has relatively poor answers, especially in humans. On the rehabilitation front, we have demonstrated that visual training can create massive adult plasticity in the ‘visual’ cortex to process functions like recognizing objects and localizing where they are located, much like the original division of labor in the visual system in which the ventral stream recognize objects and the dorsal stream help to localize them in order to orient action. Such visual cortex recruitment for ‘visual’ processing of soundscapes may greatly promote sight restoration efforts both via such technologies and by training people undergoing clinical procedures to restore vision. This approach might also be relevant, in other cases in which massive adult brain plasticity / flexibility is needed, eg, after a stroke.

The tongue display unit (TDU) is a sensory substitution device that translates visual information into electrotactile stimulation that is applied to the tongue. Blind subjects can learn to use the TDU in various visual tasks, including orientation, motion and shape identification, and spatial navigation. We used the TDU in conjunction with brain imaging techniques in order to study the cerebral correlates of cross-modal brain plasticity. The results show that when blind subjects use the TDU to perform visual tasks that are known to activate the dorsal and ventral visual streams in the sighted, they activate the same brain areas. This suggests that motion and shape processing are organized in a supramodal manner in the human brain and that vision is not necessary for the development of the functional architecture in motion and shape processing areas. We also used the TDU in spatial navigation tasks. The results showed that blind but not blindfolded sighted subjects activated their visual cortex and right parahippocampus during navigation, suggesting that in the absence of vision, cross-modal plasticity permits the recruitment of the same cortical network used for spatial navigation tasks in sighted subjects.

It has long been established that the occipital cortex of early blind animals and humans is activated by nonvisual stimuli (Rauschecker 1995). Using functional magnetic resonance imaging, we have demonstrated recently that the middle occipital gyrus (MOG) of early blind humans retains its function in spatial localization and is activated in auditory and tactile spatial tasks (Renier et al 2010). Sound localization performance was directly correlated with amount of activation in MOG. Currently we are using a visual-to-auditory substitution device to investigate whether cortical areas in the visual ventral stream also retain their function in face and object recognition when they are stimulated by auditory and tactile “objects”. The results should make a valuable contribution not only to further understanding of this fundamental question regarding cortical plasticity but also to aid blind persons in everyday practical situations.

A number of studies have documented that changes in visual cognition following profound deafness are highly specific. Among these, deaf individuals have proved to have enhanced visual ability compared with hearing controls when asked to rapidly detect the onset of abrupt visual stimuli, particularly when presented towards the periphery of the visual field. However, no study to date has addressed the question of whether visual compensatory changes may be reversed after reafferentation of the auditory system through a cochlear implant. To address this question, we measured reaction times to visually presented stimuli appearing in central and peripheral locations in two groups of adult cochlear implant recipients, who experienced auditory loss either early or late in life. Results showed that prelingually deafened recipients were faster than postlingually deafened recipients for stimuli appearing in the periphery of the visual field. On the contrary, postlingually deafened individuals paid a cost for stimuli presented in the periphery of the visual field, as typically found in hearing controls adopting identical task and stimuli. These findings lead to the suggestion that compensatory changes that occur early in life cannot be reversed in adulthood in case of sensory reafferentation.

Flight simulators which provide visual, auditory, and kinematic (physical motion) cues are increasingly used for pilot training. We have previously shown that kinematic cues, but not auditory cues, representing aircraft motion improve target tracking performance for novice ‘pilots’ in a simulated flying task (Meyer et al IMRF 2010). Here we explore the effect of learning on task performance. Our subjects were first tested on a target tracking task in a helicopter flight simulation. They were then trained in a simulator-simulator, which provided full audio, simplified visuals, but not kinematic signals to test whether learning of auditory cues is possible. After training we evaluated flight performance in the full simulator again. We show that after 2 hours training auditory cues are used by our participants as efficiently as kinematic cues to improve target tracking performance. The performance improvement relative to a condition where no audio signals are presented is robust if the sound environment used during training is replaced by a very different audio signal that is modulated in amplitude and pitch in the same way as the training signal. This shows that training is not signal specific but that our participants learn to extract transferrable information on sound pitch and amplitude to improve their flying performance.

Sometimes we feel animacy for artificial objects and their motion. Animals usually interact with environments through multiple sensory modalities. Here we investigated how the sensory responsiveness of artificial objects to the environment would contribute to animacy judgment for them. In a 90-s trial, observers freely viewed four objects moving in a virtual 3D space. The objects, whose position and motion were determined following Perlin-noise series, kept drifting independently in the space. Visual flashes, auditory bursts, or synchronous flashes and bursts appeared with 1–2 s intervals. The first object abruptly accelerated their motion just after visual flashes, giving an impression of responding to the flash. The second object responded to bursts. The third object responded to synchronous flashes and bursts. The forth object accelerated at a random timing independent of flashes and bursts. The observers rated how strongly they felt animacy for each object. The results showed that the object responding to the auditory bursts was rated as having weaker animacy compared to the other objects. This implies that sensory modality through which an object interacts with the environment may be a factor for animacy perception in the object and may serve as the basis of multi-modal and cross-modal inference of animacy.

Visually induced illusory self-motion (vection) is generally facilitated by consistent information of self-motion from other modalities. We provide three examples that consistent information between vision and other proprioception enhances vection, ie, locomotion, air flow, and sounds. We used an optic flow of expansion or contraction created by positioning 16,000 dots at random inside a simulated cube (length 20 m), and moving the observer's viewpoint to simulate forward or backward self-motion of 16 m/s. First, We measured the strength of forward or backward vection with or without forward locomotion on a treadmill (2 km/h). The results revealed that forward vection was facilitated by the consistent locomotion whereas vections in the other directions were inhibited by the inconsistent locomotion. Second, we found that forward vection intensity increased when the air flow to subjects' faces produced by an electric fan (the wind speed was 6.37 m/s) was provided. On the contrary, the air flow did not enhance backward vection. Finally, we demonstrated that sounds which increased in loudness facilitated forward vection and the sounds which ascended (descended) in pitch facilitated upward (downward) vection.

Virtual reality in various forms has been around for about 40 years. It has been considered mainly as a technology that can be used to generate the illusion of a transformation of place. However, it has recently been shown that it can successfully be used to transcend the self, through illusions of body ownership and transformation. Several papers have shown that it is possible to generate in people the illusory sense of ownership of a fake body using virtual reality techniques [1-5]. This can be achieved through synchronous multisensory stimulation with respect to the real and virtual body. For example, the participant sees his or her virtual body touched, while feeling the touch synchronously and in the same place on the real body. This can also lead to illusions of body transformation, such as a thin person having the illusion of being fat [6]. Our research suggests the prime importance of a first person perspective for the illusion of ownership, however, in [7] we also found that a representation of a person in a virtual mirror with synchronous visual-motor effects also results in a body ownership illusion. Although virtual body ownership has been established, what is also of interest are the consequences of this in terms of the attitudes and behaviour of the participant who experiences the transformed body. Our very recent work suggests that the illusion of ownership of a virtual body may also result in at least short-term transformations of behaviour and attitudes of the participant towards those that are appropriate to the virtual body. This talk will describe several experiments illustrating both the illusion of body ownership and its transformative effect on attitudes and behaviour.

Anesthesia affects cortical response properties and induces changes in functional network connectivity. Many studies in sensory neurophysiology do not explicitly differentiate between anesthetic states and possibly underestimate the effect of anesthetic level on cortical response properties. To investigate this question, we performed simultaneous multisite recordings in V1 of C57bl/6 mice under continuously changing inspiratory isoflurane concentration from low to high dosage. During the continuous change of anesthesia level we presented visual and auditory stimuli. At all levels of anesthesia, visual stimulation evoked responses in V1. However, during light anesthesia auditory stimulation slightly suppressed ongoing LFP activity in V1. With increasing anesthesia levels this suppression vanished, and at a specific point a discrete change in sensory responsiveness of V1 to auditory stimulation appeared. At this discrete point auditory suppression was reversed into auditory induced activation of V1. This transition in properties was coincident with the appearance of burst-suppression. The responses during burst suppression state had long latency and resembled spontaneous bursts. This presumably unspecific auditory effect in V1 disappeared as soon as isoflurane concentration level dropped below a certain level. The results demonstrate that in the extreme case heteromodal effects can be induced by varying anesthesia level.

We can identify objects by sight or by sound, yet far less is known about auditory object recognition than about visual recognition. Any exemplar of a dog (eg, a picture) can be recognized on multiple categorical levels (eg, animal, dog, poodle). Using fMRI combined with machine-learning techniques, we studied these levels of categorization with sounds rather than images. Subjects heard sounds of various animate and inanimate objects, and unrecognizable control sounds. We report four primary findings: (1) some distinct brain regions selectively coded for basic (“dog”) versus superordinate (“animal”) categorization; (2) classification at the basic level entailed more extended cortical networks than those for superordinate categorization; (3) human voices were recognized far better by multiple brain regions than were any other sound categories; (4) regions beyond temporal lobe auditory areas were able to distinguish and categorize auditory objects. We conclude that multiple representations of an object exist at different categorical levels. This neural instantiation of object categories is distributed across multiple brain regions, including so-called “visual association areas,” indicating that these regions support object knowledge even when the input is auditory. Moreover, our findings appear to conflict with prior well-established theories of category-specific modules in the brain.

We investigate whether the loss of one eye leads to enhanced multisensory processing. Previously, we measured speed detection and discrimination of auditory, visual, and audiovisual targets presented as a stream of paired objects and familiar sounds in people with one eye compared to controls viewing binocularly or with one eye patched. All participants were equally able to detect the presence of auditory, visual, or bimodal targets; however, when discriminating between the unimodal and bimodal targets, both control groups demonstrated the Colavita visual dominance effect, preferential processing of visual over auditory information for the bimodal stimuli. People with one eye, however, showed no Colavita effect but rather equal preference for processing visual and auditory stimuli. In the current experiment we increased the temporal processing load in an attempt to favour auditory processing and thereby reverse Colavita visual dominance with a one-back stimulus repetition detection and discrimination paradigm. The Colavita effect was reduced in both control groups, but people with one eye continued to show no Colavita effect. People with one eye display equal auditory and visual processing, suggesting better multisensory processing, likely as a form of cross-modal adaptation and compensation for their loss of binocularity.

An informative performance measure of the brain's integration across different sensory attributes/modalities is the critical temporal rate of feature alternation (between, eg, red and green) beyond which observers could not identify the feature value specified by a timing signal from another attribute (eg, a pitch change). Interestingly, this limit, which we called the critical crowding frequency (CCF), is fairly low and nearly constant (∼2.5 Hz) regardless of the combination of attributes and modalities (Fujisaki & Nishida, 2010, IMRF). One may consider that the CCF reflects the processing time required for the brain to identify the specified feature value on the fly. According to this idea, the similarity in CCF could be ascribed to the similarity in identification time for the attributes we used (luminance, color, orientation, pitch, vibration). To test this idea, we estimated the identification time of each attribute from [Go/ No-Go choice reaction time – simple reaction time]. In disagreement with the prediction, we found significant differences among attributes (eg, ∼160 ms for orientation, ∼70 ms for pitch). The results are more consistent with our proposal (Fujisaki & Nishida, Proc Roy Soc B) that the CCF reflects the common rate limit of specifying what happens when (timing-content binding) by a central, presumably postdictive, mechanism.

Autism spectrum disorders (ASD) form a continuum of neurodevelopmental disorders characterized by deficits in communication and reciprocal social interaction, repetitive behaviors, and restricted interests. Sensory disturbances are also frequently reported in clinical and autobiographical accounts. However, few empirical studies have characterized the fundamental features of sensory and multisensory processing in ASD. Recently published studies have shown that children with ASD are able to integrate low-level multisensory stimuli, but do so over an enlarged temporal window when compared with typically developing (TD) children. The current study sought to expand upon these previous findings by examining differences in the temporal processing of low-level multisensory stimuli in high-functioning (HFA) and low-functioning (LFA) children with ASD in the context of a simple reaction time task. Contrary to these previous findings, children with both HFA and LFA showed smaller gains in performance under multisensory (ie, combined visual-auditory) conditions when compared with their TD peers. Additionally, the pattern of performance gains as a function of SOA was similar across groups, suggesting similarities in the temporal processing of these cues that run counter to previous studies that have shown an enlarged “temporal window.” These findings add complexity to our understanding of the multisensory processing of low-level stimuli in ASD and may hold promise for the development of more sensitive diagnostic measures and improved remediation strategies in autism.

The process of integrating information across sensory modalities is highly dependent upon the temporal coincidence of the inputs. Audiovisual information is integrated within a range of temporal offsets, known as the temporal binding window (TBW), which varies between individuals. Three particular findings relating to TBW have led us to a novel approach to address sensory integration impairments in children with autism. The first is that autistic children have an atypically wide TBW, as measured through manipulations of audiovisual illusions. Second, an individual's TBW is related to their ability to perceptually fuse audiovisual inputs, particularly as seen in the McGurk effect; the narrower the right TBW, the stronger the McGurk effect. The third finding is that the TBW is plastic. Through perceptual feedback training, our lab showed that individuals' right TBW can be narrowed. These three findings, which we will present, lead to a study of perceptual feedback training in autistic children, who may have the ability to narrow their TBW, with a possible positive impact on their ability to integrate multisensory information, specifically speech. We will conclude with the presentation of behavioral and electrophysiological data illustrating an atypical relationship between the TBW and perceptual fusion in ASD.

Repeated exposure to apparent motion in one direction may cause the observer to perceive a subsequently presented stationary/moving stimulus as moving in the opposite direction (Mather et al 1998; Konkle et al 2009). Here we demonstrate that aftereffect of adaptation take place not only in the spatial dimension but also in the temporal dimension. We employed a visual Ternus display and randomly varied the inter-stimulus-interval (ISI) between the two frames (from 50 to 200 ms) in the display to determine the transition threshold between “group motion” and “element motion”. In Experiment 1, participants saw 7∼9 presentations of Ternus display with an ISI of either 50 ms (for element motion) or 200 ms (for group motion) followed by a probe Ternus display with an ISI equivalent to the transition threshold. In Experiment 2, participants heard 7∼9 presentations of paired beeps with an ISI of either 50 or 200 ms and saw the same probe as in Experiment 1. Adaptation to the visual or auditory stimuli with short ISI led to more reports of “group motion” for the probe while adaptation to stimuli with long ISI had no effect. This temporal adaptation contrasts with the usual bi-directional effects for spatial adaptation.

We investigated the effect of prior conditioning of an auditory stimulus on audiovisual integration in a series of four psychophysical experiments. The experiments factorially manipulated the conditioning procedure (picture vs monetary conditioning) and multisensory paradigm (2AFC visual detection vs redundant target paradigm). In the conditioning sessions, subjects were presented with three pure tones (= conditioned stimulus, CS) that were paired with neutral, positive, or negative unconditioned stimuli (US, monetary: +50 euro cents,.–50 cents, 0 cents; pictures: highly pleasant, unpleasant, and neutral IAPS). In a 2AFC visual selective attention paradigm, detection of near-threshold Gabors was improved by concurrent sounds that had previously been paired with a positive (monetary) or negative (picture) outcome relative to neutral sounds. In the redundant target paradigm, sounds previously paired with positive (monetary) or negative (picture) outcomes increased response speed to both auditory and audiovisual targets similarly. Importantly, prior conditioning did not increase the multisensory response facilitation (ie, (A + V)/2 – AV) or the race model violation. Collectively, our results suggest that prior conditioning primarily increases the saliency of the auditory stimulus per se rather than influencing audiovisual integration directly. In turn, conditioned sounds are rendered more potent for increasing response accuracy or speed in detection of visual targets.

Perceived congruence between the changing patterns of a visual image and pitch of a sound was examined by a series of rating experiments. Vertical correspondence of direction between the movement of an image and the pitch shift of a sound had a strong effect on the perceived congruence; the combination of a rising image and an ascending pitch, and that of a falling image and a descending pitch, created higher congruence than the alternative combinations. The combination of an enlarging image and an ascending pitch, and that of a reducing image and a descending pitch, also created higher congruence. Furthermore, the combination of a movement sliding from left to right and an ascending pitch, and that of a movement sliding from right to left and a descending pitch, created higher congruence. When the visual transformation pattern was a complex of these three kinds of movements, the vertical direction transformation was the most effective factor in determining congruent pitch pattern. The next most effective transformation factor was the enlarging and reducing image. The transformation of right and left direction movement was not strongly effective in determining the perceived congruence between complex visual movements and pitch patterns.

People obtain a lot of information from visual and auditory sensation on daily life. Regarding the effect of visual stimuli on perception of auditory stimuli, studies of phonological perception and sound localization have been made in great numbers. This study examined the effect of visual stimuli on perception in loudness and pitch of auditory stimuli. We used the image of figures whose size or brightness was changed as visual stimuli, and the sound of pure tone whose loudness or pitch was changed as auditory stimuli. Those visual and auditory stimuli were combined independently to make four types of audio-visual multisensory stimuli for psychophysical experiments. In the experiments, participants judged change in loudness or pitch of auditory stimuli, while they judged the direction of size change or the kind of a presented figure in visual stimuli. Therefore they cannot neglect visual stimuli while they judged auditory stimuli. As a result, perception in loudness and pitch were promoted significantly around their difference limen, when the image was getting bigger or brighter, compared with the case in which the image had no changes. This indicates that perception in loudness and pitch were affected by change in size and brightness of visual stimuli.

Although, multisensory signals often occur in noise in natural settings, few studies have compared the effect of noise on multisensory enhancement over unisensory stimulation. Twenty-two young adults completed an audiovisual discrimination task in which targets were presented in quiet and in auditory, visual, and audiovisual noise conditions. Stimuli comprised auditory, visual, and audiovisual domestic animal exemplars. Motor reaction times (MRTs) and error rates (< 7% for all stimulus types) were recorded. Multisensory enhancement was evident in all noise conditions. MRTs to audiovisual targets were slowed in visual and audiovisual noise compared to quiet. Percentage gain in audiovisual over unisensory MRTs was higher in visual and audiovisual noise than in auditory noise alone. However, there were fewer violations of the predicted inequality in the race model in audiovisual (.15–.25 probabilities) compared to visual (.15–.65 probabilities) and auditory noise and quiet (both .05–.65 probabilities). Increased variability could not explain reduced coactivation in audiovisual noise, suggesting that targets are processed differently in multisensory than unisensory noise or quiet. The multisensory advantage in visual and audiovisual noise can be explained by a greater increase in reaction times to unisensory than audiovisual targets. The findings of the study suggest an increasing interaction between multisensory integration and crossmodal attention in noise with a high visuoperceptual load.

In localizing simultaneous auditory and visual events, the brain should map the audiovisual events onto a unified perceptual space in a subsequent spatial process for integrating and/or comparing multisensory information. However, there is little qualitative and quantitative psychological data for estimating multisensory localization in peripheral visual fields. We measured the relative perceptual direction of a sound to a flash when they were simultaneously presented in peripheral visual fields. The results demonstrated that the sound and flash were perceptually located at the same position when the sound was presented in 5 deg-periphery from the flash. This phenomenon occurred even excluding the trial in which the participants' eyes moved. The measurement of the location of each sound and flash in a pointing task showed that the perceptual location of the sound shifted toward the frontal direction and conversely the perceptual location of the flash shifted toward the periphery. Our findings suggest that unisensory perceptual spaces of audition and vision have deviations in peripheral visual fields and, when the brain remaps unisensory locations of auditory and visual events into unified perceptual space, the unisensory spatial information of the events can be suitably maintained.

According to recent ERP (event-related potentials) studies, the visual speech facilitates the neural processing of auditory speech for speakers of European languages in audiovisual speech perception. We examined whether this visual facilitation is also the case for Japanese speakers for whom the weaker susceptibility of the visual influence has been behaviorally reported. We conducted a cross-linguistic experiment comparing ERPs of Japanese and English language groups (JL and EL) when they were presented with audiovisual congruent as well as audio-only speech stimuli. The temporal facilitation by the additional visual speech was observed only for native speech stimuli, suggesting a role of articulating experiences for early ERP components. For native stimuli, the EL showed sustained visual facilitation for about 300 ms from audio onset. On the other hand, the visual facilitation was limited to the first 100 ms for the JL, and they rather showed a visual inhibitory effect at 300 ms from the audio onset. Thus the type of native language affects neural processing of visual speech in audiovisual speech perception. This inhibition is consistent with behaviorally reported weaker visual influence for the JL.

In the best-known example of the McGurk effect, an auditory consonant /b/ that is presented with a face articulating /g/ is heard as a fusion /d/. However, sometimes this kind of stimulus is heard as /g/, ie, a visual-dominant percept. We explored the stimulus features giving rise to these percepts by using two different stimuli at various levels of acoustic noise. The stimulus auditory /apa/ presented with visual /aka/ was most often heard as /aka/ even without noise, and the proportion of visual-dominant percepts increased with noise level. The stimulus auditory/epe/ presented with visual /eke/ was heard mostly as a fusion /ete/, except at the highest noise level, where also /eke/ was heard. The differences in the quality of the McGurk effect were accounted for by the features of the unisensory stimuli. A phonetic analysis showed that the auditory and visual stimulus features were close to those of /t/ in the /e/-context, but not in the /a/-context stimuli. Thus, the type of the McGurk effect—fusion or visual dominance— depended on the quality of the constituent stimulus components, obeying a modality-precision rule: the more reliable the unisensory stimulus, the greater its contribution to the final percept.

Spatial ventriloquism refers to a shift of perceptual location of a sound toward a synchronized visual stimulus. It has been assumed to reflect early processes uninfluenced by cognitive factors such as syllable congruency between audio-visual speech stimuli. Conventional experiments have examined compelling situations which typically entail pairs of single audio and visual stimuli to be bound. However, for natural environments our multisensory system is designed to select relevant sensory signals to be bound among adjacent stimuli. This selection process may depend upon higher (cognitive) mechanisms. We investigated whether a cognitive factor affects the size of the ventriloquism when an additional visual stimulus is presented with a conventional audio-visual pair. Participants were presented with a set of audio-visual speech stimuli, comprising one or two bilateral movies of a person uttering single syllables together with recordings of this person speaking the same syllables. One of movies and the speech sound were combined in either congruent or incongruent ways. Participants had to identify sound locations. Results show that syllable congruency affected the size of the ventriloquism only when two movies were presented simultaneously. The selection of a relevant stimulus pair among two or more candidates can be regulated by some higher processes.

We examined whether the McGurk effect depends on context. A syllable that is expected to produce the effect was embedded in Japanese simple sentences containing a three-syllable noun. The noun was either a real word or a non-word, including /ba/, /da/, or /ga/ as the second syllable. A stimulus consisted of an auditory speech sentence with a simultaneous video of its production by a speaker. Participants were asked to report the noun as they heard it by filling in the blanks in printed sentences. The error rates in reporting the nouns were low when only the audio stimuli were presented. In the critical conditions, a voice and a video were congruently or incongruently combined and shown to participants, who judged the noun word by hearing. McGurk effect occurred in both real- and non-word conditions, but the error rates were larger for the real words. For the real words, the error rate was higher when the visual syllable matched the context. The rate of this context-matched errors was higher than the rate of comparable errors for non-words. This finding suggests that audio-visual integration in speech recognition is influenced by higher cognitive processes that are sensitive to the semantic context.

Previous studies suggested that attention modulates how well we integrate sounds with visual cues. In the present study, we examined attention effects on integration of written and heard syllables in fluent readers. Subjects were presented with auditory stimuli (consonant-vowel syllables) together with synchronized visual stimuli, which differed between conditions. The auditory stimuli included consonant and pitch changes. Visual stimuli were sound-congruent written syllables or their scrambled images. In four different attention conditions, subjects pressed a button to duration changes in 1) the visual modality, 2) auditory modality or 3) either modality, or 4) counted mentally backwards and ignored the auditory and visual stimuli. Preliminary analysis showed in event-related potentials to spoken syllables larger mismatch negativities (MMNs) to pitch changes during audiovisual, auditory, and visual tasks than during mental counting indicating that attention to either one or both modalities facilitated MMN elicitation, that is, even during the visual task, attention spread covertly to coinciding speech sounds, integrating written syllables with their spoken counterparts. Additionally, these MMNs were larger in all four conditions when the visual stimuli were syllables than when they were scrambled images, indicating that written syllables facilitated processing of coinciding spoken syllables independent of attention.

The McGurk effect represents a perceptual illusion resulting from the integration of an auditory syllable dubbed onto an incongruous visual syllable. The involuntary and impenetrable nature of the illusion is frequently used to support the multisensory nature of audiovisual speech perception. Here we show that both self-speech and familiarized speech reduce the effect. When self-speech was separated into self-voice and self-face mismatched with different faces and voices, only self-voice weakened the illusion. Thus, a familiar vocal identity automatically confers a processing advantage to multisensory speech, while a familiar facial identity does not. When another group of participants were familiarized with the speakers, participants' ability to take advantage of that familiarization was inversely correlated with their overall susceptibility to the McGurk illusion.

We can discriminate face from non-face even in infancy, but it is not clear based on what kind of features we discriminate them. One hypothesis, termed CONSPEC, posits that infants possess innate information concerning the structure of faces, specifically, three blobs constituting the corners of an inverted triangle (Morton & Johnson, 1991). However, an alternative view, top-heavy bias, posits that the preference for faces can be explained by a general visual preference of more features on the upper part of an object (Simion et al, 2001). In this experiment, four blobs constituting a virtual diamond were presented with and without a beep, and the looking time to each blob was measured. The ratio of looking time to blobs were not significantly different between beep-present/-absent conditions, except for the blob at the lowest position, where the ratio of looking time increased with a beep. This result suggests that the lowest blob is regarded as the mouth, favoring CONSPEC and against top-heavy bias hypothesis. Our result also indicates the importance of auditory information for the representation of face, although facial representation is usually discussed in terms of visual perception.

Recent technology provides us with realistic looking virtual characters. Motion capture and elaborate mathematical models supply data for natural looking, controllable facial and bodily animations. With the help of computational linguistics and artificial intelligence, we can automatically assign emotional categories to appropriate stretches of text for a simulation of those social scenarios where verbal communication is important. All this makes virtual characters a valuable tool for creation of versatile stimuli for research on the integration of emotion information from different modalities. We conducted an audio-visual experiment to investigate the differential contributions of emotional speech and facial expressions on emotion identification. We used recorded and synthesized speech as well as dynamic virtual faces, all enhanced for seven emotional categories. The participants were asked to recognize the prevalent emotion of paired faces and audio. Results showed that when the voice was recorded, the vocalized emotion influenced participants' emotion identification more than the facial expression. However, when the voice was synthesized, facial expression influenced participants' emotion identification more than vocalized emotion. Additionally, individuals did worse on identifying either the facial expression or vocalized emotion when the voice was synthesized. Our experimental method can help to determine how to improve synthesized emotional speech.

This study aims to further examine the cross-cultural differences in multisensory emotion perception between Western and East Asian people. In this study, we recorded the audiovisual stimulus video of Japanese and Dutch actors saying neutral phrase with one of the basic emotions. Then we conducted a validation experiment of the stimuli. In the first part (facial expression), participants watched a silent video of actors and judged what kind of emotion the actor is expressing by choosing among 6 options (ie, happiness, anger, disgust, sadness, surprise, and fear). In the second part (vocal expression), they listened to the audio part of the same videos without video images while the task was the same. We analyzed their categorization responses based on accuracy and confusion matrix and created a controlled audiovisual stimulus set.

Earlier research has shown that, in a dot-probe experiment, simultaneously presented vocal utterances (one with emotional prosody, one neutral) showed faster responses to probes replacing the location of emotional prosody, indicating a cross-modal modulation of attention (Brosch et al, 2008). We designed a multimodal dot-probe experiment in which: (a) fearful and neutral face pairs were simultaneously accompanied by fearful and neutral paired vocalisations or (b) the fearful and neutral vocalisations without face pictures preceded a visual target probe. A unimodal visual block was run as a control. In addition to the expected visual-only effect, we found spatial attentional bias towards fearful vocalisations followed by a visual probe, replicating the crossmodal modulation of attention shown in the Brosch et al experiment. However, no such effects were found for audiovisual face-voice pairs. This absence of an audiovisual effect with simultaneous face-voice presentation might be the consequence of the double-conflict situation; the fuzziness of two competing stimulus presentations might have ruled out or cancelled potential attentional biases towards fearful auditory and visual information.

Anxious individuals have been shown to interpret others' facial expressions negatively. However, whether this negative interpretation bias depends on the modality and familiarity of emotional cues remains largely unknown. We examined whether trait-anxiety affects recognition of multisensory emotional cues (ie, face and voice), which were expressed by actors from either the same or different cultural background as the participants (ie, familiar in-group and unfamiliar out-group). The dynamic face and voice cues of the same actors were synchronized, and conveyed either congruent (eg, happy face and voice) or incongruent emotions (eg, happy face and angry voice). Participants were to indicate the perceived emotion in one of the cues, while ignoring the other. The results showed that when recognizing emotions of in-group actors, highly anxious individuals, compared with low anxious ones, were more likely to interpret others' emotions in a negative manner, putting more weight on the to-be-ignored angry cues. This interpretation bias was found regardless of the cue modality. However, when recognizing emotions of out-group actors, low and high anxious individuals showed no difference in the interpretation of emotions irrespective of modality. These results suggest that trait-anxiety affects recognition of emotional expressions in a modality independent yet cultural familiarity dependent manner.

In this study, we examined the effects of the temporal frequency of the audio–visual stimuli on the impression formation and perception of the temporal congruency. In our previous study (2010) with two temporal frequency conditions (108, and 216bpm), we found that both the processing which underlies the impression formation of the temporal congruency and the processing which underlies the perceptual judgment depend not only upon the time lag between the audio and visual stimuli but also upon the temporal frequency of the stimuli. As visual stimulus, we used repetitive luminance change of the computer graphic image with a constant temporal frequency (87, 108, 146, or 216bpm). As audio stimulus, we used periodic drum sounds consisting of a low tam and three cymbals. There were nine conditions for asynchrony between the visual and audio stimuli (±0, 96, 192, 288, and 384ms). We found that the range of the audio-visual asynchrony for the temporal congruent impression is much narrower than the range for the temporal congruent perception. These results suggest that the perception of the temporal congruency does not determine the impression formation for the temporal congruency.

We examined how the denotative congruency (Masakura & Ichikawa, 2003) in audio-visual stimuli affects the pleasant and restful impressions for the stimuli. In Experiment 1, a single sound was combined with a single motion picture. Results showed that the pleasant and restful impressions for the combined audio-visual stimuli determined by averaging the impressions obtained from each of audio and visual stimuli. In Experiment 2, two sounds were combined with a single motion picture. Results showed that the pleasant and restful impressions positively shifted when one of the sounds was combined with a denotatively congruent motion picture, even if the unpleasant and unrestful sound was combined with unpleasant and unrestful motion picture. These results suggested that the denotative congruency between the audio and visual stimuli would exaggerate pleasant and restful impressions. Reduction of stimulus complexity is expected to lead to exaggeration of pleasant impression (Berlyne, 1970). We are proposing that the reduction of the stimulus complexity underlie the exaggerating effect of denotative congruency on the pleasant and restful impressions in observing the audio-visual stimuli.

The McGurk effect is a perceptual phenomenon that demonstrates interaction between hearing and vision in speech perception. This effect may be experienced when a visual of the production of a phoneme is dubbed with a sound recording of a different phoneme being spoken wherein the perceived phoneme is often the third intermediate phoneme. In the present study we examined the potential individual differences in the McGurk effect among 51 healthy students. The results suggested that people with higher scores for schizophrenic or autistic traits, respectively, might report less /ka/ responses (visually captured responses) but more /ta/ responses (vision-audio mixed responses) in the McGurk condition (visually /pa/ but auditor/ka/). This indicates that such people might show a preference for auditory information over visual information. Both schizophrenia and autism might have deficits in social functioning. If an individual has a poor level of interpersonal skills this would reflect in the result since he/she might not be able to read others' lips automatically.

Emotions are central to our perception of the environment surrounding us (Berlyne, 1971). An important aspect in the emotional response to a sound is dependent on the meaning of the sound, ie, it is not the physical parameter per se that determines our emotional response to the sound but rather the source of the sound (Genell, 2008), and the relevance it has to the self (Tajadura-Jiménez et al 2010). When exposed to sound together with visual information, the information from both modalities is integrated, altering the perception of each modality, in order to generate a coherent experience. In emotional information this integration is rapid and without requirements of attentional processes (De Gelder, 1999). The present experiment investigates perception of pink noise in two visual settings in a within-subjects design. Nineteen participants rated the same sound twice in terms of pleasantness and arousal in either a pleasant or an unpleasant visual setting. The results showed that pleasantness of the sound decreased in the negative visual setting, thus suggesting an audio-visual integration, where the affective information in the visual modality is translated to the auditory modality when information-markers are lacking in it. The results are discussed in relation to theories of emotion perception.

Effects of facial expressions on recognizing emotions expressed in dance movements were investigated. Dancers expressed three emotions: joy, sadness, and anger through dance movements. We used digital video cameras and a 3D motion capturing system to record and capture the movements. We then created full-video displays with an expressive face, full-video displays with an unexpressive face, stick figure displays (no face), or point-light displays (no face) from these data using 3D animation software. To make point-light displays, 13 markers were attached to the body of each dancer. We examined how accurately observers were able to identify the expression that the dancers intended to create through their dance movements. Dance experienced and inexperienced observers participated in the experiment. They watched the movements and rated the compatibility of each emotion with each movement on a 5-point Likert scale. The results indicated that both experienced and inexperienced observers could identify all the emotions that dancers intended to express. Identification scores for dance movements with an expressive face were higher than for other expressions. This finding indicates that facial expressions affect the identification of emotions in dance movements, whereas only bodily expressions provide sufficient information to recognize emotions.

The present study examined the impact of implicit mood manipulation on the scope of visual processing. Previous studies have shown that the scope expands when participants explicitly receive positive mood manipulation. Thus it is unclear whether the same principle applies to the case in which participants' mood is induced implicitly. We adopted an implicit mood manipulation in which participants held a pen in their teeth so that the muscles associated with smiling are activated without explicitly requiring to pose in a smiling face (Strack, Martin, & Stepper, 1988). Under the control condition, participants held the pen with lips to inhibit those muscles. Before and after the pen-holding, participants viewed bi-stable figures (eg, Rubin Vase) and reported the timings which their percept (narrow or broad perspective) flipped. The result indicated successful mood induction: the teeth group rated comic-stimuli funnier than the control group. None of the participants were unaware of the purpose of the manipulation. Most importantly, the dominance of the broad percepts of the bi-stable figures was greater under the teeth condition. No such effect was found under the lip condition. These results suggest that the positive mood with implicit peripheral facial-muscle control broadens the scope of visual processing.

This study examined the relationships between the cognitive functions and mood of elders in an institution. Nine elders received the POMS (Profile of Mood Scale: Tense-Anxiety, Depression, Angry, Vigor, Fatigue, Confusion) to measure their mood and the Stroop test and the reverse-Stroop test to measure their cognitive functions. The Stroop interference shows that elders' responses are influenced by word when they name the ink-color of an interfered color-word combination. The reverse-Stroop interference shows that they are interfered by color when they name the word of an incongruent color-word combination. POMS scores were under the cut-off points, and the total correct response score related with Vigor (r=0.62, p=0.04). The Stroop interference score was at the standard level, but the reverse-Stroop interference score was lower than that. The Stroop interference score related with Vigor (r=−0.65, p=0.03); the inverse-Stroop interference score related with Tense-Anxiety (r=−0.70, p=0.02) and Confusion (r=−0.61, p=0.04). These results show that 1) the mood of elders in the institution were at a healthy level and elders with high vigor responded correctly, 2) they could read word with high level, 3)elders with high vigor treated color correctly without interference by word, and 4) elders with high tension and confusion treated with word without inference by color.

We have been engaged in the development of a brain-computer interface (BCI) based on the cognitive P300 event-related potentials (ERPs) evoked by simultaneous presentation of visual and auditory stimuli in order to assist with the communication in severe physical limitation persons. The purpose of the simultaneous presentation of these stimuli is to give the user more choices as commands. First, we extracted P300 ERPs by either visual oddball paradigm or auditory oddball paradigm. Then amplitude and latency of the P300 ERPs were measured. Second, visual and auditory stimuli were presented simultaneously, we measured the P300 ERPs varying the condition of combinations of these stimuli. In this report, we used 3 colors as visual stimuli and 3 types of MIDI sounds as auditory stimuli. Two types of simultaneous presentations were examined. The one was conducted with random combination. The other was called group stimulation, combining one color, such as red, and one MIDI sound, such as piano, in order to make a group; three groups were made. Each group was presented to users randomly. We evaluated the possibility of BCI using these stimuli from the amplitudes and the latencies of P300 ERPs.

We employed a novel cross-modal non-spatial inhibition of return (IOR) paradigm with fMRI to investigate whether object concept is organized by supramodal or modality-specific systems. A precue-neutral cue-target sequence was presented and participants were asked to discriminate whether the target was a dog or a cat. The precue and the target could be either a picture or vocalization of a dog or a cat. The neutral cue (bird) was always from the same modality as the precue. Behaviorally, for both visual and auditory targets, the main effect of cue validity was the only significant effect, p<0.01, with equivalent effects for within- and cross-modal IOR. Neurally, for visual targets, left inferior frontal gyrus and left medial temporal gyrus showed significantly higher neural activity in cued than uncued condition, irrespective of the precue-target relationship, indicating that the two areas are involved in inhibiting a supramodal representation of previously attended object concept. For auditory targets, left lateral occipital gyrus and right postcentral gyrus showed significantly higher neural activity in uncued than cued condition irrespective of the cue-target relationship, indicating that the two areas are involved in creating a new supramodal representation when a novel object concept appears.

Previous neuroimaging studies have reported that learning multisensory associations involves the superior temporal regions (Tanabe et al, 2005). However, the neural mechanisms underlying the retrieval of multi-sensory associations were unclear. This functional MRI (fMRI) study investigated brain activations during the retrieval of multi-sensory associations. Eighteen right-handed college-aged Japanese participants learned associations between meaningless pictures and words (Vw), meaningless sounds and words (Aw), and meaningless sounds and visual words (W). During fMRI scanning, participants were presented with old and new words and were required to judge whether the words were included in the conditions of Vw, Aw, W or New. We found that the left superior temporal region showed greater activity during the retrieval of words learned in Aw than in Vw, whereas no region showed greater activity for the Vw condition versus the Aw condition (k > 10, p < .001, uncorrected). Taken together, the left superior temporal region could play an essential role in the retrieval process of auditory-word associations.

Phonetic symbolism is the phenomenon of speech sounds evoking images based on sensory experiences; it is often discussed with cross-modal correspondence. By using Garner's task, Hirata, Kita, and Ukita (2009) showed the cross-modal congruence between brightness and voiced/voiceless consonants in Japanese speech sound, which is known as phonetic symbolism. In the present study, we examined the effect of the meaning of mimetics (lexical words whose sound reflects its meaning, like “ding-dong”) in Japanese language on the cross-modal correspondence. We conducted an experiment with Chinese speech sounds with or without aspiration using Chinese people. Chinese vocabulary also contains mimetics but the existence of aspiration doesn't relate to the meaning of Chinese mimetics. As a result, Chinese speech sounds with aspiration, which resemble voiceless consonants, were matched with white color, whereas those without aspiration were matched with black. This result is identical to its pattern in Japanese people and consequently suggests that cross-modal correspondence occurs without the effect of the meaning of mimetics. The problem that whether these cross-modal correspondences are purely based on physical properties of speech sound or affected from phonetic properties remains for further study.

Realization of high-definition multimodal displays is keenly required for the advancement of information and communications technologies. As an index of high-definition display systems, the sense of presence has been widely investigated. Both theoretically and empirically such sense has been found to relate more dominantly to background components contained in a scene. In contrast, the appreciative role of foreground components in multimodal contents has not been investigated in detail. Therefore, we have been focusing on the sense of verisimilitude as another index. We recently studied how the sense of verisimilitude and the sense of presence were affected by temporal asynchrony between foreground audio-visual components of a Japanese garden and suggested that the sense of verisimilitude has significantly different characteristics from the sense of presence. To investigate whether this result would be valid more generally, we conducted an experiment using other audio-visual content, namely, a clip of western orchestral music. Results showed the sense of verisimilitude is more sensitive to audiovisual synchronicity than to display size, while the sense of presence is more sensitive to spatial size than the temporal property. Hence, the sense of verisimilitude can be another useful index, distinguishable from the sense of presence.

Perception in natural environments is inseparably linked to motor action. In fact, we consider action an essential component of perceptual representation. But these representations are inherently difficult to investigate: Traditional experimental setups are limited by the lack of flexibility in manipulating spatial features. To overcome these problems, virtual reality (VR) experiments seem to be a feasible alternative, but these setups typically lack ecological realism due to the use of “unnatural” interface-devices (joystick). Thus, we propose an experimental apparatus which combines multisensory perception and action in an ecologically realistic way. The basis is a 10-foot hollow sphere (VirtuSphere) placed on a platform that allows free rotation. A subject inside can walk in any direction for any distance immersed into virtual environment. Both the rotation of the sphere and movement of the subject's head are tracked to process the subject's view within the VR-environment presented on a head-mounted display. Moreover, auditory features are dynamically processed taking greatest care of exact alignment of sound-sources and visual objects using ambisonic-encoded audio processed by a HRTF-filterbank. We present empirical data that confirm ecological realism of this setup and discuss its suitability for multi-sensory-motor research.

A novel video conference system is developed. Suppose that three people A, B, and C attend the video conference, the proposed system enables eye contact among every pair. Furthermore, when B and C chat, A feels as if B and C were facing each other (eye contact seems to be kept among B and C). In the case of a triangle video conference, the respective video system is composed of a half mirror, two video cameras, and two monitors. Each participant watches other participants' images that are reflected by the half mirror. Cameras are set behind the half mirror. Since participants' image (face) and the camera position are adjusted to be the same direction, eye contact is kept and conversation becomes very natural compared with conventional video conference systems where participants' eyes do not point to the other participant. When 3 participants sit at the vertex of an equilateral triangle, eyes can be kept even for the situation mentioned above (eye contact between B and C from the aspect of A). Eye contact can be kept not only for 2 or 3 participants but also any number of participants as far as they sit at the vertex of a regular polygon.

Previous studies have shown that a similar straightforward relationship exists between lightness and pitch in synaesthetes and nonsynaesthetes (Mark, 1974; Hubbard, 1996; Ward et al., 2006). These results indicated that nonsynaesthetes have similar lightness-pitch mapping with synaesthetes. However, in their experimental paradigm such a similarity seems to stem from not so much sensory phenomena as high-level memory associations in nonsynaesthetes. So, we examined what process in perception and/or cognition relates lightness-pitch synaesthesia-like phenomena in nonsynaethetes. In our study we targeted perceived colors (luminance) per se rather than the imagery of color (luminance) via memory association. Results indicated that the performance in color selection were affected by task-irrelevant stimuli (auditory information), but there was little significant correlation between color and auditory stimuli (pitch) in simple color conditions. However, in subjective figures conditions, results showed a different tendency and partly showed correlations. Recent work indicates synaesthesia needs selective attention (Rich et al., 2010) and some research shows perception of subjective contours need attention (Gurnsey, 1996). We conjecture that lightness-pitch synaesthesia-like phenomena may need some kind of attention or the other higher brain activity (eg, memory association, cognition).

Several recent studies have examined an interaction between talker's acoustic characteristics and spoken word recognition in speech perception and have shown that listener's familiarity about a talker influences an easiness of spoken word processing. The present study examined the effect of listener's familiarity about talkers on the free recall task of words spoken by two talkers. Subjects participated in three conditions of the task: the listener has (1) explicit knowledge, (2) implicit knowledge, and (3) no knowledge of the talker. In condition (1), subjects were familiar with talker's voices and were initially informed whose voices they would hear. In condition (2), subjects were familiar with talkers' voices but were not informed whose voices they would hear. In condition (3), subjects were entirely unfamiliar with talker's voices and were not informed whose voices they would hear. We analyzed the percentage of correct answers and compared these results across three conditions. We will discuss the possibility of whether a listener's knowledge about the individual talker's acoustic characteristics stored in long term memory could reduce the quantity of the cognitive resources required in the verbal information processing.

Change blindness studies have revealed people's inability to detect changes between two consecutively presented scenes when they are separated by a distractor. This failure has been reported within vision, audition, and touch but also crossmodally. In particular, participants' performance in detecting changes in position between two tactile scenes is impaired not only when a tactile mask is introduced between the two to-be-compared displays but also when a visual mask is used instead. Interestingly, with similar procedure, there is no effect of auditory masks on a tactile task or of tactile masks on a visual task (Auvray et al., 2007, 2008; Gallace et al., 2006). Such crossmodal change blindness effect also occurs when participants perform a different task. In a recent experiment, participants had to detect changes in the frequency of presentation of tactile stimuli. The two to-be-compared sequences of three tactile stimuli were presented either at a same or at a different rate. They were presented either consecutively, separated by an empty interval, a tactile, visual, or auditory mask. The visual and tactile masks significantly impaired participants' performance whereas the auditory mask had no effect on performance. These findings are discussed in relation to the crossmodal nature of attention.

Perceptual segregation in rapidly changing visual displays can be facilitated by a synchronized salient sound that segregates itself from other sounds in the sequence (Vroomen & de Gelder, 2000). We examined whether this “freezing” phenomenon can also be found in tactile perception. Three vibrators were placed on the participant's palm to produce four different tactile patterns. Four sounds were presented separately and simultaneously with each of the four tactile patterns. Among the three same-pitch tones, an abrupt high-pitch tone was presented simultaneously with the designated temporal position of the target pattern in the sequence of tactual stimuli that was presented rapidly and repeatedly. The task was to identify the tactile pattern of the target. Results showed that participants responded faster and more accurately with the high-pitch tone, compared to the condition when all the tones were of the same pitch. However, the result reversed when an extra tactile cue was presented on the wrist. This suggests that a salient auditory signal can improve perceptual segregation not only in vision but also in touch. That is, it is a cross-modal facilitation, not an alerting or attentional cueing effect.

In recent work Thomas et al (under review) examined the subjective effects of action observation on sensory perception. They found that when we watch the actions of an actor, subjective sensory experience is enhanced at the equivalent location on the subject's body. In the present study we looked at whether this sensory transfer might also occur for more specific sensations such as roughness/unpleasantness and softness/pleasantness. We found that when participants watched another person being stroked with a soft/pleasant object (eg, velvet), their sensation of pleasantness was increased when they were simultaneously stimulated with more pleasant (ie,, higher frequency) stimuli on the corresponding area of their own body. We concluded that seeing a person being touched with stimuli perceived as pleasant or unpleasant amplifies judgements about stimuli when they are presented to the same body location on another person, and therefore suggests a cross modal interaction between vision and touch. This mechanism might allow transfer of affective dimensions of touch, a mechanism we suggest could impact on the nature of advertising and influence how organisations produce and market their products.

Some researches revealed the relationship between multisensory integration and EEG oscillations. Previous studies revealed that the visuotactile integration process could be explained by gamma and theta band oscillation. In addition, recent studies have showed the possibility that a coupling between oscillations at the different frequency bands plays an important role on the multisensory integration system. This study aimed to investigate whether the gamma and theta oscillations show the coupling during the visuotactile integration. Using congruency effect paradigm only for left hand, we measured scalp EEG during simultaneous presentation of “spatially congruent” or “spatially incongruent” visuotactile stimuli. In Experiment 1, the proportion of the spatially congruent trials (80% vs 20%) was changed across the experimental blocks. The results showed that the relationship between gamma power and theta phase at the parietal area was modulated by the proportion. In Experiment 2, the saliency of the vibration stimulus (0dB vs −20dB) was changed across trials. The results showed that the relationship between gamma power and theta phase was immune to the saliency. These results suggest that multisensory integration process has a plasticity, which is modulated by the proportion of congruent trial, and the process could be explained by the coupling between gamma/theta oscillations.

We investigated a technique that can be used to place tactile stimuli on a continuum between two points using two tactors placed several centimeters apart. As von Békésy first described in 1959, when two pieces of skin several centimeters apart are simultaneously stimulated, a single touch is perceived at an intermediate location. When the relative intensity of the vibration of the two tactors is slowly adjusted from 1:0 to 0:1 a sensation is created of a touch moving across the skin. This has come to be known as the funneling illusion. We have characterized the funneling illusion on a variety of body parts including the hand, arm, wrist, head, torso, and legs. We show that the illusion is robust to a variety of vibration parameters, and demonstrate how it may be used for within modality touch discrimination and adjustment procedures.

To investigate flexible adaptation of visual system, we developed a system to provide a user an artificial oculomotor function to control directions of both eyes. The system named “Virtual Chameleon” consists of two CCD cameras independently controlled and a head-mounted display. The user can control each tracking directions of two cameras with sensors set to both hands so that the user can get independent arbitrary view fields for both eyes. We performed fundamental experiments to evaluate capability to evaluate adaptation to use of Virtual Chameleon and effects on the user's capabilities. Eleven healthy volunteers with normal and corrected-to-normal vision participated in the experiments. The experiments were tests to find out each position of targets put in both side of a subject. In the experiments, a condition using Virtual Chameleon and a condition without it was adopted. We obtained accuracy rates and time intervals to find out target positions as experimental results. The experiments showed all of volunteers became able to actively control independent visual axes and correctly understood two different views by using Virtual Chameleon, even though two independent view fields yielded binocular rivalry to volunteers and binocular rivalry reduced human capabilities compared to cases without Virtual Chameleon.

We are able to make a saccade toward a tactile stimuli to one hand, but trajectories of many saccades curved markedly when the arms were crossed (Groh & Sparks, 2006). However, it remains unknown why some curved and others did not. We therefore examined critical factors for inducing the curved somatosensory saccades. Participants made a saccade as soon as possible from a central fixation point toward a tactile stimulus delivered to one of the two hands, and switched between arms-crossed and arms-uncrossed postures every 6 trials. Trajectories were generally straight when the arms were uncrossed, but all participants made curved saccades when the arms were crossed (12–64%). We found that the probability of curved saccades depended critically on the onset latency: the probability was less than 5% when the latency was larger than 250 ms, but the probability increased up to 70–80% when the onset latency was 160 ms. This relationship was shared across participants. The results suggest that a touch in the arms-crossed posture was always mapped to the wrong hand in the initial phase up to 160 ms, and then remapped to the correct hand during the next 100 ms by some fundamental neural mechanisms shared across participants.

The spatial modulation effect has been reported in somatosensory spatial judgments when the task-irrelevant auditory stimuli are given from the opposite direction. Two experiments examined how the spatial modulation effect on somatosensory spatial judgments is altered in various body regions and their spatial locations. In experiment 1, air-puffs were presented randomly to either the left or right cheeks, hands (palm versus back) and knees while auditory stimuli were presented from just behind ear on either the same or opposite side. In experiment 2, air-puffs were presented to hands which were aside of cheeks or placed on the knees. The participants were instructed to make speeded discrimination responses regarding the side (left versus right) of the somatosensory targets by using two footpedals. In all conditions, reaction times significantry increased when the irrelevant stimuli were presented from the opposite side rather than from the same side. We found that the back of the hands were more influenced by incongruent auditory stimuli than cheeks, knees and palms, and that the hands were more influenced by incongruent auditory stimuli when placed at the side of cheeks than on the knees. These results indicate that the auditory-somatosensory interaction differs in various body regions and their spatial locations.

Sounds often accompany tactile stimulation. The present study investigated if the temporal coherent auditory and tactile stimuli can be grouped together to form a perceptual stream. The experiments measured listeners' sensitivity to detect synchrony between a pair of auditory or tactile stimuli (in a 2I-2AFC task), either in isolation or embedded in a sequence of stimuli in the same modality (ie, ‘captors’) that was designed to perceptually ‘capture’ the target stimulus into a sequential stream. In some conditions, the captors were presented together with stimuli from the same or different modality. Results show that these accompanying tones did improve auditory synchrony detection when the captors were presented, but the accompanying taps did not, no matter whether the test tone pairs were separated by 6 or 11 semitones. On the other hand, the accompanying tones did not improve tactile synchrony detection when the captors were presented, no matter whether the test tap pairs were on the same hand or different hands. The observations did not find significant audio-tactile interactions in streaming at or below the level of synchrony detection across auditory filters and across hands. Nevertheless, the interactions may occur at higher-level processing of perceptual organization.

An interaction between orofacial somatosensation and the perception of speech was demonstrated in recent psychophysical studies (Ito et al. 2009; Ito and Ostry 2009). To explore further the neural mechanisms of the speech-related somatosensory-auditory interaction, we assessed to what extent multisensory evoked potentials reflect multisensory interaction during speech perception. We also examined the dynamic modulation of multisensory integration resulting from relative timing differences between the onsets of the two sensory stimuli. We recorded event-related potentials from 64 scalp sites in response to somatosensory stimulation alone, auditory stimulation alone, and combined somatosensory and auditory stimulation. In the multisensory condition, the timing of the two stimuli was either simultaneous or offset by 90 ms (lead and lag). We found evidence of multisensory interaction with the amplitude of the multisensory evoked potential reliably different than the sum of the two unisensory potentials around the first peak of multisensory response (100–200 ms). The magnitude of the evoked potential difference varied as a function of the relative timing between the stimuli in the interval from 170 to 200 ms following somatosensory stimulation. The results demonstrate clear multisensory convergence and suggest a dynamic modulation of multisensory interaction during speech.

We have been engaged in the development of a new support device for subjects whose independent daily activities are limited due to severe progressive neuromuscular disease. In this study, we examined the possibility of developing a P300-type BCI (brain–computer interface) using orientation discrimination of stimulus sound with auditory lateralisation using a virtual sound source for air-conducted sound presented to the subject bitemporally. Sound stimulation with sound localization used in the study had a level difference of approximately 10 dB and time difference of approximately 0.6 ms, and was fixed at 60° to the left and to the right from the median. It was suggested that a virtual sound source with sound image fixed in left and right directions with added difference between the left and right ears can be used for development of a P300-type BCI by applying auditory stimulation. To further proceed to the development of a P300-type BCI system using auditory stimulation with comfort and convenience with the aim of reducing user burden in a clinical setting, we will reexamine the extraction algorithm for the target waveform, and examine the effects of the interval between stimulus presentation to make this a daily life supporting system to realise a faster and more efficient multiple command sending mechanism.

Various studies have highlighted plasticity of the auditory system from visual stimuli, limiting the trained field of perception. The aim of the present study is to investigate auditory system adaptation using an audio-kinesthetic platform. Participants were placed in a Virtual Auditory Environment allowing the association of the physical position of a virtual sound source with an alternate set of acoustic spectral cues or Head-Related Transfer Function (HRTF) through the use of a tracked ball manipulated by the subject. This set-up has the advantage to be not being limited to the visual field while also offering a natural perception-action coupling through the constant awareness of one's hand position. Adaptation process to non-individualized HRTF was realized through a spatial search game application. A total of 25 subjects participated, consisting of subjects presented with modified cues using non-individualized HRTF and a control group using individual measured HRTFs to account for any learning effect due to the game itself. The training game lasted 12 minutes and was repeated over 3 consecutive days. Adaptation effects were measured with repeated localization tests. Results showed a significant performance improvement for vertical localization and a significant reduction in the front/back confusion rate after 3 sessions.

This study investigated how auditory space was represented during linear self-motion. Participants were passively transported forward or backward at constant accelerations by a robotic wheelchair. A short noise burst (30 ms) was presented during self-motion via a loudspeaker aligned parallel with the traveling direction. In Experiment 1, the participants judged in which direction (incoming or outgoing) the noise burst was presented (ie,, two alternative forced choice task). The auditory stimulus perceived to be aligned with the subjective coronal plane shifted in the traveling direction only during forward self-motion. The amount of shift increased with increasing acceleration. In Experiment 2, we examined the accuracy of sound localization during forward self-motion by employing a pointing method. Whereas the auditory stimulus located on the physical coronal plane was almost accurately perceived, those well located in the incoming space were perceived closer to the participants than the physical positions during forward self-motion. These results suggest that the representation of auditory space in the traveling direction was compressed during forward accelerations.

Although a sound position without head movement localized, front-back confusion frequently occurs. Moreover, sound localization accuracy, especially front-back confusion, can be dramatically improved by listener head movement. This clearly shows that in sound localization both static cues involved in the sound signal input to the two ears and dynamic cues caused by listener motion are used. However, there have been few studies concerning spatial hearing dynamic situations. In this study, therefore, listener detection thresholds of movement of a sound stimulus during a sound localization task with head rotation were measured. Participants were first trained to rotate their heads at an indicated speed. Then during a sound localization trial, they were instructed to rotate their heads the direction of a sound stimulus at the speed. As a 2AFC paradigm was used, in one of two successive trials, the sound position (azimuthal angle) slightly moved during the rotation. Participants were asked to judge in which trial the sound stimuli moved. Results revealed that detection thresholds were dynamically raised when participants rotated their heads. Moreover, this effect did not depend on the velocities. These findings may suggest that a process similar to saccadic suppression in vision exists in dynamic sound localization.

While many studies have shown that visual information affects perception in the other modalities, little is known about how auditory and haptic information affect visual perception. In this study, we investigated how auditory, haptic, or auditory and haptic stimulation affects visual perception. We used a behavioral task based on the subjects observing the phenomenon of two identical visual objects moving toward each other, overlapping and then continuing their original motion. Subjects may perceive the objects as either streaming each other or bouncing and reversing their direction of motion. With only visual motion stimulus, subjects usually report the objects as streaming, whereas if a sound or flash is played when the objects touch each other, subjects report the objects as bouncing (Bounce-Inducing Effect). In this study, “auditory stimulation”, “haptic stimulation” or “haptic and auditory stimulation” were presented at various times relative to the visual overlap of objects. Our result shows the bouncing rate when haptic and auditory stimulation were presented were the highest. This result suggests that the Bounce-Inducing Effect is enhanced by simultaneous modality presentation to visual motion. In the future, a neuroscience approach (eg, TMS, fMRI) may be required to elucidate the brain mechanism in this study.

Humans recognize other persons' motion and sometimes move our own body including arms, foots and face, imitating the other's motion. A previous study[1] reported that it was more natural for humans to imitate others' ipsilateral hand than their contralateral hand. Previous studies reported that the reaction times for identifying objects depend on perceivers' viewpoint, suggesting that human imitation can also be considered as the process of mental rotation. The purpose of this study is to investigate the spatiotemporal kinetic information in imitated action. We focused on the arm-movement of ‘darts’ action while people were moving their body after learning a specific model's darts action. All the motion data was obtained by the motion capture system. Right handed participants observed videos of a model and imitated the model with their right hand. Learned videos had following two types. The one was the original video direction, and the other was a horizontally-rotated video of the original clips. The data analysis reported here was focused on the time length of whole/part of each movement. The result suggests that the time of imitated others' contralateral hand is longer than that of ipsilateral hand, due to the process of mental rotation of their arm direction.

We ordinarily perceive our voice sound as occurring simultaneously with vocal production, but the sense of simultaneity in vocalization can be easily interrupted by delayed auditory feedback (DAF). DAF causes normal people to have difficulty speaking fluently but helps people with stuttering to improve speech fluency. However, the underlying temporal mechanism for integrating the motor production of voice and the auditory perception of vocal sound remains unclear. In this study, we investigated the temporal tuning mechanism integrating vocal sensory and voice sounds under DAF with an adaptation technique. Participants read some sentences with specific delay times of DAF (0, 30, 75, 120 ms) during three minutes to induce ‘Lag Adaptation’. After the adaptation, they then judged the simultaneity between motor sensation and vocal sound given feedback in producing simple voice but not speech. We found that speech production with lag adaptation induced a shift in simultaneity responses toward the adapted auditory delays. This indicates that the temporal tuning mechanism in vocalization can be temporally recalibrated after prolonged exposure to delayed vocal sounds. These findings suggest vocalization is finely tuned by the temporal recalibration mechanism, which acutely monitors the integration of temporal delays between motor sensation and vocal sound.

Two experiments were conducted to examine the relationship between the objective and subjective effects of disruption caused by auditory stimuli on daily cognitive tasks. Reading with/without proofreading task (for Experiment 1) and reading with/without map task (for Experiment 2) were used as cognitive activities. The speech and office noise were used as auditory stimuli for Experiment 1 and 2. The increase of recall error rate under each auditory condition was used as an objective index of disruption. Two subjective indexes, self estimated performance and self rated annoyance under each auditory stimulus were used as a subjective index to assess the perceived adverse effect of the auditory stimuli. The error rate increased significantly under the auditory stimuli conditions in the reading with proofreading task (Experiment 1) and reading with/without map tasks (Experiment 2). The error rate increased significantly in the reading with proofreading task only under the office noise condition. The results show that estimated performance was generally in agreement with the recall error rate pattern. On the other hand, the annoyance ratings did not match recall performance. The agreement and discrepancy of objective and subjective indexes should be investigated in future experiments.

Many studies indicated that factors such as attention and motion play a critical role in time perception. However, it is not clear how subjective time for an unexpected event will be changed, compared with that for an expected event. The present study investigated this question by using two kinds of stimuli, one of them is the low-frequency oddball as the unexpected event and the other is the high-frequency standard as the expected event. In all trials, the standard was a square in line drawing and the duration was fixed to 1000 ms, whereas the oddball was a circle and the duration was set to one of seven durations from 500ms to 1100ms. After the standard was presented successively 4 times to 8 times (6 times on average), the oddball was presented once. Therefore, one session consisted of 34 oddballs intermixed with the 204 standards. Participants were required to estimate the duration for each oddball by using numeric keypad based on the magnitude estimation method. The results showed that durations for oddballs were estimated longer than those for standards. These suggest that an unexpected event causes subjective expansion of time.

Though human sensations, such as the senses of hearing, sight, etc., are independent each other, the interference between two of them is sometimes observed, and is called cross-modal perception[1]. Hitherto we studied unimodal perception of visual sensation[2] and auditory sensation[3] respectively by differential geometry[4]. We interpreted the parallel alley and the distance alley as two geodesics under different conditions in a visual space, and depicted the trace of continuous vowel speech as the geodesics through phonemes on a vowel plane. In this work, cross-modal perception is similarly treated from the standpoint of non-Riemannian geometry, where each axis of a cross-modal sensory space represents unimodal sensation. The geometry allows us to treat asymmetric metric tensor and hence a non-Euclidean concept of anholonomic objects, representing unidirectional property of cross-modal perception. The McGurk effect in audiovisual perception[5] and ‘rubber hand’ illusion in visual tactile perception[6] can afford experimental evidence of torsion tensor. The origin of ‘bouncing balls’ illusion[7] is discussed from the standpoint of an audiovisual cross-modal sensory space in a qualitative manner.

The brain integrates information from multiple sensory systems for signal enhancement, and the mammalian superior colliculus (SC) is a popular model system in which many of the principles of this phenomenon have been described. Recent observations suggest that multisensory enhancement is not uniform throughout the response, but can speed physiological response latencies and commonly show “superadditive” (greater than the predicted sum) enhancements near response onset (“Initial Response Enhancement”). Closer examination of the temporal profiles of the responses suggests two underlying principles: incoming signals appear to be integrated as soon as they arrive at the target neuron, and most significant multisensory enhancements occur when one or both of the signals are near threshold. This analysis supports a simple neural network model in which time-varying sensory signals interact with each other and other random inputs to produce unisensory and multisensory responses that can account for a great deal of the extant observations. Supported by NIH Grants NS036916 and EY016716 and the Tab Williams Family Foundation.

Different cortical and subcortical structures present neurons able to integrate stimuli of different sensory modalities. Among the others, one of the most investigated integrative regions is the Superior Colliculus (SC), a midbrain structure whose aim is to guide attentive behaviour and motor responses toward external events. Despite the large amount of experimental data in the literature, the neural mechanisms underlying the SC response are not completely understood. Moreover, recent data indicate that multisensory integration ability is the result of maturation after birth, depending on sensory experience. Mathematical models and computer simulations can be of value to investigate and clarify these phenomena. In the last few years, several models have been implemented to shed light on these mechanisms and to gain a deeper comprehension of the SC capabilities. Here, a neural network model (Cuppini et al., 2010) is extensively discussed. The model considers visual-auditory interaction, and is able to reproduce and explain the main physiological features of multisensory integration in SC neurons, and their acquisition during postnatal life. To reproduce a neonatal condition, the model assumes that during early life: 1) cortical-SC synapses are present but not active; 2) in this phase, responses are driven by non-cortical inputs with very large receptive fields (RFs) and little spatial tuning; 3) a slight spatial preference for the visual inputs is present. Sensory experience is modeled by a “training phase” in which the network is repeatedly exposed to modality-specific and cross-modal stimuli at different locations. As results, Cortical-SC synapses are crafted during this period thanks to the Hebbian rules of potentiation and depression, RFs are reduced in size, and neurons exhibit integrative capabilities to cross-modal stimuli, such as multisensory enhancement, inverse effectiveness, and multisensory depression. The utility of the modelling approach relies on several aspects: i) By postulating plausible biological mechanisms to complement those that are already known, the model provides a basis for understanding how SC neurons are capable of engaging in this remarkable process. ii) The model generates testable predictions that can guide future experiments in order to validate, reject, or modify these main assumptions. iii) The model may help the interpretation of behavioural and psychophysical responses in terms of neural activity and synaptic connections.

The spatiotemporal window of integration has become a widely accepted concept in multisensory research: crossmodal information falling within this window is highly likely to be integrated, whereas information falling outside is not. Making explicit assumptions about the arrival time difference between peripheral sensory processing times triggered by a crossmodal stimulus set, we derive a decision rule that determines an optimal window width as a function of (i) the prior odds in favor of a common multisensory source, (ii) the likelihood of arrival time differences, and (iii) the payoff for making correct or wrong decisions. Empirical support for this approach will be presented.

The time-window-of-integration (TWIN) model (Diederich & Colonius, 2008) describes and predicts multisensory integration effects in (saccadic) reaction time as a function of the spatial, temporal, and intensity conditions of the stimulus set. Here we show that using identical stimulus conditions but different instructions about target/nontarget modality (redundant target vs. focused attention paradigm) TWIN makes different predictions that are supported empirically. Thus, TWIN is able to account for both bottom-up and tow-down effects.

We usually think of well-learned daily limb movements, such as reaching and walking, as being fully controlled by our action intentions. Typically, the sensory outcomes of our movements would be recognized as self-actions (as opposed to movements of the external world) due to an internal model prediction using ‘efference copy’ of our motor commands. Here we describe a series of exceptions to this rule, which can be observed in implicit motor processes. One is the ‘manual following response’ that is induced by visual motion applied during reaching movements of the arm. This implicit visuomotor response alters our perception of self-movement. Even though the manual following response is self-generated, subjects tend to perceive it as an externally generated movement. Another example is the clumsy body and leg movements that occur when we step onto a stationary escalator, and the associated odd sensation that subjects frequently report. Even though we completely understand the environmental state of the stopped escalator, we cannot avoid behaving clumsily, and feeling an odd sensation before adaptation. Experiments involving the systematic manipulation of visual information of an escalator and the surrounding visual motion further indicate that endogenous motor responses may be triggered by the expectation that the escalator is in fact moving. In addition, exogenous postural changes elicited by visual motion can induce a similarly odd sensation, suggesting that the implicit behavior itself is a key factor in the perception of movement.

Auditory input is essential for normal speech development and plays a key role in speech production throughout the life span. In traditional models, auditory input plays two critical roles: 1) establishing the acoustic correlates of speech sounds that serve, in part, as the targets of speech production, and 2) as a source of feedback about a talker's own speech outcomes. This talk will focus on both of these roles, describing a series of studies that examine the capacity of children and adults to adapt to real-time manipulations of auditory feedback during speech production. In one study, we examined sensory and motor adaptation to a manipulation of auditory feedback during production of the fricative “s”. In contrast to prior accounts, adaptive changes were observed not only in speech motor output but also in subjects' perception of the sound. In a second study, speech adaptation was examined following a period of auditory–perceptual training targeting the perception of vowels. The perceptual training was found to systematically improve subjects' motor adaptation response to altered auditory feedback during speech production. The results of both studies support the idea that perceptual and motor processes are tightly coupled in speech production learning, and that the degree and nature of this coupling may change with development.

We describe experiments that test the hypothesis that changes in somatosensory function accompany motor learning. We estimated psychophysical functions relating actual and perceived limb position before, and after two kinds of motor learning: directional motor learning (learning to reach in the presence of novel forces applied by a robot), and non-directional learning (learning to reach quickly and accurately to visual targets, without forces). Following force-field learning, sensed limb position shifted reliably in the direction of the applied force. No sensory change was observed when the robot passively moved the hand through the same trajectories as subjects produced during active learning. Perceptual shifts are reflected in subsequent movements: following learning, movements deviate from their pre-learning paths by an amount similar in magnitude and in the same direction as the perceptual shift. After non–directional motor learning in the absence of forces, we observed improvements in somatosensory acuity following learning. Acuity improvement was seen only in the region of the workspace explored during learning, and not in other locations. No acuity changes were observed when subjects were passively moved through limb trajectories produced during active learning. Taken together, our findings support the idea that sensory changes occur in parallel with changes to motor commands during motor learning, and that the type of sensory change observed depends on the characteristics of the motor task during learning.

The perceptual changes induced by motor learning are important in understanding the adaptive mechanisms and global functions of the human brain. In the present study, we document the neural substrates of this sensory plasticity by combining work on motor learning using a robotic manipulandum with resting-state fMRI measures of learning and psychophysical measures of perceptual function. We show that motor learning results in long-lasting changes to somatosensory areas of the brain. We have developed a new technique for incorporating behavioral measures into resting-state connectivity analyses. The method allows us to identify networks whose functional connectivity changes with learning and specifically to dissociate changes in connectivity that are related to motor learning from those that are related perceptual changes that occur in conjunction with learning. Using this technique we identify a new network in motor learning involving second somatosensory cortex, ventral premotor and supplementary motor cortex whose activation is specifically related to sensory changes that occur in association with learning. The sensory networks that are strengthened in motor learning are similar to those involved in perceptual learning and decision making, which suggests that the process of motor learning engages the perceptual learning network.

Categorization has traditionally been studied in the visual domain with only a few studies focusing on the abilities of the haptic system in object categorization. During the first years of development, however, touch and vision are closely coupled in the exploratory procedures used by the infant to gather information about objects. Here, we investigate how well shape information can be transferred between those two modalities in a categorization task. Our stimuli consisted of amoeba-like objects that were parametrically morphed in well-defined steps. Participants explored the objects in a categorization task either visually or haptically. Interestingly, both modalities led to similar categorization behavior suggesting that similar shape processing might occur in vision and haptics. Next, participants received training on specific categories in one of the two modalities. As would be expected, training increased performance in the trained modality; however, we also found significant transfer of training to the other, untrained modality after only relatively few training trials. Taken together, our results demonstrate that complex shape information can be transferred efficiently across the two modalities, which speaks in favor of multisensory, higher-level representations of shape.

It is well known that visual information can be retained in several types of memory systems. Haptic information can also be retained in a memory because we can repeat a hand movement. There may be a common memory system for vision and action. On the one hand, it may be convenient to have a common system for acting with visual information. On the other hand, different modalities may have their own memory and use retained information without transforming specific to the modality. We compared memory properties of visual and haptic information. There is a phenomenon known as mental rotation, which is possibly unique to visual representation. The mental rotation is a phenomenon where reaction time increases with the angle of visual target (eg,, a letter) to identify. The phenomenon is explained by the difference in time to rotate the representation of the target in the visual sytem. In this study, we compared the effect of stimulus angle on visual and haptic shape identification (two-line shapes were used). We found that a typical effect of mental rotation for the visual stimulus. However, no such effect was found for the haptic stimulus. This difference cannot be explained by the modality differences in response because similar difference was found even when haptical response was used for visual representation and visual response was used for haptic representation. These results indicate that there are independent systems for visual and haptic representations.

Information from different sensory modalities influences perception and attention in tasks such as visual search. We have previously reported identity-based crossmodal influences of audition on visual search (Iordanescu, Guzman-Martinez, Grabowecky, & Suzuki, 2008; Iordanescu, Grabowecky, Franconeri, Theeuwes, and Suzuki, 2010; Iordanescu, Grabowecky and Suzuki, 2011). Here, we extend those results and demonstrate a novel crossmodal interaction between haptic shape information and visual attention. Manually-explored, but unseen, shapes facilitated visual search for similarly-shaped objects. This effect manifests as a reduction in both overall search times and initial saccade latencies when the haptic shape (eg, a sphere) is consistent with a visual target (eg, an orange) compared to when it is inconsistent with a visual target (eg, a hockey puck)]. This haptic-visual interaction occurred even though the manually held shapes were not predictive of the visual target's shape or location, suggesting that the interaction occurs automatically. Furthermore, when the haptic shape was consistent with a distracter in the visual search array (instead of with the target), initial saccades toward the target were disrupted. Together, these results demonstrate a robust shape-specific haptic influence on visual search.

The perceived location of a touch shifts with eye or head position, suggesting touch is coded in a visual reference frame. Our lab has previously reported shifts in the direction of eccentricity, but Ho and Spence (2007, Brain Res. 1144: 136) reported head-related shifts in the opposite direction. These studies differed by body part touched (arm vs. torso), the nature of the touch (tap vs. vibration) and whether perceived touch location was recorded with head centered or eccentric. We performed a series of experiments applying vibration to the torso and measuring perceived location using a visual scale. We replicated Ho and Spence using trials blocked by head position and recording perceived position with the head displaced. A second experiment used randomized head displacements with the head centered before responding. In a control experiment head was centered during the touch and eccentric for response. Perceived touch location shifted in the direction opposite to head position under the blocked design, in the same direction as head position in the randomized design and not at all in the control. Results are discussed in terms of allocentric and egocentric coding mechanisms needed to complete each condition.

The haptic perception of 2D images is believed to make heavy demands on working memory. During active exploration, we need to store not only the current sensory information, but also to integrate this with kinesthetic information of the hand and fingers in order to generate a coherent percept. The question that arises is how much tactile memory we have for tactile stimuli that are no longer in contact with the skin during active touch? We examined working memory using a tactile change detection task with active exploration. Each trial contained two stimulation arrays. Participants engaged in unconstrained active tactile exploration of an array of vibrotactile stimulators. In half of the trials, one of the vibrating tactors that was active in the first stimulation turned off and another started vibrating in the second stimulation. Participants had to report whether the arrays were the same or different. Performance was near-perfect when up to two tactors were used and dropped linearly as the number of the vibrating tactors increased. These results suggest that the tactile working memory off the hand is limited and there is little or no memory integration across hand movements.

What does the visual cortex of the blind do during Braille reading? This process involves converting simple tactile information into meaningful patterns that have lexical and semantic properties. The perceptual processing of Braille might be mediated by the somatosensory system, whereas visual letter identity is accomplished within the visual system in sighted people. Recent advances in functional neuroimaging techniques have enabled exploration of the neural substrates of Braille reading (Sadato et al. 1996, 1998, 2002, Cohen et al. 1997, 1999). The primary visual cortex of early-onset blind subjects is functionally relevant to Braille reading, suggesting that the brain shows remarkable plasticity that potentially permits the additional processing of tactile information in the visual cortical areas. Similar cross-modal plasticity is observed by the auditory deprivation: Sign language activates the auditory cortex of deaf subjects (Neville et al. 1999, Nishimura et al. 1999, Sadato et al. 2004). Cross-modal activation can be seen in the sighted and hearing subjects. For example, the tactile shape discrimination of two dimensional (2D) shapes (Mah-Jong tiles) activated the visual cortex by expert players (Saito et al. 2006), and the lip-reading (visual phonetics) (Sadato et al. 2004) or key touch reading by pianists (Hasegawa et al. 2004) activates the auditory cortex of hearing subjects. Thus the cross-modal plasticity by sensory deprivation and cross-modal integration through the learning may share their neural substrates. To clarify the distribution of the neural substrates and their dynamics during cross-modal association learning within several hours, we conducted audio-visual paired association learning of delayed-matching-to-sample type tasks (Tanabe et al. 2005). Each trial consisted of the successive presentation of a pair of stimuli. Subjects had to find pre-defined audio-visual or visuo-visual pairs in a trial and error manner with feedback in each trial. During the delay period, MRI signal of unimodal and polymodal areas increased as cross-modal association learning proceeded, suggesting that cross-modal associations might be formed by binding unimodal sensory areas via polymodal regions. These studies showed that sensory deprivation and long- and short-term learning dynamically modify the brain organization for the multisensory integration.

Previous research has described several core principles of multisensory integration. These include the spatial principle, which relates the integrative product to the physical location of the stimuli and the principle of inverse effectiveness, in which minimally effective stimuli elicit the greatest multisensory gains when combined. In the vast majority of prior studies, these principles have been studied in isolation, with little attention to their interrelationships and possible interactions. Recent neurophysiological studies in our laboratory have begun to examine these interactions within individual neurons in animal models, work that we extend here into the realm of human performance and perception. To test this, we conducted a psychophysical experiment in which 51 participants were tasked with judging the location of a target stimulus. Target stimuli were visual flashes and auditory noise bursts presented either alone or in combination at four locations and at two intensities. Multisensory combinations were always spatially coincident. A significant effect was found for response times and a marginal effect was found for accuracy, such that the degree of multisensory gain changed as a function of the interaction between space and effectiveness. These results provide further evidence for a strong interrelationship between the multisensory principles in dictating human performance.

Which one, ‘looking without seeing’ or ‘listening without hearing’, is more effective? There have been ongoing debates on the direction of sensory dominance in the case of cross-modal distraction. In the present study, we assumed that the specific direction of sensory dominance depends on the locus of attentional selection. Visual dominance may occur at the earlier stages while auditory dominance may occur at the later stages of cognitive processing. In order to test this hypothesis, we designed the present two (modality attended) by three (cross-modal congruency) experiment. Participants were asked to attend to either visual or auditory modality while ignore stimuli from the other modality during bimodal (visual and auditory) stimulation. More importantly, we explicitly differentiated the cross-modal distraction into the pre-response and the response level by manipulating three levels of cross-modal congruency: congruent, incongruent response-ineligible (II), and incongruent response-eligible (IE). Our behavioural data suggested that visual distractors caused more interferences to auditory processing than vice versa at the pre-response level while auditory distractors caused more interferences to visual processing than vice versa at the response level, indicating visual dominance at the pre-response level while auditory dominance at the response level. At the neural level, dissociable neural networks were also revealed. The default mode network of the human brain underlies visual dominance at the pre-response level, while the prefrontal executive regions, left posterior superior temporal sulcus and left inferior occipital cortex underlie auditory dominance at the response level.

The process of integrating information across sensory modalities is highly dependent upon a number of stimulus characteristics, including spatial and temporal coincidence, as well as effectiveness. Typically, these properties have been studied in isolation, but recent evidence suggests that they are interactive. This study focuses on interactions between the spatial location and temporal synchrony of stimuli. Participants were presented with simple audiovisual in parametrically varied locations, and with parametrically varied stimulus onset asynchronies (SOAs). Participants performed spatial location and perceived simultaneity tasks (PSS). Accuracies and response times were measured. Accuracies of spatial localization were dependent upon spatial location, with no effect of SOA and interaction seen, however, RT analysis showed an effect of SOA and an interaction; more peripheral presentations showed greater slowing of RT in asynchronous conditions, and fewer violations of the race model. With the PSS tasks, effects of SOA and spatial location were found in the responses, as well as an interaction between the two. Peripheral stimuli were more likely to be judged as synchronous, a difference seen particularly with long SOAs. These results suggest that the commonly studied principles of integration are indeed interactive, and that these interactions have measureable behavioral implications.

We investigated how subjects perceive the temporal relationship of a light-flash and a complex acoustic signal. The stimulus mimics ubiquitous events in busy scenes which are manifested as a change in the pattern of ongoing fluctuation. Detecting pattern emergence inherently requires integration over time; resulting in such events being detected later than when they occurred. How does delayed detection-time affect the perception of such events relative to other events in the scene? To model these situations, we use sequences of tone-pips with a time-frequency pattern that changes from random to regular (‘REG-RAND’) or vice versa (‘RAND-REG’). REG-RAND transitions are detected rapidly, but RAND-REG take longer to detect (∼880ms post nominal-transition). Using a Temporal Order Judgment task, we instructed subjects to indicate whether the flash appeared before or after the acoustic transition. The point of subjective simultaneity between the flash and RAND-REG does not occur at the point of detection (∼880ms post nominal-transition) but ∼470ms closer to the nominal acoustic-transition. This indicates that the brain possesses mechanisms that survey the proximal history of an ongoing stimulus and automatically adjust perception to compensate for prolonged detection time, thus producing more accurate representations of cross-modal scene dynamics. However, this re-adjustment is not complete.

The sound-induced flash illusion (SIFI) is thought to be due to early audiovisual connections which allow rapid interplay between early auditory and visual cortices. However, the temporal relationship between the perception of physically presented and additional illusory flashes has not yet been directly addressed. We presented participants with a stream of four brief auditory white noise bursts separated by gaps of 60ms (SOA=70ms) and one or two brief flashes which could appear in any of the gaps between noise bursts (positions 1–4) or before the auditory stream (position 0). All possible combinations were randomized and presented equally often. First, participants were asked to indicate the perceived number of flashes, and second, to indicate the perceived position of the flash(es). Thus we could compare the physical separation of the flashes with the perceived separation. We report the following results: 1) In illusory-fission trials the average perceived distance between the perceived flashes is ∼100ms. 2) SIFI is strongest when the flash is presented at position 1 and 2. 3) The extent of experienced fission-illusion depends on performance in aligning flashes with auditory gaps. 4) We observed auditory induced de-fusion which is strongest at position 1 and again correlated to overall task performance.

We examined the cross-modal effect of irrelevant sound (or disk) on the perceived visual (or auditory) duration, and how visual and auditory signals are integrated when perceiving the duration. Participants conducted a duration discrimination task with a 2-Interval-Forced-Choice procedure, with one interval containing the standard duration and the other the comparison duration. In study 1, the standard and comparison durations were either in the same modality or with another modality added. The point-of-subjective-equality and threshold were measured from the psychometric functions. Results showed that sound expanded the perceived visual duration at the intermediate durations but there was no effect of disk on the perceived auditory duration. In study 2, bimodal signals were used in both the standard and comparison durations and the Maximum-Likelihood-Estimation (MLE) model was used to predict bimodal performance from the observed unimodal results. The contribution of auditory signals to the bimodal estimate of duration was greater than that predicted by the MLE model, and so was the contribution of visual signals when these signals were temporally informative (ie, looming disks). We propose a hybrid model that considers both the prior bias for auditory signal and the reliability of both auditory and visual signals to explain the results.

In a stream/bounce display in which two identical visual objects move toward each other, coincide (completely overlap), and then move apart, the objects can be perceived as either streaming through or bouncing off each other. Despite the perceptual ambiguity in this display, the streaming percept is dominant. However, a sound burst presented at the time that the objects coincide facilitates the bouncing percept. Herein, we report a perceptual phenomenon in which the overlap between objects is illusorily perceived as a non-overlap in the stream/bounce display accompanied with sound. In the experiment, the amount of overlap between two objects was systematically manipulated in the presence/absence of a sound. Observers were asked to judge whether the two objects overlapped with each other and then asked whether the objects appeared to stream through or bounce off each other. The results were consistent with those of previous studies showing that sound promoted the bouncing percept. Most importantly, the sound presentation facilitated the perception of a non-overlap between the objects instead of a physical overlap, suggesting that the momentary overlap was inadequately perceived. We discuss the possibility that an abrupt sound temporally interrupts visual processing such as the formation of dynamic object representations.

Do vision and audition have a similar manner in the perceptual organization? Previous research demonstrated that a brief sound changes the perceptual organization in auditory stream/bounce phenomenon as well as in vision (Yasuhara, Hongoh, Kita, 2010). In order to investigate another similarity between vision and audition, we here examined whether a flash event alters the perceptual organization in audition. In the present study, we used mixture of two glide tones ascending and descending in frequency that could be perceived as either streaming or bouncing. The experimental condition employed a flash event when the ascending and descending tones crossed, whereas the control condition did not present flash. As a result, a flash event did not increase rates of bouncing perception, of which the direction of pitch change switched in the mid-course but slightly decreased rates of streaming perception, of which the direction of pitch change stayed constantly. These results suggest that a flash event has a subtler effect on the change of the perceptual organization in audition than the auditory event.

We investigated the effects of auditory stimuli on the perceived velocity of a moving visual stimulus. Previous studies have reported that the duration of visual events is perceived as being longer for events filled with auditory stimuli than for events not filled with auditory stimuli, ie, the so-called “filled-duration illusion.” In this study, we have shown that auditory stimuli also affect the perceived velocity of a moving visual stimulus. In Experiment 1, a moving comparison stimulus (4.2∼5.8 deg/s) was presented together with filled (or unfilled) white-noise bursts or with no sound. The standard stimulus was a moving visual stimulus (5 deg/s) presented before or after the comparison stimulus. The participants had to judge which stimulus was moving faster. The results showed that the perceived velocity in the auditory-filled condition was lower than that in the auditory-unfilled and no-sound conditions. In Experiment 2, we investigated the effects of auditory stimuli on velocity adaptation. The results showed that the effects of velocity adaptation in the auditory-filled condition were weaker than those in the no-sound condition. These results indicate that auditory stimuli tend to decrease the perceived velocity of a moving visual stimulus.

As a basic study on the influence of the dynamic properties of the audio-visual stimuli upon interaction between audition and vision, the effect of the simple movement involved in the visual stimulus on the loudness perception of the audio stimulus was investigated via psychophysical experiment. In this experiment, the visual stimulus given to subjects along with the audio stimulus is a bar appeared on a display, one side of which is flexibly expanding and contracting. The loudness of the audio stimulus with such a visual effect concurrently presented was rated as an absolute numerical value by using the Magnitude Estimation method. The reference of the bar length is determined so as to correspond to the Zwicker's loudness calculated for the given audio stimulus. As a result, the visual stimulus did not affect the loudness perception, when the bar was presented with its length same as the reference. On the other hand, the rating of the loudness for the same audio stimulus was significantly increased when the bar length was longer than the reference. This indicates that the change in the correspondence between the audio and the visual stimuli affect the loudness perception.

Visual allocentric references frames (contextual cues) affect visual space perception (Diedrichsen et al., 2004; Walter et al., 2006). On the other hand, experiments have shown a change of visual perception induced by binaural stimuli (Chandler, 1961; Carlile et al., 2001). In the present study we investigate the effect of visual and audiovisual allocentred reference frame on visual localization and straight ahead pointing. Participant faced a black part-spherical screen (92cm radius). The head was maintained aligned with the body. Participant wore headphone and a glove with motion capture markers. A red laser point was displayed straight ahead as fixation point. The visual target was a 100ms green laser point. After a short delay, the green laser reappeared and participant had to localize target with a trackball. Straight ahead blind pointing was required before and after series of 48 trials. Visual part of the bimodal allocentred reference frame was provided by a vertical red laser line (15° left or 15° right), auditory part was provided by 3D sound. Five conditions were tested, no-reference, visual reference (left/right), audiovisual reference (left/right). Results show that the significant effect of bimodal audiovisual reference is not different from the visual reference one.

Although multistable perception has long been studied, in recent years, paradigms involving ambiguous visual stimuli have been used to assess phenomenal awareness. Indeed, for such stimuli, although the sensory input is invariant, a perceptual decision occurs bringing to awareness only one of the possible interpretations. Moreover, the observer experiences inevitable oscillations between these percepts. Only few studies have tackled bistability for multimodal percepts. Conrad et al. [1] presented dot fields going in opposite direction to each eye, they found that lateral sound motion prolongs the dominance duration of the coherent visual motion percept and not the other. In this study, the volitional control—the capacity to promote a given percept—was not assessed. Van Ee et al. [2] tested how sensory congruency influences volitional control using looming motion stimuli that could be seen, heard or sensed on the hand. They found that attention to the additional modality (sound or touch) has to be engaged to promote visual dominance and that only temporal congruency is required in the end. In both studies such low-level stimuli could have reduced the possibility of finding passive interactions and limited the congruency to temporal features, which motivated the use of higher-level audio-visual speech processing in our experiments. Accordingly, we used the McGurk effect [4] involving robust audio-visual integration to investigate multimodal rivalry. After measuring the perceptual oscillation dynamics of rivaling videos (lips pronouncing /aba/ vs. /aga/), we added the sound /aba/ in order to assess the influence of audio-visual integration on the perceptual dynamics. Adding the sound /aba/ increased the dominance durations of both percepts when viewed passively. For McGurk sensitive participants, it also increased the capacity to promote lips uttering /aba/ percept as compared to the same situation without sound; but not for lips uttering /aga/ although it could be equivalent to promoting lips uttering /ada/. Our findings suggest that at higher-level processing stages, auditory cues do interact with the perceptual decision and with the dominance mechanism involved during visual rivalry. These results are discussed according to the individual differences in the audio-visual integration for speech perception. We propose a descriptive model based on known characteristics of binocular rivalry, which accounts for most of these findings. In this model, the top-down attentional control (volition) is modulated by lower-level audio-visual matching.

Despite several demonstrations of crossmodal semantic-congruency effect, it remains controversial as to whether it is a genuine perceptual phenomenon or instead it actually results from post-perceptual response bias such as decision or strategies (de Gelder and Bertelson, 2003). Here we combine the invisible stimuli with sounds to exclude the participants' awareness of the relation between visual and auditory stimuli. We render the visual events invisible by adopting the continuous flash suppression paradigm (Tsuchiya and Koch, 2005) in which the dynamic high-contrast visual patches were presented in one eye to suppress the target that was presented in the other eye. The semantic congruency between visual and auditory stimuli was manipulated and participants had to detect any parts of visual target. The results showed that the time needed to detect the visual target (ie, the release from suppression) was faster when it was accompanied by a semantically congruent sound than with an incongruent one. This study therefore demonstrates genuine multisensory integration at the semantic level. Furthermore, it also extends from previous studies with neglect blindsight patients (eg, de Gelder, Pourtois, and Weiskrantz, 2002) to normal participants based on their unawareness of the relation between visual and auditory information.

The sense of presence is crucial for evaluating audio-visual equipment and content. To clarify the multidimensional attributes of the sense, we conducted three experiments on audio, visual, and audio-visual content items. Initially 345 adjectives, which express the sense of presence, were collected and the number of adjectives was reduced to 40 pairs based on the KJ method. Forty scenes were recorded with a high-definition video camera while their sounds were recorded using a dummy head. Each content item was reproduced with a 65-inch display and headphones in three conditions of audio-only, visual-only and audio-visual. Twenty-one subjects evaluated them using the 40 pairs of adjectives by the Semantic Differential method with seven-point scales. The sense of presence in each content item was also evaluated using a Likert scale. The experimental data was analyzed by the factor analysis and four, five and five factors were extracted for audio, visual, and audio-visual conditions, respectively. The multiple regression analysis revealed that audio and audio-visual presences were explained by the extracted factors, although further consideration is required for the visual presence. These results indicated that the factors of psychological loading and activity are relevant for the sense of presence.

While reading texts orally, we process the multisensory language information. Accordingly, in the context of reading aloud, we process the visually presented text and produce the auditory information of the text through articulatory movement. These multisensory processing activities are assumed to facilitate the memory and comprehension of textual information. Conversely, while reading silently, we process only the visual information of the text. Although we cannot use the multisensory language information while reading silently, several researchers have found that there is little difference between the degree of comprehension based on silent and oral reading for adult readers. The purpose of this study is to explain how we compensate the loss of multisensory process during silent reading by comparing the visual processing process during silent and oral reading. By conducting two experiments, we measured and compared the eye movement during silent and oral reading. The results showed that silent reading took shorter time for comprehension than oral reading, and readers had more visual fixation points and read back frequently during reading silently than orally. These reading strategies during silent reading seemed to compensate the loss of multisensory process and support the text comprehension.

Dual-coding theory (Paivio, 1986) postulates that the human mind represents objects not just with an analogous, or semantic code, but with a perceptual representation as well. Previous studies (eg, Fiebach & Friederici, 2004) indicated that the modality of this representation is not necessarily the one that triggers the representation. The human visual cortex contains several regions, such as the Lateral Occipital Complex (LOC), that respond specifically to object stimuli. To investigate whether these principally visual representations regions are also recruited for auditory stimuli, we presented subjects with spoken words with specific, concrete meanings (‘car’) as well as words with abstract meanings (‘hope’). Their brain activity was measured with functional magnetic resonance imaging. Whole-brain contrasts showed overlap between regions differentially activated by words for concrete objects compared to words for abstract concepts with visual regions activated by a contrast of object versus non-object visual stimuli. We functionally localized LOC for individual subjects and a preliminary analysis showed a trend for a concreteness effect in this region-of-interest on the group level. Appropriate further analysis might include connectivity and classification measures. These results can shed light on the role of crossmodal representations in cognition.

The conventional view that perceived emotion in music is derived mainly from auditory signals has led to neglect of the contribution of visual image. In this study, we manipulated mode (major vs. minor) and examined the influence of a video image on emotional judgment in music. Melodies in either major or minor mode were controlled for tempo and rhythm and played to the participants. We found that Taiwanese participants, like Westerners, judged major melodies as expressing positive, and minor melodies negative, emotions. The major or minor melodies were then paired with video images of the singers, which were either emotionally congruent or incongruent with their modes. Results showed that participants perceived stronger positive or negative emotions with congruent audio-visual stimuli. Compared to listening to music alone, stronger emotions were perceived when an emotionally congruent video image was added and weaker emotions were perceived when an incongruent image was added. We therefore demonstrate that mode is important to perceive the emotional valence in music and that treating musical art as a purely auditory event might lose the enhanced emotional strength perceived in music, since going to a concert may lead to stronger perceived emotion than listening to the CD at home.

In an event-related potential (ERP) experiment, laser dots were either projected onto a table such that they appeared in a participant's peripheral field of view (FOV) or they were projected onto the participant's hands that were also placed within their peripheral FOV. An oddball paradigm was utilized where participants had to detect a laser deviant that appeared in a stream of standard laser stimuli. Participants were instructed to fixate on a central crosshair and to selectively attend to their left or right hand. In the hands-off condition, participants' ERPs waveform and topographical iso-potential map analyses revealed a typical contra-lateral primary visual activation. In the hands-on condition, the visual activation obtained in the hands-off condition shifted significantly toward anterior primary somatosensory areas. Localization techniques such as independent components analysis, dipole fitting, and sLORETA yielded results that were convincingly convergent with the ERP waveform and topographical map data. That is, visual laser stimuli appeared to possess tactile qualities when they were projected onto visible haptic regions of participant observers.

It is well known that detection of a central Gabor stimulus is modulated by the presence of collinear flanker stimuli. Target detection is facilitated when flanker stimuli are iso-oriented and aligned with the target at adequate spatial separations, suggesting the involvement of lateral interactions in early visual processing. Can such contextual effects of flankers on the target detection occur in a cross-modal setting (a central visual target and tactile flankers)? To address this, I investigated a variant of the lateral flanker paradigm using the visually presented line targets and tactile flanker lines. The results showed that tactile flankers facilitated the detection of visual target when the tactile flankers were collinear with the visual target, indicating the occurrence of cross-modal contextual modulation between vision and touch. The findings suggest that the integration process could occur at a relatively early perceptual processing stage of perceptual grouping.

Animals are always integrating environmental information from multiple sensory modalities, but the mechanisms underneath are highly underexploited. Crossmodal interactions in animal perception have been found in several species including human, mice and flies. Here we subjected zebrafish as model because its genetic effects on brain and sense organ development are well studied, but the attentional processes are mainly unexplored. Zebrafish show impressive behaviour capabilities with relatively small or “simple” brains which make their nervous system relatively more accessible to experimentation than large-brained mammals. When conditioned with both vision and vibration signals, zebrafish were able to make higher correct choices than only one sensation. After multimodal training, zebrafish were also able to transfer the memory to unimodal conditioning when only given vision or vibration signals. This study provided basic findings for how animals process multimodal sensory from the environment, and showed crossmodal interactions in zebrafish for the first time.

Cross-talk between visual and tactile modalities influences on very different levels from neural activities, perception to higher cognition and behaviour. Many researches showed that one modality alters the other modality perception. There are some plausible explanations for those perceptual changes by multisensory inputs. The viewpoint of “Pre-attention effect” which is attention drawn by another modality(vision) facilitates the tactile encoding is more advocated by many researches. However, the timing issue between two modalities is critical in order to insist that attention across modality enhances perceptual sensitivity(detection threshold). So, the aim of this study is how vision influences on tactile sensitivity according to temporal synchrony. Total 15 subjects were given to perform the 2FAC task whether tactile pulse is felt. Even 6 points of tactile sub-threshold including none condition are randomly set to observe psychophysical detection performance. Visual stimuli are simultaneously provided with SOAs (0,±50,±100,±300) and no visual stimulus was given to Tactile Only condition. Discriminability between visual and tactile stimulus shows better when two stimuli are apart so does not interrupt each other(SOA±300 or Tactile Only). However, the tactile threshold is more sensitive when providing with visual stimulus, especially when visual stimulus precedes 50ms than tactile one. The change of detection sensitivity at a certain time period(vision is presented forward 50ms) implies that attention is drawn across sensory modality and it facilitates perceptual sensitivity. Therefore, it is hard to say that multisensory input leads perception of other sensory modality and would rather say attention across modality enhances perceptual sensitivity.

Participants were simultaneously presented with a colour square on a monitor (red, black or blue) and a temperature at their fingertip using a peltier tile (warm, room temperature or cool). Each stimulus pair was presented in congruent (red-warm and blue-cool) and incongruent (red-cool and blue-warm) combinations. Latencies were recorded for each participant's verbal response when naming the colour and the temperature. The semantic coding hypothesis proposes that although input patterns and types are different for different sense organs, interactions can occur after these inputs have been recoded at post-perceptual levels. For example, reaction times to a simultaneously presented high-pitched tone and light source above head height might be shorter than those to a low-pitched tone and light source above head height because the recoded inputs of the former share the post-perceptual format “high”, whereas the later do not (ie, the later are incongruent). In our experiment, response times were similar regardless of whether the stimulus pair was “congruent” or “incongruent”, suggesting that while it is commonly believed that red is a warm colour and blue is cold, this kind of correspondence did not facilitate shorter latencies as it should according to the semantic coding hypothesis.

On a non-spatial task, individuals respond more slowly when the stimulus and response are on different sides of space versus the same side of space (the Simon effect). Past Simon effect studies have shown that visual stimuli are represented based on external reference frames. In contrast, we have found that tactile stimuli in a Simon effect task are encoded based on a somatotopic reference frame (eg, stimuli presented to the left hand are always encoded as left regardless of the hand's location in space). In order to understand how visual stimuli are represented on or near the body, we presented individuals with a visual Simon effect experiment. Using foot pedals, participants responded to visual stimuli projected on, near, or far from the hands in three conditions—hands uncrossed, hand crossed, and no hands. In the no hands condition, visual stimuli were encoded based on an external reference frame. However, we find that as with tactile stimuli, visual stimuli presented on or near the hands (crossed condition) were encoded based on a somatotopic frame of reference. These novel results provide evidence that visual stimuli on or near the hands can be encoded based on body representations.

In our research, we aim to construct the visual-haptic system which generate haptic feedback as if they were touching on the surface of various virtual objects, even though they are touching only a static shape. We focus on the cross modal effect between our vision and haptic sense, which is generally called Pseudo-Haptics, and make up a illusionary haptics only using visual feedback, without any complicated physical devices. In our system, user touches the physical object behind the LCD, and sees the virtual object through the LCD using video see-through methodology. We put two web cameras to provide the different image for each eye and maintain the optical conjugation between viewpoint and cameras using the mirror in the back of the LCD. From images captured by these cameras, we compose visual feedback as if they were touching another virtual shape, based on the detection of the point where they are touching the physical object. This makes up a haptic illusion as if they were touching the virtual shape. By our proposed system, users perceive visual and haptic sensation as if they were touching various virtual shapes, without any complicated physical devices.

We report a multi-sensory illusion observed in a tram climbing up the Peak of Hong Kong. The skyscrapers nearby appear to spectacularly fall towards the Peak at night as the tram ascends or descends the slope. We investigated the contribution to this illusion from the vestibular system. Five observers sat in a row of a compartment and marked the perceived tilt of the skyscrapers with a rotary pitch (which reads the angle relative to gravity). The slope of the mountain was measured simultaneously with the observers' report by an assistant with an identical pitch. In two separate trips, observers sat with a wedge of 18.7 degrees at their back or stood up thereby changing the self-position information from the otolith system. The skyscrapers appeared to tilt away by an average of 12 degrees during normal seating in the tram. This subjective tilt illusion was reduced by an average of 20% when the wedge was inserted and by more than 50% when observers stood up. Our results show that self-position information profoundly modulates the perceived tilt of buildings observed from a moving tram. The noticeable individual differences we observed also suggested the weighting from each sensory modality may differ for different observers.

Tools such as pliers systematically change the relationship between an object's size and the hand opening required to grasp it. Previous work suggests the brain takes this into account, integrating visual and haptic size information that refers to the same object, independent of the similarity of the ‘raw’ visual and haptic signals (Takahashi et al., VSS 2009). Variations in tool geometry also affect the reliability (precision) of haptic size estimates, however, because they alter the change in hand opening caused by a given change in object size. Here, we examine whether the brain appropriately adjusts the weights given to visual and haptic size signals when tool geometry changes. We first estimated each cue's reliability by measuring size-discrimination thresholds in vision-alone and haptics-alone conditions. We varied haptic reliability using tools with different object-size:hand-opening ratios (1:1, 0.7:1, and 1.4:1). We then measured the weights given to vision and haptics with each tool, using a cue-conflict paradigm. The weight given to haptics varied with tool type in a manner that was well predicted by the single-cue reliabilities (MLE model; Ernst and Banks, 2002). This suggests that the process of visual-haptic integration appropriately accounts for variations in haptic reliability introduced by different tool geometries.

In this research, we focus on symmetry to investigate the difference between visual and haptic motional representations. It is known in vision that symmetry about vertical or horizontal axis is easier to detect comparing with the other axis. We conducted a psychophysical experiment to examine whether symmetry process in haptic motion depends on axis orientation as in visual process. The observer's task was to respond symmetry pattern of a memorized image and the accuracy of the response was measured. Results showed similar effect of axis orientation on symmetry judgment both in visual and haptic motional perception, suggesting similarity in the two modalities. However, there were also a clear difference in axis orientation effect between visual and haptic motion conditions. Less superiority of horizontal axis was found in the haptic motion condition. This suggests that haptic motion system has a representation process independently of the one for vision.

Do humans integrate visual and vestibular information in a statistically optimal fashion when discriminating rotational self-motion stimuli? Recent studies are inconclusive as to whether such integration occurs when discriminating heading direction. In the present study eight participants were consecutively rotated twice (2s sinusoidal acceleration) on a chair about an earth-vertical axis in vestibular-only, visual-only and visual-vestibular trials. The visual stimulus was a video of a moving stripe pattern, synchronized with the inertial motion. Peak acceleration of the reference stimulus was varied and participants reported which rotation was perceived as faster. Just-noticeable differences (JND) were estimated by fitting psychometric functions. The visual-vestibular JND measurements are too high compared to the predictions based on the unimodal JND estimates and there is no JND reduction between visual-vestibular and visual-alone estimates. These findings may be explained by visual capture. Alternatively, the visual precision may not be equal between visual-vestibular and visual-alone conditions, since it has been shown that visual motion sensitivity is reduced during inertial self-motion. Therefore, measuring visual-alone JNDs with an underlying uncorrelated inertial motion might yield higher visual-alone JNDs compared to the stationary measurement. Theoretical calculations show that higher visual-alone JNDs would result in predictions consistent with the JND measurements for the visual-vestibular condition.

Since all motion bases of simulator involve intrinsic physical limits, the lack of coupling between visual and inertial cues may be responsible for visuo-vestibular conflict, lowering presence in the virtual environment and increasing simulator sickness. In order to proportion the movements of the motion base and of the visual scenery, the present study aimed at characterizing the coupling between visual and inertial cues that generates a believable and realistic illusion of roll movement in a motorcycle simulator. In the experiment, participants (n=29) actively tuned the visual and physical tilt to achieve the best sensation of leaning, while the theoretical tilt of a real motorcycle (in a similar situation), the road curvature as well as the horizontal field of view (ie,, 60deg vs. 180deg) were manipulated. The results revealed different patterns of use of the visual scenery among riders (eg,, in tilting the visual horizon in the same or in the opposite direction of the displayed curve, or in keeping the visual horizon straight). The results are discussed in terms of the differential roles of tilting the visual and physical dimensions in creating a reliable illusion of riding among motorcyclists.

Spatial cognition requires integration of visual perception and self-motion. The lack of motion information in vision like image flows on TV often causes confusion of the spatial cognition and it becomes hard to understand where surrounding objects are. Our aim of this work is to investigate the mechanism of constructing spatial cognition from image flows and self-motion. In our experiments, we evaluated the performance of the spatial cognition abilities in VR environments according to the amounts of visual and motion information. Motion and visual information are closely related to each other since it is possible to estimate motion from the image flows of background. To modulate visual information, two kinds of head-mounted displays (32times24 and 90times45 deg.) were used. To compensate for motion information explicitly, a marker that indicates viewpoint movement of images was added in display and the effect was tested. As a result it is found that the maker significantly improved the performance and larger FOV could compensate for the lack of motion information however the effect was not significant when the marker was added. We will discuss these results with the image stabilization method to maintain a consistency between motion and image flows.

Previous studies have shown that modifying visual information contributes to the on-line control of self-motion, and that vection can be induced by modifying the velocity of optical flow. In this study, we investigate whether velocity modulation of optical flow affects self-motion perception during whole-body motion. In our experiments, visual stimuli were provided by a virtual wall consisting of dots on a screen. The participants are asked to move their own head from a starting point to a goal in 1.5 sec (about 33 cm/s). We compared three conditions of visual feedback (optical flow) with/without modifications, ie,, accelerated, decelerated, and no change in optical flow. The rates of change in velocity were between 0.5 and 2 times the control condition. As a result, we found that the accelerated condition induced under-shooting, while the decelerated condition induced over-shooting relative to the control condition's goal point. Moreover, the positioning errors became largest with a change rate of 1.5 times. Our findings suggest that self-motion perception during body motion is influenced by the velocity change of the optical flow, and that there is an optimal rate of velocity change for perceiving a self-motion. Finally, we discuss the mechanism of integrating visual and proprioceptive information.

Our previous study (Takahashi, Fukuda, Kaneko, 2010) reported that for the perception of the gravitational vertical (up-down) of visual images, the information processed in the early stage of visual system, such as the luminance distribution and edges, seemed to have large effects when the stimulus was presented for a short time. When the presentation time increased, this tendency decreased with the increase of the influence of the information processed in a relatively higher stage of visual system such as knowledge or inference. The information regarding the gravitational vertical is also important in our actions such as walking and standing upright. In this study, we aimed to identify the visual factors to affect our action related to the gravitational vertical. We manipulated the factors such as the luminance distribution, motion, edges and meaning in natural pictures. As an indicator of action, we measured the observers' body sway induced by rotating the pictures periodically clockwise and counterclockwise. The results showed the motion and edges had large effects and the luminance distribution had little effect on body sway. We discuss the difference between perception and action in the processing of the information regarding the gravitational vertical.

Mixed reality (MR) is the technology which merges real and virtual worlds in real-time. In MR space, a real object can be changed its visual appearance by superimposing a CG image (CGI) on it. Because it is said that the feeling of the weight is affected strongly by visual stimulation, we believe that it is affected similarly, in the case of presenting MR visual stimulation. If the behavior and extent of such an influence are well investigated, one real object can be perceived differently. In this study, we focus on the center-of-gravity (COG), and verify the influence of MR visual stimulation on sense of COG in MR environment. In this paper, we describe the systematic experiments of the influence. As the result, we obtained the interesting and promising result: (1) the sense of COG can be changed by MR visual stimulation, and (2) although the different feelings of COG between vision and sense of force, the feeling of COG can be represented by MR visual stimulation under certain conditions, (3) the influence from MR visual stimulation becomes smaller when COG positions of a real object and a CG image is distant.

A virtual environment (VE) presents several kinds of sensory stimuli for creating a virtual reality. Some sensory stimuli presented in the VE have been reported to provoke cybersickness, which is caused by conflicts between sensory stimuli, especially conflicts between visual and vestibular sensations. Application of a rest frame has been known to be effective on reducing cybersickness by alleviating sensory conflict. The form and the way rest frames are presented in 3D VEs have different effects on reducing cybersickness. In this study, two different types of 3D rest frames were created. For verifying the rest frames' effects in reducing cybersickness, twenty subjects were exposed to two different rest frame conditions and a non-rest frame condition after an interval of three days in 3D VE. We observed the characteristic changes in the physiology of cybersickness in terms of autonomic regulation. Psychophysiological signals including EEG, EGG, and HRV were recorded and a simulator sickness questionnaire (SSQ) was used for measuring the intensity of the sickness before and after the exposure to the different conditions. In the results, the SSQ was reduced significantly in the rest frame conditions. Psychophysiological responses changed significantly in the rest frame conditions compared to the non-rest frame condition. The results suggest that the rest frame conditions have condition-specific effects on reducing cybersickness by differentially alleviating aspects of visual and vestibular sensory conflicts in 3D VE.

We investigated how human perceives forward self-motion from the information produced by the vestibular, somatosensory and visual systems. In the experiments, we manipulated vestibular and somatosensory information by changing the direction of the subject's body. The subject sat in a drum which was tilted about the pitch axis of the subject. We also manipulated the visual information by presenting optic flow of dots on the display in front of the subject. In Exp.1, the pitch of subject was fixed at a certain angle for each trial. In Exp.2, the pitch angle was changed temporally, and the acceleration of the optic flow was synchronized with the angle to simulate the flow produced by the forward self-motion of the acceleration. During the trials, the subject continuously responded the gravitational vertical. In the condition of fixed pitch angle (Exp.1), subjects' responses of the gravitational vertical almost corresponded to the real direction and were not affected by the visual stimulus. In the condition of changing pitch angle (Exp.2), the responses were close to the direction of their foot. These results suggest that, the temporal correspondence between vestibular, somatosensory and visual information is important for the perception of forward self-motion.

Binaural recordings can simulate externalized auditory space perception over headphones. However, if the orientation of the recorder's head and the orientation of the listener's head are incongruent, the simulated auditory space is not realistic. For example, if a person lying flat on a bed listens to an environmental sound that was recorded by microphones inserted in ears of a person who was in an upright position, the sound simulates an auditory space rotated 90 degrees to the real-world horizontal axis. Our question is whether brain activation patterns are different between the unrealistic auditory space (ie, the orientation of the listener's head and the orientation of the recorder's head are incongruent) and the realistic auditory space (ie, the orientations are congruent). River sounds that were binaurally recorded either in a supine position or in an upright body position were served as auditory stimuli. During fMRI experiments, participants listen to the stimuli and pressed one of two buttons indicating the direction of the water flow (horizontal/vertical). Behavioral results indicated that participants could not differentiate between the congruent and the incongruent conditions. However, neuroimaging results showed that the congruent condition activated the planum temporale significantly more than the incongruent condition.

The somatic sensation and the special sensation have some relation each other. For example, some kind of visually-induced self-motion perceptions are caused as an influence of the visual sensation on the somatic sensation that monitors posture. Meanwhile, some acoustically-induced self-motion perceptions have been investigated. For linearly moving sound images receding from listener, Sakamoto et al. (Acoust. Sci. & Tech., 25, 1, 2004) reported that the perceived self-motion was generally opposite to the direction of the moving sound images. In this study, we investigated an acoustically-induced self-motion perception for linearly moving sound images crossing in front of listener. In this experiment, the track of the sound image was 5m far from listener in the front. Subjects were instructed to keep stand upright and listen carefully moving sounds through headphone. Self-motion was evaluated subjectively by the rating scale method for perceived direction and level, and objectively by measuring the center-of-pressure through the experiment. As a result, auditory-induced self-motion was observed in the same direction of the moving sound images especially when the sound image was receding. The above self-motion settled in the initial position after the sound image disappeared. This may mean that auditory vection is sensitive to sounds receding from listener.

We investigated the effects of listeners' head movement and proprioceptive feedback on the accuracy of sound localization. The effects were examined under the conditions that their head movement was restricted (study 1) or unrestricted (study 2) on the sound localization training. In both experiments, participants divided into two groups: training group performed the sound localization training with accurate proprioceptive feedback, while control group conducted the training with no-feedback. The results indicated that: (1) sound localization training under active head movements increased the accuracy of sound localization; (2) accurate proprioceptive feedback facilitated the sound localization during the initial training phase; and (3) training with the feedback dominantly decreased the rate of vertical localization errors in sound localization performance.

During movement, the position of a sound object relative to an observer continuously changes. Nevertheless, the sound source position can be localized. Thus, self-motion information can possibly be used to perceive stable sound space. We investigated the effect of self-motion perception induced by visual stimuli, ie,, vection and/or vestibular information, on sound localization. To enable perception of vection, random dots moving laterally on a wide screen were presented. For presentation of vestibular stimuli, a three-degree of freedom (3 DOF) motion platform which inclines right or left was employed. Sound stimuli were presented behind a screen when an observer perceived self-motion induced by visual stimuli and/or the platform. The observer's task was to point to the position of the sound image on the screen. Experimental results showed that the perceived sound position shifted to the direction opposite the perceived self-motion induced by visual information, regardless of the direction of vestibular information. Moreover, this tendency was observed only on the side of the median sagittal plane whose direction was the same as that of the movement of visual information. Thus, auditory spatial perception is possibly changed by self-motion due to the coexistence of visually induced self-motion and vestibular information.

We report the modulation of tactile motion perception by presenting static sounds with two alternately and repeatedly presented vibrotactile stimuli for the perception of tactile apparent motion. Previous research on tactile motion perception has used direction judgment tasks for apparent motion that consist of two non-repeating, or more than two repeating stimuli. However, the direction of two repeating apparent motion stimuli has been considered too ambiguous to be judged. The present study shows that the additional presentation of sounds with manipulated timings could help to determine the perceived direction of tactile motion despite the ambiguity in the interpretation of tactile stimuli at ipsilateral locations. Furthermore, we found that there is a limited alternation rate for tactile stimuli that can be used to achieve significant modulation using sound. We relate the temporal properties observed during crossmodal effects in tactile motion perception, to those observed during some other crossmodal phenomena.

We examined whether a consistent haptic cue suggesting forward self-motion facilitated vection. We used a fan with no blades (Dyson, AM01) providing a wind of constant strength and direction (wind speed was 6.37 m/s) to the subjects' faces with the visual stimuli visible through the fan. We used an optic flow of expansion or contraction created by positioning 16,000 dots at random inside a simulated cube (length 20 m), and moving the observer's viewpoint to simulate forward or backward self-motion of 16 m/s. we tested three conditions for fan operation, which were normal operation, normal operation with the fan reversed (ie, no wind), and no operation (no wind and no sound). Vection was facilitated by the wind (shorter latency, longer duration and larger magnitude values) with the expansion stimuli. The fan noise did not facilitate vection. The wind neither facilitated nor inhibited vection with the contraction stimuli, perhaps because a headwind is not consistent with backward self-motion. We speculate that the consistency between multi modalities is a key factor in facilitating vection.

Three-dimensional (3D) television sets are available in the market and are becoming increasingly popular among consumers. However, viewing 3D films has potential adverse effects such as asthenopia and visually induced motion sickness (VIMS). The phenomenon of VIMS is not fully understood yet. One cause of VIMS has been considered to be sensory conflict, ie,, the disagreement between vergence and visual accommodation, while viewing a stereoscopic film. VIMS can be analyzed in subjective and physiological terms. The aim of this study is to evaluate the effect of viewing a long stereoscopic film on the human body. An electrocardiogram was obtained for subjects viewing an hour-long stereoscopic film. We also conducted stabilometric analysis for the subjects in the Romberg posture, flicker tests, and subjective questionnaires to detect fatigue and eye strain every 20 minutes. Statistical significance was observed for subjective averages of the test results obtained before exposure to the film and 40 minutes after the initiation of the measurement (p < 0.05). Symptoms of VIMS could be detected during the exposure to the hour-long 3D film. Based on this result, guidelines will be devised to ensure safety in viewing 3D movies.

There are many researches showing interactions between olfaction and taste. Many of them supported that the interactions are not innate, but are learned through our daily eating experiences. Stevenson (2009) called this phenomenon as “learned synesthesia”. The authors also showed the interactions between flavor and taste are learned and processed by higher cognitive systems in rats and humans (Sakai et al., 2001; Sakai and Imada, 2003). Here the interactions between umami taste and dashi flavors are developed by the daily eating experience of Japanese traditional cuisine. Twenty flavors (such as sea weed, bonito, onion, garlic, ginger etc. by courtesy of YAMAHO CO. Ltd.) were used as flavor stimuli. Taste stimuli are monosodium glutamate (umami substance, MSG), miso soup, and Katsuo Dashi (bonito soup stock). Participants tasted these stimuli, 12∼20 stimuli in a day, and evaluated the strength of umami taste, the palatability, congruity between taste and flavor with 100 mm visual analogue scales. The results of evaluations analyzed with the participants' daily eating experience showed the interactions between taste and flavor are developed by their own daily intake of traditional Japanese cuisine, especially dashi soup.

In our previous experiments, it was shown that odors (rose, sandalwood) might influence the haptic perception (stiffness, roughness). In particular, subjects felt a smooth surface even smoother with rose odor and a stiff surface even harder with sandalwood odor. This study investigated whether the effects of odors could be explained by their categories that are derived from the verbal descriptions of odors. We used jasmine that is categorized as “flower”, which includes rose, and peppermint that is categorized as “herb” close to sandalwood for our experiments. Subjects were requested to adjust the roughness or stiffness of a test surface to match that of a standard surface just shown before it using a force-feedback device. Standard surfaces were presented with different odors (jasmine, peppermint, odorless) which was projected to the noses of subjects. The results showed that jasmine made subjects feel the surfaces softer than those in odorless condition, but had no effect on roughness perception. Meanwhile peppermint did not show significant effect on either stiffness or roughness perception. According to the inconsistency between the results of present and previous experiments, we argue that odors in similar categories do not necessarily influence haptic perception in the same way.

The determinants of synesthetic colors for Japanese scripts were studied in six Japanese grapheme-color synesthetes. We investigated the influence of linguistic properties such as phonology, orthography, and meaning on synesthetic colors for logographic characters (Kanji), phonetic characters (hiragana and katakana), and digits. From a palette of 138 colors, the synesthetes selected colors for 79 Kanji, 71 hiragana, and 71 katakana characters, and 9 digits. The results revealed that the color choices for hiragana and katakana characters representing the same sound were remarkably consistent, indicating that color selection depended on phonology and not visual form. On the other hand, synesthetic colors for Kanji characters, which are usually learned later, depended on meaning and phonology. Kanji characters representing concepts that are highly related to colors (eg, names of objects with typical colors) were associated with those colors. Digits and corresponding Kanji numerals elicited strikingly similar colors. Colors for Kanji and hiragana characters sharing the same sound were likely to be similar. These results suggest that synesthetic colors are generalized from digits and phonetic scripts to Kanji characters via meaning and phonology. This study provides insights into the generalization of synesthetic colors to later acquired sets of graphemes within a language.

It has been reported that the presence of others facilitates or inhibits an individual's performance. To investigate the neural bases of the effect of the presence of others, using near-infrared spectroscopy (NIRS), we measured bilateral prefrontal activation (BA 10) in humans while they performed a simulated driving task. Participants were divided into four groups according to the combination of the presence of others (Single vs. Paired) and the game experience (High vs. Low): S-H, S-L, P-H, and P-L groups. The participant's task was to drive from start to goal using a route map either driving alone without a paired partner (single condition) or driving with a paired partner as an observer (paired condition). The results revealed that the single groups (S-H, S-L) yielded more errors than the paired groups (P-H, P-L) regardless of the game experience. However, the NIRS data demonstrated an interaction between the presence of others (S, P) and the game experience (H, L): The P-H showed higher brain activity than the S-H, whereas the P-L showed lower brain activity than the S-L. These results suggest that the presence of others functions distinctively in the prefrontal cortex in participants according to their prior experience in the relevant task.

We examined whether lighting direction, left or right, has an influence on the impression of faces and objects, and the role of gaze direction in the impression-forming. In the first experiment, we examined how lighting directions influenced the impression of faces and objects. On each trial, a pair of faces or objects was presented on top of each other. Left side was brighter in one and right side was brighter in the other. The participants were asked to answer which face or object was more preferable. The results showed that the participants preferred left-brighter faces and objects significantly more frequently than right-brighter stimuli (p < .05, chi-square test). The effect was especially strong in the upright faces. The results suggested that faces and objects give better impressions when they are lit from the left. In the second experiment, we examined whether eye-movement would play a role in our preference for left-brighter faces and objects. We recorded eye-movements of the participants while doing the same task as the first experiment. The results showed that the participants' preference for left-brighter faces were stronger when the participant started viewing from the left. The gaze direction may modulate our impression-formation (Shimojo et al. 2003).

Imagine that a 3D object is illuminated from behind, and its shadow is cast into a translucent screen viewed by an observer. When stationary, it looks like a flat 2D figure, but when rotated, it appears as a 3D rotating object (depth-from-rotation). The relative motion of the points of shadow gives depth information but perspective information is not present. So, the relative depth of points can be perceived but not their depth order. It is impossible to tell which is the front and which is the back of the objects. The 3D object is ambiguous with respect to its reflection in the projection plane. As a result the object appears to reverse in depth periodically as well as direction of rotation. This study investigated if manipulating intent of subject alters perception of depth-from-rotation display. In the experiment, the stimulus change was coincided with the rotation of a crank handle rotated rightward or leftward by the observer. The result showed that the perceived direction of rotation from the display coincided the manual control at a high rate. Prolonged viewing, however, made reverse the apparent rotation. We measured duration of initial rotation. The manual control extended the mean duration.

Typically, arousal increases when telling a lie, as indicated in psychophysiological studies about lie detection. But the emotional valence induced by lying is unknown, though intuition indicates that it may be negative. Indeed, the Electrodermal Activity (EDA), used in such studies, only shows arousal changes during an emotional response. In this study, we examined the emotional valence induced by lying using two tasks. First, in the deceptive task, participants answered “no” to every question regarding the nature of displayed playing cards. Therefore, they told a lie about specific cards. During the task, their EDA was recorded. Secondly, in the figure estimation task, they assessed pictures by “like” or “dislike” after looking at playing cards visibly or subliminally as prime stimuli. We expected them to tend to estimate figures by “dislike” when cards relevant to deception were previously shown. This would mean that an affective priming effect due to telling a lie happened. Actually, this effect was found only when prime stimuli were displayed visibly. This result suggests that lying per se induces negative emotions even without motivation or punishment due to lying. Furthermore, we found that such effect was more blatant in participants whose EDA changes were salient while lying.

Audiovisual capture happens when information across modalities get fused into a coherent percept. Ambiguous multi-modal stimuli have the potential to be powerful tools to observe such effects. We used such stimuli made of temporally synchronized and spatially co-localized visual flashes and auditory tones. The flashes produced bistable apparent motion and the tones produced ambiguous streaming. We measured strong interferences between perceptual decisions in each modality, a case of audiovisual capture. However, does this mean that audiovisual capture occurs before bistable decision? We argue that this is not the case, as the interference had a slow temporal dynamics and was modulated by audiovisual congruence, suggestive of high-level factors such as attention or intention. We propose a framework to integrate bistability and audiovisual capture, which distinguishes between “what” competes and “how” it competes (Hupé et al., 2008). The audiovisual interactions may be the result of contextual influences on neural representations (“what” competes), quite independent from the causal mechanisms of perceptual switches (“how” it competes). This framework predicts that audiovisual capture can bias bistability especially if modalities are congruent (Sato et al., 2007), but that is fundamentally distinct in nature from the bistable competition mechanism.

Given ambiguity in visual inputs, human chooses one among diverse possible interpretation. Perceptual decision in visual competition is susceptible to inputs into the modalities other than vision. Here we will review recent investigation conducted in our laboratory. The studies mainly aimed to examine how auditory inputs implicitly and/or explicitly affect temporal characteristics of perceptual decision. Here is the summary of our findings: (1) Synchronization: perceptual alternation for bi-stable visual patterns can be synchronized with implicit auditory events after experiencing synchronized audio-visual event. (2) Destabilization: auditory transients immediately collapse current percepts of bi-stable visual patterns. (3) Suppression/restoration: auditory transients immediately reveal and/or conceal masked visual patterns in continuous flash suppression. These results would imply that temporal patterns of perceptual decision in vision might be, implicitly and explicitly, related to other sensory modalities.

Previous studies on perception have yet to establish that psychophysical adaptation effects transfer from one sense to another. To test for this phenomenon, the current study examines the possible crossmodal transfer of temporal rate adaptation from vision to audition (VA) and from audition to vision (AV). Participants were trained, using feedback, to discriminate the perceived rapidity of either auditory or visual stimuli presented at a range of randomly-ordered frequencies (3.25–4.75 Hz) as compared to that of stimuli (of the same modality) at a familiar average frequency (4 Hz). Afterwards, subjects were repeatedly exposed to stimuli (of the other modality) at a specific rate (3 Hz or 5 Hz). To test whether adaptation resulted from this exposure, subjects again completed the task previously used for training, but now without feedback. After the initial training and adaptation phases, these test and adaptation tasks were presented in 20 alternating blocks. A comparison of the pre- and post-adaptation responses showed crossmodal changes in subjects' perception of temporal rate, such that adaptation to 5 Hz led to the subsequent stimuli seeming slower than they had before adaptation. On the other hand, after exposure to 3 Hz stimuli, the opposite effect was seen. This shift occurred in both VA and AV conditions. As audition and vision were never simultaneously presented, this is suggestive of a strong linkage between the two modalities in perceiving rate. We propose that this is due to the presence of early, distributed, within-modal clocks, that can vigorously modulate each other cross-modally.

Sounds containing no motion or positional cues could induce illusory visual motion perception for static visual stimuli. Two identical visual stimuli placed side by side were presented in alternation producing apparent motion perception and each stimulus was accompanied by a tone burst of a specific and unique frequency. After prolonged exposure to the apparent motion, the tones acquired driving effects for motion perception; a visual stimulus blinking at a fixed location was perceived as lateral motion. The effect lasted at least for a few days and was only observed at the retinal position that was previously exposed to apparent motion with the tones. Furthermore, the effect was specific to ear and sound frequency presented in the exposure period. These results indicate that strong association between visual motion and sound sequence is easily formed within a short period and that very early stages of sensory processing might be responsive loci for the current phenomenon.

Early visual cortex has been thought to be mainly involved in the detection of low-level visual features. Here we show that complex natural sounds can be decoded from early visual cortex activity, in the absence of visual stimulation and both when sounds are actually displayed and when they are merely imagined. Blindfolded subjects listened to three complex natural sounds (bird singing, people talking, traffic noise; Exp. 1) or received word cues (“forest”, “people”, “traffic”; Exp 2) to imagine the associated scene. fMRI BOLD activation patterns from retinotopically defined early visual areas were fed into a multivariate pattern classification algorithm (a linear support vector machine). Actual sounds were discriminated above chance in V2 and V3 and imagined sounds were decoded in V1. Also cross-classification, ie, training the classifier to real sounds and testing it to imagined sounds and vice versa, was successful. Two further experiments showed that an orthogonal working memory task does not interfere with sound classification in early visual cortex (Exp. 3), however, an orthogonal visuo-spatial imagery task does (Exp. 4). These results demonstrate that early visual cortex activity contains content-specific information from hearing and from imagery, challenging the view of a strict modality-specific function of early visual cortex.

Observer's active control of visual stimulus movement by the use of computer mouse reduces the flash-lag effect (Ichikawa & Masakura, 2006, Vision Research). This reduction of the flash-lag effect in visual motion depends upon directional relationship between hand movement and visual stimulus movement (Ichikawa & Masakura, 2010 AP&P). These studies indicate that the reduction of the flash-lag effect depends upon the sensory-motor learning of the directional consistency between the hand movement and stimulus movement in everyday-computer use. In this study, we examined how directional relationship between hand movement and stimulus change, which is not involved in every-computer use, affects the reduction of the flash-lag effect if there is no inevitable directional relationship between hand movement and stimulus change. While the luminance of the visual stimulus was controlled by the use of computer-mouse, a flash was presented. Observer judged which of the flash and luminance change stimulus is lighter. We found significant reduction of the flash-lag effect only when the directional relationship between hand movement and stimulus change was fixed. The results suggest that the proprioceptive signal to control the visual stimulus would facilitate the visual processing if the directional relationship between hand movement and stimulus change is consistent.

Active control over visual targets has been shown to influence spatial aspects of visual perception (eg, flash lag effect: Ichikawa & Masakura, 2006). We investigated whether or not observers' manual control over target motion affects the pattern of perceptual biases in audio/visual stream/bounce (Sekuler, Sekuler & Lau, 1997) displays. Participants completed four conditions: 1) zero control 2) full manual control over the targets via a concurrent left to right movement of a computer mouse 3) participants thought they were controlling the targets but were not 4) participants moved the mouse to accompany the target motion but were aware this motion was unrelated to the motion on the display. Participants responded “stream” or “bounce” at the end of each sequence when a sound was present or absent when the targets superimposed. Bouncing dominated when a sound was presented in the zero control condition replicated previous stream/bounce studies. The effect was reduced in the remaining conditions, though no sound trials elicited significantly fewer bounce reports than sound trials. These results support and extend our previous work (Grove & Sakurai, 2009) showing that the influence of a sound on the resolution of a motion sequence persists when proprioceptive information is consistent with streaming.

When we manipulate an object by hand, the movements of the object are produced with the visual and haptic movements of our hands. Studies of multimodal perception show the interaction between touch and vision in visual motion perception(1,2). The influence of touch on visual motion perception is shown by the fact that adaptation to tactile motion across the observer's hand induces a visual motion aftereffect, which is a visual illusion in which exposure to a moving visual pattern makes a subsequently viewed stationary visual pattern appear to move in the opposite direction(2). This visuo-tactile interaction plays an important role in skillful manipulation(3). However, it is not clear how haptic information influences visual motion perception. We measured the strength of a visual motion aftereffect after visuo-haptic adaptation to a windmill rotated by observers. We found that the visual motion aftereffect was enhanced when observers actively rotated the windmill. The motion aftereffect was not enhanced when the observer's hand was passively moved. Our results suggest the presence of a visual motion system that is linked to the intended haptic movements.

The perception of one's body includes distinguishing one's body from other objects and creating an internal body representation. In this talk I will review evidence suggesting that localizing a tactile stimulus on the body surface involves a body representation that is at least partially coded in visual coordinates and that the view of one's body needs to be synchronous in time and aligned in space with non-visual information to be treated as one's own. The former claim comes from the effect of eye and head position on localizing a touch on the arm or torso. The latter claim comes from experiments in which the perspective from which the hand or head was viewed was altered using an electronic mirror. When the perspective was consistent with viewing one's own body (either directly or in a mirror) subjects were more sensitive at detecting temporal asynchronies between visual and non-visual cues to movement of that body part. These results will be discussed in terms of allocentric and egocentric coding of the body.

Perceiving the direction of self-motion is typically a multisensory process. The most effective cue to detect the observer's heading direction is the visual optic-flow pattern. In the initial period of body movement, however, the vestibular sense is another effective cue to detect the direction in which one's body started to move. Here I report our recent research on the perception of self-motion, 1) when observers are passively experiencing their real somatic motion in different body posture, and 2) when observers are actively moving their bodies forward and backward. Previously, we reported that when upright observers passively experience real linear oscillatory somatic motion (leftward/rightward or forward/backward) while viewing orthogonal visual optic flow patterns (translating or expanding/contracting), their perceived body motion direction is intermediate to those specified by visual and vestibular information individually (Sakurai et al., 2002, ACV; 2003, ECVP; Sakurai et al., 2010, VSS; Kubodera et al., 2010, APCV). We then generalized those findings exploring other visual/vestibular combinations, investigating when the vertical axis of body coordinates is orthogonal to the gravity axis. Observers lay supinely and reported their perceived direction of self-motion, experiencing real upward/downward or leftward/rightward motion in body coordinates, while viewing orthogonal optic-flow patterns that were phase-locked to the swing motion. The results are very similar to our previous reports, but for combinations of body motion with visual expanding/contracting optic-flow, some observers' judgments were vision-only or vestibular-only, suggesting that multimodal integration in this context is an either-or process for these observers. Compared to our previous reports, one possible reason for this weighted combination failure is the discrepancy between body coordinates and gravity coordinates (Sakurai et al., 2011, ECVP). We recently extended those studies to active somatic motion, measuring the angular shift in body direction after active body motion while viewing synchronized orthogonal optic flow. Observers viewed the stimulus of translating leftward (rightward) random-dots for 30 seconds through a face-mounted display, while actively stepping forward and backward such that their forward body movement was synchronized with the random-dot translational motion. Observers' body direction was measured before and after each trial, and was compared with that of control condition (random noise). Translational optic flow induced shifts in body direction that were opposite to shifts in perceived direction with passive viewing in our previous reports. Observers may have compensated their body motion in response to perceived direction shifts similar to those we reported for passive viewing.

In many everyday situations, our senses are bombarded by numerous different unisensory signals at any given time. In order to gain the most veridical, and least variable, estimate of environmental stimuli/properties, we need to combine the individual noisy unisensory perceptual estimates that refer to the same object, while keeping those estimates belonging to different objects or events separate. How, though, does the brain ‘know’ which stimuli to combine? Traditionally, researchers interested in the crossmodal binding problem have focused on the role that spatial and temporal factors play in modulating multisensory integration. However, crossmodal correspondences between various unisensory features (such as between auditory pitch and visual size) may provide yet another important means of constraining the crossmodal binding problem. A large body of research now shows that people exhibit consistent crossmodal correspondences between many stimulus features in different sensory modalities. So, for example, people will consistently match high-pitched sounds with small, bright objects that are located high up in space. In this talk, the latest literature is reviewed. We will argue that crossmodal correspondences need to be considered alongside semantic and spatiotemporal congruency, among the key constraints that help our brains to solve the crossmodal binding problem. Crossmodal correspondences will also be distinguished from synaesthesia.

Although conceptually similar, sensory binding theory is based on oscillatory synchronization while multisensory integration is largely grounded on multisensory neurons. Many multisensory neurons show strong enhancement of response when both senses are stimulated, but the mechanism is obscure. This is a potential bridge between binding and sensory integration because oscillatory synchronization can enhance synchronized firing rates. Consider two independent excitatory neurons, each driven by a different sensory input. For synaptic coupling with a small delay, they can synchronize at a firing rate that lies well above either of their independent activities. We use this model to simulate behavior of a pair of “Enhanced Single Modality” (ESM) neurons like those in rattlesnake optic tectum, driven by stimulation in the eyes and infrared pits (Newman & Hartline, 1981). If these units both drive a common third cell, it will behave like an “Enhanced OR” neuron: it could be driven by either infrared or visible, but will fire at a higher rate when both of its inputs are present. Although the rattlesnake binds two visual modalities, the binding model behaves like superadditive multisensory units in Superior Colliculus (Stein & Meredith, 1993) and exhibits the intensity-dependent enhancement of Meredith & Stein's (1983) Principle of Inverse Effectiveness.

In vision, it is known that humans use summary statistics to efficiently perceive and encode the ‘gist’ of groups of features. For instance, after viewing a set of differently sized circles, people can reliably estimate the average circle size, often more accurately than they can identify an individual member of the set (Ariely, 2001). Summary statistical encoding (ie, ensemble coding) is common in visual processing, having been demonstrated not only for low-level visual features (eg, size and orientation) but also for high-level features such as facial expression (Haberman & Whitney, 2007). Here, we present evidence that ensemble coding is operative in audition. Specifically, participants were able to estimate the mean frequency of a set of logarithmically spaced pure tones presented in a temporal sequence, even though they performed poorly when asked to identify an individual member of the set (an identification task) or identify a tone's position in the set (a localization task). This suggests that ensemble coding is not limited to visual processing and instead is an important, multisensory mechanism for extracting useful information from groups of objects.

We have recently demonstrated that alternating left-right sound sources induce motion perception to static visual stimuli along the horizontal plane (SIVM: sound-induced visual motion perception, Hidaka et al., 2009). The aim of the current study was to elucidate whether auditory motion signals, rather than auditory positional signals, can directly contribute to the SIVM. We presented static visual flashes at retinal locations outside the fovea together with a lateral auditory motion provided by a virtual stereo noise source smoothly shifting in the horizontal plane. The flashes appeared to move in the situation where auditory positional information would have little influence on the perceived position of visual stimuli; the spatiotemporal position of the flashes was in the middle of the auditory motion trajectory. Furthermore, the auditory motion altered visual motion perception in a global motion display; in this display, different localized motion signals of multiple visual stimuli were combined to produce a coherent visual motion perception so that there was no clear one-to-one correspondence between the auditory stimuli and each visual stimulus. These findings suggest the existence of direct interactions between the auditory and visual modalities in motion processing and motion perception.

Ranging of auditory objects relies on several acoustic cues and is possibly modulated by additional visual information. Sound pressure level can serve as a cue for distance perception because it decreases with increasing distance. In this agnetoencephalography (MEG) experiment, we tested whether psychophysical loudness judgment and N1m MEG responses are modulated by visual distance cues. To this end, we paired noise bursts at different sound pressure levels with synchronous visual cues at different distances. We hypothesized that noise bursts paired with far visual cues will be perceived louder and result in increased N1m amplitudes compared to a pairing with close visual cues. The rationale behind this was that listeners might compensate the visually induced object distance when processing loudness. Psychophysically, we observed no significant modulation of loudness judgments by visual cues. However, N1m MEG responses at about 100 ms after stimulus onset were significantly stronger for far versus close visual cues in the left auditory cortex. N1m responses in the right auditory cortex increased with increasing sound pressure level, but were not modulated by visual distance cues. Thus, our results suggest an audio-visual interaction in the left auditory cortex that is possibly related to cue integration for auditory distance processing.

Our ability to maintain orientation to the upright is a result of multi-sensory integration, but little is known about how our subjective vertical interacts with our perceived world. We measured observers' subjective vertical in the same environment where we observed the Hong Kong Peak Tram Illusion. We tested whether the perceived tilt of a physically vertical building is a result of observers' misjudgment of their subjective vertical position. Six observers sat in the tram and held a stick on the bench to indicate their subjective vertical on both upward and downward trips. In a separate trip, they did the same measurement with eyes closed. An assistant marked the angle of the stick relative to gravity with a rotary pitch. Observers' reclining position, which matched with the slope of the mountain, was measured simultaneously by another pitch. We found observers' subjective vertical was biased away from gravitational vertical, which linearly increased with mountain slope. A blind-fold reduced this bias to half, suggesting visual information is crucial for our judgment of subjective vertical. However, observers' illusory tilt of buildings did not reflect the same trend, implying the illusion cannot be fully accounted for by observers' misjudgment of subjective vertical.

In a previous behavioural study, we reported that a particular context of stimulus congruency influenced audiovisual interactions. In particular, audiovisual interaction, measured in terms of congruency effect, was reduced when a high proportion of incongruent trials was presented. We argued that this modulation was due to changes in participants' control set as a function of the context of congruency, with greater control applied when most of the trials were incongruent. Since behavioural data do not allow to specify the level at which control was affecting audiovisual interaction, we conducted an event-related potentials (ERPs) study to further investigate each context of audiovisual congruency. Participants performed an audiovisual congruency task, where the stimulus onset could be present on two different contexts mixed at random: a high proportion congruent context and a low proportion congruent context. The context manipulation was found to modulate brain ERPs related to perceptual and response selection processes, ie, the N2 and P3 components. The N2 amplitude was larger for the less common trials on both high and low congruent proportion contexts, while the P3 amplitude and latency were differentially modulated by incongruent trials on the two contexts.

In active echolocation, reflections from self-generated acoustic pulses are used to represent the external environment. This ability has been described in some blind humans to aid in navigation and obstacle perception[1-4]. Echoic object representation has been described in echolocating bats and dolphins[5,6], but most prior work in humans has focused on navigation or other basic spatial tasks[4,7,8]. Thus, the nature of echoic object information received by human practitioners remains poorly understood. In two match-to-sample experiments, we tested the ability of five experienced blind echolocators to identify objects haptically which they had previously sampled only echoically. In each trial, a target object was presented on a platform and subjects sampled it using echolocation clicks. The target object was then removed and re-presented along with a distractor object. Only tactile sampling was allowed in identifying the target. Subjects were able to identify targets at greater than chance levels among both common household objects (p < .001) and novel objects constructed from plastic blocks (p = .018). While overall accuracy was indicative of high task difficulty, our results suggest that objects sampled by echolocation are recognizable by shape, and that this representation is available across sensory modalities.

Integrating emotional information from multiple sensory modalities is generally assumed to be a pre-attentive process (de Gelder et al., 1999). This assumption, however, presupposes that the integrative process occurs independent of attention. Using event-potentials (ERP) the present study investigated whether the neural processes underlying the integration of dynamic facial expression and emotional prosody is indeed unaffected by attentional manipulations. To this end, participants were presented with congruent and incongruent face-voice combinations (eg, an angry face combined with a neutral voice) and performed different two-choice tasks in four consecutive blocks. Three of the tasks directed the participants' attention to emotion expressions in the face, the voice or both. The fourth task required participants to attend to the synchronicity between voice and lip movements. The results show divergent modulations of early ERP components by the different attentional manipulations. For example, when attention was directed to the face (or the voice), incongruent stimuli elicited a reduced N1 as compared to congruent stimuli. This effect was absent, when attention was diverted away from the emotionality in both face and voice suggesting that the detection of emotional incongruence already requires attention. Based on these findings, we question whether multisensory integration of emotion occurs indeed pre-attentively.

The Visual Word Form Area (VWFA) is a ventral-temporal-visual area that develops expertise for visual reading. It encodes letter-strings irrespective of case, font, or location in the visual-field, with striking anatomical reproducibility across individuals. In the blind, reading can be achieved using Braille, with a comparable level-of-expertise to that of sighted readers. We investigated which area plays the role of the VWFA in the blind. One would expect it to be at either parietal or bilateral occipital cortex, reflecting the tactile nature of the task and crossmodal plasticity, respectively. However, according to the notion that brain areas are task specific rather than sensory-modality specific, we predicted recruitment of the left-hemispheric VWFA, identically to the sighted and independent of visual experience. Using fMRI we showed that activation during Braille reading in congenitally blind individuals peaked in the VWFA, with striking anatomical consistency within and between blind and sighted. The VWFA was reading-selective when contrasted to high-level language and low-level sensory controls. Further preliminary results show that the VWFA is selectively activated also when people learn to read in a new language or using a different modality. Thus, the VWFA is a mutlisensory area specialized for reading regardless of visual experience.

Several studies using visual objects defined by luminance have reported that the auditory event must be presented 30 to 40 ms after the visual stimulus to perceive audiovisual synchrony. In the present study, we used visual objects defined only by their binocular disparity. We measured the optimal latency between visual and auditory stimuli for the perception of synchrony using a method introduced by Moutoussis & Zeki (1997). Visual stimuli were defined either by luminance and disparity or by disparity only. They moved either back and forth between 6 and 12 arcmin or from left to right at a constant disparity of 9 arcmin. This visual modulation was presented together with an amplitude-modulated 500 Hz tone. Both modulations were sinusoidal (frequency: 0.7 Hz). We found no difference between 2D and 3D motion for luminance stimuli: a 40 ms auditory lag was necessary for perceived synchrony. Surprisingly, even though stereopsis is often thought to be slow, we found a similar optimal latency in the disparity 3D motion condition (55 ms). However, when participants had to judge simultaneity for disparity 2D motion stimuli, it led to larger latencies (170 ms), suggesting that stereo motion detectors are poorly suited to track 2D motion.

Auditory and visual spatial representations are produced by distinct processes, drawing on separate neural inputs and occurring in different regions of the brain. We tested for a bimodal spatial representation using a spatial increment discrimination task. Discrimination thresholds for synchronously presented but spatially separated audiovisual stimuli were measured for base separations ranging from 0° to 45°. In a dark anechoic chamber, the spatial interval was defined by azimuthal separation of a white-noise burst from a speaker on a movable robotic arm and a checkerboard patch 5° wide projected onto an acoustically transparent screen. When plotted as a function of base interval, spatial increment thresholds exhibited a J-shaped pattern. Thresholds initially declined, the minimum occurring at base separations approximately equal to the individual observer's detection threshold and thereafter rose log-linearly according to Weber's law. This pattern of results, known as the ‘dipper function’, would be expected if the auditory and visual signals defining the spatial interval converged onto an early sensory filter encoding audiovisual space. This mechanism could be used to encode spatial separation of auditory and visual stimuli.

Can auditory input influence the sensory processing of visual information? Many studies have reported cross-modal enhancement in visual tasks, but the nature of such gain is still unclear. Some authors argue for ‘high-order’ expectancy or attention effects, whereas others propose ‘low-order’ stimulus-driven multisensory integration. The present study applies a psychophysical analysis of reaction time distributions in order to disentangle sensory changes from other kind of high-order (not sensory-specific) effects. Observers performed a speeded simple detection task on Gabor patches of different spatial frequencies and contrasts, with and without accompanying sounds. The data were adjusted using chronometric functions in order to separate changes is sensory evidence from changes in decision or motor times. The results supported the existence of a stimulus unspecific auditory-induced enhancement in RTs across all types of visual stimuli, probably mediated by higher-order effects (eg, reduction of temporal uncertainty). Critically, we also singled out a sensory gain that was selective to low spatial frequency stimuli, highlighting the role of the magno-cellular visual pathway in multisensory integration for fast detection. The present findings help clarify previous mixed findings in the area, and introduce a novel form to evaluate cross-modal enhancement.

Environments typically convey contextual information via several different sensory modalities. Here, we report a study designed to investigate the crossmodal semantic modulation of visual perception using the binocular rivalry paradigm. The participants viewed a dichoptic figure consisting of a bird and a car presented to each eye, while also listening to either a bird singing or car engine revving. Participants' dominant percepts were modulated by the presentation of a soundtrack associated with either bird or car, as compared to the presentation of a soundtrack irrelevant to both visual figures (tableware clattering together in a restaurant). No such crossmodal semantic effect was observed when the participants maintained an abstract semantic cue in memory. We then further demonstrate that crossmodal semantic modulation can be dissociated from the effects of high-level attentional control over the dichoptic figures and of low-level luminance contrast of the figures. In sum, we demonstrate a novel crossmodal effect in terms of crossmodal semantic congruency on binocular rivalry. This effect can be considered a perceptual grouping or contextual constraint on human visual awareness through mid-level crossmodal excitatory connections embedded in the multisensory semantic network.

Humans are equipped with multiple sensory channels, jointly providing both redundant and complementary information. A primary challenge for the brain is therefore to make sense of these multiple sources of information and bind together those signals originating from the same source while segregating them from other inputs. Whether multiple signals have a common origin or not, however, must be inferred from the signals themselves (causal inference, cf. “the correspondence problem”). Previous studies have demonstrated that spatial coincidence, temporal simultaneity, and prior knowledge are exploited to solve the correspondence problem. Here we demonstrate that cross-correlation, a measure of similarity between signals, constitutes an additional cue to solve the correspondence problem. Capitalizing on the well-known fact that sensitivity to crossmodal conflicts is inversely proportional to the strength of coupling between the signals, we measured sensitivity to crossmodal spatial conflicts as a function of the cross-correlation between audiovisual signals. Cross-correlation (time-lag 0ms) modulated observers' performance, with lower sensitivity to crossmodal conflicts being measured for correlated than for uncorrelated audiovisual signals. The current results demonstrate that cross-correlation promotes multisensory integration. A Bayesian framework is proposed to interpret the present results whereby stimulus correlation is represented on the prior distribution of expected crossmodal co-occurrence.

The ability of our sensory systems to merge sensory information from distinct modalities is remarkable. One stimulus characteristic utilized in this operation is temporal coincidence. Auditory and visual information are integrated within a narrow range of temporal offsets, known as the temporal binding window (TBW), which varies between individuals, stimulus type, and task. In this series of experiments, we assessed the relationship within individuals between the width of their TBW and their ability to integrate audiovisual information. The TBW was measured through a perceived subjective simultaneity task. In conjunction with this, we measured each individual's ability to integrate auditory and visual information with two multisensory illusions, the McGurk effect and Flashbeep illusion. The results from these studies demonstrate that the TBW is highly correlated with the individual's ability to integrate. These relationships were seen in only the right TBW, in which visual presentations preceded auditory presentations, a finding that is ecologically logical. However, differences were seen between the two illusory conditions, where the McGurk effect was stronger in individuals with narrow TBWs, again, an ecologically logical finding. The opposite relationship was seen with flashbeep illusion, possibly due to inherent asynchronies in the illusion.

A fission illusion (also named a double—flash illusion) is a famous phenomenon of audio-visual interaction, in which a single brief flash is perceived as two flashes when presented simultaneously with two brief beeps (Shames, Kamitani, & Shimojo, 2000; 2002). The fission illusion has been investigated using relatively simple visual stimuli like single circle. Thus the illusion has not been examined by using complex visual stimuli. Markovic & Gvozdenovic (2001) reported that the processing of complex visual stimuli tends to be delayed. Therefore, the complexity of visual stimuli may affect the occurrence rate of the fission illusion, since this illusion is generated in the process that copes with visual and auditory stimuli in a short time. The present study examined the differences in illusory occurrence rates by manipulating the complexity of visual stimuli. We used the patterns proposed by Garner & Clement (1963) to control the complexity. The results indicated that it was more difficult to induce the fission illusion by using complex visual stimuli than it was by using simple stimuli. Thus, the present study suggested that the occurrence rate of the fission illusion differed depending on the perceptual efficiency in the coding process of visual stimuli. This study was supported by Grant-in-Aid for Specifically Promoted Research (No. 19001004).

These experiments examined the ability to track a moving target with our heads under various stimulus conditions and modalities. While previous studies [1,2] have concentrated on eye tracking within the frontal region; we extended the modes of tracking to include auditory and auditory-visual stimuli and in a significantly wider locus of space. Using a newly developed system that combines high fidelity virtual auditory space with a high speed LED strip we were able to examine head tracking behaviour in a 100° arc around a subject with velocities from ±20°/s to ±110 °/s. This allows us to derive behavioural norms for head tracking, compare differences in tracking ability across the modalities, and determine if cross modal facilitation occurs. Preliminary results show that subjects were better able to track a visual and bimodal target than an auditory one, as evident in the smaller average RMS error (visual = 4°, bimodal = 4.6°, auditory = 7°) and shorter lag (visual = 5.5°, bimodal = 5.9° and, auditory = 8.9°). Furthermore, tracking ability was influenced by stimulus speed, especially in the unimodal auditory situation where a significant increase in both RMS error and lag for speeds >80°/s was observed.

The onset position of a visual moving object is mislocalized in the direction of subsequent motion. Although recent studies have proved that visual and auditory motion processing is closely related to each other, the interaction of motion and position coding in a cross-modal situation was largely unknown. The present study explored whether the position of a visual flash was mislocalized in the direction of subsequent auditory motion. Observers first memorized the position of a probe and then compared this position with the position of a visual flash, which acted as a target; the target temporally lagged the probe by 600 ms. The auditory motion was defined on the basis of the interaural level difference, and lasted for 100 ms. The onset of the target was synchronized with the onset of the auditory motion. As a result, the target location was significantly mislocalized in the direction of subsequent auditory motion. The results indicate that the interaction of motion and position coding occurs beyond the sensory modality.

In metacontrast masking, target visibility is impaired by a subsequent non-overlapping contour-matched mask, a phenomenon attributed to low-level processing. Previously we found that sound could reduce metacontrast masking (Yeh & Chen, 2010), and yet how it exerts its effect and whether the sound-triggered attention system plays a major role remains unsolved. Here we examine whether the sound-facilitatory effect is caused by alertness, attentional cueing, or audiovisual integration. Two sounds were either presented simultaneously with the target and the mask respectively, or one preceded the target by 100 ms and the other followed the mask 100 ms afterwards. No-sound and one-sound conditions were used for comparison. Participants discriminated the truncated part (up or down) of the target, with four target-to-mask SOAs (14ms, 43ms, 114ms, and 157ms) mixedly presented. Results showed that the attentional cueing effect was evident when compared to the one-leading sound condition. Additionally, selective (rather than overall) improvement in accuracy and RT as a function of SOA was found with synchronized sound than without, suggesting audiovisual integration but not alertness. The audio-visual integration effect is attributed to enhanced temporal resolution but not temporal ventriloquism.

We examined the novel crossmodal semantic congruency effect on bistable figures in which a static stimulus gives rise to two competing percepts that alternate over time. Participants viewed the bistable figure “my wife or my mother-in-law” while listening to the voice of an old woman or a young lady speaking in an unfamiliar language. They had to report whether they saw the old woman, the young lady, or a mixed percept. Robust crossmodal semantic congruency effects in the measures of the first percept and the predominance duration were observed. The possibilities that the participants simply responded to, and/or that they fixed at the location in favor of, the percept congruent with the sound that they happened to hear were ruled out. When the participants were instructed to maintain their attention to a specific view, a strong top-down modulation on the perception of bistable figure was observed, although the audiovisual semantic congruency effect still remained. These results thus demonstrate that top-down attention (ie,, selection and/or voluntary control) modulates the audiovisual semantic congruency effect. As the alternating percepts in bistable figures indicate competition for conscious perception, this study has important implications for the multifaceted interactions between multisensory processing, attention, and consciousness.

Using a crossmodal priming paradigm, this study investigated how the brain bound the spatial and semantic features in multisensory processing. The visual stimuli (pictures of animals) were presented after the auditory stimuli (sounds of animals), and the stimuli from different modalities may match spatially (or semantically) or not. Participants were required to detect the head orientation of the visual target (an oddball paradigm). The event-related potentials (ERPs) to the visual stimuli was enhanced by spatial attention (150–170 ms) irrespectively of semantic information. The early crossmodal attention effect for the visual stimuli was more negative in the spatial-congruent condition than in the spatial-incongruent condition. By contrast, the later effects of spatial ERPs were significant only for the semantic- congruent condition (250–300 ms). These findings indicated that spatial attention modulated early visual processing, and semantic and spatial features were simultaneously used to orient attention and modulate later processing stages.

Spatial frequency is a visual feature coded in V1, and temporal amplitude-modulation (AM) rate is an auditory feature coded in the primary auditory cortex; both are fundamental building blocks of perception. We demonstrate that processing of the auditory AM rate and visual spatial frequency are closely associated. Observers consistently and linearly matched a specific visual spatial frequency to a specific auditory AM rate. In addition, AM rates modulate visual attention and awareness in a visual frequency-specific manner. When a pair of Gabors with different spatial frequencies was simultaneously presented with an AM rate perceptually matched to one of the Gabors, observers detected a change more rapidly when it occurred on the matched Gabor, suggesting that the AM rate guided attention to the matched spatial frequency. When a pair of Gabors with different spatial frequencies competed to generate binocular rivalry, presenting an AM rate matched to one of the Gabors increased the proportion of perceptual dominance of the matched Gabor, suggesting that auditory amplitude modulation boosts signal strength for the matched visual spatial frequency. Additional results suggest that the association between auditory AM rate and visual spatial frequency develops through the multisensory experience of manually exploring surfaces.

We examined the crossmodal effect of the presentation of a simultaneous sound on visual detection and discrimination sensitivity using the equivalent noise paradigm (Dosher & Lu, 1998). In each trial, a tilted Gabor patch was presented in either the first or second of two intervals consisting of dynamic 2D white noise with one of seven possible contrast levels. The results revealed that the sensitivity of participants' visual detection and discrimination performance were both enhanced by the presentation of a simultaneous sound, though only close to the noise level at which participants' target contrast thresholds started to increase with the increasing noise contrast. A further analysis of the psychometric function at this noise level revealed that the increase in sensitivity could not be explained by the reduction of participants' uncertainty regarding the onset time of the visual target. We suggest that this crossmodal facilitatory effect may be accounted for by perceptual enhancement elicited by a simultaneously-presented sound, and that the crossmodal facilitation was easier to observe when the visual system encountered a level of noise that happened to be close to the level of internal noise embedded within the system.

Multisensory integration has seldom been examined in adolescents. Thus, performances on an audiovisual discrimination task were compared between 19 adolescents (M=15:00 SD=0:04 years), 20 primary school-aged children (M=11:04 years, SD=0:05 years) and 22 young adults (M=26:02 years, SD=2:07 years) in quiet and in auditory, visual and audiovisual noise conditions. Stimuli comprised domestic animal exemplars. Motor reaction times (MRTs) and accuracy were recorded. Adolescents had equivalent false positive error rates but significantly lower false negative error rates than children. Children and adolescents had higher error rates than adults. MRTs differed significantly between all groups, being fastest for adults and slowest for children. Adults and children showed facilitation in all noise conditions, while adolescents showed equivalent, faster MRTs to visual and audiovisual than auditory targets in auditory noise. Coactivation was evident in all noise conditions for adults (probabilities of .05–.65 in quiet and auditory noise, .15–.65 in visual noise and .15–.25 in audiovisual noise) but was limited to quiet and auditory noise conditions for children (probabilities of .05–.15) and adolescents (probabilities .05–.65 and .15–.45, respectively). The results indicate that adult-like levels of multisensory integration develop by adolescence in quiet conditions but remain immature in unisensory and audiovisual noise.

Previous studies suggest that subjects gain modality-specific information across vision and tactile (Newell, Ernst, Tjan, & Bülthoff, 2001) and form a coherent percept (Ernst & Bülthoff, 2004). This study explored potential difference in perceiving real-world objects (cellular phones) uni-modally (visual or haptic) and bimodally. In phase 1, subjects provided verbal descriptions of 9 different phones while interacting in a visual, haptic, or cross-modal manner. In phase 2, a new group of subjects performed a card-sorting task using on the descriptions obtained in phase 1, with the instructions to sort descriptions based on similarity. A multidimensional scaling (MDS) analysis was applied to similarity matrices produced in phase 2. Stress values for MDS fits of varying dimensionality were similar for haptic, visual, and bimodal conditions, suggesting that different inspection conditions produced mental representations of similar complexity. Inspection of 2D plots of descriptors across the three condition suggests that dimensions of usability and style. Results imply that people extract largely redundant information across different sensory modalities while evaluating cellular phones.

The purpose of the present study was to clarify a long-term potentiation like effect in the corticomotor area after anodal transcranial direct current stimulation (tDCS), which had been done with motor imagery. Anodal tDCS was applied transcranially as an intervention to the left hand motor area in the right hemisphere for 15 min with the intensity of 1.0 mA during resting or motor imagery (MI) conditions. In the MI condition, subjects performed motor imagery of index finger abduction. Motor-evoked potentials (MEPs) were recorded from the first dorsal interossei (FDI) of the left hand before the intervention and at 0 min, 15 min, 30 min and 60 min after the intervention. The stimulus intensities of TMS were set at 1.05, 1.15, and 1.25 times the strength of the resting motor threshold (RMth). Increases of MEPs were detected with all intensities of TMS at 0 min and 15 min after tDCS with motor imagery. However, in the resting condition, MEP amplitudes were elevated only at 15 min after tDCS and at the highest TMS intensity of 1.25×RMth. The facilitatory effect induced by tDCS with motor imagery was significantly long-lasting and definite compared to that after tDCS without motor imagery.