New Directions in Mental-Imagery Research

Mental imagery, the voluntary retrieval and representation of sensory information from memory, has a fascinating biography. Historically, mental-imagery research suffered criticism because of methodological constraints caused by imagery’s inherent private nature. Recently, many objective research methods have been introduced that allow a more direct investigation into the mechanisms and neural substrates of mental imagery. These new methods have spurred numerous new discoveries, culminating in a flurry of impactful publications over the past few years.

Parallel to the exciting new work investigating imagery itself, many recent developments are illuminating the role of mental imagery in almost everything we do. With research on imagery’s role in everything from visual working memory, eyewitness memory, and moral dilemmas, to learning, mental disorders, and episodic memory, the field of mental imagery is rapidly expanding in both an intra- and interdisciplinary manner.

Although imagery played a distinct role in discussions of mental function for thousands of years, empirical work on imagery did not gain strong traction until the last 30 or 40 years. Despite this recent traction, mental-imagery research has still not enjoyed the same degree of investigative attention that other psychological topics have. For example, Figure 1 shows that the number of articles published each year that include the phrase “mental imagery” in the title, compared with those that include “visual attention” or “visual working memory,” is relatively low.

OPEN IN VIEWER

Fig. 1.

The number of published articles per year including the phrase “mental imagery,” “visual working memory,” or “visual attention” in the title (data sourced using PubMed; http://www.ncbi.nlm.nih.gov/pubmed).

This comparative lack of research publications is somewhat surprising, considering the ubiquity and functional relevance of metal imagery in so many of our everyday cognitive processes. One possible cause of the lack of scientific publications investigating imagery is the criticism aimed at the methodological constraints due to imagery’s subjective internal nature. An alternate explanation might simply be that authors prefer to use substitution phrases instead of “mental imagery.” Either way, such data speak to the acceptance and/or unpopularity of imagery research.

In the 1970s, cognitive psychologists started to develop clever methods to measure and study mental imagery objectively. Some of the early discoveries demonstrated a clear relationship between the content of mental images and the time it took to generate or manipulate them (Kosslyn, Ball, & Reiser, 1978; Shepard & Metzler, 1971). The larger the imagery manipulation, the longer it took to complete, suggesting a correspondence between imagery and physical space.

More recently, there has been a jump in brain-imaging work investigating mental imagery. A recent trend of analyzing the information content of fMRI patterns (instead of the mean amplitude change) has yielded interesting results. This work is often described as decoding because one of the more popular methods trains an algorithm to decode, or make a prediction about, the experimental condition or task, on the basis of the spatial pattern of the fMRI signal across a brain area (Tong & Pratte, 2012).

For example, Stokes, Thompson, Cusack, and Duncan (2009) could decode or predict which of two patterns subjects were imagining well above chance level, using fMRI patterns in sensory brain areas. They could even decode the content of imagery when the algorithm was “trained” on perceptual versions of the stimuli only, and not on the imagery data. Earlier imagery-decoding work demonstrated large variance between individuals in the degree of decoding (O’Craven & Kanwisher, 2000; Thirion et al., 2006), which has been discussed in terms of individual differences in imagery strength. More recent work has extended these principles to demonstrate the overlap in sensory representation during a visual working memory task and voluntary imagery (Albers, Kok, Toni, Dijkerman, & de Lange, 2013). Here, the content of either visual working memory or imagery could be decoded above chance level by training on either, or on perceptual representations of the same stimuli. This provides strong evidence that both of these voluntary processes (imagery and visual working memory) can involve overlapping sensory representations in early visual cortex.

An intriguing question regarding imagery is not only how the brain voluntarily represents or maintains sensory representations, but also how these representations can be manipulated and utilized. A recent fMRI study demonstrated that different cortical networks are involved in maintaining and manipulating imagined sensory stimuli (Schlegel et al., 2013; but see also Mellet, Tzourio, Denis, & Mazoyer, 1995). The particular form of manipulation the researchers investigated included both constructive and destructive dynamic imagery manipulations, the type that might be utilized during creative problem solving. Such work tantalizingly hints at how we might bring together different imagined elements to form compound scenes or objects in the mind’s eye.

In recent work from 2013, researchers combined multiple techniques to decode visual dream imagery as individuals were falling asleep (Horikawa, Tamaki, Miyawaki, & Kamitani, 2013). This science-fiction-like result demonstrates that humans have sensory representations of visual objects in early perceptual cortex during sleep. In other words, visual experience, whether wakeful or during sleep, involves overlapping activity in early sensory brain regions. This work points to the possibility of perhaps one day decoding people’s entire dreams. This research is not limited to fundamental science; similar techniques have already been used in an applied manner to detect consciousness in vegetative-state patients (Owen & Coleman, 2008). Here, researchers asked patients to imagine one of two scenarios, and in some cases, they could decode sustained fMRI response as a means to communicate with the patients. These are all exciting discoveries with potentially impactful implications, but they also speak to the nature of imagery itself. Together, these findings suggest that the neural representation of voluntary visual imagery can overlap with sensory representations in early visual cortex. Hence, voluntary imagery might quite literally take on a pictorial format, as such early brain areas maintain the pictorial format of visual information.

While the “decoding” brain-imaging work mentioned above has made valuable contributions to our understanding of mental imagery, it is also worth noting the inherent limitations of its methods. For example, the brain mechanisms that produce higher levels of decoding in fMRI data are often unclear, given that an increase in decoding accuracy really only demonstrates that representations are more separable. In other words, without a clear hypothesis regarding the encoding of a stimulus, mapping decoding accuracy to the underlying mechanisms can be a tricky business (for a taxonomy of the pros and cons, see Serences & Saproo, 2012).

There is a long history of research in behavioral psychophysics showing that voluntary imagery can interfere with perception and perceptual processes. More recent work has demonstrated that imagery can facilitate subsequent perception (Pearson, Clifford, & Tong, 2008). By separating the period of imagery generation and perception in time, the effects of imagery can be examined without the potential confounds of attention (Carrasco, Ling, & Read, 2004). This work has demonstrated that when individuals imagine one of two patterns, that pattern has a much higher probability of being perceptually dominant in a subsequent brief binocular-rivalry presentation (Pearson et al., 2008; Pearson, Rademaker, & Tong, 2011). In other words, the content of the mental image primes subsequent dominance in binocular rivalry—it changes visual awareness of the rivalry display (see Fig. 2a). Binocular rivalry is a visual phenomenon that occurs when two different visual stimuli are presented, one to each eye, such that they appear to coexist at the same visual location. One pattern tends to be dominant over the other, forcing it out of awareness. Binocular rivalry has been a hugely popular tool to study visual awareness in recent times (Tong, Meng, & Blake, 2006). However, this work on imagery has used rivalry as a tool to measure the sensory strength, or “visual energy,” of mental imagery, enabling individual episodes of imagery to be assessed in an indirect and objective sensory manner. This discovery is also interesting in its own right because it demonstrates that what we imagine can literally change how we see the world.

OPEN IN VIEWER

Fig. 2.

An illustration of the binocular-rivalry technique used to measure mental imagery and the findings gleaned from using it. In the binocular-rivalry technique (a), the subject imagines one of two possible stimuli. After a period of time, the two stimuli are presented, one to each eye, such that they appear to coexist at the same visual location. The imagined stimulus tends to be dominant over the other, forcing it out of awareness. The table (b) shows some of the empirical findings from work using the binocular-rivalry technique. Red Xs indicate that the process is not associated with the characteristic; the yellow X indicates that strong evidence for association is lacking. (Please note that this table is intended to serve as a summary guide for the research findings for this specific technique and is in no way representative of the entire literature for imagery, perception, or attention.)

These methods have established that visual imagery can be local in visual retinotopic space, becomes stronger with longer durations, is disrupted by the presence of uniform background luminance, and is specific to a spatial orientation (Chang, Lewis, & Pearson, 2013; Pearson et al., 2008; Sherwood & Pearson, 2010). Imagery as measured using the binocular-rivalry technique was stable over time, even when performed for 1 hour per day over 5 days (Rademaker & Pearson, 2012). Together, these findings suggest that imagery’s priming of subsequent rivalry is not due to the cognitive penetrability of the process. In other words, the aforementioned patterns of data (e.g., precision in visual space and spatial orientation, stronger priming after longer durations) cannot be accounted for by any knowledge “naïve” participants might have regarding the aim of the experiments. The priming effects of imagery on subsequent rivalry were still observed when participants performed a challenging cognitive letter-detection task for 5 seconds after generating the mental image. This suggests that actively attending to or imagining the pattern at the time of the binocular-rivalry presentation is not necessary for these effects to occur. In summary, it seems that this method can be used to assess the sensory and cognitive characteristics of imagery, and these findings show that imagery can resemble weak perceptual stimulation.

Subsequent work has demonstrated that the sensory strength of imagery, measured using binocular rivalry, predicted subjective self-reports of imagery vividness (Pearson et al., 2011). Moreover, this relationship held on a trial-by-trial basis. When participants reported their feeling of imagery vividness for each individual trial, the degree of reported vividness predicted the likelihood that each period of imagery would bias subsequent rivalry. This demonstrated two interesting phenomena regarding visual imagery. First, the predictive relationship between vividness and the sensory strength of individual episodes of imagery suggests that we have good metacognition of our visual imagery. We know—or can feel—when we have generated perceptually strong imagery. Second, imagery strength varied within each participant (i.e., intra-individually) over time. We have known that imagery shows large differences between individuals for centuries. However, this work suggests that within a single individual, imagery strength can vary over time. In the same study, subjective reports of effort on each trial did not predict the likelihood that imagery would bias subsequent rivalry. Hence, the intra-individual variance does not seem to be due simply to variance of effort from one trial to the next.

The binocular-rivalry method’s assessment of visual imagery is indirect: The dependent measure is not imagery per se, but binocular-rivalry dominance. When participants are tested using this method, they do not need to report anything directly about their imagery. This is an important point to consider when studying the mechanisms of psychological or cognitive phenomena because people do not, or cannot, always accurately report on subjective sensations when asked to. While not requiring any direct reports of imagery, this method does require the subjective report of binocular-rivalry dominance. Hence, when using this method, we always include fake binocular-rivalry “catch trials” to test participants’ reporting criteria. For example, a catch trial might involve a nonambiguous or nonrivalrous (i.e., the same in each eye) fake rivalry stimulus made from blending the two rivalry patterns together, so that subjects see both patterns simultaneously. When participants see a perceptual blend of the two rivalry patterns like this—on any trial, not just on catch trials—they are required to report the percept as perceptually mixed. These catch trials provide a way to test for any nonperceptual bias in how these mixed stimuli are reported. If a participant reports a catch trial as red after imagining red, we then know that this participant has a nonperceptual or decisional bias, and hence their data will not speak directly to the sensory strength of imagery.

An important and perhaps underappreciated factor in utilizing binocular rivalry to assess mental imagery is sensory eye dominance and its effects on rivalry perception. Most people tend to have one eye that is slightly dominant, although sometimes the dominance can be quite strong. When someone has a strongly dominant eye, they will tend to see the binocular-rivalry stimulus that is presented to that eye for more time than the other pattern. Accordingly, we developed a method to assess eye dominance, using the rivalry stimuli, and then nullify the imbalance—for example, by adjusting stimulus contrast between the two eyes (Pearson et al., 2008). This is an important point, given that without such a method anyone with strong eye dominance would primarily see the rivalry pattern presented to that dominant eye and not the stimulus they had imagined. This could lead to a confound between eye dominance and imagery, and result in erroneous conclusions that those with strong eye dominance have weak or nonexistent imagery.

Recent imagery research has shed new light not only on the mechanisms of imagery itself, but also on the role imagery plays in other behavioral processes. For example, we recently utilized the binocular-rivalry method to investigate the role of visual imagery in visual working memory (Keogh & Pearson, 2011; but see also Borst, Ganis, Thompson, & Kosslyn, 2011; Borst, Niven, & Logie, 2011). We found that the sensory strength of imagery predicted visual working memory performance, but not iconic memory. Further, individuals with strong imagery were susceptible to passive sensory disruption during working memory storage (as they were during imagery generation), whereas those with weak imagery could maintain working memory performance during the attempts at sensory perturbation. These data suggest that individuals with strong imagery might use it to aid active visual mnemonic storage. This finding was supported by the recent fMRI study mentioned above, in which the content of mental imagery could be decoded when an algorithm was trained on the content of visual working memory (Albers et al., 2013).

It might be worthwhile here to differentiate visual imagery from other sensory-cognitive functions such as visual attention and visual working memory. A process is typically classified as visual imagery when someone voluntarily retrieves a sensory representation from memory that does not directly correspond to the concurrent incoming visual stimulus. This can be differentiated from visual attention, in which the individual is required to voluntarily attend to a given visual stimulus or part of it. Once an individual starts selectively attending to elements within a mental image, the semantics to differentiate imagery and attention become much more ambiguous. Likewise, research suggests that individuals who have strong imagery will use it to perform a visual working memory task, but individuals without imagery can still perform the memory task using different strategies and mechanisms. Accordingly, this definition classifies imagery as one of potentially many sensory-cognitive tools in our arsenal to aid in short-term memory function.

In many mental disorders, mental imagery plays both a symptomatic and a therapeutic role. For example, in disorders of anxiety, individuals suffer from traumatic and uncontrollable mental imagery. Imagery is often used to treat such disorders during forms of cognitive behavioral therapy as a means of imagined exposure or rescripting of the order and outcome of events (Foa, Steketee, Turner, & Fischer, 1980; Holmes, Arntz, & Smucker, 2007). However, we know relatively little about how imagery might interact with the mechanisms of such disorders. We recently demonstrated that the content of voluntary imagery can undergo associative learning, considered the backbone mechanism of disorders such as PTSD (Lewis, O’Reilly, Khuu, & Pearson, 2013). These data suggest that voluntary imagery can interact with the mechanisms of learning, much like it does with normal visual perception. Further experiments demonstrated that associative learning with voluntary imagery is specific to the spatial orientation of the imagery content. Just like the priming effect of imagery on subsequent binocular rivalry, associative learning with imagery seems to be orientation selective. We know from cross-species and multimethod research that only cells early in the visual-processing hierarchy are selective for spatial orientation. Hence, the orientation precision of associative learning with imagery indirectly suggests that the early visual cortex is important for such learning.

The above finding suggests that the effectiveness of imagery-based treatments for anxiety might be enhanced by focusing on the low-level sensory aspects of the distressing imagery symptoms as opposed to, say, the semantic content. Such sensory-focused implications fit well with current theories of intrusive imagery in psychopathologies (Brewin, Gregory, Lipton, & Burgess, 2010).

The recent advances in methods to measure mental imagery and the possibilities for predicting or even controlling imagery strength in clinical disorders are exciting prospects. In other areas of potential application, recent work has diagnosed mental imagery as potentially playing an important simulatory role in making moral decisions regarding the worth of sacrifice (Amit & Greene, 2012). This suggests that we might rely on imagined representations of scenarios to aid us in such dilemmas. It follows that tactical manipulation of imagery strength in some people might indirectly change moral judgments.

Together, the interplay between fundamental mechanistic research and the potential applications of mental imagery in functional and dysfunctional everyday life is an exciting current area of discovery. The recent upsurge in empirical work on mental imagery will hopefully continue, with theoretical and computational work bridging empirical research across multiple traditional disciplines.