What is an Emotion? A Connectionist Perspective

Are Emotions Generally Consistent or Variable?

Consider two instances of “fear” – either in the same person at two different times, or in two different people. Some researchers have noted that such instances often feature consistent responses – such as a widening of the eyes or an increased pulse rate – that are characteristic of many instances of fear (Fullana et al., 2016; Öhman & Wiens, 2003). More generally, researchers with basic emotion perspectives (Cowen & Keltner, 2017; Ekman, 1992; Ekman & Cordaro, 2011; Izard, 2007; Lench et al., 2011; Panksepp, 2004, 2007; Tracy & Randles, 2011; Tracy & Robins, 2004) have emphasized the ubiquitous consistency in response patterns in emotions of the same type across time and across people.

Other researchers have noted that experience and context often create variability in emotional responses. According to this view, two instances of “fear” may be dissimilar from one another due to differences in the specific contexts that produce these emotions (e.g. Crivelli et al., 2016; Russell, 1991). For example, some instances of fear may involve freezing in place, whereas others may involve rapid flight. More generally, researchers with constructed emotion perspectives (Averill, 1983; Barrett, 2006; Lindquist, 2013; Lindquist & Barrett, 2008; Mesquita & Frijda, 2016; Russell, 2003, 2005, 2009) have emphasized the ubiquitous variability of emotion responses. This variability is thought to arise from the influence of context-specific variables and prior-experience.

Many basic emotion and constructed emotion researchers refrain from endorsing the most-extreme, absolute version of their respective perspectives. For example, some basic emotion researchers have noted that emotion-instances are distinct from emotion-schemas which refer to the dynamic interplay of basic emotions and higher order cognition (Izard, 2007). This distinction allows for learning-based effects to cause variances in otherwise consistent emotion instances. Similarly, some constructed emotion researchers have noted that there may be emotion concepts shared across a broad range of human populations (Mesquita & Frijda, 1992). This consistency of concepts may result in some consistency in otherwise varying emotion instances.

Thus, even when agreeing about deviations from the absolute versions of their respective viewpoints (as described above), basic emotion and constructed emotion researchers differ on the underlying mechanics responsible for this deviation.

Do Emotions Generally Unfold Serially or in Parallel?

Consider how the various response types in an instance of “fear” may unfold. Some researchers have noted that such an instance typically arises with an initial appraisal of a situation (Ellsworth, 2013) and may involve a sequential unfolding of sub-component processes (e.g. Aue et al., 2007; Grandjean & Scherer, 2008). More generally, researchers with the appraisal-based perspective (Arnold, 1960; Lazarus, 1991; Smith & Ellsworth, 1985; Sander et al., 2005; Scherer, 1988) have emphasized that emotion instances often (but not necessarily always) feature an initial appraisal and other responses follow that initial appraisal.

Other researchers have noted that emotion instances may begin without an appraisal (Parkinson, 2007) and often involve interactive and reciprocal influences between response features. For example, researchers studying facial feedback (e.g., Coles et al., 2019) have suggested that appraisal-less facial expressions may initiate and profoundly affect emotional responding. Similarly, researchers have found that merely directing participants to assume facial expressions related to certain emotion categories (i.e. without appraisal) can result in autonomic responses that typically occur in emotion instances of that category (Ekman et al., 1983).

Towards an Integrative Framework

If one accepts that emotions are either consistent or variable, or that they are either appraisal-led and sequential or diversely led and interactive, one must reject a large corpus of empirical data that is not consistent with the ‘correct’ perspective. Another possibility, however, and the one we develop here, is that emotions are consistent in some circumstances and variable in others. Similarly, they unfold serially from appraisals in some circumstances, and not in others. On this view, the key question is not whether a particular perspective is correct or incorrect, but rather when (i.e. in what circumstances) a particular perspective is applicable and when it is not. Developing such a possibility requires an integrative framework that can identify the particular circumstances in which an emotion instance has some properties but not others.

To create such a framework, we draw upon the Parallel Distributed Processing (PDP) tradition (Rumelhart & McClelland, 1986), which features local computation, heterarchical processing, and the emergence of behavior. In particular, we rely upon the Interactive Activation and Competition (IAC) architecture (McClelland, 1981; McClelland & Rumelhart, 1981) that embodies a specific set of assumptions within the broader PDP tradition.

We build our argument over two sections. In Section 1, we describe the key principles of the IAC framework for Emotion (IAC-E) and describe the occurrence of an emotion instance in the IAC-E. In Section 2, we explore how emotion instances represented in the IAC may be consistent in some circumstances and variable in others and how they may be appraisal-initiated and sequential in some circumstances and input-agnostic and interactive/reciprocal in others. We use IAC-E based simulations to provide algorithmic explanations for several empirical phenomena related to emotion. We conclude by inviting consideration of the IAC-E as integrative and generative framework for emotion research.

Key Principles of the IAC-E

Like other artificial neural networks, the IAC-E framework assumes that all processing occurs within neuron-like elements called units. These units influence each other via weighted connections. All connections in the IAC-E are bidirectional, and may be positive or negative.

All knowledge is resident within these weighted connections. Learning in the network occurs either by creating new connections between units or by updating existing connection weights.

The units (depicted by small circles in Figure 1) are organized into pools. There are two types of pools: input pools and hidden pools. Input pools can receive input from outside the network – however their final activation reflects both the input provided and the activation of other units that they are connected with. The activation of hidden units is completely determined by the activation of other units that they are connected with (they cannot receive external input). Positive weights tend to increase activation in the receiving unit, whereas negative tend to decrease activation in the receiving unit. Positive weights connect units belonging to different pools (i.e. input and hidden). Negative weights may exist within units of the same pool.

OPEN IN VIEWER

Figure 1.

A representation of the interactive activation and competition framework for emotion (IAC-E). The parallelograms are input pools that represent feature sets of an emotion instance. Six pools are shown for illustrative purposes, but the framework is agnostic to which particular feature pools are used. The units are represented by white circles within each input pool and are organized by dimensions (2 dimensions – rows and columns – are shown, but any number of dimensions are possible). The circle in the middle is the hidden pool that contains two (conjunction) hidden units. Connections between hidden units and feature units may be innate or learned.

For illustrative purposes, we shall assume six input pools (denoted as parallelograms in Figure 1), however the number of pools is a free parameter in the framework (i.e. one may have any specified number of pools greater than one). The units of the input pool represent the features associated with an instance of an emotion. In Figure 1, for illustrative purposes, we use pools representing features related to cognition, motivation, behavior, neural activation in particular circuits, somatic physiology, and experience. We do not claim that this list – adapted from prior reviews on the nature of emotion (Moors, 2017) – is exhaustive, or that its components are mutually exclusive. It is also evident that the features we have listed are not all at a similar level of complexity or granularity. Different theorists may therefore prefer different lists of features, or may regard one or more of the input pools we include to be better represented as hidden pools rather than input pools (e.g., emotion experience). Our intent is not to suggest a definitive list of feature-sets. We merely seek to offer it as one example of what a list of emotion features might look like. The IAC framework for emotion is agnostic to the particular list of features that are included in the network.

Within each feature pool, the units are organized along one or more dimensions. For example, the ‘somatic’ pool might be composed of a dimension representing blood pressure and heart rate. Figure 1 features units organized along two dimensions (depicted via 3 rows and 8 columns), but in principle, a pool could have any number of dimensions. Once again, the IAC framework for emotion is agnostic to the particular dimensions that are included in each pool.

We next consider four core principles of the IAC-E. Similar principles are common to most IAC networks, but our description and examples emphasize their usage in contexts involving emotion.

The Connectivity Principle. Connections between units may be excitatory or inhibitory. Excitatory connections result in the amplification of activation in the involved units, whereas inhibitory connections lessen the activation of the involved units. In the IAC-E, all connections are bi-directional and equal. Bi-directional networks are guaranteed to converge (Perfetti, 1993) such that for any input, the network always attains a stable state. Connections between units in different pools are always excitatory and connections with units in the same pool are always inhibitory. These properties are summarized in the Connectivity Principle.

Principle 1 – Connectivity: In the IAC-E connections between units in different pools are always excitatory, and connection with pools are always inhibitory. All connections are bidirectional and equal.

Inhibitory connections are a core element of mammalian brains (e.g. Kisvárday et al., 1997) and they are important enablers of competition between units in the IAC-E. Competition (when present) allows the network to represent and act upon the feature with the highest activation rather than responding to an array of potentially contradictory feature units. For example, assume that some elements in an environment cause approach-related motivation units to activate while others cause avoidance-related motivation units to activate. Further, assume that activation related to approach-related motivation units is higher than activation related to avoidance-related motivation units. If there were no inhibitory connections between the approach units and the avoid units, then the network's response would be, at best, weakly oriented towards an overall approach response. If, however, the approach units and avoid units have inhibitory connections, then the slight lead of the approach units will suppress the less activated avoid units. In this case, due to the “rich get richer” phenomenon in IAC networks (McLeod et al., 2006), units corresponding to the approach motivation units (and not the avoid units) would be activated and the network would be robustly oriented towards an approach response.

The Input Principle. The network cannot receive direct input into the conjunction units in the hidden pool (hidden in the sense that this pool cannot receive external input). Only the feature units in the input pools can receive external input.

Principle 2 – Input: External input into the network can only be provided into any feature units in any input pool (i.e. conjunction units in the hidden pool cannot directly receive external input). Conjunction units, once activated, can reciprocally influence any feature unit.

The input principle ensures input into emotion networks could potentially arise from any feature unit in any feature pool. Importantly, feature units in the input pools are not merely activated by external input. Rather, they could receive activation from conjunction units via reciprocal connections. Activation in feature units rises and falls during the course an emotion, and collectively these activation patterns represent an emotion instance.

The Interactivity Principle. The occurrence of simultaneous changes in different features is a central aspect of an instance of emotion. For example, an instance of fear may involve simultaneous changes in units representing peripheral physiology (in the somatic pool) and units representing avoidance tendencies (in the motivational pool), and other pools as well. It is unlikely such simultaneity is exclusively attributable to different features independently responding to an emotion-eliciting stimulus at the same time. This would require each feature to be exclusively responsive to emotion-eliciting stimuli and exclusively unresponsive to co-occurring features. It is more plausible that changes in some features – represented by activations of corresponding units in the IAC network – cause changes in other feature units via the influence of other features. In the IAC-E (like all neural networks), we capture such influences via a transference of activation. One possible way to ensure such transference to postulate direct connections between some feature units. Another option – and one we develop in the present work – is to postulate that feature units are connected to each other via units in the hidden pool.

Principle 3 – Interactivity: Feature units in input pools are connected to each other via conjunction units in the hidden pool.

The interactivity principle entails that a feature unit that is activated by an emotion-eliciting stimulus would in turn activate the hidden units that it is connected with, and these activated hidden units would then activate other feature units that are connected with these hidden units. Thus, hidden units ensure interactivity between feature units in the IAC-E and enable network flexibility and context sensitivity. If feature units were directly connected with each other, the activation of one feature unit would reliably cause activation in the feature units it is connected with which would decrease the context sensitivity of the network.

The Learning Principle. Connections between feature units and hidden units may be innate or learned. Innate connections are assumed to be present from birth. Learned connections between feature units are acquired via experiences in which an emotion-eliciting stimulus causes simultaneous activations in different sets of feature units which are then connected to each other via sets of hidden units. Hidden units enabling such learned conjunctions between feature units become functional with experience and are not present from birth. Some connections may start out as being innate and may be transformed with experience.

Principle 4 – Learning: Connection weights between conjunction units and feature units may be innate (present from birth) or learned. Learned connection weights are formed with experience when two or more feature units co-occur due to statistical regularities in the environment.

Importantly, some (innate or learned) connections may be numerous and (indirectly) connect similar sets of feature units. Collectively such populations will exert a strong influence on network dynamics. Other conjunction units that are not participating in large populations connecting similar sets of feature units will exert less influence on network dynamics.

The instance-based approach that we have taken here is an algorithmic claim about the functional behavior of our brains, and not a claim that the brain literally represents each individual emotion experience separately (in different units). Ideas about how distributed representations, in which a small set of units, each participating in the representation of many individual instances, could capture the properties of instance-based models were first introduced by Marr (1982) and have subsequently been explored by many authors as alternatives to the idea that we actually allocate individual neurons or distinct non-overlapping sets of neurons to individual items (overview provided in Plaut & McClelland, 2010).

Notably, some of the key principles of the IAC-E were anticipated by Gordon Bower and his colleagues (Bower, 1981, 1992; Bower & Forgas, 2000) in their Human Associative Memory (HAM) model. According to the HAM model, emotions could be represented as units within a network, and could form associations with coincident events (which were also represented by units). The spread of activation in the HAM model began with external input and was thought to be akin to the flow of water or electricity. The HAM model was able to explain various effects including mood-state-dependent memories, emotional/social judgements, and some types of categorizations. However, the links in HAM network models were treated in an ad-hoc manner and could have a range of labels (such as “has a”, “is not a”, or “name” etc.). Such unexplained flexibility proved difficult to justify, and was finally resolved via PDP models that featured interactive and hierarchical processing of the type included in the IAC-E.

An Instance of Emotion in the IAC-E

In the IAC-E, an emotion instance begins when one or more emotion-related feature units receive input from outside the network. As an example, consider the case in which input begins in the somatic pool (e.g. due to an infant experiencing discomfort). This feature unit may be connected to a behavior unit (e.g. crying) via a (conjunction) hidden unit. Input into the somatic unit produces activation in the unit in the hidden pool. Rising activation in the hidden pool unit then activates the feature unit in the behavioral pool. Activations from the behavioral unit (and the somatic unit) then reinforce activation in the hidden unit (and are reinforced by it). Over time, the input into the somatic feature unit may decrease and/or the activation of the units may lessen via an in-built decay function. This rise and fall of activation in the network – enabled by an external input into one or more emotion feature units, its interactive spread to other emotion feature units, and its decay back to baseline – constitutes an instance of emotion in the IAC-E.

Other instances of emotion may be far more complex. Consider the case of an adult walking into a job interview. Several emotion-related feature units may be relevant and they may have many associative links with each other via conjunctive hidden units.

The hidden units are illustratively shown as two clusters because all units within each cluster have prior connections to the same input units in the cognitive, somatic, and behavioral pools (these pools were chosen as illustrative pools; any other combination of pools could have been specified). For example, the light-colored cluster units may be connected with units representing pleasant anticipation at the prospect of getting the job. The cluster of darker units – fewer in number – might be connected with feature units representing anxiety related to being interviewed. The light and dark colored clusters are connected to different somatic and behavioral units.

It is instructive to analyze the dynamics of this more complex situation. Let's say an event in the world occurs (e.g. the prospect of a job interview) and is represented by input in unit C18 (first row, 8th column with respect to the bottom-right corner). Input in this unit activates both collections of units in the hidden pool. These collections compete with each other (since they have inhibitory weights not shown in Figure 2). The overall activation of the darker collection is reduced in comparison to units in the lighter collection – because the latter has more units. Units in the lighter collection in turn activate units S33 and B11 in the somatic and behavioral pools respectively. Units in the darker collection in turn activate S21 and B31 – however these latter units are activated less strongly than activations in S33 and B11 since a greater number of hidden units were voting for them (i.e. sending activation to them). The units in each input pool (e.g. S33 and S21) are competitive with each other (i.e. they have inhibitory weights). As the interactivity builds, it displays a compromise level of activation (e.g. between S33 and S21 but weighted towards S33 since it received a greater activation flow). Finally, activation levels reduce (due to reduced levels of input and activation decay) and the emotion instance is completed.

OPEN IN VIEWER

Figure 2.

Illustrative network dynamics of an instance of an emotion in the IAC-E. This emotion instance begins in the cognitive pool with input into the unit labeled C18 (an emotion instance could begin with input into any feature unit in any pool). This input activates two populations of units in the hidden pool, which in turn activate competing units in the behavioral and somatic pool. See text for details.

The IAC-E Applies at Different Levels of Granularity

Thus far, we have worked with individual feature units that are connected to each other via individual conjunction units. All positive connection weights equal some small constant value (say + ε), and all negative connection weights equal some small negative constant value (say -ε). However, the principles of the IAC-E described above are equally applicable to collections of feature units. In this case, multiple feature units would be combined and represented as a single summary unit. The connection weight between two summary units is simply the sum of the individual weights of the units contained in the summary units. For example, if each of two summary units contain 5 individuals each, each of which are connected with a corresponding individual unit in the other summary unit, then the weight between the two summary units would be 5ε. In practice (for the simulations described in the next section), we shall cap the maximum positive weight between units at + 1 and the maximum negative weights at −1. This is a common assumption in most IAC frameworks.

The granularity of the collection chosen for a particular summary unit depends on the goals of the modeler performing the simulation. For example, it is, in principle, possible to specify two feature units, each representing an electrodermal response of 2.241 and 2.242 Micro Siemens respectively. If an experimenter seeks to model such small differences, then her IAC-E network would feature separate units representing these quantities. However, if she is interested in more macro-properties (e.g. electrodermal responses of less than 2 vs. responses greater than 5) and not in small differences in emotion features, then she may collect many granular units into fewer summary units. This approach is in line with exemplar models used in many domains of psychology (e.g. Ashby & Rosedahl, 2017; Hintzman & Ludlam, 1980; Medin & Schaffer, 1978).

An individual feature unit represents a particular response feature of an instance of an emotion. A granular feature represents an approximate summary of a property relevant to an emotion instance. Similarly, an individual conjunction unit represents the summary of a single instance of an emotion. A granular conjunction unit represents an approximate summary of past experiences with a collection of instances of an emotion.

Types of Connections in the IAC-E

Thus far we have described how connections between units in the IAC-E may be innate or they may be learned. As discussed above, we have allowed for the possibility that each type of connection may have varying strengths (between −1 and +1, by convention).

We assume that it is possible to specify a set of emotion instances such that each one of them feature a high percentage of strong and innate connection weights. We also assume that it is possible to specify a set of emotion instances such that each one of them feature a high percentage of learned connection weights of varying strengths. This assumption does not specify a particular threshold for what constitutes a ‘high percentage’ because doing so requires observers to agree on a canonical list of emotion features.

In the absence of such an agreed-to canonical list, the spirit of our assumption is that both innate and strong connections as well as learned connections of varying strength are ubiquitous in what people generally refer to as instances of emotion.

The IAC-E is Falsifiable

As we have discussed above, The IAC-E is capacious enough to support emotion instances of different types without sacrificing its internal consistency. However, this does not mean that the IAC-E supports any pattern of emotion instances and cannot be disproved. Rather a number of empirical observations could falsify the IAC-E's core assumptions. For example, if most populations of neurons that are active in representing particular emotion features are directly connected to each other (i.e. not via conjunction neurons), then the interactivity assumption would not hold. In such a situation, context effects in emotion (which have often been empirically observed) would have to have a different neural mechanism other than the interactivity-based mechanisms offered by the IAC-E. Similarly, consistency of responses in the IAC-E is based on feature units that are strongly connected with other feature units. We have proposed that such connections must either be innate or developed via universal statistical regularities. Once again, this is a testable (and falsifiable) claim.

The potential generativity of the IAC-E stems from its implications. We turn to these next.

The IAC-E as an Integrative Framework

We propose that under certain conditions the IAC-E can represent instances of emotions comprised of consistent responses to similar inputs and in other situations the IAC-E can represent instances of emotion comprised of variable responses that are shaped by experience and time. Additionally, under certain conditions, the IAC-E represents emotion instances that are not exclusively initiated by any single input pool (input agnosticism) and feature interactive and reciprocal effects. However, under certain (different) conditions, the IAC-E represents emotion instances that are appraisal-led and may proceed sequentially.

Representing Consistent as well as Variable Emotion Instances. The IAC-E enables consistency of emotion response patterns via at least two mechanisms. In both mechanisms, consistency of response patterns occurs due to the strong influence of a large number of conjunction units exerting their collective influence.

The first mechanism enabling consistency is the presence of a large population of hidden units that innately connect selected feature input units to each other. In such a situation if a feature pool unit (say unit A) receives external input, it activates the population of hidden units it is innately connected with; these units in turn activate the other feature units they are innately connected with (say units B and C). If there are enough such similarly-connected conjunction units they will collectively overcome the influence of (presumably fewer) context-related feature units. In such cases, activation in unit A will consistently produce activation in units B and C.

The second mechanism enabling consistency relies upon statistical regularities in the environment to produce a population of well-connected learned hidden units (postulated by the learning principle). Here the hidden units were not innate, but developed over time in consistently co-occurring populations of input units. Such units will produce similar consistency as innate units.

Related to variability, connecting feature units to each other via conjunctive hidden units (postulated by interactivity principle) affords the opportunity to explain the effects of experience and context of emotion-related responses over time. For example, consider the case of a person who has experienced many instances of emotions in which a higher than normal level of somatic arousal is associated with withdraw motivation (corresponding, for example, to fear) and only a few instances of emotions in which a higher than normal level of somatic arousal is associated with approach motivation (corresponding, for example, to excitement). If such a person experiences a high heart rate (an activation of units in the somatic input pool), she is likely to exhibit increased activation in withdraw motivation units. In the IAC framework for emotion, this occurs because the ‘increased heart rate’ input unit activates many prior hidden units – each of which send activation to the ‘withdraw’ motivation unit. Contrastingly, fewer hidden units send activation to the ‘approach’ motivation unit. Eventually the activation in the ‘withdraw’ feature unit will inhibit the activation in the ‘approach’ feature unit (postulated by the competitive feature unit principle).

However, this association between ‘increased heart rate’ and ‘withdraw’ motivation units can be modified by experience: imagine that this person next encounters several other instances of emotion that are associated with a high heart rate, but are also associated with an approach motivation (e.g. because she frequently rides on an enjoyable Ferris wheel). These emotion instances connect via new conjunction units in the hidden pool (due to learning). Now the ‘high heart rate’ feature units would activate conjunction units that send activation to ‘approach’ (as well as earlier conjunctions sending activation to ‘withdraw’). This would effectively weaken the association between experiencing a high heart rate and wanting to withdraw. With enough new hidden unit conjunctions, a high heart rate may even become associated with approach motivation. Hidden units thus play the role of casting top down votes that determine the effective associative strength between different emotion features.

Variability may also arise due to contextual variables because associations involving hidden units are also subject to selective activation based on combinations of input cues (Medin & Schaffer, 1978; McClelland, 1981). Increasing specificity of input can alter the pattern of activated conjunctions in the hidden pool. Continuing with our example above, providing input into the ‘high heart rate’ and ‘withdraw’ feature units will activate hidden units corresponding to instances of fear and disgust; this activation may in turn activate the feature units associated with the subjective experience of fear or disgust. On the other hand, input into the ‘high heart rate’ and ‘approach’ feature units will activate the hidden unit corresponding to instances of excitement; this activation may in turn, activate the feature units associated with the subjective experience of excitement.

Representing Input-Agnostic as well as Appraisal-led Emotion Instances. The input principle postulates that any feature unit in any input pool may receive external input. Thus, in principle, emotion instances need not be initiated by activation in a single pool (e.g. the appraisal pool). Once a set of feature units activate a population of conjunction units, those units may activate a different set of feature units, which may dynamically influence the original set of feature units. Thus, the IAC-E supports interactive and reciprocal influences in the unfolding of an emotion instance.

While the IAC-E framework has no requirements that emotions begin with external input in the appraisal pool, it supports the possibility that many emotion instances begin with an appraisal. Further while interactivity and reciprocity are modal in the IAC-E, it supports the possibility of observing emotion instances that are appraisal-led and appear to unfold sequentially. For example, consider the case in which one feature unit, say F1, receives external activation and activates a set of conjunction units, say C1, that are connected with a feature unit say F2. Once F2 is activated, it may activate a different set of conjunction units say C2, which may then activate a third set of feature units F3. Such an activation pattern will cause a sequential spread from F1 to F2 to F3.

More generally, appraisal led emotion instances require a large number of strongly connected units in the appraisal feature pool. Such units may arise with repeated input into one or more appraisal units – which may, for example, be the case for human adults. Such repeated external input will cause strengthened connections between the appraisal feature pool and the conjunction pool (due to the learning principle).

Simulations of Empirical Findings Using the IAC Framework

In this section, we seek to test whether the IAC-E can simulate well known empirical findings from the emotion literature. We reasoned that such simulations, if successful, could strengthen the claim for the IAC-E to be a general framework for emotion.

Our goal in these simulations is to test the consequences of the principles of the IAC-E. If diverse empirical phenomena in the emotion literature can be explained as logical consequences of these principled (and only these principles), then one must increase one's confidence that the IAC-E offers a useful algorithmic account for emotion phenomena. We shall detail two simulations below to illustrate the IAC-E model. Further simulations are described in the Supplementary Materials.

The first simulation is concerned with the influence of context on emotion perception. The second simulation is concerned with consistency of emotion responses across people. Other simulations, using similar techniques as those showcased below, related to the facial-feedback effect (Noah et al., 2018) and how emotion words shape emotion percepts (Gendron et al., 2012), are described in the Supplementary Materials. The particular feature units chosen for the simulation, while always obeying the IAC-E assumptions, depend on the particular empirical phenomena being modeled. All specific algorithmic details of the IAC-E network are identical to other interactive activation networks and have been described elsewhere (Suri et al., 2019).

Simulation #1: Malleability of Emotion Perception. A large number of empirical studies (e.g. Hess et al., 2016; Matsumoto & Hwang, 2013) have examined the effects of context (e.g. making a fist) on facial emotion recognition (e.g. recognizing that a face depicts the emotion of anger). This simulation proposes an IAC-based mechanism for such effects. Here we take the perspective of an emotion observer estimating emotion dynamic using her own IAC-E.

In a well-known study, Aviezer et al. (2008) presented participants with images of an individual with various facial expression/context combinations and asked to choose the emotion that “best describes the facial expression” from a list of six basic-emotion labels (disgust, anger, sadness, fear, surprise, and happiness). The stimulus set consisted of posed disgust faces superimposed onto models in four emotional contexts: disgust, anger, fear, and sadness. Context, here, refers to all information external to the face, including body posture and the surrounding scene. For example, the disgust context depicted the person holding a soiled diaper, and the anger context depicted the person making a fist. Importantly, while the context variables varied across images, the face did not (i.e. it was the same face in all images – always showing a disgust face).

Categorization of facial expressions matched the context emotion most often when the emotional context matched the facial expression typically associated with such context (i.e., a disgusted expression in a disgust-suggestive context; see empirical results in Figure 3a and 3b for details). For example, in a disgust context, respondents accurately classified the face as representing disgust in 91% of trails. However, in an anger context, respondents only classified the face representing disgust in 11% of trials (as depicted in Figure 3a); in 87% of trials, respondents were influenced by context and classified the face as representing anger (as depicted in Figure 3b). The effects of context were largest for anger, second for sadness, and weakest for fear.

OPEN IN VIEWER

Figure 3.

Empirical and simulation results for simulation 1. Panel (a) shows empirical data and simulation results for categorization accuracy in different contexts. Here a disgust face in a disgust context is accurately classified in most (91%) of trials, but a disgust face in an anger context is infrequently classified as disgust. Panel (b) shows the empirical data and simulation results of the frequency with the disgust face is labeled with the context emotion. For example, in the anger context, the disgust face is labeled as anger in 87% of trials. Panel (c) shows the network structure used in the simulation.

Figure 3c represents the network structure used in the experiment. There are two feature pools – one for the face and one for the context. Both feature pools contain units for disgust, fear, sadness, and anger – the face feature pool receives input from face-related features, and the context feature pool receives inputs from context-related features. The hidden pool (depicted as a circle in Figure 3c) contains an integrative representation of disgust, fear, sadness, and anger. The units of the hidden unit receive activation from – and provide activation to – units in the face feature pool and context feature pools.

Since integrative representations of the emotion categories in this empirical context are well differentiated from each other, connection weights between units in the hidden pool are fully inhibitory (i.e. the connection strength between each hidden unit is −1). This is diagramed using the loop on the top right of the hidden pool in Figure 3c. Similarly, since contexts depicted in the experiment (e.g. a diaper for disgust and a fist for anger) are well-differentiated connection weights between units in context feature pool are also fully inhibitory (i.e. the connection strength between each unit in the context feature pool is −1). Units in the face feature pool are unevenly differentiated from each other – some units share common features whereas others are highly differentiated. Following a computational model developed for assessing similarities between facial expressions (Susskind et al., 2008) facial expressions of disgust are highly differentiated from facial expressions of fear, somewhat distinct from facial expressions of sadness, and relatively similar to facial expressions of anger. Constrained by this model, we assigned an inhibitory weight of −1 between “face” feature units of disgust and fear, a weight of 0.8 between disgust and sadness, and a weight of −0.5 between disgust and anger.

There were excitatory weights (+1) between units in the context feature pool and corresponding units in the hidden pool. For example, the ‘sadness’ context unit connected with the ‘sadness’ hidden unit, but not to any other hidden units. Facial feature units were also connected with their corresponding feature units (+1 connection weight). However, they could also be connected with other hidden units that they were somewhat consistent with. For example, the disgust facial unit was connected to the disgust hidden unit with +1 weight, with the fear unit with 0 weight (i.e. unconnected) with the sadness hidden unit with +0.2 weight, and with the anger hidden unit with +0.5 weight. These weights reflected the similarity between a facial feature and an emotion (Susskind et al., 2008).

In all conditions, the disgust facial feature unit was provided with an input of 1. The context units were provided activation depending on the context being modeled. In the disgust context, the disgust context unit was given input equal to 1, in the anger context, the anger context unit was given input equal to 1 (and similarly for the sadness and fear contexts). The input activation from the feature units flowed to the hidden units, and then back to the feature units (since all weights in the network are bi-directional) until the network converged. A Softmax function (λ = 7) was used to calculate the probability that a face was classified as disgust in each condition (Figure 3a) and the probability that the face was classified in line with the context emotion (Figure 3b).

This simulation showcases how context effects might change the perception of emotion faces. Activation was the only currency in the network – it did not rely on affect programs to instantiate emotions, nor was it solely dependent on conceptual knowledge related to emotion contexts. It showcased a mechanism which allowed context to affect faces and also made the prediction that facial expressions would impact the perception of ambiguous contexts.

Simulation #2: Consistent Autonomic Activity in Response to Consistent Input. In this simulation, we propose a mechanistic explanation of how autonomic activity produced via facial prototypes or via reliving past experiences displays consistent footprints in response to consistent network input the categories of anger, fear, and disgust.

Ekman et al. (1983) measured autonomic activity in two tasks. In the directed facial action task, in a series of trials, participants were asked to construct facial prototypes of emotion by arranging facial muscles into an emotion-prototypic expression. They were not required to produce an emotional expression but instead were told which facial muscles they should contract. Video-records were used to only include those trials in which participants successfully followed instructions. In the relived emotion task, in a series of trials, participants were asked to experience emotions by reliving a past emotional episode. Participants rated the intensity of felt emotion after every trial, and only those trials in which their reported intensity was greater than the mid-point, were used for analysis.

The experimenters reported findings in two autonomic categories: changes in heart rate, and changes in (right) finger temperature. They found consistent differences in the emotion categories of fear, anger, disgust, surprise, happiness, and sadness. For simplicity, in this simulation, we illustrated the first three emotion categories only (an identical mechanism can be posited for the other three), and we therefore report on the results for those categories only. Across both tasks, anger and fear were associated with higher heart rates, and disgust was associated with lower heart rates. Additionally, across both tasks, anger was associated with a higher temperature, whereas fear and disgust were associated with lower temperatures.

The network for this simulation (Figure 4) consists of a Face feature pool and a Relived emotion feature pool – each of which have units corresponding to the emotion categories of disgust, fear, and anger. There is a Hidden pool representing summary units for the three categories. Finally, there were feature pools containing units for higher or lower heart rates and higher or lower temperatures.

OPEN IN VIEWER

Figure 4.

Network structure and activation dynamics for simulation 2.

Figure 4 contains the connection strengths for Simulation 2. The connections involving the Face feature pool and the Hidden pool are constrained with those described in Simulation 1. All other connections are fully excitatory (+1, between pools) or fully inhibitory (−1, within pools).

We simulated 6 conditions: in the first three conditions we provided +1 input, respectively, to the disgust, fear, and anger face feature units. In the next three conditions, we provided +1 input, respectively, to the disgust, fear, and anger relived emotion feature units. In each condition, the feature unit activated the hidden unit it was connected, which in turn activated the appropriate feature heart rate and temperature feature units. As expected, activation values upon convergence were very similar in Conditions 1 and 4, 2 and 5, and 3 and 6. The values for Condition 1 (input into Disgust Face unit), Condition 2 (input into Fear Face unit), and Condition 3 (input into Anger Face unit) are shown in Table 1.

OPEN IN VIEWER

Table 1.

Activation values for input into face feature units in simulation 2.

	Low HR	High HR	Low Temperature	High Temperature
Disgust	0.31	−0.12	0.31	−0.12
Fear	−0.13	0.41	0.30	−0.10
Anger	−0.13	0.35	−0.13	0.35

The IAC-E can display consistent response patterns in response to consistent network inputs. Further, this simulation is an instance of an emotion that does not begin with an appraisal (rather, some conditions are initiated via posed facial expression causing activation in the face feature pool).