The Inversion Problem: Why Algorithms Should Infer Mental State and Not Just Predict Behavior

Sonar

Imagine a submarine commander, encased in metal, submerged in the deepest, darkest part of the ocean, with neither windows nor lights. The commander ideally wants what a video camera could provide: a picture of the ocean depths surrounding them. Instead all they have is a string of data that by themselves are meaningless: how long it takes sound waves to return to the submarine.

To help the submarine commander use these sonar pings to “see,” we rely on our knowledge of physics. The speed of sound waves in water turns out to depend on (among other things) the water’s temperature, pressure, and salinity. By using a structural model from physics, sonar converts data on sound-wave return times to a measure of distance to the nearest object. By modeling these physical processes, we can invert something useful (e.g., whether there is an enemy sub nearby) from a string of not directly helpful sonar-ping data. These sorts of inversion problems are rampant in natural sciences extending far beyond sonar (see, e.g., Tarantola, 2005).

Often, behavioral data are like sonar data, except less transparently so. Making sense of sonar data obviously requires physics. Yet behavioral data seem like they can be directly interpreted. We argue that in many applications that sense of direct interpretability is an illusion.

An understanding of behavioral science quickly reveals that when it comes to analyzing human behavior, we are much closer to submarine commanders; our data on people are much closer to sonar pings than to videos. That is because, in many applications, we are not interested in the data that are observed, such as behavior, but rather in unmeasured mental states, such as preferences or knowledge. Like inferring surroundings from sonar pings, we must infer these mental states from behavioral data.

We illustrate these inversion challenges with two canonical algorithmic applications: curation and automating expertise. These are not meant to be exhaustive but merely illustrative of the breadth of such problems.

Curation

What we call curation algorithms are ubiquitous in modern life. In a world with so many options, we need some way to sift and rank them for customers. Retailers have many items, entertainment companies have oceans of content, social media companies have many posts, and so on.

Although such algorithms can differ dramatically in how they are implemented, their construction typically has a shared foundation. Take the case of content on social media, which of course has itself been the subject of a great deal of research in behavioral science (see, e.g., Allcott et al., 2022; Bayer et al., 2020; Hunt et al., 2018; Neubaum & Krämer, 2017; Vogel et al., 2014). When you log on, the content could in principle just be ordered by, say, when it was posted, and then it is left up to you to scroll through the hundreds or even thousands of posts since you last logged on. There is a reason few social media platforms work like that: It is very cumbersome and not very helpful.

A curation algorithm attempts to rank your social media feed to put the posts you would like the most nearest to the top. Although the mechanics of the algorithm can vary dramatically across companies, in essence they all have a shared foundation. They take data on you (e.g., which posts you have previously liked or interacted with) and use that data to build a predictor of what kind of post you are likely to interact with in the future. That prediction of how you will engage with a post is how the algorithm then ranks the ocean of content.

Looking for inversion problems makes one question transparent: What is the goal of a curation algorithm? If the deeper goal is simply to maximize engagement then there is no problem. But if the goal were to maximize user satisfaction, then this raises a behavioral (inversion) question: In what ways does someone’s observed engagement level with a post not fully reveal whether they like that post?

Of course, a substantial amount of research in psychology has shown numerous ways in which behaviors deviate from preferences. For example, we have self-control problems (Ainslie, 1992; Baumeister et al., 2007; Fujita et al., 2006; Inzlicht et al., 2021; Laibson, 1997; Loewenstein & Prelec, 1992; Milkman et al., 2009; Muraven & Slessareva, 2003; O’Donoghue & Rabin, 2000; Thaler & Shefrin, 1981; Ward & Mann, 2022); our aspirations can differ from our momentary impulses (our “shoulds” are not our “wants”). That means our actions can deviate from our intentions.

As another example, we might make our choices without much conscious deliberation, almost automatically (see, e.g., Trueblood et al., 2018). Those automatic behaviors are generally of enormous value to us. They help us deal with common situations we encounter in our daily lives over and over again in a way that is usually adaptive (see, e.g., Bargh, 1994; Chaiken & Trope 1999; Gawronski & Creighton, 2013; Gilbert & Hixon, 1991; Haidt, 2001; Jacoby, 1991; Kahneman, 2011; Kahneman & Frederick, 2005; Nisbett & Wilson, 1977; Wilson, 2002). In situations in which the environment is fairly predictable and people have a chance to learn situational regularities through repeated exposure and practice, automatic responses can be better—sometimes even substantially better—than more deliberate ones (see, e.g., Gigerenzer & Gaissmaier, 2011; Kahneman & Klein, 2009).

But as a large body of literature in psychology has documented, automatic responses can sometimes also lead to problems—to lead us to act in ways that our more deliberate selves do not necessarily want. For example, we may stereotype; in-group bias tends to be more likely when we are—roughly speaking—acting automatically (e.g., Cunningham et al., 2004; Devine, 1989; Gilbert & Hixon, 1991; Greenwald et al., 1998). Our body language can be inadvertently more open toward people like us (in-group members) than toward people not like us (out-group members). So our past choices might include subconscious bias that our deliberate selves wish were not there, which can then lead to algorithmic bias (see, e.g., Barocas & Selbst, 2016; Chouldechova, 2016; Dwork et al., 2012; Executive Office of the President, 2016; Gebru, 2020; Kleinberg et al., 2017; Lum, 2017; Obermeyer et al., 2019).

There are, in other words, countless inversion problems here that we are neglecting.

Automating expertise

Another category of algorithms are those that attempt to automate human experts: grading student essays, administering college admissions, screening resumes, deciding legal cases, diagnosing illnesses, and so on. A very commonly discussed example from medicine makes the problem clear: automating the reading of an X-ray. One might think such algorithms are built to read X-rays using biomedical ground truths. In fact, in almost all cases, they are instead built using data on the behavior of the expert. Physiological ground truth is rarely available: What is available is typically a diagnosis provided by an expert. (For example, even something that seems like a physical ground truth, like the results of a biopsy, is often actually an expert judgment—in this case, that of the pathologist reading the biopsy data.) So when we automate, we are automating that human expertise.

To build such an algorithm, typically we would collect data on, say, X-rays, along with a radiologist’s judgments of the patient’s conditions for those X-rays. We would then use those data to build a predictor of what a radiologist would say about any given X-ray. Using data never before seen by the algorithm, we could then validate the algorithm’s predictions; that is, how do its predictions about new X-rays compare with the radiologist’s statements about these X-rays?

Thinking about inversion here again forces us to think about our goal in building this algorithm. Is our goal to predict what a radiologist will say? On first blush, of course. But on second blush we can see the answer is of course not. Our goal is to understand what the radiologist’s expert opinion would be on this X-ray. Those two things—what a radiologist says on any given X-ray versus their expert judgment of that X-ray—seem deceptively similar. Those untrained in psychology may not even see the distinction.

Yet psychologists have long recognized that in any given instance an expert may behave in ways that are at variance with their expertise (see, e.g., Berthet, 2022; Payne et al., 1993; Shanteau, 1992; for how actuarial models outperform experts, see, e.g., Dawes et al., 1989; Kleinberg et al., 2018; for how far automation may go and the implications for the future workforce, see, e.g., Brynjolfsson & Mitchell, 2017).

For example, fatigue affects judgments. Studies suggest that radiologist errors may be as much as 25% more common at the end of a doctor’s shift than at the beginning (Taylor-Phillips & Stinton, 2019). Fatigue is so intuitive that it seems almost obvious—painfully obvious. Yet to our knowledge none of the algorithms that automate medical judgments account for fatigue. In fact, as far as we know they typically do not even collect the key piece of data we would need to address fatigue (something such as how many hours the radiologist had worked before seeing the X-ray). The difference between specific judgments and expertise is not limited to fatigue. In coming up with a diagnosis the radiologist may ignore the base rate of different conditions. Or radiologists may be prone to the so-called gambler’s fallacy: After several negative X-rays in a row, the radiologist will be inclined to think that a positive X-ray is due or vice versa (Chen et al., 2016).

An algorithm that simply predicts the radiologist’s behavior will automate all of these biases alongside the variable we really want, which is the radiologist’s true expertise. When we automate what radiologists say—or teachers, college-admissions officers, resume screeners, or judges—we are neglecting important inversion problems.

Curation

To see the real-world consequences of treating inversion problems like pure prediction problems, consider an audit study of one of the world’s most widely used social media platforms: Facebook (Agan et al., 2022). Two of the most widely used curation algorithms on Facebook are People You May Know (PYMK), which ranks potential new “friends” by a prediction of your likelihood of connecting with them, and Feed, which ranks posts by people you have already friended on the basis of the algorithm’s prediction of whether you would engage with a given post.

The Facebook audit revealed a striking fact: Feed recommendations are subject to out-group bias. That is, Feed systematically “hides” (down-ranks) posts by friends that are of a different ethnicity or race: In the United States, for a White user, their Black friends’ posts are ranked systematically lower (and vice versa). A second audit in India found the same pattern: For a Hindu user, posts by Muslim friends are much lower in the feed. Because users often read only the first few posts this means that even once a user has befriended someone from a different group, the algorithm implicitly works to weaken the tie to the “out-group” friend.

Importantly, these patterns hold true even when controlling for how much the user is interested in the content that their friends have posted. In the Agan et al. (2022) study, the authors also were able to survey study subjects about their explicit, deliberate preferences about how much they would like to see different Feed posts. People’s deliberate, stated preferences seemed to value the posts of Black versus White friends (or Muslim vs. Hindu) friends equally well. But somehow the algorithm, built using people’s past choices, gave them something different.

The psychological model of automaticity gives us an explanation for why Feed shows significant in-group bias. How do we know it is automaticity that is contributing to this bias? Partly because we do not see the same pattern with the PYMK algorithm, for which the data show us users are acting much more deliberately when making their choices, and partly because in a laboratory experiment that presents users with a stylized recommender system, adding time pressure to make subjects’ behavior more automatic seems to increase bias.

Ignoring the role of automaticity in people’s past Feed choices (i.e., treating the construction of the Feed algorithm as a pure prediction problem rather than an inversion problem) leads to a large-scale algorithm that may impede social interactions across lines of race or religion in two of the largest social media markets in the world.

This problem is of course not limited to Facebook; it is endemic to curation algorithms.

Consider just one other example: The algorithms that help people surf the Web try to learn our preferences from our past choices, assuming those are one and the same. But that ignores the psychological insight that all of us tend to underinvest in exploring new things; our past choices will understate the degree to which we like variety rather than just familiarity. ³ This provides an explanation for the following blogger’s post, which captures something surely most of us feel to some degree:

Obviously this isn’t objectively true, the internet is always adding content . . . but it doesn’t really give me that open, untamed frontier feeling that it gave me when I was younger. These days, it feels like 99% of my internet usage is confined to a handful of the same websites that I never really venture away from (WestEgg, 2020). ⁴

The algorithm that mistakenly thinks we really only crave the familiar by looking at our past choices winds up making the Internet feel small.

Automating expertise

Although we do not have the same sort of direct evidence of the real-world consequences of confusing inversion problems for pure prediction problems in the case of automating expertise, there is every reason to believe the consequences here may be substantial as well.

When we try to automate expert judgment, from the expert’s behavior the algorithm will learn not just their expertise but also their biases. One study asked radiologists to review a set of X-rays and then on the last X-ray inserted a superimposed image of a gorilla, an homage to a classic study in psychology showing how attention gets allocated automatically and the limits of the human attentional spotlight (see Drew et al., 2013). A shockingly large share of radiologists did not see the gorilla at all. Those are the types of choices that are getting incorporated and learned by the algorithm behind the automated radiologist.

A different hypothetical example makes the same point more vividly (and visually): an algorithm trained to automatically call balls and strikes by watching baseball umpires. The result is shown in Figure 1 (taken from Walsh, 2010). The actual strike zone is the rectangle. Yet umpires—and hence a hypothetical robot umpire that was trained on their calls rather than on the formal rectangular boundaries—end up with a called strike zone that is shorter and wider, an irregular oval rather than the rectangle that the rule book says it should be.

OPEN IN VIEWER

Fig. 1.

Umpire-called balls and strikes (shown by the heat map) relative to the official strike zone according to the rule book (white rectangle). From Walsh (2010).

One empirical example that we do have for the real-world consequences of confusing inversion for pure prediction problems comes from medicine. An algorithm trained to detect knee pain is usually trained not to predict knee pain but to predict instead a clinician’s judgment of knee problems. A psychologist would look at this and ask, “Don’t doctors—like all people—sometimes suffer from implicit biases?” Those implicit biases will in turn get baked into an algorithm that is built using data from the clinician’s past choices. Pierson et al. (2021) showed the consequences—clinicians are more likely to fail to recognize physical problems in the knees of Black and low-income patients.

By treating inversion problems as if they are pure prediction problems, we are inadvertently building algorithms that do not achieve the goals we set out for them—and sometimes accomplish the opposite of what we really want.

Having multiple behavioral measures helps triangulate

How would we ever invert in practice? In our simple examples so far we have assumed that there is just a single behavioral measure in the data. But in reality there are often different types of behavioral measures available in the data. That is a boon for inversion because psychological theory is quick to point out that not all behaviors are equally automatic.

Consider curation algorithms. Most social media platforms have a variety of engagement measures—whereas clicking may be more automatic, reading an article all the way through, or even commenting on it, may be more deliberate. Moreover, psychology has even produced validated measures of automaticity that can be used to quantitatively measure the relative degree of automaticity of different engagement measures that might be captured in available data. ⁵ These two kinds of content (more vs. less automatic) will behave differently on the different types of behavioral measures we have available. That is, for the sort of content more prone to automaticity, we will see greater divergence between behaviors that tend to be more versus less automatic, whereas for other content we will see less divergence. That degree of divergence can be a road map to the regions where inversion is needed more than pure prediction.

It is very possible that solving the inversion problems relevant to those segments of the data may require formal structural models of psychological phenomena of a sort that currently do not exist. Although we could in principle simply put more weight on those behavioral engagement measures that are relatively less automatic, that leaves us short of the target: “Less prone to bias” is better than “more prone to bias” but not equal to “unbiased.” More sophisticated variants are also possible of this basic idea. For example, the more automatic measures are likely to be the ones that are available in higher frequency and volume than the less automatic measures (because things people do relatively more of are, all else equal, likely to be things they do relatively more quickly). Some weighting scheme could be used to optimally weight together the higher frequency biased engagement measures with the lower frequency, less biased but noisier engagement measures. The key point is that a formal psychological model is needed to extrapolate beyond the range of automaticity captured in the data.

That is, the variation in automaticity of the engagement measures captured in the data can be used to learn the relationship between the level of automaticity and out-group bias in the support of the data. But data alone cannot let us say anything beyond the support of the data; we need some additional information or assumptions to do that (i.e., a model). Specifically we need some explicit mathematical model of automaticity that makes some assumptions about the nature of automaticity and out-group bias on the basis of psychological theory. The key question is one of functional form: Does the gradient between automaticity and out-group bias flatten or steepen as we move beyond the support of the data toward lower and lower levels of automaticity?

This need for quantification in solving inversion problems is what requires an extra degree of precision and formality that seems to remain fairly rare with psychological models. The field currently works largely in directional terms: Automaticity makes people relatively more prone to rely on stereotypes. But for solving inversion problems we need to understand something about magnitudes, not just directional effects, to be incorporated into the design of training algorithms. That requires formalizing psychological ideas in terms of mathematical models, akin to what we have started to see as part of the development of behavioral economics. For example, time inconsistency had been a phenomenon that psychologists had studied and documented for decades (I would prefer $5 today over $10 tomorrow, but I would prefer $10 next Sunday rather than get $5 the day before on Saturday). It was the development of formal models of quasi-hyperbolic discount functions (see, e.g., Laibson, 1997) that allowed economists (and in principle psychologists for that matter) to empirically estimate the value for the key beta and delta parameters of the hyperbolic discount function. Those parameter values help behavioral economists determine, for example, which specific commitment devices or tax policies or financial-market innovations make consumers better versus worse off. Much more work along those lines remains to be done, however, because behavioral economics has incorporated and formalized just a vanishingly small share of all the psychological insights that are likely to be important for the development of trainer algorithms.

Inverting on small data sets can help build algorithms on larger data sets

The main challenge of treating inversion problems for pure prediction problems is that the algorithm builder has no way to judge whether any given candidate algorithm is doing what we want because the outcome being examined, behavior, does not correspond to the thing we really want to know, mental state. Sometimes inversion may involve costly solutions—like the collection of additional data—to learn the mapping between behavior and mental state. The good news is that learning that mapping for a small data set can sometimes be useful for constructing industrial-scale algorithms on much larger data sets.

On social media, for instance, one way to learn what people really want may be to ask them in ways that try to elicit the preferences of their more deliberate selves. Such explicit data collection cannot reach the scale of passive collection—surveys are more expensive than simply tracking clicks. Still, they can be done: Social media companies regularly survey user satisfaction. Such survey data can be immensely helpful for learning the structure of inversion. Behavioral science can help us craft what to survey, and these data can be used (albeit in a small sample) to learn the mapping between behavior and desired mental state; in essence, behavioral science can help us learn the best way to aggregate large-scale data to best proxy for mental state.

Notice why it is not necessary to collect preference or satisfaction data for literally every user, site, post, and product that will be “touched” by the at-scale algorithm because algorithms are essentially all about grouping things together by observable characteristics. So long as we have “enough” preference or satisfaction data for the relevant user-site-post-product “cells,” defined by observable characteristics of users, sites, posts, and products, we can extrapolate to other users, sites, posts, and products with similar characteristics (subject to the usual assumption that the data-generating process out in the world that relates user preferences to those characteristics is stable). Learning from a sample of data that 50-year-old smart-pantry users deeply regret eating more than a handful of Doritos a week whereas 18-year-old high school athletes have less ambiguous feelings is a useful if-then rule that can be applied more broadly to guide the construction of larger scale algorithms.

That same logic can highlight where small amounts of additional data collected at large scale can be of disproportionate utility. For example, in the Facebook case, automaticity manifests itself in bias against Feed posts by out-group friends. We know that because an audit study collected costly user preference data at small scale (as in Agan et al., 2022). But with that information at hand we now know what the key variable is to ensure we collect at large scale as part of any attempt to debias the Feed algorithm: the race (or religiosity, etc.) of the friend posting.

One need not be able to invert perfectly

Perhaps the only thing we are willing to confidently predict about the new science of inversion, whatever that winds up looking like, is that it will not be perfect. There will surely be many applications for which it will not be possible to confidently invert the mental state of interest from the behavior observed in the data. Does that render the idea of inversion irrelevant? Our answer is: So what? Inversion can still be enormously useful.

There are past success stories that illustrate this principle for online content. For example, an early insight in the ranking of search results was the notion of “position bias”—that people are more likely to click on search results that are higher up in a ranking, even if other results lower down might be better responses to their query. Position bias arises naturally from psychological insights into how people browse for information—that they may engage in satisficing behavior and stop early—and it was measured through eye-tracking studies (Joachims et al., 2017), leading to improved ranking algorithms that take into account the possibility that highly ranked items might receive more clicks even when they are worse. In this case, an inversion strategy created important improvements even though we are far from inverting all aspects of a searcher’s mental state.

Considering other applications prospectively, think again, for example, of curation algorithms for social media. Some content (posts, tweets, etc.) is likely to lead to more automatic behavior than others. Some content causes us to pause and be more considered; other content simply provokes automatic responses. For many current applications in content curation, each piece of content is essentially treated the same from a psychological perspective—as equally reflective of the user’s true preferences. But learning that some type of content might be engaging to people perhaps because they really like it or perhaps because it is a temptation good is enormously valuable. The smart pantry that gives you all the fruits and vegetables you want but says “Hmm, let’s be open to multiple possible motivations here with the Doritos” has just done you a great favor.

Put differently, data scientists and behavioral scientists alike are used to living with the idea of a statistical uncertainty interval around our estimates. Inversion, even when it cannot “point identify” our mental states, can still be useful by giving us something analogous to a psychological uncertainty interval.

Inversion is not the same as having multiple objectives

Data scientists have long been used to solving problems with multiple objectives: We want an algorithm that, for example, recommends the shortest driving route but also tries to minimize fuel consumption to help address climate change. That might sound like inversion. It is not.

Inversion problems are those for which the thing we really want to know—whether that is a single objective or multiple objectives—are not represented in our data, and there is no transparent way to go from here to there using the available behavioral data alone. Machine-learning engineering tools to design algorithms that balance competing goals against one another cannot solve the problem that the goals that we really care about are not measured in data. Multiple-objective optimization algorithms are no substitute for the development of new models that combine psychological insights with machine learning.

The one common thread across all of these objections is that algorithms—and everyone who relies on algorithms (which is to say, everyone)—needs psychologists to be centrally involved in the solving of inversion problems.