Correcting Political and Consumer Misperceptions: The Effectiveness and Effects of Rating Scale Versus Contextual Correction Formats

Abstract

While fact-checking has grown dramatically in the last decade, little is known about the relative effectiveness of different formats in correcting false beliefs or overcoming partisan resistance to new information. This article addresses that gap by using theories from communication and psychology to compare two prevailing approaches: An online experiment examined how the use of visual “truth scales” interacts with partisanship to shape the effectiveness of corrections. We find that truth scales make fact-checks more effective in some conditions. Contrary to theoretical predictions and the fears of some journalists, their use does not increase partisan backlash against the correction or the organization that produced it.

Keywords

fact-checking journalism political communication media effects

While misinformation—about policies, politics, and even consumer goods—has always been a part of the media landscape, the last decade has seen the emergence of dedicated fact-checking organizations aimed at correcting these inaccuracies (Amazeen, 2012; Graves, 2016; Kessler, 2014). These fact-checking organizations vary in organizational structure, research methods, and story presentation; one of the biggest divides concerns the use of ratings systems to assess the truth of public claims. Sites like FactCheck.org and TruthInAdvertising.org provide readers with a nuanced analysis of the contested claim, but stop short of systematically rating the degree of truth in a statement. In contrast, groups like PolitiFact.com and the Washington Post’s Fact Checker also add an ordinal “truth scale” that provides a clear visual indicator (Figure 1) of their judgment (Amazeen, 2012; Graves & Glaisyer, 2012).

Research in political science, communication, and psychology suggests that the inclusion of a clear true/false indicator may affect whether a person chooses to read the correction, how that person processes it, and ultimately how successful it is in correcting misinformation. This article presents the results of an experimental study designed to assess how including a rating scale shapes the effectiveness of a correction and whether this effect varies depending on the type of misinformation (political vs. non-political) and the party affiliation of the reader. We also examine how the inclusion of a rating scale affects readers’ attitudes toward public figures and the media. Overall, we find strong evidence that truth scales can be effective tools in countering misinformation and offer few drawbacks. In a non-political context, the addition of a truth scale increases the effectiveness of a correction. In a political context, while the truth scale does not significantly increase the correction’s effectiveness, it also does not have the “backfire effect” that theories of motivated reasoning might predict. Even when a correction runs counter to a person’s partisanship, the inclusion of a truth scale does not increase the likelihood that the reader will reject the correction or negatively evaluate the outlet that published it.

Fact-Checking, Motivated Reasoning, and Truth Scales

Fact-checking as a genre of journalism consists of evaluating the truth of public claims, typically but not exclusively political claims. This style of reporting has grown tremendously in the United States and overseas over the last decade, and especially since 2010 (Amazeen, 2013; Fridkin, Kenney, & Wintersieck, 2015; Graves, Nyhan, & Reifler, 2015; Kessler, 2014; Stencel, 2016). It is separate from traditional, internal fact-checking by news organizations seeking to weed out errors before publishing a story or to correct past mistakes, although all these reflect journalism’s defining professional preoccupation with factual accuracy (Chalaby, 1996; Karlsson, Clerwall, & Nord, 2016). Contemporary fact-checking has precursors in the muckraking journalism of the early 20th century, which exposed deception in both the political and business worlds—for instance, challenging false claims of patent-medicine producers. However, fact-checking emerged as a distinct genre of reporting in the 1980s and 1990s as part of a broader turn toward interpretive, critical reporting on politics (Fink & Schudson, 2014; Hallin, 1992). So-called “ad watch” stories evaluating campaign commercials took hold in both newspapers and broadcast news in the wake of the contentious 1988 U.S. presidential election (Amazeen, 2012; Graves, 2016). Since 2000, the fact-checking landscape has expanded considerably: FactCheck.org launched in 2003, while PolitiFact.com and the Washington Post’s Fact Checker began in 2007. Non-political fact-checkers have also proliferated: Snopes.com, launched in 1995, was joined by The Consumerist Blog in 2005 and TruthInAdvertising.org in 2013.

These organizations share the broad goal of informing the public and promoting fact-based public discourse (Amazeen, 2013; Graves, 2016) but differ in their approaches. The question of whether to formally rate the accuracy of the claims they investigate has been particularly contentious. By one recent measure, 75% of fact-checking outlets in the United States deploy some sort of truth meter (Stencel, 2016); those that do argue that these scales make fact-checks more engaging and easier to understand. However, other fact-checkers have questioned the use of “inflexible ratings systems” (Jackson, 2012; see also Kessler, 2014) that seem to promise scientific precision, a critique echoed by scholars and media critics (e.g., Hemingway, 2011; Uscinski & Butler, 2013).

These different views raise the question of how the use of ratings affects the impact of fact-checks. Research to date has shown that corrections are more effective at reducing misperceptions when they are presented as fact-checks than when they follow the “he-said/she-said” style of reporting that eschews journalistic adjudication (Pingree, Brossard, & McLeod, 2014; Thorson, 2013). However, their effectiveness at reducing misperceptions depends on the type of misinformation being corrected. Specifically, when fact-checks include partisan cues, they are often less successful at reducing misperceptions (Garrett, Nisbet, & Lynch, 2013) and can even backfire (Nyhan & Reifler, 2010). Partisan-driven motivated reasoning consistently leads people to reject corrections that run counter to their partisan predispositions, which may substantially reduce the overall effectiveness of fact-checking. Examining how rating scales may mitigate or exacerbate such motivated reasoning is thus critical not only for designing better fact-checks, but also for understanding how readers process political information.

Truth Scales’ Effect on Processing Ability

Fact-checks can be understood as a type of persuasion (Garrett & Weeks, 2013)—in other words, a catalyst for attitude change. According to the elaboration likelihood model (ELM), attitude change can occur through two different modes of information processing. In the central route to persuasion, cognitive processing involves careful and thoughtful consideration of messages for quality of arguments. Conversely, the peripheral route involves attitude formation based on non-argument signals or heuristic inferences (Cacioppo & Petty, 1982; Petty & Cacioppo, 1986). Although both routes can result in persuasion, central processing tends to produce stronger and more lasting attitude change.

Whether or not a person processes a correction centrally depends on both her ability and motivation. Contextual corrections—those without rating scales—force readers to rely on the context of the article and may require greater processing skills than corrections that do use scales (MacInnis & Jaworski, 1989). Simple and straightforward disclosures, such as those offered by a rating scale, may be more accessible and thus easier for people to understand. Instead of reading an entire article—as may be necessary for a contextual correction—readers can quickly glance at the rating scale to determine if a given statement is true or false. Rating scales may also make it easier for those who do not have the background knowledge necessary to engage with and understand a contextual correction. Moreover, other research has shown that graphical representations of information in corrections are more convincing than contextual corrections (Nyhan & Reifler, 2011). Thus, we expect that by positively affecting ability, truth scales will make the correction more effective.

H1: Contextual corrections that include rating scales will be more effective at reducing misperceptions than those that do not.

Truth Scales’ Effect on Motivation

H1 predicted that on the whole, the inclusion of truth scales would increase the effectiveness of the correction by increasing the audience’s ability to process the information. However, a truth scale may also affect processing through a second channel, by shaping a reader’s motivation—and this effect might vary depending on partisanship. Substantial research suggests that the success of a correction is strongly affected by a person’s prior beliefs (Johar, 1996; Lewandowsky, Ecker, Seifert, Schwarz, & Cook, 2012; Nyhan & Reifler, 2010, 2012). Psychological reactance can motivate individuals to engage in biased processing or simply ignore a message as invalid (Brehm, 1966; Fennis & Stroebe, 2016). People are biased information processors and often reject information that runs counter to their views and cling to that which reinforces those views, even in the face of contradictory evidence (Garrett & Weeks, 2013; Kunda, 1990; Nyhan & Reifler, 2010; Taber & Lodge, 2006; Thorson, 2013). For example, Democrats may be more resistant to a fact-check challenging the claims of a Democratic candidate and more accepting of a fact-check that refutes the claims of a Republican candidate (Nyhan & Reifler, 2010; Pfau & Louden, 1994).

We expect that the truth scale will serve as a clear, strong partisan cue. If it confirms a person’s pre-existing beliefs, she will be more motivated to centrally process the rest of the message. If, however, it contradicts these beliefs (e.g., by classifying a statement from a politician from her own party as “false”), reactance may be triggered so that she is less willing to cognitively engage with the argument that follows. Thus, we anticipate that when a fact-check is inconsistent with a person’s partisan beliefs, a rating scale will decrease its effectiveness.

H2: Corrections that include rating scales will be less effective than those that do not when the correction is inconsistent with the reader’s prior partisan beliefs.

Truth Scales’ Effect on Reader Attitudes

The content of a fact-check can affect a reader’s evaluation of the sponsoring organization (e.g., a newspaper publisher or fact-checking organization). Rendering judgment opens fact-checkers to accusations of bias (Amazeen, 2013; Graves & Glaisyer, 2012), partly because taking sides contradicts the journalistic norm of objectivity (Cunningham, 2003). Indeed, when the Cleveland Plain Dealer ended its relationship with PolitiFact in 2014, its “reader representative” said it was in part because of concerns about the impact of the rating scale on media trust. He worried that the rating scale “had the effect of making readers suspicious of the objectivity of the whole enterprise” by “[overwhelming] the objectivity of the reporting” (Diadlun, 2014). Consistent with the hostile media effect as well as with motivated reasoning (Vallone, Ross, & Lepper, 1985), Thorson (2013) found that in the fact-checking realm, perceptions of bias were driven by past partisan beliefs. When a media outlet exposed a Republican as making a false claim, Republicans saw the outlet as biased toward Democrats, and vice-versa. At the same time, however, media outlets suffer when they make mistakes, particularly among consumers with low levels of media trust (Karlsson et al., 2016); this suggests that the public holds them to high standards, and may welcome journalists taking a more active role in fact-checking. Indeed, fact-checking as a whole tends to benefit media outlets: Those that regularly engage in fact-checking are evaluated more favorably than those that do not (Pingree et al., 2014; Thorson, 2013).

We predict that the use of rating scales will exacerbate perceptions of bias. While truth scales make it easier for readers to view and process the correction, it is a double-edged sword: Readers can also more easily tell whether the media outlet is criticizing a politician from their own party. Thus, we predict that the effect of truth scales on perceptions of bias will be moderated by partisanship:

H3: When a correction runs counter to a person’s partisan predispositions, the addition of a truth scale will lower evaluations of the fact-checking organization.

Fact-checking organizations are faced with a dilemma. On one hand, rating scales may make fact-checks easier to process and understand, especially for those who are not politically knowledgeable or interested. On the other hand, they may increase the likelihood of a backfire effect in which partisans both reject the correction and view the news organization as less legitimate. Rating-free corrections (of the type used by FactCheck.org) may be less likely to elicit this backfire by encouraging people to read the piece more deeply and better understand the nuances of the issue. However, while a politically disinterested person who is forced to read a contextual correction in an experiment may learn quite a bit from it, would that person ever choose to read such a correction in the real world? In general, people make conscious and often consistent choices about what type of media to consume (Arceneaux & Johnson, 2013), and so estimating the overall effects of a given treatment (e.g., the inclusion of truth scales) also requires investigating the underlying distribution of preferences.

RQ1: Do readers prefer corrections with or without truth scales?

RQ2: Are there demographic, political, or psychological differences that affect correction format preference?

Method

Sample

We tested our hypotheses using an online experiment administered in October 2014 to a nationally representative sample of 1,020 people. Participants were recruited by YouGov, which uses matching and sampling techniques to approximate a nationally representative sample from its opt-in online panel of more than two million (Rivers, 2006). For full sample demographics, see the appendix. The survey took an average of 19 min to complete.

Design and Protocol

Table 1 shows the full 3 × 3 (Correction Type: rating scale, context only, no correction × Statement Type: same party, opposing party, non-political) experimental design. First, two batteries of questions measured participants’ need for cognition and close-mindedness. Next, through a series of branching questions, all participants, including Independents, were sorted into being either closer to the Democratic or Republican parties. Then, after being told that they would read a controversial statement by a public figure that had recently been in the news, participants were randomly assigned to one of three groups (Table 1). Those assigned to the “non-political statement” group saw a claim made by a fictitious corporate executive, Mark Glassman, the chief marketing officer of Mill Foods. Glassman’s claim was presented as follows:

On June 2, 2014, Glassman made this statement in an interview on Good Morning America: “A breakfast of Frosted Oat Loops is clinically shown to improve kids’ attentiveness by nearly 40 percent when compared to children who eat Wheat Puffers.”

Table 1.

Experimental Design.

	Correction type
	Rating scale: Contextual correction with visual rating	Context only: Contextual correction without visual rating	No correction^a
Statement type
Same party	n = 144	n = 128	n = 62
Opposing party	n = 142	n = 135	n = 66
Non-political	n = 144	n = 129	n = 70

The “no correction” condition had fewer participants because of oversampling the two correction format conditions for the purposes of a segmentation analysis associated with a separate study. The separate study examined participant choice. Half of the participants in the correction format conditions were offered a choice of the correction format they saw. We found no differences in how participants who chose their correction format and participants randomly assigned to correction format reacted to the corrections.

This scenario, based on a federal false-advertising complaint against Kellogg’s Frosted Mini Wheats (see “Kellogg,” 2009), serves as a non-partisan contrast to the fact-checks of statements by political figures.

The other two groups both saw a statement attributed to a fictitious Congressman, Daniel Stacks: “During the 2014 election, Stacks said this about his opponent, John Hunter: ‘One hundred percent of John Hunter’s ads have been negative.’” For participants in the “same party statement” group, Stacks’s party affiliation matched their own (i.e., if the respondent was a Democrat, then Stacks was also described as a Democrat). For those in the “opposing party statement” group, Stacks was described as being a member of the opposite party (i.e., if the respondent was a Democrat, then Stacks was described as a Republican). This scenario is based on a PolitiFact item from the 2008 presidential race that debunked a charge by Senator Barack Obama against Senator John McCain (Farley, 2008).

Correction type

Participants were randomly assigned to one of three groups. The “no correction” group saw no correction of the statement. The “context only” group viewed the public figure’s statement followed by a correction attributed to GetTheFacts.org (a fictional organization), which included a few paragraphs of text explaining why the statement was largely inaccurate but did not include a rating scale (Figure 2). (The fact-check debunking the claim about cereal was 270 words and the political fact-checks were 231 words. None of the context-only corrections included the words mostly false in the text of the article.) The “rating scale” group viewed the same statement and contextual correction with the addition of a visual rating scale clearly labeling the statement as mostly false (Figure 3). Thus, the only difference between the correction stimuli in either the political or non-political conditions was the presence or absence of the visual rating scale identifying the statement as mostly false. The 5-point rating scale, borrowed with permission from an overseas fact-checking site (Georgia’s Reforms Associates [GRASS] FactCheck), conforms to the design of many real-world rating systems, which typically combine a scale graphic and a word or phrase indicating the level of truth (Figure 1, for examples).

The fact-check was followed by a brief distractor task, after which participants evaluated four people using a 0 to 100 feeling thermometer: Mark Glassman, Daniel Stacks, Katie Couric, and Anderson Cooper. A feeling thermometer was again used for evaluating five constructs: frosted oat loops cereal, the advertising industry, journalists, GetTheFacts.org, and Congress. Next, all participants except those in the “no correction” condition evaluated GetTheFacts.org along five dimensions: unfair/fair, biased/unbiased, does not tell the whole story/tells the whole story, inaccurate/accurate, and untrustworthy/trustworthy.

Two manipulation checks measured whether participants could recall the party of Daniel Stacks and the identity of GetTheFacts.org. Participants then answered whether, in their own opinion, Stacks’s or Glassman’s statement was true, mostly true, half true, mostly false, or false, after which they completed an open-ended question asking them to “please list every argument that makes you think the statement is [previous answer].” Finally, all participants except those in the uncorrected condition were asked to “set aside [their] own opinion” and indicate what GetTheFacts concluded about the statement.

Measures

Manipulation checks

The final two questions of the experiment served as manipulation checks. Participants were asked to identify the party of the candidate, Stacks. When Stacks was described as a member of the opposing party, 81.5% of respondents correctly identified his party, and when he was the same party as the participant, 80.7% were correct. Those who saw a correction were also asked to identify GetTheFacts.org (the choices were “a non-partisan fact-checking organization,” “a fact-checking website run by MSNBC,” and “a fact-checking website run by FOX News”). In total, 86.0% of participants correctly identified GetTheFacts.org as a non-partisan fact-checking organization.

Preference for context or rating scales

A random subset of the participants (n = 403) was asked whether they preferred corrections with rating scales. The question explained that some fact-checking organizations present readers with evidence and rank that evidence using truth scales and others present evidence without using scales so that readers can judge the evidence for themselves, then asked participants to indicate which type of fact-check they would prefer to see.

Effectiveness of correction

The survey included both a question asking respondents to give their own opinion on whether the statement was true or false as well as a request to then set aside that opinion and recall how GetTheFacts.org ruled on the statement. Studies of explicit memory versus implicit memory demonstrate that attitude change is a better measure of message effectiveness than are recall measures (Fennis & Stroebe, 2016). Thus, to test whether the corrections were effective, we examined whether participants believed the correction (in effect, attitude change) rather than whether they remembered what the evaluation was. Participants were asked to report, in their own opinion, what they thought about the statements by Daniel Stacks (political candidate) or Mark Glassman (businessman). They could respond that the statement was true (1), mostly true (2), half true (3), mostly false (4), or false (5) for Stacks or Glassman. The responses were then recoded into a “fact-checking distance” measure. People who answered mostly false (i.e., the same as GetTheFact.org’s rating) received a high score of 4 on this measure. People who answered either false or half true received a score of 3 because they were one scale point off from the fact-checking evaluation. People who answered mostly true received a score of 2, and those who answered true received a low score of 1. This led to a measure that ranged from 1 (least accurate) to 4 (most accurate) for Stacks (M = 3.05, SD = 0.69) or Glassman (M = 3.32, SD = 0.59). (As discussed in the results below, alternative approaches to coding these answers yielded substantively similar findings.)

Other dependent variables

To investigate the effects of correction formats on feelings toward public figures, GetTheFacts.org, and other institutions, participants responded to several feeling thermometer questions. They reported whether they felt cool/unfavorable (0), warm/favorable (100), or somewhere in between toward Stacks (M = 41.30, SD = 22.21), Glassman (M = 43.10, SD = 19.23), GetTheFacts.org (M = 60.88, SD = 22.07), Frosted Oat Loops cereal (M = 41.27, SD = 25.76), advertising (M = 36.17, SD = 22.41), Congress (M = 27.97, SD = 22.42), and journalists (M = 46.98, SD = 25.36).

To measure an individual’s close-mindedness, participants responded to a four-item scale adapted from Pingree et al. (2014) ranging from strongly disagree (1) to strongly agree (6). Statements consisted of (a) “I consider as many different options on a problem as possible,” (b) “In conflict situations, I can see how both sides could be right,” (c) “I see many possible solutions to problems,” and (d) I do not usually consult many different opinions before forming my own view (reverse coded). The four items were found to be low on internal consistency (Cronbach’s α = .53). Dropping statement (d) led to a minimally acceptable level of consistency with an α score of .68 between the remaining three items that were averaged together (M = 4.62, SD = 0.88).

Participants responded to two items to measure political interest. First, they reported their level of interest in politics generally (M = 3.10, SD = 0.99), ranging from not at all interested (1) to very interested (4). Second, they reported how much they follow politics (M = 3.33, SD = 0.92), ranging from hardly at all (1) to most of the time (4). The two measures were significantly correlated (Pearson’s r = .75, Spearman’s r = .75, p < .001), so they were averaged to form a two item political interest measure (M = 3.23, SD = 0.89).

Finally, participants responded to general and specific measures of media trust and credibility. A “trust in media” variable (M = 2.47, SD = 0.96) was based on a 5-point scale ranging from never (1) to all the time (5). A credibility index was based on a five item, 7-point bi-polar scale asking respondents to assess whether GetTheFacts.org was fair, unbiased, tells the whole story, accurate, and trustworthy (M = 4.65, SD = 1.34, α = .940).

Results

In both the non-political and political conditions (pooled to include both opposing and same-party affiliated participants for this initial analysis), correcting the misinformation was effective: combining both correction formats, those who saw a correction were significantly more accurate in their assessment of the controversial statement than those who did not see a correction.¹ H1 predicted that compared with a context-only correction, a correction that included a rating scale would be more effective. Figure 4 shows the average accuracy of respondents by the type of correction.

There was a significant effect of correction format type on non-political beliefs, F(2, 340) = 4.74, p < .01. In the non-political group, 39.7% of those who saw the correction with the rating scale correctly said that the statement was mostly false, compared with 34.1% of those who read only the correction without the rating scale. Planned contrasts revealed that respondents who saw a correction with both context and a rating scale (M = 3.43, SD = 0.54) were significantly more accurate in their beliefs (t = −2.59, p = .01) than those who saw only a contextual correction (M = 3.25, SD = 0.64). Indeed, the context-only correction was not statistically different than people who saw no correction at all (M = 3.21, SD = 0.56).² Thus, for the non-political group, H1 is confirmed: Adding a rating scale to a contextual correction increased its effectiveness. There was no significant difference between the average survey time of those who saw the rating scale (M = 152 s, SD = 1,028 s) and those who did not (M = 213 s, SD = 835 s), suggesting that the rating scale did not lead people to engage less deeply with the content. And those who saw the rating scale were also more accurate in recalling GetTheFacts.org’s ruling on the issue (t = −4.36, p < .001), suggesting that the rating scale was effective at changing attitudes partly because it improved respondents’ ability to process and then later recall the relevant information.

Among participants exposed to the pooled political misinformation, there was a significant effect of correction format on political beliefs, as well, F(2, 674) = 10.35, p < .0001. Planned contrasts revealed that both the context only (M = 3.05, SD = 0.65) and context with ratings (M = 3.14, SD = 0.70) correction formats were equally successful at correcting beliefs compared with those receiving no correction (M = 2.81, SD = 0.72). Unlike in the non-political condition, the presence of rating scales did not significantly increase the correction’s effectiveness as predicted by H1. Thus, participant reactions to the rating scale in the political misinformation condition does not support H1.

H2 predicted that the effectiveness of a truth scale would be affected by partisanship: Specifically, those contextual corrections which include rating scales will trigger reactance and thus be less effective than those that do not when the correction is inconsistent with the reader’s prior partisan beliefs. Figure 5 shows the effectiveness of the correction for those who have the same-party affiliation as the candidate (i.e., those predisposed to react with distrust toward the correction) and those who have the opposing affiliation (predisposed to believe the correction).

When a candidate was from the same party as the participant, there was a significant effect of correction format on pro-attitudinal beliefs, F(2, 331) = 11.24, p < .0001. Planned contrasts indicated that both context only (M = 2.91, SD = 0.68) and context with ratings (M = 3.04, SD = 0.76) correction formats were successful at increasing the likelihood that respondents correctly understood the statement was incorrect (or mostly false) compared with no correction (M = 2.51, SD = 0.78). In contrast, the correction format manipulation had no effect on counter-attitudinal beliefs, F(2, 340) = 1.56, p = n.s. When a candidate was from the opposing party, the type of correction format made no difference: Neither context only (M = 3.19, SD = 0.60) nor context plus ratings (M = 3.25, SD = 0.62) correction formats were statistically better at correcting misinformation compared with no correction (M = 3.09, SD = 0.52). Thus, H2 is not supported: Rating scales do not decrease the effectiveness of a correction by triggering reactance when it runs counter to a person’s pre-existing beliefs. These results suggest that contrary to some critics’ concerns, adding a rating scale does not decrease the effectiveness of the correction, even when the participant’s partisanship disinclines them to believe the correction in the first place.

H3 predicted that when a correction runs counter to a person’s partisan predispositions, the addition of a truth scale will make them evaluate the fact-checking organization more negatively than if the correction supports their political views. When GetTheFacts.org fact-checked a candidate who shared the respondent’s party (M = 4.53, SD = 1.31) it was viewed as less credible than when it fact-checked the opposition (M = 4.79, SD = 1.27). This difference was significant, F(3, 634) = 5.34, p < .05. The organization was also viewed as significantly more credible, F(3, 634) = 6.07, p < .05, when a ratings scale was included (M = 4.87, SD = 1.28) than when it was omitted (M = 4.49, SD = 1.37). However, the lack of significant interaction, F(3, 634) = 2.52, p = n.s., between the partisanship of the fact-check and the addition of the rating scale suggests that H3 is not supported: The inclusion of a rating scale does not significantly increase the backlash incurred by fact-checking a person’s favored candidate. The addition of a rating scale thus appears to offer substantial benefits to perceived credibility but no measurable cost.³

Finally, to determine which correction format participants prefer (RQ1), a random subset of respondents was allowed to choose which type of correction they wanted to see. A one-way χ² test, χ²(1, n = 403) = 5.03, p < .05, indicated that the rating plus context condition was selected more frequently (56%) than the context-only condition (44%), suggesting that the public has a slight preference for ratings scales. A logistic regression assessed the demographic (age, sex, education), political (ideology, political interest), and psychological (need for cognition, close-mindedness) variables that predicted participants’ preference for the rating scale (RQ2). Overall, the only significant predictor was close-mindedness (B = .267, SE = .127, p = .035): Those who were more open-minded were more likely to prefer the version of the correction with the truth scale.

Discussion

The results of this study suggest that truth scales can substantially increase people’s understanding of the world around them and that their use comes with few disadvantages. The effectiveness of rating scales is especially pronounced when they are used to correct non-political misinformation. This outcome may be due in part to participant unfamiliarity with the fictitious brand and the low-involvement nature of the product category, which, in this case, was a breakfast cereal. Consistent with the ELM theory of persuasion, the cues provided by the ratings icon may have facilitated the ability of participants to process the information given the low-involvement nature of the stimulus: cereal. Since most consumer advertising is geared toward low-involvement purchases (Fennis & Stroebe, 2016), corrections of consumer-related misinformation may be most effective at informing the public when fact-checkers use a truth scale in conjunction with a contextual correction.

In contrast, varying the type of correction format had no effect when political misinformation was involved. Most critically, for this study, psychological reactance was not triggered by the use of rating scales. However, consistent with prior research, the effects of partisanship appear to dampen the effects of corrections. When the candidate was of the same party as the respondent, both correction formats were equally effective at correcting misinformation; no reactance effects were observed. However, when the candidate was from the opposing party, neither correction format was effective. These findings may at first seem to run counter to the theory of motivated reasoning. From this perspective, participants should be less likely to accept a correction that discredits a candidate from their own party and more likely to accept a correction that discredits the opposing party. However, the observed results may be attributable to a “ceiling effect” when it comes to believing the worst about the opposing party. When people are presented with a statement made by someone in the opposing party, their baseline skepticism of the statement is substantially higher than if the statement had been by someone of their own party. Looking only at the “uncorrected” group, 50% of people in the opposing-party condition said Stacks’s statement was either false or probably false, versus just 9.7% of people in the same-party condition. Because 50% of people in the opposing-party condition already believed that he was lying, this leaves relatively little room for the correction to have an effect. Overall, these results reinforce the difficulty of overcoming partisan-driven motivated reasoning. Partisanship consistently moderated the political correction’s effectiveness in educating the public.

Beyond effectiveness, this study indicates that attitudes toward fact-checkers are also affected by partisanship—a finding consistent with the hostile media effect and motivated reasoning (Vallone et al., 1985). If a tertiary goal of fact-checking is to have positive effects on journalism, it is noteworthy—yet concerning—that people feel more favorably toward fact-checkers when they correct the opposition and less so when they correct one’s own party. However, the findings also suggest that adding a rating scale may help to mitigate some of these effects. Although people are affected by their predisposed biases, they do seem to show some appreciation for making corrections more accessible via the use of ratings.

Finally, while a majority of respondents preferred to see a ratings icon, many also expressed a preference for the contextual correction without the rating scale. The only individual attribute associated with a preference for a ratings scale was being open-minded. Interestingly, the politically uninvolved did not express an explicit preference for a rating scale, despite it potentially increasing their ability to process complex information.

Limitations and Future Research

As with any experimental study, it is important to examine the limitations of these findings. Looking first at the generalizability of the treatment, to what extent are the treatments used in this experiment (both the nature of the misinformation and the content of the fact-check) representative of their real-world analogues? The product selected for the non-political information, breakfast cereal, is likely a low-involvement category (Ratchford, 1987). Furthermore, a fictitious brand of cereal was utilized. Future studies should examine whether the current results hold with the use of a nationally recognized brand. Reactance effects may be more likely if a popular brand is exposed as misinforming the public. Higher-involvement categories of products (like automobiles) where people have strong pre-existing opinions may also produce a pattern of results more similar to those seen in the political misinformation condition.

The political misinformation concerned a false accusation made against an opposing candidate. While the political statements examined by fact-checking organizations vary widely from policy statements to biographical claims, accusations about the opposition are consistently a staple. However, this choice may have contributed to the observed ceiling effect—when it comes to politicians lying about their opponents’ record, people may simply assume the worst. It is also possible that the observed effects of motivated reasoning would be greater for fact-checks involving particular issues in which respondents are deeply invested, such as abortion or gun control.

The correction manipulation included only one type of rating scale evaluation: mostly false. This particular judgment is the most theoretically interesting, as it creates the ideal context for the type of motivated reasoning that confounds so many attempts to correct political misinformation. However, it is worth noting that we expect fact-checks of true or mostly true to have opposite effects—in other words, when a politician is of their own party, a fact-check of true will lead readers to be more likely to accept the correction and evaluate the fact-checking organization positively. Furthermore, ratings that indicate a higher degree of uncertainty, such as a half true rating, might be more difficult for readers to process. It is unclear how ambiguous evaluations would affect acceptance and attitudes. Therefore, although our study intentionally focused on fact-checks of inaccurate statements, future research should address the inverse relationship that may exist with more accurate statements as well as those that are more ambiguous.

Conclusion

The enterprise of fact-checking continues to grow. In the United States, fact-checking references in newspapers have increased more than 900% since 2001 and increased above 2,000% in broadcast media (Amazeen, 2013). Worldwide, at least 96 active fact-checking organizations have been documented in locations such as Turkey, Uruguay, and South Korea (Stencel, 2016). Moreover, the practice of fact-checking now extends beyond just politics. The accuracy of blockbuster movies such as Argo and Selma has been checked (“Fact-Checking ‘Argo,’” 2012; Lockett, 2014). Sites such as TruthInAdvertising.org and HealthNewsReview.org have emerged to verify the accuracy of marketing and advertising claims about consumer products and health services. Even the political fact-checkers have begun expanding their targets beyond politics: In 2015, PolitiFact fact-checked the claim of a national insurance advertiser during Super Bowl XLIX, and FactCheck.org introduced its SciCheck feature to focus on scientific claims (Contorno, 2015; Kiely, 2015). As the practice of fact-checking expands, so too must our understanding of whether and how it is achieving its intended goals of improving public knowledge and promoting fact-based political discourse. The answers to these questions are not only practically important for fact-checkers, but can also shed light on our understanding of how people process corrective interventions, including the role of motivated reasoning. Overall, the results of this study provide some room for optimism: At best, “truth scales” appear to increase the effectiveness of corrections—and this comes with little downside.

Footnotes

Appendix

Acknowledgements

The authors thank the American Press Institute and the Democracy Fund for supporting this research. The authors also extend thanks to Kevin Arceneaux for his suggestions and Paata Gaprindashvili and Georgia’s Reforms Associates (GRASS) FactCheck () for permission to use its rating icons in the experiment.

Authors’ Note

A previous version of this article was presented at the Midwest Political Science Association annual meeting in Chicago, Illinois, on April 16, 2015. The analysis and any errors herein are authors’ own.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) declared receipt of the following financial support for the research, authorship, and/or publication of this article: Funding was provided by a grant from the Democracy Fund via the American Press Institute.

Notes

Author Biographies

Michelle A. Amazeen (PhD, Temple University) is an assistant professor at Boston University. Her research interests are cross-disciplinary at the intersection of advertising, journalism, and political communication, often exploring the effects of mis/disinformation and the effectiveness of corrections. She has examined the prevalence and corrections of inaccuracies in political advertising and how political actors are held accountable for their claims. She has also explored the authenticity of strategic communication efforts in the mediated information environment including the alignment of corporate social responsibility campaigns with advertising.

Emily Thorson joined the Boston College political science department in 2015. Before that, she was an assistant professor at the School of Media and Public Affairs at George Washington University. She received a dual PhD in communications and political science at the Annenberg School and at the Department of Political Science at the University of Pennsylvania. Her research focuses on information effects in politics, both on the aggregate and individual level. Specifically, she traces how political information and misinformation reach citizens (through traditional and new forms of media as well as through informal interactions) and the conditions under which this information affects their attitudes and behavior.

Ashley Muddiman earned her PhD from the Department of Communication Studies at the University of Texas at Austin. Her research broadly focuses on the content and effects of political media. More specifically, she studies political incivility, including citizens’ perceptions of incivility as well as the effects of political incivility in online settings. She also works with grant-funded research groups, including the Engaging News Project, to explore ways to encourage more citizen engagement with digital news. She teaches courses in political media and research methods, including rhetoric, politics, and mass media; new media and politics; and introduction to behavioral research methods in communication.

Lucas Graves (PhD, Columbia) an assistant professor in the School of Journalism and Mass Communication at the University of Wisconsin - Madison. His work examines new journalistic norms, practices, and organizations in the digital age. His book Deciding What’s True: The Rise of Political Fact-Checking in American Journalism was published in September 2016 by Columbia University Press. His writing has appeared in the New York Times, the Columbia Journalism Review, the Nieman Journalism Lab at Harvard University, and many other outlets.

References

Amazeen

M. A.

(2012). Blind spots: Examining political advertising misinformation and how U.S. news media hold political actors accountable. Dissertation Abstracts International, 74(02). Retrieved from http://gradworks.umi.com/35/39/3539244.html

Amazeen

M. A.

(2013). Making a difference: A critical assessment of fact-checking in 2012 (New America Foundation Media Policy Initiative Research Paper). Retrieved from https://www.newamerica.org/new-america/making-a-difference/

Arceneaux

Johnson

(2013). Changing minds or changing channels? Partisan news in an age of choice. Chicago, IL: The University of Chicago Press.

Brehm

J. W.

(1966). A theory of psychological reactance. Oxford, UK: Academic Press.

Cacioppo

J. T.

Petty

R. E.

(1982). The need for cognition. Journal of Personality and Social Psychology, 42, 116-131.

Chalaby

J. K.

(1996). Journalism as an Anglo-American invention: A comparison of the development of French and Anglo-American journalism, 1830s-1920s. European Journal of Communication, 11, 303-326.

Contorno

(2015, February 2). Nationwide super bowl ad claims accidents are the leading cause of death among children. PolitiFact. Retrieved from http://www.politifact.com/truth-o-meter/statements/2015/feb/02/nationwide/nationwide-super-bowl-ad-claims-accidents-are-lead/

Cunningham

(2003, July 8). Rethinking objective journalism. AlterNet. Retrieved from http://www.alternet.org/story/16348/rethinking_objective_journalism

Darke

P. R.

Ashworth

Ritchie

R. J. B.

(2008). Damage from corrective advertising: Causes and cures. Journal of Marketing, 72, 81-97.

10.

Diadlun

(2014, January 24). Coming: Truth-squad reporting without the gimmicks. The Plain Dealer. Retrieved from http://www.cleveland.com/readers/index.ssf/2014/01/coming_truth_squad_reporting_w.html#incart_river

11.

Fact-checking “Argo”: A great escape that takes some leaps. (2012, December 18). National Public Radio. Retrieved from http://www.npr.org/2012/12/25/167537259/fact-checking-argo-a-great-escape-that-takes-some-leaps

12.

Farley

(2008, October 15). No, McCain’s ads haven’t all been negative. PolitiFact. Retrieved from http://www.politifact.com/truth-o-meter/statements/2008/dec/03/barack-obama/no-mccains-ads-havent-all-been-negative/

13.

Fennis

B. M.

Stroebe

(2016). The psychology of advertising (2nd ed.). New York, NY: Routledge.

14.

Fink

Schudson

(2014). The rise of contextual journalism, 1950s–2000s. Journalism, 15, 3-20.

15.

Fridkin

Kenney

P. J.

Wintersieck

(2015). Liar, liar, pants on fire: How fact-checking influences citizens’ reactions to negative advertising. Political Communication, 32, 127-151.

16.

Garrett

R. K.

Nisbet

E. C.

Lynch

E. K.

(2013). Undermining the corrective effects of media-based political fact checking? The role of contextual cues and naïve theory. Journal of Communication, 63, 617-637.

17.

Garrett

R. K.

Weeks

B. E.

(2013, February 23-27). The promise and peril of real-time corrections to political misperceptions. Paper presented at the ACM Conference of Computer Supported Cooperative Work, San Antonio, TX.

18.

Graves

(2016). Deciding what’s true: The rise of political fact-checking in American journalism. New York, NY: Columbia University Press.

19.

Graves

Glaisyer

(2012). The fact-checking universe in spring 2012: An overview (New America Foundation Media Policy Initiative Research Paper). Retrieved from http://newamerica.net/sites/newamerica.net/files/policydocs/The_Fact-checking_Universe_in_2012.pdf

20.

Graves

Nyhan

Reifler

(2015). The diffusion of fact-checking: Understanding the growth of a journalistic innovation (American Press Institute Research Paper). Retrieved from http://www.americanpressinstitute.org/wp-content/uploads/2015/04/The-Growth-of-Fact-Checking.pdf

21.

Hallin

D. C.

(1992). The passing of the “high modernism” of American journalism. Journal of Communication, 42(3), 14-25.

22.

Hemingway

(2011, December 19). Lies, damned lies, and “fact checking.” The Weekly Standard. Retrieved from http://www.weeklystandard.com/lies-damned-lies-and-fact-checking/article/611854

23.

Jackson

(2012, December 21). Firefighters, fact-checking and American journalism. FactCheck.org. Retrieved from http://www.factcheck.org/2012/12/firefighters-fact-checking-and-american-journalism/

24.

Johar

G. V.

(1996). Intended and unintended effects of corrective advertising on beliefs and evaluations: An exploratory analysis. Journal of Consumer Psychology, 5, 209-230.

25.

Karlsson

Clerwall

Nord

(2016). Do not stand corrected: Transparency and users’ attitudes to inaccurate news and corrections in online journalism. Journalism & Mass Communication Quarterly. Advance online publication. doi:10.1177/1077699016654680

26.

Kellogg, FTC settle false advertising charges. (2009, April 20). NBCnews.com. Retrieved from http://www.nbcnews.com/id/30311211/ns/business-consumer_news/t/kellogg-ftc-settle-false-advertising-charges/#.VkeSBzqM-yN

27.

Kessler

(2014, June 13). The global boom in political fact checking. Washington Post. Retrieved from http://www.washingtonpost.com/blogs/fact-checker/wp/2014/06/13/the-global-boom-in-fact-checking/

28.

Kiely

(2015, January 28). FactChecking science-based claims. FactCheck.org. Retrieved from http://www.factcheck.org/2015/01/factchecking-science-based-claims/

29.

Kunda

(1990). The case for motivated reasoning. Psychological Bulletin, 108, 480-498.

30.

Lewandowsky

Ecker

U. K. H.

Seifert

C. M.

Schwarz

Cook

(2012). Misinformation and its correction: Continued influence and successful debiasing. Psychological Sciences in the Public Interest, 13, 106-131.

31.

Lockett

(2014, December 24). How accurate is Selma? Slate. Retrieved from http://www.slate.com/blogs/browbeat/2014/12/24/selma_fact_vs_fiction_how_true_ava_duvernay_s_new_movie_is_to_the_1965_marches.html

32.

MacInnis

D. J.

Jaworski

B. J.

(1989). Information processing from advertisements. Journal of Marketing, 53, 1-23.

33.

Nyhan

Reifler

(2010). When corrections fail: The persistence of political misperceptions. Political Behavior, 32, 303-330.

34.

Nyhan

Reifler

(2011, April). “Opening the political mind? The effects of self-affirmation and graphical information on factual misperceptions. Paper presented at the annual convention of the Midwest Political Science Association, Chicago, IL.

35.

Nyhan

Reifler

(2012). Misinformation and fact-checking: Research findings from social science (New America Foundation Media Policy Initiative Research Paper). Retrieved from https://www.newamerica.org/oti/policy-papers/misinformation-and-fact-checking/

36.

Petty

R. E.

Cacioppo

J. T.

(1986). Communication and persuasion: Central and peripheral routes to attitude change. New York, NY: Springer.

37.

Pfau

Louden

(1994). Effectiveness of adwatch formats in deflecting political attack ads. Communication Research, 21(3), 325-341.

38.

Pingree

R. J.

Brossard

McLeod

D. M.

(2014). Effects of journalistic adjudication on factual beliefs, news evaluations, information seeking, and epistemic political efficacy. Mass Communication and Society, 17, 615-638.

39.

Ratchford

(1987). New insights about the FCB grid. Journal of Advertising Research, 27(4), 24-38.

40.

Rivers

(2006). Sample matching: Representative sampling from internet panels (Polimetrix White Paper Series). Retrieved from http://psfaculty.ucdavis.edu/bsjjones/rivers.pdf

41.

Stencel

(2016, February 16). Global fact-checking up 50% in past year. Duke Reporters Lab. Retrieved from http://reporterslab.org/global-fact-checking-up-50-percent/

42.

Taber

C. S.

Lodge

(2006). Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science, 50, 755-769.

43.

Thorson

(2013, April). The consequences of misinformation and fact-checking for citizens, politicians, and the media. Paper presented at the annual meeting of the Midwest Political Science Association, Chicago, IL.

44.

Uscinski

J. E.

Butler

R. W.

(2013). The epistemology of fact checking. Critical Review: A Journal of Politics and Society, 25(2), 162-180.

45.

Vallone

R. P.

Ross

Lepper

M. R.

(1985). The hostile media phenomenon: Biased perception and perceptions of media bias in coverage of the Beirut massacre. Journal of Personality and Social Psychology, 47, 577-585.