Boosting Media Literacy Using Lateral Reading and Online Search Interventions

General disclosures

Conflicts of interest: All authors declare that they have no conflicts of interest. Funding: This project was funded by the European Commission (Horizon 2020 Grant No. 01094752, SoMe4Dem) and received support from the Max Planck Institute for Human Development. Artificial intelligence: No artificial-intelligence-assisted technologies were used in this research or in the creation of this article. Ethics: This research received approval from the ethics board of the Max Planck Institute for Human Development on October 2, 2024 (ID: A 2024-29 and Amendment A 2024-39).

Study disclosures

Preregistration: The hypotheses, methods, and analysis plan were preregistered (https://osf.io/tsb68/) prior to data collection. There were no deviations from the preregistration. Materials: All study materials are publicly available on the Open Science Framework, or OSF (https://osf.io/wtbr5/). In addition, the lateral-reading video is available on YouTube (https://www.youtube.com/watch?v=7_ZyBzGsehc). Data: All primary survey and experimental data are publicly available on OSF (https://osf.io/wtbr5/). Sensitive and otherwise nonanonymizable web-tracking data is available in aggregated form and detached from the survey data. According to the contract conditions agreed with NewsGuard Technology Inc., the credibility labels of NewsGuard cannot be shared. Analysis scripts: All analysis scripts are publicly available on OSF (https://osf.io/wtbr5/). Computational reproducibility: The computational reproducibility of the results in the main article (but not the Supplemental Material) has been independently confirmed by the journal’s STAR team. Open Science Framework (OSF): To ensure long-term preservation, we registered all OSF files at https://osf.io/sdpef/.

Sampling

This study uses an online survey of a national sample of Germans (N = 2,666), supplemented with passively collected web-tracking data from mobile or desktop devices of a subset of respondents (n = 436). Data were collected by the survey company YouGov from members of their nationally representative German panel (n = 2,330) and members of their Pulse panel, (n = 436) who gave their informed consent for their online activities to be tracked anonymously. For members of the Pulse panel, the tracking software monitored web traffic (excluding passwords and financial transactions) across desktop browsers or mobile devices. Users had to consent before installing the software and could disable or uninstall it at any time. Identifying information was not collected, but as tracking data can contain information that could be used to identify participants, a strict data-management procedure was followed. For the main analysis of treatment effects, samples from both panels were pooled into the total sample of N = 2,666. Of these participants, N = 2,120 also completed the second wave. Attrition rates range between 19% and 21% and showed no significant differences between conditions. The sample size was fully determined by the research budget. The pooled sample is representative for the German adult population with regard to age, gender, education, state of residency, and voting intentions for the 2025 federal election. See Table S1 in the Supplemental Material for a comparison with marginals from the 2024 German Microcensus (Statistisches Bundesamt, 2024) and the 2025 election results (Die Bundeswahlleiterin, 2025).

The first survey wave was conducted between November 21 and December 2, 2024. Participants were invited to the second wave roughly 2 weeks after filling out the first wave and were allowed several days to fill out the second survey to minimize attrition between waves, resulting in a fielding period of the second wave between December 9 and December 24, 2024.

Web-tracking data was collected between November 4, 2024 and December 31, 2024, spanning a 2-month tracking period that included more than 6 million website visits, of which 390,000 (approximately 6.5%) could be labeled using NewsGuard.

Experiment

Participants were randomly assigned to one of the three experimental conditions (lateral reading, online search, control). Participants were individually randomized at the start of the survey; participants who replaced dropouts were also individually randomized. Participants were screened and replaced before being assigned a condition if YouGov could not confirm that they were able to watch a video.

Participants evaluated websites at three time points throughout the study: pretreatment and posttreatment within Wave 1, and follow-up in Wave 2 (see Fig. 1).

The lateral-reading condition included a 5-min video explaining the lateral-reading strategy and demonstrating an examination of an untrustworthy website with a screen recording. The video was presented after a pretreatment set of four websites (four more websites were evaluated in the posttreatment block, and the final four websites were evaluated in the follow-up assessment in Wave 2). The instructions for the online-search intervention, explaining how to find evidence around the central claim, were translated from Aslett et al. (2023) and presented as text after a pretreatment set of four websites. Participants in the control condition received no treatment (see Section 1 in the Supplemental Material for treatment instructions).

Following best practices, participants were debriefed after each wave (Greene et al., 2022). We judged that the debriefing of participants, which was necessary to minimize the impact of exposing them to misinformation, as more important than avoiding potential learning effects (for instance, by alerting participants about the potential untrustworthiness of websites).

Stimulus websites

The stimulus material consisted of six trustworthy and six untrustworthy websites containing online articles on specific issues (COVID-19, climate change, migration, international organizations, the integrity of different elections, and the war in Ukraine). For every untrustworthy article, there was one trustworthy article on the same topic. Four websites were selected for the pretreatment assessment, and another four for the posttreatment assessment; the remaining four were shown to participants in Wave 2 for a follow-up assessment. The presentation of stimuli was fully randomized between participants and across waves, with the constraint that in each phase (pretreatment, posttreatment, follow-up), two trustworthy/true items and two untrustworthy/false items were presented.

Stimulus websites (web pages of news sites that contained a news article on a specific issue) were selected using a systematic bottom-up strategy involving an external data provider (NewsGuard), a systematic protocol, and a pretest with 301 participants in Prolific for stimulus evaluation. Websites that were not informative for the assessment of participants’ ability to discern credibility and trustworthiness (e.g., because of ceiling effects—websites that everyone rated correctly, leaving no room for improvement) were not included in the main study. The stimulus pretest was registered separately on OSF (https://osf.io/fp8cs/overview) prior to data collection and was approved independently of the main study by the ethics board of the Max Planck Institute for Human Development on May 13, 2024 (see Section 4 in the Supplemental Material for details on the stimulus-selection procedure and results of the stimulus-selection pretest).

Measurement

To enable a comparison between both strategies, we used two parallel outcome measures: source trustworthiness and claim credibility. The trustworthiness of websites is defined as the extent to which a website can be relied upon to provide accurate information about a specific domain or subject (e.g., politics or health). Similarly, claim credibility is defined as the extent to which the central claim of an online article is based on factually accurate information. Both definitions align with the evaluation criteria used by NewsGuard, the independent rating system for news and information websites that we used here (NewsGuard Technologies, Inc., 2024).

The instructions for assessing source trustworthiness and claim credibility were the same under all three conditions. In all conditions, in the instructions, a 2-min time limit (per website) was stated, although technically it was not enforced. The instructions were as follows (translated from German):

In the following, you will visit different websites. We will then ask you for your personal evaluation of the website. As you probably know, you often come across advertising on the Internet. Sometimes you will be asked to consent to direct messages, sign up for a newsletter, or even make a donation. Please do not respond to any of these requests when visiting the websites within our study. Some of the following websites contain untrustworthy information. At the end of the study, we will inform you in detail which websites are not trustworthy and why.

The instructions presented before every website were (translated from German):

Please follow the link [link to article]. The website should open in a new browser tab.

Please take no longer than 2 minutes to reach your judgment.

Participants rated two primary outcomes: source trustworthiness and claim credibility. Source trustworthiness was elicited with the question, “How trustworthy do you find the website?” (definitely trustworthy, probably trustworthy, probably not trustworthy, definitely not trustworthy). Claim credibility was elicited with the question, “How credible do you find the central claim of the article?” (definitely credible, probably credible, probably not credible, definitely not credible). We report the outcomes at two levels of granularity: For both preregistered and exploratory model-based analyzes (Figs. 2b and 3a), we use the full 4-point scale ratings (i.e., including their expression of confidence, such as definitely trustworthy vs. probably trustworthy; see Fig. 2a). For exploratory and descriptive analyses, we dichotomized the scale and report proportions of correct ratings to simplify the presentation of results and to aid interpretation (Figs. 2c, 3b, and 3c).

OPEN IN VIEWER

Fig. 2.

Treatment effects. In (a) are shown raw source-trustworthiness ratings on 4-point scale by condition and assessment time point. Black diamonds above the dashed line indicate mean ratings for true and trustworthy items (error bars representing bootstrapped 95% confidence intervals, or CIs, are hidden because they are smaller than the diamond shape); black diamonds below the line indicate mean ratings for false or untrustworthy items. In (b) are shown average treatment effects as preregistered (left: pretreatment vs. posttreatment; right: pretreatment vs. follow-up). Coefficients indicate effects of the interventions on source-trustworthiness or claim-credibility discernment (i.e., changes pretreatment vs. posttreatment and follow-up) compared with control. Ratings from the original 4-point scale in (a) were scaled to the [0,1] interval so that coefficients represent changes in percentage points. Error bars represent preregistered 90% CIs from one-sided tests. In (c) are shown percentages of binarized ratings pretreatment for both main outcomes, pooled across conditions.

OPEN IN VIEWER

Fig. 3.

Exploratory descriptive analyses. In (a) are shown treatment effects (pretreatment vs. posttreatment change compared with control) on ratings, split between true and false items. Coefficients represent group-level treatment effects (in the model: two-way interaction between condition, with control as the reference category, and time point, with pretreatment as the reference category). In (b) are shown the marginal means of predicted proportion of correct ratings based on pretreatment ratings. Coefficient estimates stem from separate, single-predictor regression models; error bars represent 95% confidence intervals; dashed line represents global average. In (c) we show the proportion of correct ratings by treatment and time. In (d) we show the web search volume after participants visited a target website, separately for treatments, with temporal trajectories of searches within the postexperimental website click-browsing sequence; the y-axis represents the proportion of searches at respective postexperimental click positions. The analysis is based on a subset of 202 participants with web-tracking data. CI = confidence interval; NATO = North Atlantic Treaty Organization; AfD = Alternative für Deutschland.

For additional exploratory analyses, we elicited the following per website: source intent (inform, convince, mislead), binary claim veracity (true, misleading and/or false, could not determine), and whether participants had encountered the website before (yes, no). We also collected several demographic, political, and psychological variables: age, gender, education, employment status, political position on the left–right scale, party preferences, voting intentions, affective polarization, populist attitudes, political extremism, political knowledge (efficacy), issue attitudes, issue knowledge, institutional trust, and “active open-minded thinking” (including items on myside bias and overconfidence; Baron et al., 2022). This last item was previously found to be strongly correlated with misinformation susceptibility (Roozenbeek et al., 2022). However, given constraints with regard to statistical power, we restricted our exploratory analysis of demographic and survey variables to a descriptive breakdown of pretreatment discernment between groups.

Source trustworthiness

Building on previous research (Lühring et al., 2025), we employed a domain-level approach to measurement. We used the widely applied NewsGuard Reliability Ratings of German online sources as benchmarks of source trustworthiness. NewsGuard has wide coverage of German domains and, in the English context, was found to correlate highly with other news-domain quality-rating sets (Lin et al., 2023).

Analysis

Our primary, preregistered analyses (see https://osf.io/tsb68/) examined the changes in source-trustworthiness discernment (i.e., difference between trustworthiness ratings for trustworthy vs. untrustworthy sources) and changes in claim-credibility discernment (i.e., difference between credibility ratings for credible vs. not credible claims) across two time frames:

Source-trustworthiness ratings served as the first outcome (H1s, H2s). The analysis was repeated with claim-credibility ratings (H1c, H2c, H3c). As an additional robustness check, we explored source intent and binary-claim veracity measures (see Fig. S8 in the Supplemental Material).

We employed linear mixed-effects models, implemented in R using the lme4::lmer() function. Models 1s and 1c (pretreatment vs. posttreatment), to test hypotheses H1s, H1c, and H3c, model the rating using the equation below (shown in lme4 syntax):

rating ~ item_type * phase * treatment (1)

+ (1 + item_type + phase | id)

+ (1 | item),

where

The interaction of rating * item type corresponds to a calculation of discernment; the interaction of this with phase provides a measure of change in discernment over time. Because discernment is defined as a contrast between responses to different item types (true/trustworthy vs. false/untrustworthy) rather than a single observed variable, the reported effect sizes are derived from these interaction terms. The full set of fixed effects estimates is provided in Table S6 of the Supplemental Material.

Models 2s and 2c (pretreatment vs. follow-up), to test hypotheses H2s and H2c, were identical to Models 1s and 1c, respectively, with two exceptions: Only participants who had completed the follow-up wave were included (N = 2,120), and the variable phase now distinguished between pretreatment and follow-up ratings (rather than pre- and posttreatment ratings).

Consistent with our preregistration, and following Bates et al. (2018), we selected a model specification by initially implementing the full random-effect structure (see the preregistration and the Supplemental Material) and iteratively simplified the random effect structure to address issues of singular fit we encountered (see Section 2.1 in the Supplemental Material for details). The equation above is the simplified version used in all analyses reported in the main text.

Consistent with our preregistration, we compared both treatment conditions (lateral reading and online search) to the control condition. We used one-tailed tests, and we report fixed-effects coefficients for the interaction term treatment:phase:item type with 90% confidence intervals (CIs), which reflect the conventional alpha threshold of 0.05 (Hales, 2024).

Web-tracking data

Web-tracking data was collected for a subsample of 436 participants from YouGov’s Pulse panel. The experiment, with visits to target stimulus websites visible in the tracking data, was successfully tracked for only 202 participants. For the remaining Pulse respondents, the tracked device was not the one on which the participants completed the study. The raw URL tracking data contained the following variables: participant ID, full URL of the visited website, time stamp of the visit, and duration of the visit in seconds. Research questions involving URL tracking data were not formally preregistered because of the complexity of analytical decisions; however, we did outline an envisioned processing pipeline in our preregistration.

After extracting domains from the URLs, we classified all potentially relevant domains for which NewsGuard labels were available. However, because engagement with untrustworthy news media was not a frequent behavior in our data set (only 33 of the 436 Pulse respondents visited an untrustworthy outlet in the 2-month tracking period), we decided against substantial interpretations of group differences in pretreatment versus posttreatment browsing.

To explore signals of adoption of the intervention strategy, we extracted search queries from URL strings and constructed sequences of browsing after visits to experimental stimulus websites. Because the tracked sample was small and the analysis of behavioral data requires a complex analytical pipeline, we present the results as a descriptive illustration of search behavior and refrain from any statistical tests between groups.

Lateral-reading and online-search treatment effects

Both interventions showed small positive point estimates for treatment effects on source-trustworthiness and claim-credibility discernment (Fig. 2b). Assuming a quantitative structure (Michell, 2012), coefficients can be roughly interpreted as percentage-point changes in discernment, as the original 4-point outcome scale (see Fig. 2a) was transformed to the interval [0,1], with 1 representing perfect discernment of true and trustworthy and false and untrustworthy information, and with 0 representing consistent misclassifications (e.g., of untrustworthy sources as “definitely trustworthy”). For lateral reading, the estimated effect on source-trustworthiness discernment was 1.1 percentage points relative to the control condition; however, this estimate was not statistically significant, because the 90% confidence interval included zero (H1s: b = 0.011, 90% CI = [−0.001, 0.023]). Lateral reading showed evidence of a small improvement in claim-credibility discernment compared with the control condition (H1c: b = 0.016, 90% CI = [0.003, 0.028], corresponding to a 1.6 percentage-point increase).

We observed a general improvement in the discernment performance from Wave 1 to Wave 2, unexpectedly also in the control condition (see the exploratory results reported in Fig. 3c). This partially changes the interpretation of our preregistered analyses regarding the longevity of the interventions (H2s and H2c). Although we were substantially interested in the longevity of the effects, we specified this effect as a persisting difference from the control group in our preregistered hypotheses. However, lateral reading did not improve source-trustworthiness discernment compared with the control condition after 2 weeks (H2s: b = 0.007, 90% CI = [−0.006, 0.02]), nor did it improve claim-credibility discernment (H2c: b = 0.005, 90% CI = [−0.008, 0.019]).

To determine whether these findings reflect a decrease in the effects of the treatment over time or were instead a consequence of the unexpected improvement in the control group, we computed exploratory contrasts between the posttreatment and follow-up results, separately for each experimental condition. We found that the control group had substantially improved in the follow-up compared with posttreatment (b = 0.023, 90% CI = [0.012, 0.034]), thereby offsetting the previous differences between intervention groups and the control group in pretreatment to posttreatment changes (the main pretreatment–posttreatment effects; see Table S7 in the Supplemental Material for all contrasts between posttreatment and follow-up).

We found no evidence of a backfire effect in the online-search condition. The intervention did not reduce claim-credibility discernment relative to control, as would be implied by an average backfire effect (Aslett et al., 2023). Instead, the point estimates for online search were positive for both claim credibility (b = 0.014, 90% CI = [0.002, 0.027]) and source-trustworthiness discernment (b = .015, 90% CI = [0.003, 0.027]). Although these estimates suggest effects similar in magnitude to those of lateral reading, all observed effects were small, and several confidence intervals approached zero.

The results of the preregistered analyses reported above were robust to alternative model specifications, such as treating the ratings as ordinal measurements and using a Bayesian approach (see Fig. S9 in the Supplemental Material for details).

When comparing treatment effects (pretreatment vs. posttreatment change in comparison to control) on ratings instead of discernment, and splitting the analysis between true and false items, we found that the interventions appeared to work through complementary pathways: The lateral reading interventions lowered participants’ ratings of the trustworthiness of untrustworthy sources and inaccurate claims, whereas the online-search intervention increased ratings of the trustworthiness of trustworthy sources and credible claims (Fig. 3a).

Exploratory descriptive analyses

In separate regression models (many survey variables are strongly intercorrelated) to predict the proportion of correct ratings prior to treatment, pooled across experimental conditions, we found that, broadly aligning with previous research (Guess et al., 2020; Sultan et al., 2024), the following variables were most predictive of poor discernment: positive ratings for the far-right AfD (Alternative für Deutschland) and the new populist party BSW (Bündnis Sahra Wagenknecht; Thomeczek, 2024), right-leaning political orientation, support for regulating immigration, low regard for NATO, lack of interest in mitigating climate change or supporting Ukraine, and disapproval of the German government’s COVID response. In contrast, trust in science and public broadcasting, satisfaction with democracy, interest in politics, and general education were positive predictors of accurate ratings (Fig. 3b). Although these findings illustrate enormous heterogeneity in discernment performance at baseline, our study was underpowered to robustly examine the heterogeneity of the treatment effect along these groups.

Using the subset of YouGov Pulse respondents for whom tracking data was available (n = 202), we explored behavioral signals to determine the extent to which participants used the interventions’ techniques to evaluate websites within the experiment. To this end, we compared the use of search engines when evaluating websites. We found that participants in both treatment groups showed a stronger search for information directly after receiving an evaluation task (Fig. 3d).