Clickbait news and algorithmic curation: A game theory framework of the relation between journalism,users,and platforms

Abstract

Algorithmic curation of social media platforms is considered to create a clickbait media environment. Although clickbait practices can be risky especially for legacy news outlets, clickbait is widely applied. We conceptualize clickbait content supply as a revision game with an unknown threshold. Combining supervised machine learning with time series analysis of Facebook posts and Twitter messages of 37 German legacy news outlets over 54 months, we observe outlets’ behavior following algorithm changes. Results reveal (1) an infrequent use of clickbait with few heavier-using outlets and (2) turning points of clickbait performance as clickbait supply and user interaction form a reversed U-shaped relationship. News outlets (3) collectively adjust toward an industry clickbait standard. While we (4) cannot prove that algorithmic curation increases clickbait, (5) Facebook’s regulative intervention to decrease clickbait disperses heterogeneous tendencies in clickbait supply. We contribute to an understanding of editorial decision-making in competitive environments facing platforms’ regulative intervention.

Keywords

Algorithms digital journalism Facebook game theory legacy media news social media platforms supervised machine learning Twitter user interaction

Introduction

Social media platforms provide an infinitely scrollable and personalized news feed intending to tie users in. To be part of the feed, Caplan and boyd (2018) argue that news outlets must construct their content with algorithmic curation in mind. Therefore, social interaction metrics such as likes or shares denote an online currency (Myllylahti, 2020) and serve as popularity cues for editors and users alike (Haim et al., 2018; Munger, 2020). By rewarding popularity, algorithmic curation can lead to conformity in the news industry (Caplan and boyd, 2018). News outlets collectively design “addictive distractions” (Naughton, 2018: 387) to grab user attention, maximize popularity, and ultimately profits (Sismeiro and Mahmood, 2018).

Clickbait headlines signify one example of such addictive distractions created to succeed in a high-choice algorithmic-curation environment of social media. Clickbait represents a linguistic strategy to articulate a message in a curiosity-arousing way that entices readers to click on the referring article (Potthast et al., 2018). Clickbait headlines originate from new entrants to journalism such as Upworthy or BuzzFeed (Tandoc, 2018). Meanwhile, they are used by tabloids (Blom and Hansen, 2015), broadsheets (Potthast et al., 2018; Rony et al., 2017), and by hoax outlets to spread disinformation (Braun and Eklund, 2019). Particularly, legacy news outlets find themselves in a dilemma of jeopardizing their reputation and user trust when providing too much clickbait (Molyneux and Coddington, 2020; Zhang et al., 2020).

Against the background of algorithmic curation and competition for user attention, we ask, what are determinants of legacy news outlets’ clickbait headline supply on social media platforms over time? Going beyond previous research focusing on clickbait performance, this study sheds light on the collective clickbait provision behavior of legacy news outlets facing algorithmic curation. Our goals are (a) to contribute to an understanding of journalistic decision-making in competitive social media environments and (b) to empirically assess “how algorithms and data-driven technologies, enacted by an organization like Facebook, can induce similarity across an industry” (Caplan and boyd, 2018: 1).

Empirically, we focus on the news industry level and analyze 856,252 Facebook posts and 1,305,297 tweets of n = 37 national and regional legacy news outlets in Germany, a country with a strong ethical-professional news industry (Hanitzsch et al., 2011) and comparatively modest social media use for news (Newman et al., 2018), representing a conservative example for the present research goals. The observation period begins in 2013, when the German news industry was widely active on social media and ends in 2017, when social media use for news had stagnated in Germany (Newman et al., 2018). During this period, Facebook provided an algorithmic-curation environment and intended to reduce clickbait supply twice. Twitter shifted from non-curation to algorithmic curation, potentially inviting clickbait.

Literature review

Clickbait news making

Journalists consider various aspects for news selection and editing, including audience metrics (Fürst, 2020; Lamot and van Aelst, 2020), competitor content (Boczkowski, 2010), social media content (Cagé et al., 2020), and outlet reputation (Andina-Díaz and García-Martínez, 2020; Siegert et al., 2011). Social media editors or members of the editorial team are involved in editing news for social media (Cornia et al., 2018). These editors develop conceptions of social media algorithms and “feel the need to optimize their content” (Peterson-Salahuddin and Diakopoulos, 2020: 27) for platforms. Consequently, social media editors recognize that certain topics, for example, health, work better on Facebook than foreign politics (Lischka, 2021). On social media, editors depart from the traditional text norms and add subjective views (Hågvar, 2019) or highlight surprise and emotions (Peterson-Salahuddin and Diakopoulos, 2020). Therefore, news outlet’s social media teasers are more tabloid-like (Magin et al., 2021) and include emojis and more question marks (Haim et al., 2021) than news on their websites. Peterson-Salahuddin and Diakopoulos (2020) conclude that social media editors constantly negotiate newsworthiness with interaction worthiness to be promoted through curation algorithms.

Clickbait represents one way to edit clickworthy teasers. Clickbait headlines highlight gossip, sensational, or provoking aspects or create curiosity gaps by withholding information, which includes listicle, forward-referencing, and question-based headlines (Scacco and Muddiman, 2020). Traditional (summary) headlines also intend to attract readers’ attention by giving more weight to certain aspects of a story while remaining precise (Andrew, 2007). In contrast, clickbait headlines exaggerate arousal and lack precision (Scacco and Muddiman, 2020). Clickbait is regarded as deviating from professional traditions (Hågvar, 2019), and journalists employ clickbait even if they do not favor it (Potthast et al., 2018). Since clickbait originates from digital-native news outlets (Tandoc, 2018), especially legacy news editors may face a dilemma when writing clickbait headlines.

Game-theoretic framework to clickbait news making

Clickbait headlines could be conceptualized resulting from heteronomous market forces in journalism based on field theory (Bourdieu, 1983), which deviate from the dominant view, the journalistic “doxa.” Field theory is applied to explain new entrances in journalism (Tandoc, 2018) and changes in journalistic culture and practices (Schultz, 2007). Moreover, to explain institutional pressures and homogeneity within journalism facing algorithmic curation, the institutionalist perspective of isomorphism (DiMaggio and Powell, 1983) is useful, as adopted in Caplan and boyd (2018). The strength of field theory and isomorphism lies in connecting actors to their environment. Game theory focuses on decision-making while considering the environment of actors. Game theory provides a suitable framework for describing clickbait news supply, acknowledging the interests and actions of editors, competitors, users, and platforms.

A game is an abstract, usually mathematical, description of possible choices that players can make given their information about available actions and consequences in a given setting (Osborne and Rubinstein, 1994). Players can be cooperative or may not exchange information (Nash, 1951; Neumann and Morgenstern, 1944). Game-theoretic approaches are rarely applied to journalism. One exception demonstrates a leader–follower game, in which reputational news outlets withhold questionable scoops fearing reputation loss, while less reputable outlets cover such scoops (Andina-Díaz and García-Martínez, 2020). We describe the roles of journalists, users, competitors, and social media platforms for clickbait news supply in four scenes that can occur subsequently or simultaneously.

Scene 1: game invitation

The online environment of “clickbait media” (Munger, 2020), enabled by interaction-rewarding algorithmic curation (Caplan and boyd, 2018), signals cues to news outlets, civic users, and hoax publishers alike to join an attention maximization game on social media platforms (Braun and Eklund, 2019; Myllylahti, 2020). Previous research describes platforms as powerful intermediaries (Nielsen and Ganter, 2017), while placing journalism into dilemmas with professional standards (Peterson-Salahuddin and Diakopoulos, 2020). Journalists regard Facebook as a central distributer they invest resources into and depend on (Meese and Hurcombe, 2020). Nielsen and Ganter (2017) argue that “digital intermediaries act, leaving news media organizations to react” (p. 1613). If social media platforms efficiently initiate an attention maximization game by algorithmic curation, clickbait supply of news outlets should increase.

H1. Clickbait supply increases in algorithmic curation but not in a non-curative setting.

Scene 2: collective outcomes through stigmergy

While competing news organizations rarely coordinate news supply, news outlets observe each other’s news coverage leading to greater news homogeneity online (Boczkowski, 2010). Through mutual monitoring, news formats, and style collectively adopt over time across the news industry, especially under fierce competition (Arbaouit al., 2020), and even if individual journalists disapprove (Potthast et al., 2018). Accordingly, Haim et al. (2021) show that news outlets’ language is more homogeneous on social media than across their websites. Collective outcomes in journalism thus may involve coordination through environmental cues, which can also explain the universal usage of clickbait.

The concept of stigmergy describes this kind of coordination. Originally proposed to explain collective behavior of termites (Grassé, 1959), stigmergy is a way of indirect communication to coordinate actions through changes to the environment, which actors mutually perceive and respond to (Vrancx et al., 2010). While termites leave pheromone trails that coordinate nest building, human actors’ communicative cues within networks of users instigate stigmergy (Cimino et al., 2018). If clickbait dissemination in journalism is stigmergic, the following should hold:

H2. Outlets adjust their clickbait supply to an industry standard.

Scene 3: playing against an unknown threshold

Whether clickbait effectively maximizes interaction is questioned by studies showing that clickbait may either receive more (Rony et al., 2017) or less clicks (Kuiken et al., 2017; Mourão and Robertson, 2019; Scacco and Muddiman, 2020) than non-clickbait posts. Zhang et al. (2020) find an inverted U-shaped relationship between the number of clickbait headlines and page views. That is, traffic increases up to certain level of clickbait, but decreases if used too extensively.

An inverted U-shape can be justified with regret game theories (Starmer, 2000). Accordingly, the decision of whether to click on a piece of content depends on users’ expected utility. If users frequently experience disappointment with clickbait from any source, because of the lower quality these articles have than summary headlines (Scacco and Muddiman, 2020), regretful users may avoid the type of content or source for future decisions. In this regard, Molyneux and Coddington (2020) argue that negative perceptions of clickbait headlines may aggregate over time, whenever a clickbait headline disappoints users, and ultimately, risks losing their trust. Interaction with clickbait may decrease considerably, which is reflected in popularity cues including likes and shares (Haim et al., 2018). Disappointment follows a steep convex negative function (little disappointment is related to strong utility loss), whereas exceeding expected utility is characterized by a flatter concave positive utility function (a lot more utility than expected is required for extra utility gain; Starmer, 2000). The combination of concave and convex utility functions provides an explanation for the reverse U-shaped relation between clickbait and clicks.

To learn about the optimal amount of clickbait, editors need to experiment. Chen’s (2020) revision game of experimentation on an unknown threshold represents a risk-control game to explore an optimum. Chen (2020) uses the example of a food maker that tries to find out how much of a cheaper substance can be added to a food product without decreasing its quality or, worse, causing health issues. Similarly, news organizations may post clickbait headlines to increase traffic. But if a news outlet adds too many clickbait headlines over time, users’ utility and the outlet’s reputation may be compromised (Molyneux and Coddington, 2020). However, news outlets do not exactly know how much of their own and other clickbait is too much for users. By revising their own actions while observing user behavior, news outlets can incrementally approach the unknown clickbait threshold. Meese and Hurcombe (2020) describe such behavior as “riding the waves of engagement” (p. 5). We propose,

H3. The more user interaction with past clickbait, the more clickbait is supplied at present.

Scene 4: interfering in the game

Outlets benefit from platforms with audience traffic (Sismeiro and Mahmood, 2018) and platforms from engaging content with user loyalty (Chen and Pain, 2019), suggesting a strategic alliance relationship between platforms and outlets. In an alliance, the threat of opportunistic behavior of each party is small (Hansen et al., 2008). Yet, a downward slope of utility initiated by user regret can be harmful for social media platforms due to decreasing user loyalty. Platforms will be interested to maximize utility for users and punish outlets for crossing users’ clickbait threshold. Being providers for the outlet-user interaction, platforms can transform from an ally to a regulator. Their curation algorithms serve as bureaucratic mechanisms (Caplan and boyd, 2018) to enforce regulation. Facebook has intended to penalize clickbait in 2014 and 2016 due to displeased users (Peysakhovich and Hendrix, 2016), revealing a form of responsive regulator interference (Etienne, 2013). To outlets, regulation implies opportunistic behavior of the outlet’s former ally and increases uncertainty toward the future behavior of the platform. Since journalistic routines are used to adapt to frequent algorithm shifts (dos Santos et al., 2018), news outlets may adhere to platform’s algorithm changes, as shown for increases in ideological news diversity (Garz and Szucs, 2021) and social video supply (Tandoc and Maitra, 2018). Alternatively, outlets may not comply facing unpredictable algorithm changes (Cornia et al., 2018; Meese and Hurcombe, 2020). We ask,

RQ1. How does clickbait supply change after a platform’s regulative algorithm change?

Method

Sample

Platform selection

Facebook and Twitter were selected as information-rich cases stratified regarding news outlets’ dependency (high vs low), algorithmic curation (permanent vs incremental), intention to reduce clickbait (twice vs none), and a rather general versus information-elitist audience, respectively (Cornia et al., 2018; Myllylahti, 2020; Newman et al., 2018). Given Facebook’s and Twitter’s accessibility, the platform selection strategy combines stratified purposeful and convenience sampling (Suri, 2011).

Regarding Facebook’s News Feed, two algorithm updates explicitly targeted clickbait. In August 2014, Facebook announced that its news feed would “weed out” clickbait (El-Arini and Tang, 2014). In August 2016, Facebook refined its clickbait detection using supervised learning techniques to curb the distribution of “misleading and spammy” clickbait more effectively (Peysakhovich and Hendrix, 2016).

Twitter users were exposed to tweets in reverse chronological order until 2016. In January 2015, Twitter announced to start testing features of algorithmic content selection to “surface a few of the best Tweets” (Rosiana, 2015). These features were gradually rolled out on an opt-in basis. In March 2016, Twitter switched to algorithmic curation as users’ default option (Kamen, 2016).

Country and news outlet selection

Germany is a country with a strong ethical-professional conduct in journalism (Hanitzsch et al., 2011), strong in print revenues compared to slowly growing online revenues (Newman, 2016), comparatively low social media use for news but increasing mobile news use during the observation period (Newman et al., 2018). Facebook was used for news by 25% and Twitter by 4% of users (Newman et al., 2018).

The sample of news outlets comprises 10 national and 27 regional news outlets with a legacy in print following a traditional, two-sided model of advertising and audience revenue, which had a Facebook and Twitter account during the study’s observation period (see Figures 3 and 4). The sample thus stands for a potentially maximized tension between following traditional professional standards and adopting to algorithmic curation—while news outlets’ revenue dependency on platforms is comparatively low.

Data collection

We collected 856,252 posts and 1,305,297 tweets published between January 2013 and June 2017 by the official Facebook Pages and Twitter accounts of 37 German news outlets, which comprises the total posting activity of these outlets. The data include the date and time of publication, the post message, and the number of likes and shares. The dataset was compiled using the Facebook graph application programming interface (API) combined with the mining application Netvizz. For Twitter, we used the Python library “GetOldTweets3.” This library offers an automated approach to access and parse the browser-based Twitter search. We recovered each tweet’s date, time, message, and number of likes and retweets.

Supervised machine learning approach

We used supervised learning to obtain clickbait classifications of all content in our dataset, based on the characteristics of the posts and tweets rated by human coders.

Human coding

We drew random samples of 20,000 posts and 20,000 tweets and had humans evaluate whether these messages classify as clickbait. German-speaking coders were recruited through Fiverr.com, an online platform for freelance services. To verify the coding quality, we had each potential coder evaluate a test batch of 500 messages, which we had coded ourselves. We paid potential coders 20 Euros for rating these 500 messages and offered further work if their coding “fulfilled our quality standards.” That is, we compared their evaluation of the test batch with our own coding and selected only coders whose kappa-statistic (Landis and Koch, 1977) indicated that the ratings are systematically related to the gold standard (at the 1% level). The employed coders achieved inter-rater agreement rates between 94.8% and 98.0%, corresponding to kappa-statistics between 0.61 and 0.77. The coders received further batches of randomly selected post messages, each batch for 20 Euros incentive. We assured ongoing coding quality by spot checking the results. Importantly, we kept repeating our offer to ask for more work “as long as our quality standards were met,” which arguably created a strong incentive for the coders to take the task seriously. Each batch included a nondescript identifier, the post or tweet message, and one blank column for the evaluation. The files did not include date and outlet, to avoid priming the coders.

The coding instructions asked to evaluate whether a given post message qualifies as clickbait (“yes,” “no,” and “maybe”), considering the following instructions:

Clickbait refers to headlines designed to maximize clicks, often for commercial interests such as ad revenues. For instance, clickbait exaggerates or withholds information to arouse the attention and curiosity of the reader, so that the reader clicks on the headline and opens the linked story.

The instructions included specific examples of non-clickbait and clickbait, including forward-references, questions, listicles, and sensationalization (see Table 1).

Table 1.

Examples of non-clickbait and clickbait headlines.

Non-clickbait	Clickbait
Frau Merkel denkt noch lange nicht ans Aufhören (Ms Merkel is nowhere near considering leaving)	Traurig—und jetzt auch noch unfassbar tragisch! (Sad—and now also unbelievably tragic!)
Polizei sucht weiter nach den Buntmetall-Dieben (Police continues to look for the non-ferrous metal thieves)	Dieser Trick ist goldwert! Ihr solltet ihn unbedingt kennen (This trick is worth a mint! You definitely need to know it)
Opernhaus des Jahres steht in Köln (Opera house of the year is in Cologne)	Diese 10 Apps helfen euch beim Sparen (These 10 apps will save you money)

Clickbait classification

The classification was implemented using the R package “caret—Classification And REgression Training” (Kuhn, 2008). To evaluate the power of different classification methods, we randomly divided the samples of coded messages into training (70%) and testing observations (30%). Given the potential differences between formats, content, and outlets’ social media strategies across platforms, we used separate classifiers for each platform (see Appendix 1).

Following the literature on automated clickbait classification (Biyani et al., 2016; Chakraborty et al., 2016; Rony et al., 2017), we considered the following predictors to train the models. First, we created count variables for each 1-, 2-, and 3-gram in the messages. For that purpose, we removed punctuation and numeric characters and tokenized the texts. We did not remove stop words and refrained from stemming words to their root because differences between clickbait and non-clickbait could be subtle. The messages consist of ca. 540,000 n-grams, all of which were considered as potential predictors of clickbait. In addition, we constructed various variables based on quantifiable characteristics of the messages, including the number of words, the mean word length, as well as the number of periods, question marks, exclamation marks, colons, and numerical characters. We also constructed binary variables indicating if a message starts with a number or a 5W1H word (i.e. what, why, when, who, which, and how), contains a third person personal pronoun (i.e. he, she, his, her, him), or includes a demonstrative (this, that, those, these). Finally, we evaluated the predictive power of meta-level characteristics, including outlet, day of the week, day of the month, and hour of publication, and whether the post or tweet included a link embedded in the text.

Given the large number of potential predictors, we pre-screened the variables based on their bivariate correlation with the observed clickbait status of a post or tweet to speed up training. We retained the 500 predictors with the largest correlation coefficients to train various classification models, including (boosted) logistic regression, (penalized) discriminant analysis, generalized linear models, gradient boosting, and random forests. All models were trained using 20-fold cross validation. We used penalized discriminant analysis, because this approach outperformed the other approaches.¹ We also compared the performance for a 3-point (i.e. “yes,” “maybe,” “no”) versus a 2-point clickbait scale (i.e. “yes” and “no”; combining initial “yes” and “maybe” into one category). We decided to proceed with the 2-point scale because the coding instructions induced conservative ratings and the classification models performed better.

Analysis

Data preparation

Model-based classification does not involve a “hard” assessment of whether a headline is clickbait or not. Instead, for each post and tweet in our dataset, we obtained the probability that the corresponding headline classifies as clickbait.² For the regression analyses, the individual probabilities were then aggregated at the outlet-month level. We created two panel datasets (one for each platform), consisting of 37 outlets and 54 months. We follow Zhang et al. (2020) and use a monthly frequency because quarterly or yearly observations would imply an unnecessary loss of information, whereas a higher frequency (e.g. weekly or daily) would make the analyses vulnerable to the influence of posts or tweets with extreme interaction metrics.

Estimation

To test the hypotheses, we estimated regression models of the following form:

\begin{array}{l} c b_{i, t} = β_{1} + β_{2} i n t e r a c t i o n_{i, t - 1}^{c b} + β_{3} a b o v e_{i, t - 1} + β_{4} T_{t} + β_{5} D_{i, t}^{a l g o} \\ + β_{6} D_{i, t}^{a l g o} \times T_{t} + β_{7} D_{i, t}^{a l g o} \times a b o v e_{i, t - 1} + β_{8} c b_{i, t - 1} + u_{i} + ε_{i, t} \end{array}

(1)

where $c b$ denotes the average clickbait probability of outlet $i$ in month $t$ . User $i n t e r a c t i o n^{c b}$ is defined as the sum of all likes and shares or retweets of posts or tweets with a clickbait probability > .5. The binary variable $a b o v e$ takes the value 1 if the outlet’s average clickbait probability in the previous month was above the median clickbait probability (overall outlets) in that month. Using the lagged values, $t - 1$ of both variables allows us to evaluate the clickbait response of social media editors to the behavior of users and competitors, rather than the other way around. We also included dummy variables to capture the algorithm changes of Facebook and Twitter ( $D_{i, t}^{a l g o}$ ) and a linear time trend ( $T_{t}$ ). The interaction terms $D^{a l g o} \times T$ and $D^{a l g o} \times a b o v e$ , respectively, can be used to verify if the clickbait probability and outlets’ tendency to adjust to the industry standard varied across different algorithm phases.

We estimated Equation (1) as a fixed effects panel model through ordinary least squares, using $u_{i}$ to denote the outlet fixed effect. As shown below, some outlets were generally more likely to provide clickbait than others, presumably due to different audiences, business models, and social media strategies. The fixed effects model makes it possible to account for those factors and any time-invariant differences in clickbait supply across outlets. Equation (1) also includes the outlet’s clickbait probability in the previous month ( $c b_{i, t - 1}$ ), to control for routine behavior of social media editors.³

Results

Clickbait probability

The average probability that a post or tweet is clickbait was 5.9% (SD = 4.6, Min = 0, Max = 33.5) on Facebook during our observation period and 2.8% (SD = 2.0, Min = 0, Max = 33.3) on Twitter. On average, the monthly like and share interaction with clickbait summed up to 4712 likes and shares on Facebook and 133 on Twitter per outlet. On Facebook, the clickbait probability followed an upward trend, starting at around 5% in 2013 and reaching a peak of about 7% toward the end of 2016. Afterwards, the use of clickbait seems to have stagnated or even declined (Figure 1). The clickbait probability on Twitter appears quite stable over time (Figure 2). On Facebook, news outlets ranged in their average clickbait probability between 2% (Freie Presse) and 28% (Focus; see Figure 3). On Twitter, the clickbait probability ranged between 1% and 3% for most outlets; but four outlets stand out, with probabilities around 6% (Figure 4).

Figure 1.

Use of clickbait over time, Facebook.

Figure 2.

Use of clickbait over time, Twitter.

Figure 3.

Use of clickbait across outlets, Facebook.

Figure 4.

Use of clickbait across outlets, Twitter.

Reverse U-shape

We assume a threshold of clickbait that news outlets can supply before users grow tired. Our data reveal a reverse U-shaped relationship between clickbait supply and subsequent user interaction (see Figures 5 and 6). That is, user interaction increases when the outlets provide more clickbait—but only until a certain point, after which more clickbait results in decreasing interaction. We calculate this point at 21% clickbait probability on Facebook and 6% on Twitter.

Figure 5.

Clickbait and Facebook user interaction.

Figure 6.

Clickbait and Twitter user interaction.

Scene 1: game invitation

To evaluate if clickbait supply increases in algorithmic curation but not in a non-curative setting (H1), we compare the development of the clickbait probability on Twitter for the time before and after the platform rolled out its algorithm. The raw probability plotted in Figure 2 did not change visibly when Twitter started testing its algorithm in 2015, and there does not seem to be any increase after the global roll-out in 2016 either. To evaluate H1 formally, we consult the regression results in Table 2, Column (2), especially the coefficients on the interaction terms between the algorithm dummies and the time trend variable. The coefficient on Jan 2015 to Feb 2016 × time trend captures the development of the clickbait probability after the start of Twitter’s testing phase (relative to the pre-algorithm period from Jan 2013 to Dec 2014), whereas after Feb 2016 × time trend measures the development after the global roll-out. As the estimates show, neither the testing phase nor the implementation phase was associated with increases in clickbait probability. In contrast, we find that this probability decreased during the testing phase between Jan 2015 and Feb 2016—the corresponding coefficient is negative and significant at the 10% level. In addition, the Time trend variable—which captures the development between Jan 2013 and Dec 2014—indicates that clickbait had been growing in a non-curative setting, before Twitter started testing its algorithm. Hence, we reject H1.

Table 2.

Determinants of clickbait supply.

	(1)	(2)
	Clickbait probability (Facebook)	Clickbait probability (Twitter)
Avg. clickbait probability (t–1)	0.6521***	0.1904**
Avg. clickbait probability (t–1)	(0.0977)	(0.0870)
Combined interaction with clickbait (t–1)	0.0141**	0.4321***
Combined interaction with clickbait (t–1)	(0.0055)	(0.1161)
Above industry level, t–1 (binary)	−0.0147***	−0.0015
Above industry level, t–1 (binary)	(0.0025)	(0.0018)
Time trend	0.0002**	0.0002**
Time trend	(0.0001)	(0.0001)
Facebook algorithm changes (reference category: January 2013 to July 2014)
August 2014 to July 2016	0.0655
August 2014 to July 2016	(0.0698)
August 2014 to July 2016 × time trend	−0.0001
August 2014 to July 2016 × time trend	(0.0001)
August 2014 to July 2016 × above industry (t–1)	0.0100***
August 2014 to July 2016 × above industry (t–1)	(0.0018)
After July 2016	0.3153***
After July 2016	(0.1016)
After July 2016 × time trend	−0.0005***
After July 2016 × time trend	(0.0002)
After July 2016 × above industry (t–1)	0.0104***
After July 2016 × above industry (t–1)	(0.0019)
Twitter algorithm changes (reference category: January 2013 to December 2014)
January 2015 to February 2016		0.3474*
January 2015 to February 2016		(0.1730)
January 2015 to February 2016 × time trend		−0.0005*
January 2015 to February 2016 × time trend		(0.0003)
January 2015 to February 2016 × above industry (t–1)		0.0012
January 2015 to February 2016 × above industry (t–1)		(0.0018)
After February 2016		0.1724
After February 2016		(0.1255)
After February 2016 × time trend		−0.0003
After February 2016 × time trend		(0.0002)
After February 2016 × above industry (t–1)		0.0035
After February 2016 × above industry (t–1)		(0.0023)
Constant	−0.1271*	−0.1287*
Constant	(0.0672)	(0.0678)
Adj. R²	.5890	.0661

Notes: Estimates based on fixed effects panel data models. Robust standard errors (clustered by outlet) in parentheses.

p < .10; **p < .05; ***p < .01.

Scene 2: stigmergy

H2 is evaluated by consulting the coefficient on Above industry level, t–1 in Table 2. For Facebook, we obtain a negative significant value (Column 1). The estimate thus implies that those outlets that have a higher clickbait probability than the median outlet adjust their clickbait supply downward in the following month. The coefficient also implies that outlets below the industry level increase their clickbait supply. Thus, the monthly adjustment process can be described as one of convergence between outlets. For Twitter, the coefficient is also negative, but not significant at conventional levels. Hence, we confirm H2 for Facebook but reject it for Twitter.

Scene 3: playing against a threshold

Results in Table 2 confirm that higher amounts of past user interaction with clickbait are associated with greater levels of current clickbait supply (H3). On Facebook, this relationship is statistically significant at the 5% level. According to the point estimate of 0.0141, a one-unit increase in Combined interaction with clickbait (= 100,000 additional likes/shares) results in an increase in clickbait probability by 1.4 percentage points in the next month. To put this effect size in perspective, the combined interaction with clickbait on Facebook averaged 4712 likes/shares per month. If this number was doubled, the outlets’ clickbait probability would increase by $\frac{4712 \times 1.4}{100, 000}$ = 0.066 percentage points. Thus, the outlets reacted rather carefully to changes in user interaction. We also obtain a positive and significant coefficient for Twitter. The point estimate of 0.4321 suggests that 100,000 additional likes/shares lead to subsequent increase in clickbait probability by 43 percentage points. A change in the magnitude of the mean value of user interaction with clickbait on Twitter affects the clickbait probability by $\frac{133 \times 43.21}{100, 000}$ = 0.058 percentage points, which suggests that social media editors on Twitter adjust their clickbait supply in very small steps, too. We accept H3.

Scene 4: interfering in the game

To answer RQ1, we first test if Facebook’s modifications of the news feed algorithm to curb the proliferation of clickbait worked as intended. The interaction terms Aug 2014 to Jul 2016 × time trend and after Jul 2016 × time trend allow us to compare the development of clickbait supply over time after each modification of the algorithm in comparison to the period from Jan 2013 to Jul 2014. Accordingly, the time after the platform’s first algorithm update (Aug 2014 to Jul 2016) was not characterized by a different trend in clickbait supply. However, the coefficient on after Jul 2016 × time trend is negative and significant at the 1% level, which implies that Facebook’s second modification initiated a downward trend in clickbait. Yet, this development did not (immediately) result in a lower level of clickbait. As the coefficient on the After Jul 2016 dummy indicates, the overall clickbait supply remained on a higher level than between Jan 2013 to Jul 2014 despite the initiation of a downward trend.

Second, the interaction terms Aug 2014 to Jul 2016 × above industry and after Jul 2016 × above industry have a significantly positive sign. Thus, after Facebook’s modification, the news outlets were less inclined to revert to the industry standard, compared to the time before the platform tried to lessen the distribution of clickbait. Hence, the algorithm updates also reduced the outlets’ tendency to align their behavior with their competitors.

Discussion

This study explores the clickbait posting behavior of legacy news outlets in the German news industry. Overall, we observe a low level of clickbait during the observation period (about 6% on Facebook and 3% on Twitter). The variations in the use of clickbait across platforms as well as between news outlets could be driven by differences in audiences’ news consumption preferences. That is, Twitter’s information-elitist audience of “politicians, journalists, and news lovers” (Cornia et al., 2018: 7) could be generally less attracted to clickbait because of a greater demand for quality news. Since social media use is related to an incidental news consumption during leisure time (Boczkowski et al., 2018), consuming sensational news on Facebook may be more valued than on Twitter. Moreover, the outlets with the highest clickbait probabilities in our sample tend to be tabloids (e.g. Express, B. Z. Berlin, and Hamburger Morgenpost), whose readers may have a higher degree of acceptance of sensational content.

Beyond these structural differences, news outlets appear to play a stigmergic revision game for finding the optimal amount of clickbait. Clickbait supply reveals characteristics of collective behavior that is coordinated through environmental cues. With clickbait headlines, news outlets leave each other communicative trails that explain an industry-wide diffusion of clickbait as shown in previous research (Blom and Hansen, 2015; Rony et al., 2017). News outlets adopt toward an industry-wide clickbait supply standard as predicted by stigmergy games (Vrancx et al., 2010).

However, although platforms represent a powerful force for journalism (Caplan and boyd, 2018; Nielsen and Ganter, 2017), we cannot prove that introducing algorithmic curation increases clickbait—at least not on Twitter. In contrast, the clickbait game was already being played before Twitter introduced algorithmic curation, as we find a slight upward trend in clickbait during that period. Thus, we observe a stigmergic tendency toward interaction-cumulating content within the news industry across platforms, which may reflect a universal appeal of clickbait in an environment of competing for fleeting user attention. In addition, our results show that Facebook’s attempts to reduce clickbait were not initially successful. The platform’s first modification of its algorithm used reading time and interaction with a piece of content and thus focused on indicators for user regret. This approach may not have been sophisticated enough to identify clickbait and outlets may not have experienced lower visibility. Only the second update in 2016, in which Facebook used a supervised approach with humans identifying clickbait to train a classifier, resulted in a sustainable decrease in clickbait growth. That is, news outlets adjusted to this algorithm change but did not reduce clickbait to the lower 2013 level. Hence, the universal appeal of clickbait represents a sustainable sway in the news industry whose effect is hardly reversible.

Still, regulative intervention has an indirect effect on news outlets’ clickbait supply. Outlets’ tendency to adjust to the average industry clickbait supply declined after both algorithm changes of Facebook. That is, the news outlets’ clickbait supply deviated from each other, which indicates a replacement of stigmergic orientation with opportunistic behavior. This result complements the findings of Meese and Hurcombe (2020) and Cornia et al. (2018), who show that algorithm changes induced outlets to turn away from Facebook as a central distributer. Thus, algorithm changes may have unintended consequences leading to a decrease of news homogeneity online.

User behavior determines clickbait supply as well, and perhaps, more so than algorithm changes. We show that likes and shares of clickbait predict subsequent clickbait supply on Facebook and Twitter. However, the outlets’ reactions to user interaction are relatively small, which implies cautious supply adjustment in every step. News outlets continue their upward search for a threshold as long as the user interaction with clickbait content grows—or else, they decrease their supply accordingly, indicating a careful ride of the waves of interaction.

Regarding a fatigue threshold, our results show that a moderate level of clickbait performs better than low or high levels. This result is in line with previous research pointing at an inverted U-shaped relationship between clickbait and user interaction (Zhang et al., 2020) and suggesting a wear-out effect of these headlines (Molyneux and Coddington, 2020). We specify a wear-out threshold of clickbait at 21% probability on Facebook and 6% on Twitter over all observed news outlets, with a flat increase and a steeper decrease of clickbait performance for Twitter. The lower threshold, flat increase, and steeper decrease on Twitter may be due to an information-elitist audience whose experienced utility of clickbait news barely excels their expected utility but is easily disappointed. Moreover, the steepness of the interaction decline on Twitter may be fueled by decreasing popularity cues signaling low utility to users and lowering stigmergic user behavior as suggested by Dipple et al. (2012).

Regarding limitations, first, our data capture social media news and thus ignore a larger news environment that editors may consider. Future research on clickbait provision outside social platforms could reveal whether clickbait supply collectively changes as found here. Also, our data do not include non-journalistic content suppliers, which are however relevant as clickbait fatigue is related to users’ total clickbait exposure. Second, our sample period ends in 2017 due to Facebook’s restrictions, not including Facebook’s 2018 algorithm changes that broadly address news content. More recent data could also reveal whether legacy news outlets abstain from using clickbait since the rise of hoax outlets, as predicted by reputation games. Third, our investigation is limited to legacy media in Germany, implying a conservative context with relatively low online advertising revenue growth and modest social media news use. Future research could evaluate if our results hold in countries with more strongly growing online revenues and larger social media audiences, possibly exhibiting greater responsiveness to platforms’ regulative intentions. Fourth, likes and shares are certainly of interest to social media editors—although these popularity cues cannot be equated with engagement with journalism (Haim et al., 2018). User interaction in the form of referral traffic might have an even larger impact on media outlets’ decisions because this traffic can be directly monetized. Sismeiro and Mahmood (2018) show that social media interaction with news stories is a strong predictor of news outlets’ website traffic, which gives us confidence that the likes and shares considered here are relevant metrics. Fifth, supervised machine learning makes it possible to detect clickbait in a large corpus of text. The automated classification may not reach the same accuracy as human coding though. As shown in Appendix 1, our clickbait classifiers achieved a satisfactory performance—especially in comparison to previous clickbait detection studies (Biyani et al., 2016; Chakraborty et al., 2016; Potthast et al., 2018)—but methodological advances are desirable to improve the accuracy of automated text classification. Finally, as our data include only the editorial output of the news organizations under study, it remains unclear to what extent players consciously follow an established strategy. While audience metrics seem to gamify journalism to some extent (Ferrer-Conill, 2017), follow-up research is required to understand whether social media editors make decisions with a revision game on an unknown threshold in mind.

Conclusion

We explore the clickbait supply trends and patterns of legacy news outlets in an altering algorithmic-curation platform environment with utility-seeking users. This study is the first to determine the relation between news outlets and platforms by observing collective long-term news supply behavior during several curation algorithm changes.

Legacy outlets collectively supply interaction-inducing content on social media platforms following a revision game. They tend to underuse clickbait, except for a few heavier-using outlets with a tabloid focus, suggesting that professional norms endure on social media. An adjustment toward the optimal clickbait threshold takes place in small steps, which implies uncertainty despite the availability of detailed audience metrics. This uncertainty may lead to sharing audience metrics across outlets—potentially enhancing audience datafication in journalism (Livingstone, 2019).

Results endorse an increasing audience focus in online environments. Audience interaction serves as utility cues for users, revision cues for news outlets, and loyalty cues for platforms. Present results indicate that news outlets embrace a universal and almost irreversible focus on audience attention and popularity cues as suggested previously (Fürst, 2020; Lamot and van Aelst, 2020; Myllylahti, 2020). In a competitive environment, even legacy news outlets become “clickbait media” (Munger, 2020) that design “addictive distractions” (Naughton, 2018: 387), at least on a low level.

Finally, our findings contribute to an understanding of the relationship between social platforms and legacy media, especially concerning implications of platform policy for editorial decisions. News outlets can adhere to (Garz and Szucs, 2021; Tandoc and Maitra, 2018) and, as in our study, be indirectly responsive to changes in algorithmic curation. Both reactions acknowledge the central role of platforms not only in news distribution but also for news supply. Hence, curation algorithms serve as bureaucratic mechanisms (Caplan and boyd, 2018) that can enforce news similarity (Boczkowski, 2010; Haim et al., 2021). Our results advance these findings suggesting that algorithm changes reverse heterogeneity tendencies and unintendedly may increase news diversity online.

Footnotes

Appendix 1 Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Juliane A Lischka

Notes

Author biographies

Juliane A Lischka (PhD University of Zurich) is Professor of Communication Studies, esp. Digital Journalism at Universität Hamburg. Her research focuses on communication on social media platforms and journalism intersecting with digital technology.

Marcel Garz (PhD Universität Hamburg) is Associate Professor of Economics at Jönköping University. His work focuses on the economics of news markets, especially topics related to media bias, opinion diversity, and news consumption via social platforms.

References

Andina-Díaz

García-Martínez

(2020) Reputation and news suppression in the media industry. Games and Economic Behavior 123: 240–271.

Andrew

(2007) Media-generated shortcuts: do newspaper headlines present another roadblock for low-information rationality? Harvard International Journal of Press/politics 12(2): 24–43.

Arbaoui

Swert

de van der Brug

(2020) Sensationalism in news coverage: a comparative study in 14 television systems. Communication Research 47(2): 299–320.

Biyani

Tsioutsiouliklis

Blackmer

(eds) (2016) “8 amazing secrets for getting more clicks”: detecting clickbaits in news streams using article informality. Proceedings of the AAAI Conference on Artificial Intelligence 30(1):9966.

Blom

Hansen

(2015) Click bait: forward-reference as lure in online news headlines. Journal of Pragmatics 76: 87–100.

Boczkowski

(2010) News at Work: Imitation in an Age of Information Abundance. Chicago, IL: University of Chicago Press.

Boczkowski

Mitchelstein

Matassi

(2018) “News comes across when I’m in a moment of leisure”: understanding the practices of incidental news consumption on social media. New Media & Society 20(10): 3523–3539.

Bourdieu

(1983) The field of cultural production, or: the economic world reversed. Poetics 12(4–5): 311–356.

Braun

Eklund

(2019) Fake news, real money: ad tech platforms, profit-driven hoaxes, and the business of journalism. Digital Journalism 7(1): 1–21.

10.

Cagé

Hervé

Mazoyer

(2020) Social media and newsroom production decisions. Working paper. Available at: https://ssrn.com/abstract=3663899

11.

Caplan

boyd

(2018) Isomorphism through algorithms: institutional dependencies in the case of Facebook. Big Data & Society 5: 1–12.

12.

Chakraborty

Paranjape

Kakarla

, et al. (2016) Stop Clickbait: detecting and preventing clickbaits in online news media. In: 2016 IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM), San Francisco, CA, 18–21 August, pp. 9–16. New York: IEEE.

13.

Chen

Pain

(2019) News on Facebook: how Facebook and newspapers build mutual brand loyalty through audience engagement. Journalism & Mass Communication Quarterly 98: 366–386.

14.

Chen

(2020) A revision game of experimentation on a common threshold. Journal of Economic Theory 186: 104997.

15.

Cimino

Lazzeri

Pedrycz

, et al. (2018) Using stigmergy to distinguish event-specific topics in social discussions. Sensors 18(7): 2117.

16.

Cornia

Sehl

Levy

DAL

, et al. (2018) Private Sector News, Social Media Distribution, and Algorithm Change. Available at: https://reutersinstitute.politics.ox.ac.uk/sites/default/files/2018-10/Cornia_Private_Sector_News_FINAL.pdf

17.

DiMaggio

Powell

(1983) The iron cage revisited: institutional isomorphism and collective rationality in organizational fields. American Sociological Review 48(2): 147–160.

18.

Dipple

Raymond

Docherty

(2012) Stigmergy in web 2.0. In: Proceedings of the 3rd annual ACM web science conference on—WebSci ‘12 (eds Contractor

Uzzi

Macy

, et al.), Evanston, IL, 22–24 June, pp. 86–94. New York: ACM Press.

19.

dos Santos

Lycarião

Aquino

, et al. (2018) The virtuous cycle of news sharing on Facebook: effects of platform affordances and journalistic routines on news sharing. New Media & Society 21(2): 398–418.

20.

El-Arini

Tang

(2014) Click-baiting. Available at: https://about.fb.com/news/2014/08/news-feed-fyi-click-baiting/

21.

Etienne

(2013) Ambiguity and relational signals in regulator-regulatee relationships. Regulation & Governance 7(1): 30–47.

22.

Ferrer-Conill

(2017) Quantifying journalism? A study on the use of data and gamification to motivate journalists. Television & New Media 18(8): 706–720.

23.

Fürst

(2020) In the service of good journalism and audience interests? How audience metrics affect news quality. Media and Communication 8(3): 270–280.

24.

Garz

Szucs

(2021) Algorithmic selection and ideological diversity of news on Facebook. Working paper. Available at: https://www.marcelgarz.com/wp-content/uploads/News-Feed-Apr2021.pdf

25.

Grassé

P-P

(1959) La reconstruction du nid et les coordinations interindividuelles chez Bellicositermes natalensis et Cubitermes sp. La théorie de la stigmergie: Essai d’interprétation du comportement des termites constructeurs. Insectes Sociaux 6(1): 41–80.

26.

Hågvar

(2019) News media’s rhetoric on Facebook. Journalism Practice 13(7): 853–872.

27.

Haim

Karlsson

Ferrer-Conill

, et al. (2021) You should read this study! It investigates Scandinavian social media logics ☝. Digital Journalism 9: 406–426.

28.

Haim

Kümpel

Brosius

H-B

(2018) Popularity cues in online media: a review of conceptualizations, operationalizations, and general effects. Studies in Communication—Media 7(2): 186–207.

29.

Hanitzsch

Hanusch

Mellado

, et al. (2011) Mapping journalism cultures across nations: a comparative study of 18 countries. Journalism Studies 12(3): 273–293.

30.

Hansen

Hoskisson

Barney

(2008) Competitive advantage in alliance governance: resolving the opportunism minimization–gain maximization paradox. Managerial and Decision Economics 29(2–3): 191–208.

31.

Kamen

(2016) Twitter’s non-chronological timeline is now opt-out. WIRED, 17 March. Available at: https://www.wired.co.uk/article/twitter-non-chronological-timeline-how-to-opt-out#:~:text=Twitter’s%20non%2Dchronological%20timeline%20is%20now%20opt%2Dout,-Oli%20Scarff%2FGetty&text=The%20latest%20documentation%20change%20adds,with%2C%20and%20much%20more.%22

32.

Kuhn

(2008) Building predictive models in R using the caret package. Journal of Statistical Software 28(5): 1–26.

33.

Kuiken

Schuth

Spitters

, et al. (2017) Effective headlines of newspaper articles in a digital environment. Digital Journalism 5(10): 1300–1314.

34.

Lamot

Van Aelst

(2020) Beaten by Chartbeat? Journalism Studies 21(4): 477–493.

35.

Landis

Koch

(1977) The measurement of observer agreement for categorical data. Biometrics 33(1): 159.

36.

Lischka

(2021) Logics in social media news making: how social media editors marry the Facebook logic with journalistic standards. Journalism 22(2): 430–447.

37.

Livingstone

(2019) Audiences in an age of datafication: critical questions for media research. Television & New Media 20(2): 170–183.

38.

Magin

Steiner

Häuptli

, et al. (2021) Is Facebook driving tabloidization. In: Conboy

Eldridge

(eds) Global Tabloid: Culture and Technology. London: Routledge, pp. 56–74.

39.

Meese

Hurcombe

(2020) Facebook, news media and platform dependency: the institutional impacts of news distribution on social platforms. New Media & Society. Epub ahead of print 3 June. DOI: 10.1177/1461444820926472.

40.

Molyneux

Coddington

(2020) Aggregation, clickbait and their effect on perceptions of journalistic credibility and quality. Journalism Practice 14(4): 429–446.

41.

Mourão

Robertson

(2019) Fake news as discursive integration: an analysis of sites that publish false, misleading, hyperpartisan and sensational information. Journalism Studies 20(14): 2077–2095.

42.

Munger

(2020) All the news that’s fit to click: the economics of clickbait media. Political Communication 37(3): 376–397.

43.

Myllylahti

(2020) Paying attention to attention: a conceptual framework for studying news reader revenue models related to platforms. Digital Journalism 8(5): 567–575.

44.

Nash

(1951) Non-cooperative games. Annals of Mathematics 54: 286–295.

45.

Naughton

(2018) Platform power and responsibility in the attention economy. In: Moore

Tambini

(eds) Digital Dominance: The Power of Google, Amazon, Facebook, and Apple. Oxford: Oxford University Press, pp. 371–395.

46.

Neumann

von Morgenstern

(1944) Theory of Games and Economic Behavior. Princeton, NJ: Princeton University Press.

47.

Newman

(2016) Journalism, media, and technology: trends and predictions 2016. Digital News Project. Available at: https://reutersinstitute.politics.ox.ac.uk/sites/default/files/2017-06/Journalism%2C%20media%20and%20technology%20predictions%202016.pdf (accessed 11 January 2017).

48.

Newman

Fletcher

Kalogeropoulos

, et al. (2018) Reuters Institute Digital News Report 2018. Available at: media.digitalnewsreport.org/wp-content/uploads/2018/06/digital-news-report-2018.pdf?x89475 (accessed 22 June 2017).

49.

Nielsen

Ganter

(2017) Dealing with digital intermediaries: a case study of the relations between publishers and platforms. New Media & Society 20(4): 1600–1617.

50.

Osborne

Rubinstein

(1994) A Course in Game Theory. Cambridge, MA: MIT Press.

51.

Peterson-Salahuddin

Diakopoulos

(2020) Negotiated autonomy: the role of social media algorithms in editorial decision making. Media and Communication 8(3): 27–38.

52.

Peysakhovich

Hendrix

(2016) Further reducing clickbait in feed. Available at: https://about.fb.com/news/2016/08/news-feed-fyi-further-reducing-clickbait-in-feed/

53.

Potthast

Gollub

Komlossy

, et al. (2018) Crowdsourcing a large corpus of clickbait on Twitter. In: Proceedings of the 27th international conference on computational linguistics, Santa Fe, NM, August, pp. 1498–1507. Stroudsburg, PA: Association for Computational Linguistics.

54.

Rony

MMU

Hassan

Yousuf

(2017) Diving deep into clickbaits. In: Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017 (eds Diesner

Ferrari

), Sydney, NSW, Australia, 31 July–3 August, pp. 232–239. New York: ACM.

55.

Rosiana

(2015) While you were away. Available at: https://blog.twitter.com/official/en_us/a/2015/while-you-were-away-0.html

56.

Scacco

Muddiman

(2020) The curiosity effect: information seeking in the contemporary news environment. New Media & Society 22(3): 429–448.

57.

Schultz

(2007) The Journalistic Gut Feeling: journalistic Doxa, news habitus and orthodox news values. Journalism Practice 1(2): 190–207.

58.

Siegert

Gerth

Rademacher

(2011) Brand identity-driven decision making by journalists and media managers: the MBAC model as a theoretical framework. International Journal on Media Management 13(1): 53–70.

59.

Sismeiro

Mahmood

(2018) Competitive vs. complementary effects in online social networks and news consumption: a natural experiment. Management Science 64(11): 5014–5037.

60.

Starmer

(2000) Developments in non-expected utility theory: the hunt for a descriptive theory of choice under risk. Journal of Economic Literature 38(2): 332–382.

61.

Suri

(2011) Purposeful sampling in qualitative research synthesis. Qualitative Research Journal 11(2): 63–75.

62.

Tandoc

(2018) Five ways BuzzFeed is preserving (or transforming) the journalistic field. Journalism 19(2): 200–216.

63.

Tandoc

Maitra

(2018) News organizations’ use of Native Videos on Facebook: tweaking the journalistic field one algorithm change at a time. New Media & Society 20(5): 1679–1696.

64.

Vrancx

Verbeeck

Nowé

(2010) Analyzing the dynamics of stigmergetic interactions through pheromone games. Theoretical Computer Science 411(21): 2116–2126.

65.

Zhang

Bian

, et al. (2020) Seeing is not always believing: an exploratory study of clickbait in WeChat. Internet Research 30(3): 1043–1058.