Exploring the forces driving ChatGPT diffusion on X: An integrated theoretical perspective

Abstract

This study examines the key factors driving ChatGPT’s diffusion on X through the media effects and innovation diffusion theories. The findings reveal perceived usefulness was discussed more than topics like government regulations, with positive and neutral sentiments dominating. Theoretically, this study advances understanding of innovation diffusion via the media effects perspective. Practically, it offers guidance to AI developers and stakeholders for improving AI applications and decision-making.

Keywords

ChatGPT diffusion of innovations llama media effects social media

Introduction

ChatGPT is a large language model developed by OpenAI that has garnered significant attention in recent years. This model uses machine learning algorithms to generate human-like text, allowing it to be used for a wide range of applications, including customer service, language translation and content creation. As an emerging technology that has the potential to revolutionize the internet, ChatGPT had totted up more than 100 million users within the first 2 months of its launch in November 2022. As of March 2024, ChatGPT has around 180.5 million users, with 1.6 billion visits (Duarte, 2024). It has overtaken all previous user applications in terms of diffusion rate within such a short period. Research firm Gartner predicts that by 2026, the market for AI software will reach US$453 billion, and market growth is expected to accelerate from 14.4 per cent in 2021 to 47 percent in 2026, far outpacing overall growth in the software market. With the fastest diffusion speed, ChatGPT will play an essential role in this huge market.

ChatGPT is a type of AI-based chatbot that has rapidly gained global diffusion. The AI-based chatbots are automated conversational agents designed with user interfaces that facilitate natural language interactions (Adamopoulou and Moussiades, 2020). Since their inception, AI-based chatbots have attracted significant attention from researchers, who have explored the driving forces behind their rapid adoption and diffusion (e.g. Dobrinićč et al., 2021; Hari et al., 2022; Hwang and Kim, 2021; Kushwaha et al., 2021; Kwangsawad and Jattamart, 2022). Two well-established theoretical frameworks – the Diffusion of Innovations Theory (DOI) and the Technology Acceptance Model (TAM) – have been widely applied by these researchers to investigate how individuals across various industries adopt AI-based chatbots.

ChatGPT has rapidly emerged as a focal point in public discourse, sparking extensive discussions across traditional and social media platforms. Drawing on media effects theories such as agenda-setting and framing (Entman, 1993; McCombs and Shaw, 1972), this study explores how media agendas and frames shape public perceptions and attitudes toward emerging technologies. While these theories have been widely applied in domains like political communication, health communication and public opinion research, their application to the adoption and diffusion of innovations remains underexplored. This research bridges this gap by examining the interplay between major themes in public discourse and key constructs from the Diffusion of Innovations (DOI) and Technology Acceptance Model (TAM). By integrating these frameworks, the study offers novel theoretical insights into how media narratives influence the acceptance and diffusion of technological innovations, thereby advancing media effects research and contributing to a deeper understanding of innovation adoption processes. To achieve this, we categorize discourse into micro-level themes, which focus on ChatGPT’s characteristics, and macro-level themes, which address its broader societal impacts. By applying second-level agenda-setting theory to micro-level themes and framing theory to macro-level themes, this conceptual framework offers a comprehensive understanding of ChatGPT’s representation and societal implications.

Literature review

Diffusion of innovations theory, technology acceptance model and chatbot adoption

The Diffusion of Innovations (DOI) theory defines the rate of innovation adoption as the speed at which members of a social system adopt an innovation, typically measured over a given time period (Rogers, 1962). The theory suggests that adoption follows an S-shaped curve, with the slope representing the adoption rate. Rogers (2003) noted variations in the slope across innovations, indicating that some diffuse faster than others, and even the same innovation can exhibit different adoption rates across social systems. DOI also outlines five key characteristics of innovations: relative advantage (the perceived superiority over what it replaces), compatibility (consistency with adopters’ values and needs), complexity (ease of understanding and use), trialability (opportunity to test the innovation), and observability (visibility of tangible results; Rogers, 2003).

The Technology Acceptance Model (TAM) is a key framework for studying the factors driving technology adoption. TAM identifies two critical attributes – perceived ease of use (PEOU) and perceived usefulness (PU) – as key influences on individuals’ intentions to adopt innovations (Davis et al., 1989). PEOU refers to the effort required to use the innovation, while PU reflects the perceived utility of the innovation. Unlike the societal-level focus of the Diffusion of Innovations (DOI) theory, TAM emphasizes adopters’ psychological factors, particularly subjective attitudes and intentions.

To address the original TAM’s neglect of social systems, scholars proposed extensions like the Unified Theory of Acceptance and Use of Technology (UTAUT). UTAUT identifies four factors influencing adoption: performance expectancy (belief that the innovation enhances job performance), effort expectancy (ease of use), social influence (perceptions of others’ expectations), and facilitating conditions (availability of organizational and technical support; Venkatesh et al., 2003).

Researchers have extended the DOI and TAM/UTAUT theories by incorporating new variables, such as perceived risk, to better understand innovation diffusion and adoption. Perceived risk refers to consumers’ perceptions of risks associated with adopting an innovation and encompasses six dimensions: performance, financial, social, psychological, time and privacy (safety) risks (Faroughian et al., 2012). Many studies have built on DOI and TAM/UTAUT by integrating perceived risk. For instance, Chang et al. (2016) modified the UTAUT model by adding perceived risk to examine consumers’ online shopping intentions, while Farzin et al. (2023) extended DOI and UTAUT with perceived risk to study the acceptance of autonomous vehicles.

DOI and TAM are well-established theories widely applied to study the diffusion and adoption of digital innovations, including the internet, broadband and mobile media. Both frameworks have also been used to examine chatbot adoption. For instance, Kushwaha et al. (2021) analyzed AI-based chatbots’ impacts on B2B customer experience, finding that adoption is influenced by system design, users’ technological proficiency, and trust in the brand and system. Similarly, Dobrinićč et al. (2021) studied Croatian consumers’ attitudes toward mobile messenger chatbots, revealing that perceived usefulness, ease of use, compatibility, and attitudes toward mobile ads significantly impact chatbot adoption.

Communication, media effects and innovation diffusion/adoption

DOI emphasizes the critical role of communication in influencing the rate and extent of innovation diffusion. Communication factors such as channels, styles, source credibility, social context and opinion leaders shape perceptions and adoption of innovations (Rogers, 2003). Building on this idea, scholars have incorporated communication variables into diffusion models. Barnett (2011), for example, integrated communication variables into mathematical models to capture the complexity of innovation diffusion. Similarly, Bunea et al. (2022) distinguished between mediated and interpersonal communication, noting that the former affects adoption rates, while the latter influences imitation rates.

As social media emerge as a key communication channel with interpersonal characteristics, their role in innovation diffusion and adoption has drawn scholarly attention. Zhu et al. (2020) developed a media-based perceptions and adoption model, revealing that mass media and social media have opposite effects on perceived risks for fully autonomous vehicles—mass media increases perceived risks, while social media reduce them. Similarly, Purwiyati et al. (2020) found that platforms such as Facebook, WhatsApp and YouTube significantly influence farmers’ innovation adoption. Hu et al. (2019) demonstrated that endorsements by social media influencers boost adoption intentions for computer applications by directly enhancing trust in the applications.

While DOI theory emphasizes the role of communication in innovation diffusion, and numerous studies have shown that mass media and social media significantly shape perceptions of various innovation aspects, major media effects theories have not yet been integrated into DOI theory or these prior studies. This gap highlights a valuable new research opportunity to explore how media effects theories can deepen our understanding of the influence of media on the diffusion and adoption of innovations.

Media effects is a major research area in communication studies, encompassing the ways in which media influences individuals’ perceptions, behaviors and decision-making processes. Communication theorists have developed numerous theories to explain these effects, with agenda-setting theory and framing theory standing out as two of the most well-established and widely-applied frameworks. Agenda-setting theory, first introduced by McCombs and Shaw (1972), posits that media influences what issues the public considers important by emphasizing certain topics over others. Framing theory, as articulated by Entman (1993), explains how media shapes the way issues are presented, influencing how audiences interpret and respond to them. Both theories have been extensively tested and applied across various contexts, from political communication to health campaigns, demonstrating their enduring relevance in understanding media’s role in shaping public opinion and behavior (Scheufele and Tewksbury, 2007; Valkenburg et al., 2016).

The foundational concept of agenda setting emphasizes how the prominence of issues in media translates to public opinion (Severin and Tankard, 2001). Building on this, the second level of agenda-setting theory examines how media attributes and tones affect audience perceptions. It encompasses two dimensions: the cognitive dimension, which addresses the characteristics of an issue, and the affective dimension, which considers the valence (positive, negative or neutral) of those attributes (McCombs and Reynolds, 2009). This theory suggests that by emphasizing specific attributes with varying tones, media can shape both the perceived importance of these attributes and the public’s emotional response to them (McCombs, 2005). Additionally, agenda-setting theorists have investigated the psychological factors driving information-seeking behaviors and proposed the need for orientation hypothesis, which suggests that individuals seek media information to navigate their environment, influenced by relevance and uncertainty (Weaver, 1977; McCombs and Reynolds, 2009).

Since Entman (1993) introduced the concept of framing to communication research, framing theory has become a key framework in media research, aiding in the analysis of how news media cover various issues (e.g. de Vreese, 2005; Famulari, 2020; Pan and Kosicki, 2010) and how framing operates in social media contexts (e.g. Muralidharan et al., 2011; Russell Neuman et al., 2014; Yang et al., 2019). Research has shown that framing can significantly influence public attitudes and behaviors across different domains, including political perceptions (Lecheler et al., 2015), health-care opinions (Chong and Druckman, 2007) and environmental attitudes (Dunaway et al., 2018). Overall, framing theory provides a vital lens for understanding the complexities of communication in contemporary media landscapes (Scheufele, 1999).

Although agenda-setting theory and framing theory originate from distinct theoretical backgrounds, their relationship is complex and can be understood through three main perspectives. The first perspective equates framing with attribute agenda-setting, suggesting they are essentially the same concept (McCombs, 2005). The second perspective posits that the two theories complement each other and align in their functions (Entman, 1993). The third perspective argues that agenda-setting and framing are fundamentally different and cannot be integrated (Scheufele and Tewksbury, 2007).

An attempt to integrate innovation diffusion/adoption theories and media effects theories in the case of ChatGPT

A new theoretical framework

Although these media effect theories have been widely used in political communication, health communication and public opinion research, they have seldom been applied in the study of the diffusion and adoption of innovative technologies. The literature on innovation diffusion/adoption suggests communication plays a crucial role in the process of innovation diffusion/adoption. And there are many studies that demonstrate news media and social media have significant influence on innovation diffusion/adoption regardless of the fact that there is no strong connection between innovation diffusion/adoption theories and media effects theories in the literature. There is a potential that these two types of theories could be combined to better interpret the role of media in innovation diffusion/adoption.

This study attempts to fill this gap in the literature and build the connection between innovation diffusion/adoption theories and media effects theories. It identifies the attributes and frames in the public discourse of ChatGPT in social media and investigates the possible connections between the two types of theories. The constructs in DOI theory and those in TAM/ TUAUT have some overlaps. For example, relative advantage in DOI has an overlap with performance expectancy in TUAUT; the perceived usefulness in TAM has an overlap with compatibility in DOI, as well as performance expectancy in TUAUT. To combine these theories to study the diffusion of ChatGPT, we select the constructs that have no overlap with one another in these theories. On the other hand, the relationship between agenda-setting theory and framing theory is complex. To combine these theories, we adopt the view of Entman (1993) that agenda-setting theory and framing theory are complementary. We differentiate between two types of themes in the discourse. The first type focuses on the micro-level characteristics or aspects of ChatGPT, while the second type addresses the macro-level social influences or impacts of ChatGPT. Drawing on second-level agenda-setting theory, we examine micro-level themes and public sentiment, defining these themes as attributes of ChatGPT in the discourse. Simultaneously, we employ framing theory to analyze macro-level themes, which we define as frames within the discourse. This dual approach allows for a comprehensive understanding of ChatGPT’s representation and societal implications (see Figure 1).

Figure 1.

The conceptualization of frames and attributes of the social media discourse on ChatGPT.

Definitions of attributes and frames in the social media discourse of ChatGPT

Particularly, we define the following four micro-level attributes in the social media discourse of ChatGPT:

Perceived usefulness: how ChatGPT can help people do things more effectively or efficiently;

Relative advantage: how ChatGPT does a better job than other applications/technologies;

Observability: the extent to which ChatGPT provides tangible results; and

Peer influence: the degree to which an individual perceives that other people believe they should use ChatGPT.

We define the following four macro-level frames in the social media discourse of ChatGPT:

Facilitating conditions: the organizational and technical infrastructure, as well as the institutional policies and other forces that support or prohibit the use of ChatGPT;

Social impacts: a variety of social impacts that ChatGPT brings about. These impacts might be positive, such as promoting production at a societal level, creating new industries and job opportunities, facilitating further innovation, etc. They might be negative such as job replacement, salary and benefit reduction, bargain, protest, etc.;

Ethical issues: the concerns that people have about ChatGPT in terms of ethics such as cheating, phishing, cybercrime, etc.; and

Government regulations: how the government should regulate ChatGPT, as well as legal issues such as copyright and intellectual property.

We exclude trialability, defined as the degree to which an innovation can be tested or experimented with prior to adoption, as ChatGPT offers a free app account, making it readily accessible for experimentation. Similarly, we omit constructs such as complexity, perceived ease of use, and effort expectancy, given that ChatGPT is designed for ease of use, requiring only basic typing and search skills to operate its core functionalities. Additionally, performance expectancy is excluded due to its conceptual overlap with perceived usefulness and relative advantage.

To address the unique societal implications of ChatGPT as a disruptive technology, we introduce three new constructs: social impacts, ethical issues and government regulations. These constructs reflect the transformative potential of ChatGPT across various societal domains, its associated ethical challenges, and the need for regulatory oversight. Recent studies have extensively explored these issues (e.g. Agathokleous et al., 2023; Cao and Zhai, 2023; Chavanayarn, 2023; Chung, 2023; Delellis et al., 2023; Gursoy et al., 2023; Hung and Chen, 2023; Jeyaraman et al., 2023; Yan et al., 2023; Yang, 2023). These new constructs align with compatibility in DOI, as well as performance expectancy and perceived risk in UTAUT. Furthermore, we use peer influence instead of social influence to avoid conceptual overlap with the newly introduced construct of social impacts, which pertains to broader societal consequences rather than interpersonal influence.

Based on the literature review, the new theoretical framework, and the conceptual division of attributes and frames in the social media discourse on ChatGPT, the following research questions are proposed:

RQ1: What are the proportions and sentiment polarities of X posts addressing micro-level attributes versus macro-level frames of ChatGPT?

RQ2: Which micro-level attributes or macro-level frames of ChatGPT were most frequently discussed and most positively depicted in X posts?

Methods

We used X (formerly Twitter) as the representative of social media because X has increased its dominance in public discourse (Jones-Jang et al., 2020). The X data on ChatGPT was downloaded from Kaggle Datasets (Bhattacharjee, 2023) on 29 September 2023. The dataset contains the information of 307,219 posts in English. To reduce the noise in the dataset, we filtered this dataset by searching for the keyword ‘ChatGPT’ in the posts’ texts. Our final data sample contains the information of 49,317 posts. The time range of these posts is between 5 December 2022 to 3 April 2023, which was the period when ChatGPT started to diffuse rapidly across the world.

We applied the aspect-based sentiment analysis (ABSA) approach to conduct data analysis. ABSA is a natural language processing (NLP) technique that goes beyond traditional sentiment analysis by not only determining the overall sentiment of a piece of text but also identifying and analyzing the sentiment associated with specific aspects mentioned in the text. In this study, the specific aspects are the micro-level attributes and macro-level frames defined previously. The ABSA process typically involves two steps: (1) Aspect extraction through which the specific aspects are identified and extracted from data; and (2) Sentiment classification through which the sentiment polarity (positive, negative or neutral) associated with each identified aspect is labeled.

As the dataset we analyzed is large, it is not possible to conduct ABSA analysis using the traditional human coding approach. The cutting-edge NLP technologies, the large language models (LLMs), enable us to do such content analysis efficiently. These LLMs have been created based on the Transformer model, which was originally developed by Google Research and Google Brain in 2017 (Vaswani et al., 2017). Earlier LLMs, such as the BERT models, were fine-tuning models that require task-specific datasets to fine-tune the pre-trained models. Recently developed LLMs, such as GPT models and Llama models, are Zero-Shot models that do not need to be fine-tuned and possess the capacities of cognition and reasoning like human intelligence.

The use of Large Language Models (LLMs) in analyzing social media content has garnered significant attention in recent research. For instance, Garg et al. (2023) employed the GPT-4 model to analyze public sentiment on COVID-19 vaccines, demonstrating its effectiveness in extracting insights from large-scale social media data. Similarly, Lyu et al. (2023) explored GPT-4′s application in social media analysis, highlighting its efficacy in tasks such as sentiment analysis, hate speech detection, fake news identification and political ideology detection. Additionally, Pendyala and Hall (2024) examined the capabilities of LLMs, including Llama, Orca, Falcon and Mistral, in identifying misinformation across diverse datasets. Their findings indicate that LLMs can effectively detect misinformation by leveraging their extensive knowledge bases. Collectively, these studies underscore the transformative potential of LLMs in advancing social media content research.

We used the Llama 2 (70 billion parameters) model, a Zero-Shot LLM, to identify the aspects (i.e. micro-level attributes and macro-level frames) in the X data sample. We provide the definitions of micro-level attributes and macro-level frames as well as examples of these concepts to the model. Then, we used the following query for the model to perform this task:

Here is a tweet¹:

The tweet text. . .

Does this tweet talk about the [perceived usefulness], [relative advantage], [observability], [peer influence], [facilitating conditions], [social impacts], [ethical issues], or [government regulations] of ChatGPT? Reply with only one topic label.

We also used the Llama 2 (70 billion parameters) model to conduct sentiment analysis on each post and associated these sentiment polarities with micro-level attributes and macro-level frames. To test the validity of the model, we randomly selected 1000 posts for human annotation and reliability tests. This validation consists of two steps. The first step is to test the inter-coder reliability, in which the two coders identify the micro-level attributes and macro-level frames, as well as the sentiment polarities from post texts, and answer the question of whether they agree with the model’s predictions (1 = agree, 0 = disagree). The two coders conducted this coding process independently and their coding results were used to calculate the inter-coder reliability index. After three rounds of inter-coder reliability tests, we achieved acceptable results between the two coders (percentage of agreement = 92%) for the aspect identification and (percentage of agreement = 86%) for the sentiment polarity. We also used the agreed coding results between the two codes to test the inter-coder reliability between the coders and the model and achieved acceptable results (percentage of agreement = 86%) for aspects identification and (percentage of agreement = 96%) for the sentiment polarity.

Results

Llama 2 assigned labels of micro-level attributes and macro-level frames to 38,092 (77.24%) posts. For the other 11,225 posts, the model only repeated the question and did not provide answers. The examples of the posts for each of the micro-level attributes and macro-level frames are reported in Table 1. Based on the prediction of the Llama 2 model, the majority of the posts (77.24%) are related to the micro-level attributes and macro-level frames defined in this study. The model also classified these posts into one of the three sentiment categories: positive, neutral and negative. The examples of the posts for each sentiment category are reported in Table 2. The number of positive posts is 25,325, consisting of 66.48 per cent of the total posts in the data sample. The number of neutral posts is 12,585, consisting of 33.04 per cent of the total posts in the data sample. The number of negative posts is 182, consisting of 0.48 per cent of the total posts in the data sample.

Table 1.

Examples of X posts on micro-level attributes and macro-level frames.

Attributes and frames	Examples
Perceived usefulness	Example 1: @thesephist AI natives will develop shorthand text/emoji/audio/gesture languages that optimized their interactions with AI, and the AI will translated for them so they can talk to others. Each person can have their own language. The common language will be JSON or XML like data. #ChatGPT. Example 2: My new AI startup reads your MS Teams messages and transforms them into easy-to-understand messages. It highlights the main point of each colleague’s message to eliminate misunderstandings.
Relative advantage	Example 1: ChatGPT-4 just finalized my itinerary for Cape Town. @Tripadvisor better be working on a plugin to book the experiences from within ChatGPT, before I do it and take a commission. . . Example 2: Artificial Intelligence is moving at the speed of light, and the possibilities, it seems, are endless. Now GPT-4 offers much more than GPT-3. Check out the article to find out what it can do.
Observability	Example 1: Playing around with how #chatgpt edits its content when you give it the same prompt but with two different pop culture characters as narrator https://t.co/Yujuh3EMwk. Example 2: #ChatGPT I asked ChatGPT 3.5 to write a python script to factorize a number, a result of multiplying two primes. The result was WRONG the line.
Peer influence	Example 1: I love fiddling around with ChatGPT and trying out cool new prompts If you have any you’d like to share, drop it in the replies below, and well share with your friends if you think they’d like it. Example 2: Are other #grantwriters using #chatgpt? I’ve been finding it incredibly useful! How are you using it and what are some of the limitations you’ve come across?
Facilitating conditions	Example 1: Do you know what is chatgpt? ChatGPT is an artificial-intelligence (AI) chatbot developed by OpenAI. Join us and learn how to use it. #ChatGPT #digital_marketing_course_in_delhi #digital_marketing_course_near_me Example 2: 21 ways to use ChatGPT in Real Estate. #realestate #chatgpt #realtor #realestatetraining
Social impacts	Example 1: Is emerging new technologies and automation a threat to future jobs and tax revenue? Example 2: ChatGPT shouldn’t be given the power to write full articles and code. This can literally destroy a ton of careers. It was able to write a data loader pipeline for yolov7.
Ethical issues	Example 1: ChatGPT: A genuine cyber security threat or massively overhyped? Example 2: #chatgpt will now have access to the #internet to make it more #reliable. As a bonus, it can now also become all powerful. This is basically the plot of the move Transcendence. Maybe we should slow down just a little and have a public debate about the pros and cons of this #technology.
Government regulations	Example 1: Due to concerns about data privacy, the privacy watchdog of the Italian government has temporarily blocked ChatGPT, an artificial intelligence (AI) software chatbot. Example 2: According to the reports: After Italy now many other countries like USA, China, Pakistan, India and Japan etc., are going to Ban ( Chat GPT ) in their Country.

Table 2.

Examples of X posts of different sentiments.

Sentiment category	Examples
Positive	Example 1: This means that #chatgpt can understand the context of a conversation and respond in a more natural and appropriate way. Example 2: My new AI startup reads your MS Teams messages and transforms them into easy-to-understand messages. It highlights the main point of each colleague’s message to eliminate misunderstandings.
Neutral	Example 1: One thing playing with the #chatgpt taught me– it is certainly likely that with certain priming the true capabilities of a system could be hidden. I doubt this is what they intended is to come away with it. But that is really the biggest impression. Example 2: As of now the GPT3 output is still easily detectable as AI content from trained GPT2 models. As shown in the top half of the screenshot below.
Negative	Example 1: I’m more creative than #chatgpt, it only reproduces in a humanly style, it does not create. Example 2: ChatGPT SUCKS at local SEO. I can’t believe how much it got wrong. Watch my heated discussion with the AI in the funniest video I’ve made to this date.

Perceived usefulness was the most frequently discussed micro-level attribute, with 21,263 posts (55.82% of the sample), followed by relative advantage (4188 posts, 10.99%). Observability (146 posts, 0.38%) and peer influence (86 posts, 0.23%) were discussed far less frequently. Among the macro-level frames, facilitating conditions was the most prominent, with 9895 posts (25.98%), while ethical issues (1467 posts, 3.85%), social impacts (815 posts, 2.14%) and government regulations (232 posts, 0.61%) were less frequently discussed. Overall, perceived usefulness, facilitating conditions and relative advantage accounted for 92.79 per cent of all the posts.

Regarding sentiment polarities, perceived usefulness had the highest number of positive (14,249 posts, 56.26%), negative (111 posts, 60.99%) and neutral posts (6903 posts, 54.85%). Facilitating conditions ranked second in positive (6164 posts, 24.34%) and neutral posts (3715 posts, 29.52%), while ethical issues had the second-highest number of negative posts (43 posts, 23.63%). For positive sentiment, relative advantage had the highest percentage (77.22%), followed by ethical issues (75.26%) and perceived usefulness (67.01%). Ethical issues also had the highest percentage of negative posts (2.93%), followed by social impacts (1.10%). For neutral sentiment, government regulations had the highest percentage (79.74%), followed by observability (55.48%) and social impacts (48.47%). Detailed percentages are provided in Table 3.

Table 3.

Number of X posts on different attributes/frames and sentiment categories.

Attributes/frames	Number of posts	Number of positive posts	Number of negative posts	Number of neutral posts
Perceived usefulness	21,263	14,249	111	6903
%	55.82%	56.26% (67.01%)	60.99% (0.52%)	54.85% (32.46%)
Facilitating conditions	9895	6164	16	3715
%	25.98%	24.34% (62.29%)	8.79% (0.16%)	29.52% (37.54%)
Relative advantage	4188	3234	3	951
%	10.99%	12.77% (77.22%)	1.65% (0.07%)	7.56% (22.71%)
Ethical issues	1467	1104	43	320
%	3.85%	4.36% (75.26%)	23.63% (2.93%)	2.54% (21.81%)
Social impacts	815	411	9	395
%	2.14%	1.62% (50.43%)	4.95% (1.10%)	3.14% (48.47%)
Government regulations	232	47	0	185
%	0.61%	0.19% (20.26%)	0 0	1.47% (79.74%)
Observability	146	65	0	81
%	0.38%	0.26% (44.52%)	0 0	0.64% (55.48%)
Peer influence	86	51	0	35
%	0.23%	0.2% (59.30%)	0 0	0.28% (40.70%)
Sum	38,092	25,325	182	12,585
%	100%	66.48%	0.48%	33.04%

Note: The values of percentages in the first column are the percentages of the numbers of posts of attributes/frames in the total number of posts in the data sample. The values of the percentages without parentheses in other columns are the percentages of the number of posts of different sentiments (positive, negative, neutral) in the total number of posts of these sentiment categories in the data sample. The values of the percentages within parentheses in the other columns are the percentages of the number of posts of different sentiments (positive, negative, neutral) in the total number of posts of the respective attributes/frames in the data sample.

Discussion

As a cutting-edge and rapidly diffused innovation, ChatGPT has garnered significant attention in public discourse. This study employs Llama 2 to identify four micro-level themes – perceived usefulness, relative advantage, observability and peer influence – and four macro-level themes – facilitating conditions, social impacts, ethical issues and government regulations – in the public discourse surrounding ChatGPT on X. These themes are derived from the literature of two major innovation diffusion theories (DOI and TAM/UTAUT) and the most recent research on ChatGPT’s diffusion. Drawing on second-level agenda-setting theory, the micro-level themes are conceptualized as attributes, while the macro-level themes are defined as frames based on framing theory. This conceptual division allows for a nuanced analysis of how ChatGPT is represented in social media discourse.

The prediction of the Llama 2 model demonstrates that most of the posts in the data sample are related to these attributes and frames. The top three most frequently discussed attributes and frames include two attributes (perceived usefulness and relative advantage) and one frame (facilitating conditions). The attributes of observability and peer influence received the least attention in the social media discourse. The results of the sentiment analysis demonstrate that most posts are either positive or neutral, and only a small proportion of posts are negative.

In terms of sentiment polarities, perceived usefulness has the highest number of positive and neutral posts, as well as the most negative posts among all attributes and frames. These negative posts discussed the flaws or shortcomings of ChapGPT and other AI applications such as inaccuracy, incompetence and inconvenience. An example of these posts is: ‘I’m more creative than #chatgpt, it only reproduces in a human style, it does not create’. Another negative area in the discourse revolves around the ethical issues frame. These posts discussed the privacy, bias, conflict with ethical norms, and threats to humanity issues brought about by ChatGPT and other AI applications. An example of these posts is: ‘#chatgpt is a fine example of mankind’s failures when its programmers are biased and put in their own bullshit views. AI will always be what man wants it to be and with it. It will inherit mankind’s failures and will never have true thought’.

The attribute of relative advantage, on its own, has the highest percentage of positive posts among all attributes. The positive posts on relative advantage revolved around the advantages of ChatGPT and other AI applications in terms of enhancing, replacing or superseding other technologies or tools. Here is an example: ‘Transform your website into a user engagement powerhouse with our AI SuperApp Builder’s custom mini apps and monetization options powered by OpenAI API’.

It is somewhat surprising that most posts about ethical issues are positive. These posts discuss the potential of ChatGPT and other AI in addressing ethical issues, and advocate developing ChatGPT and AI in ethical ways, etc. Here are two examples: ‘Before #EinsteinGPT was announced I was already playing with integration with #ChatGPT API and was building a bunch of Salesforce features. What about a chatter moderator to control violence and sexual posts using the moderation api? What if chatgpt was a chatter free user?’ ‘New AI tools are emerging daily. It’s so hard to keep up! That’s why examining AI ethics/responsible use in education is important. Our multi-disciplinary panel promises lively debate and fresh perspectives ahead!’

The frame of social impacts, on its own, has the second largest percentage of negative posts. These negative posts discuss the negative impacts of ChatGPT and AI on our lives and societies. Here is one example: ‘#chatgpt takes over your job and you have to resort to eating your moms dog turd for a living’. Another frame – government regulations – has no negative posts and has the largest percentage of neutral posts on its own. This suggests that most X users are not against government regulations on ChatGPT.

It is worth noting that while the macro-level frames – ethical issues, social impacts and government regulations – have garnered significant attention from researchers (e.g. Agathokleous et al., 2023; Cao and Zhai, 2023; Chavanayarn, 2023; Chung, 2023; Delellis et al., 2023; Gursoy et al., 2023; Hung and Chen, 2023; Jeyaraman et al., 2023; Yan et al., 2023), X users discussed these topics less frequently during the initial months of ChatGPT’s global diffusion. As ChatGPT becomes more widely adopted and applied across various settings, it is reasonable to expect increased public discourse on these frames.

This study offers several theoretical contributions. While major innovation diffusion/adoption theories (DOI and TAM/UTAUT) examine how perceptions influence innovation diffusion, they do not address the origins or formation of these perceptions. Media effects theories can provide insights into these critical questions. Additionally, although innovation diffusion theories highlight the role of communication in the diffusion process, they do not explore the impact of media messages. This study bridges these gaps by integrating innovation diffusion/adoption theories with media effects theories. The proposed conceptual framework establishes a foundation for analyzing media messages related to innovation diffusion. Although the analysis of this study focuses on ChatGPT, this framework has the potential to be applied to other innovations.

The key finding that various attributes and frames have differing levels of prominence in public discourse suggests that people do not perceive all factors as equally significant. While major innovation diffusion/adoption theories propose that factors related to these attributes and frames drive diffusion, they do not account for variations in their significance. Media effects theories, however, argue that the prominence of attributes and frames in public discourse shapes perceptions, with differing levels of prominence leading to varying degrees of attention and potentially influencing perceptions or intentions. This study’s findings highlight the need to advance innovation diffusion/adoption theories by integrating media effects theories and exploring the varying significance of their key factors.

The predominance of positive sentiment reflects the general attitudes of X users toward ChatGPT, as X serves as a platform for expressing opinions. According to second-level agenda-setting theory, these sentiments shape users’ affective or emotional perceptions of ChatGPT. The overall positive attitudes align with the rapid diffusion of this innovation. The sentiment patterns revealed in this study – such as perceived usefulness having more positive posts than other attributes and frames – suggest that certain attributes elicit more favorable emotional perceptions, influencing adoption intentions and behaviors. While individuals naturally have varying emotional responses to innovations, DOI and TAM/UTAUT do not address these differences or their impact on adoption. The second-level agenda-setting theory fills this gap by explaining how media message sentiments influence affective perceptions. The findings of sentiment patterns underscore the need to integrate the second-level agenda-setting theory with DOI and TAM/UTAUT to explore the role of media effects in innovation diffusion.

This study has practical implications for OpenAI and other AI developers to improve their AI applications. Users’ feelings and experiences are especially important for them to develop their products. The discussions on X as well as other social media would reflect the users’ feelings and experiences. They should pay attention to these discussions and utilize the valuable information in the discussions to improve their current applications or guide new application development. They should focus more on the negative discussions on social media, which might reflect users’ concerns or the flaws in their applications. Other organizations, such as governments and AI investors, should also investigate the public discourse of ChatGPT and AI in social media. The attributes/frame and sentiment analysis would help them make wise policies and strategies based on the understanding of public opinion reflected in the public discourse.

As the first attempt that applies the Llama 2 model to analyze social media content, this study has several limitations. The data used in this study is from the second-hand dataset. It does not contain all the posts during the time frame of investigation. As X changed its Application Programming Interface drastically, it is very hard to extract historical data from this platform. Although many studies used the second dataset, the authors note this is the major limitation of this study. Another limitation may come from the machine learning model approach. Although large language models (LLMs), such as Llama 2, are powerful tools to analyze large datasets, they may be not as accurate as human brains in recognizing the patterns in textual data. The posts on X are usually short and sometimes messy, which add more risks to semantic analysis. Moreover, this study only identifies the attributes/frames and sentiments in the X discourse but does not investigate the influence of these factors on the adoption intentions and behaviors of ChatGPT. This limitation suggests a direction for future research.

Notwithstanding these limitations, this study suggests the potentials of the theoretical integration between media effects theories and innovation diffusion/adoption theories, as well as the utilization of LLMs in analyzing big data in social media. Future research would benefit from further exploring these potentials.

Footnotes

ORCID iDs

Xiaoqun Zhang

Lingzi Hong

Mohammed Huraysi

Sharon Lisseth Perez

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Notes

Author biographies

Dr Xiaoqun Zhang is an Associate Professor at the Mayborn School of Journalism. His research interests include media management and economics, media technology and policy, and the measurement of media coverage. His work has been published in Communication Research, Journalism & Mass Communication Quarterly, Telematics & Informatics, Telecommunications Policy, Time & Society, Journalism Studies and Corporate Reputation Review.

Dr Lingzi Hong is an Associate Professor of Data Science at the College of Information and an affiliated Associate Professor of Computer Science at the University of North Texas. Her research centers on developing AI-driven methods and systems that are responsive to diverse information needs. Hong’s work has been published in top-tier conferences and journals, including AAAI, NAACL, ICWSM, EMNLP, JASIS&T, and the International Journal of Human-Computer Interaction. She is also a recipient of multiple research grants from the Institute of Museum and Library Services (IMLS).

Dr Mohammed Huraysi is an Assistant Professor of Media and Journalism at King Khalid University. He earned his PhD and Master’s degrees in journalism from the University of North Texas, focusing on foreign news coverage and press freedom related to Saudi Arabia. His research interests include reputation management, tourism reputation, historical documentation, and issues of race, gender and class. He has received multiple international awards from organizations such as AEJMC, the Broadcast Education Association and the Arab League.

Sharon Lisseth Perez is a PhD candidate in Information Science at the University of North Texas and a Fulbright and PEO scholar. Originally from Panama, her work is grounded in a commitment to using technology, education and research to address social inequalities. Her research focuses on information access and equity, with an emphasis on supporting communities through more inclusive and effective use of information.

References

Adamopoulou

Moussiades

(2020) Chatbots: History, technology, and applications. Machine Learning and Applications: An International Journal 2: 100006.

Agathokleous

Saitanis

Fang

, et al. (2023) Use of ChatGPT: What does it mean for biology and environmental science? Science of the Total Environment 888: 164.

Barnett

(2011) Mathematical models of the diffusion process. In: Vishwanath

Barnett

(eds) The Diffusion of Innovations: A Communication Science Perspective. Peter Lang Publishing, pp. 105–124.

Bhattacharjee

(2023) Tweets on ChatGPT– #ChatGPT. Available at: https://www.kaggle.com/datasets/manishabhatt22/tweets-onchatgpt-chatgpt

Bunea

Guidolin

Manfredi

, et al. (2022) Diffusion of solar PV energy in the UK: A comparison of sectoral patterns. Forecasting 4: 456–476.

Cao

Zhai

(2023) Bridging the gap – The impact of ChatGPT on financial research. Journal of Chinese Economic and Business Studies 21(2): 177–191.

Chang

Jain

(2016) Modifying UTAUT and innovation diffusion theory to reveal online shopping behavior: Familiarity and perceived risk as mediators. Information Development 32(5): 1757–1773.

Chavanayarn

(2023) Navigating ethical complexities through epistemological analysis of ChatGPT. Bulletin of Science Technology & Society 43(3-4): 105–114.

Chong

Druckman

(2007) Framing public opinion in competitive democracies. American Political Science Review 101(4): 637–655.

10.

Chung

(2023) What is the impact of ChatGPT on education? A rapid review of the literature. MIS Quarterly 13(3): 319–340.

11.

Davis

Bagozzi

Warshaw

(1989) User acceptance of computer technology: A comparison of two theoretical models. Management Science 35: 982–1003.

12.

Delellis

Chen

Cornwell

, et al. (2023) ChatGPT media coverage metrics; initial examination media coverage metrics; initial examination. Proceedings of the Association for Information Science and Technology 60: 935–937.

13.

de Vreese

(2005) News framing: Theory and typology. Information Design Journal 13(1): 51–62.

14.

Dobrinićč

Gregurec

Dobrinić

(2021) Attitudes of Croatian consumers about mobile messenger chatbots. Journal of information and organizational sciences 45(2): 579–599.

15.

Duarte

(2024) Number of ChatGPT users (Mar 2024). Exploding topics. Available at: https://explodingtopics.com/blog/chatgpt-users

16.

Dunaway

Bystrom

Tinkham

(2018) Media framing of climate change: A meta-analysis. Environmental Communication 12(4): 471–485.

17.

Entman

(1993) Framing: Toward clarification of a fractured paradigm. Journal of Communication 43(4): 51–58.

18.

Famulari

(2020) Framing the Trump administration’s “Zero Tolerance” policy: A quantitative content analysis of news stories and visuals in US news websites. Journalism Studies 21(16): 2267–2284.

19.

Faroughian

Kalafatis

Ledden

, et al. (2012) Value and risk in business-to-business e-banking. Industrial Marketing Management 41(1): 68–81.

20.

Farzin

Mamdoohi

Ciari

(2023) Autonomous vehicles acceptance: A perceived risk extension of unified theory of acceptance and use of technology and diffusion of innovation, evidence from Tehran, Iran. International Journal of Human–Computer Interaction 39(13): 2663–2672.

21.

Garg

Al-Garadi

Yang

, et al. (2023) Leveraging large language models and weak supervision for social media data annotation: An evaluation using COVID-19 self-reported vaccination tweets. arXiv preprint arXiv:2309.06503. Available at: https://arxiv.org/abs/2309.06503

22.

Gursoy

Song

(2023) ChatGPT and the hospitality and tourism industry: An overview of current trends and future research directions. Journal of Hospitality Marketing & Management 32(5): 579–592.

23.

Hari

Iyer

Sampat

(2022) Customer brand engagement through Chatbots on bank websites– Examining the antecedents and consequences. International Journal of Human–Computer Interaction 38(13): 1212–1227.

24.

Zhang

Wang

(2019) Impact of social media influencers’ endorsement on application adoption: A trust transfer perspective. Social Behavior and Personality: An International Journal 47(11): 1–12. https://doi.org/10.2224/sbp.8518

25.

Hung

Chen

(2023) The benefits, risks and regulation of using ChatGPT in Chinese academia: A content analysis. Social Science 12(7): 380.

26.

Hwang

Kim

(2021) Toward a chatbot for financial sustainability. Sustainability 13(6): 1–18.

27.

Jeyaraman

Balaji

Jeyaraman

, et al. (2023) Unraveling the ethical enigma: Artificial intelligence in healthcare. Cureus 15(8): e43262.

28.

Jones-Jang

Hart

Feldman

, et al. (2020) Diversifying or reinforcing science communication? Examining the flow of frame contagion across media platforms. Journalism & Mass Communication Quarterly 97(1): 98–117.

29.

Kushwaha

Kumar

Kar

(2021) What impacts customer experience for B2B enterprises on using AI-enabled chatbots? Insights from big data analytics. Industrial Marketing Management 98: 207–221.

30.

Kwangsawad

Jattamart

(2022) Overcoming customer innovation resistance to the sustainable adoption of chatbot services: A community-enterprise perspective in Thailand. Journal of Innovation & Knowledge 7(3): 100211.

31.

Lecheler

Keer

Schuck

ART

, et al. (2015) The effects of repetitive news framing on political opinions over time. Communication Monographs 82(3): 339–358.

32.

Lyu

Huang

Zhang

, et al. (2023) GPT-4V(ision) as a social media analysis engine. arXiv preprint arXiv:2311.07547.

33.

McCombs

(2005) A look at agenda-setting: Past, present, and future. Journalism Studies 6(4): 543–557.

34.

McCombs

Reynolds

(2009) How the news shapes our civic agenda. In: Bryant

Oliver

(eds) Media Effects: Advances in Theory and Research. Lawrence Erlbaum Associates, Inc, pp.1–16.

35.

McCombs

Shaw

(1972) The agenda-setting function of mass media. Public Opinion Quarterly 36: 176–187.

36.

Muralidharan

Dillistone

Henson

(2011) The role of social media in building a better understanding of the 2011. Egyptian Revolution. Journal of Mass Communication & Journalism 1(1): 1–9.

37.

Pan

Kosicki

(2010) Framing analysis: An approach to news discourse. Political Communication 10(1): 55–75.

38.

Pendyala

Hall

(2024) Explaining misinformation detection using large language models. Electronics 13(9): 1673.

39.

Purwiyati

Daroini

Talkah

(2020) The influences of cyber extension and communication media (Facebook, WhatsApp, YouTube) on innovation adoption in Nganjuk regency. Habitat 31(2): 86–94.

40.

Rogers

(ed.) (1962) Diffusion of Innovations, 1st edn. Free Press of Glencoe.

41.

Rogers

(ed.) (2003) Diffusion of Innovations, 5th edn. Free Press.

42.

Russell Neuman

Guggenheim

Mo Jang

, et al. (2014) The dynamics of public attention: Agenda-setting theory meets big data. Journal of Communication 64: 193–214.

43.

Scheufele

(1999) Framing as a theory of media effects. Journal of Communication 49(1): 103–122.

44.

Scheufele

Tewksbury

(2007) Framing, agenda setting, and priming: The evolution of three media effects models. Journal of Communication 57: 9–20.

45.

Severin

Tankard

(eds) (2001) Communication Theories: Origins, Methods, and Uses in the Mass Media, 5th edn. Longman.

46.

Valkenburg

Peter

Walther

(2016) Media effects: Theory and research. Annual Review of Psychology 67: 315–338.

47.

Vaswani

Shazeer

Parmar

, et al. (2017) Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008. https://doi.org/10.48550/arXiv.1706.03762

48.

Venkatesh

Morris

Davis

, et al. (2003) User acceptance of information technology: Toward a unified view. MIS Quarterly 27: 425–478.

49.

Weaver

(1977) Political issues and voter need for orientation. In: McCombs

Shaw

Weaver

(eds) The Emergence of American Public Issues: The Agenda-Setting Function of The Press. West Publishing Company, pp.107–119.

50.

Yang

Wendorf Muhamad

Yang

(2019) Exploring environmental health on Weibo: A textual analysis of framing haze-related stories on Chinese social media. International Journal of Environmental Research and Public Health 16(13): 2374.

51.

Yang

(2023) Research on the legal regulation of generative artificial intelligence: Take ChatGPT as an example. SHS Web of Conference, 178, Article 01017, https://doi.org/10.1051/shsconf/202317802017

52.

Yan

Feng

, et al. (2023) Research on the impact of trends related to ChatGPT. Procedia Computer Science 221: 1284–1291.

53.

Zhu

Chen

Zheng

(2020) Modelling the acceptance of fully autonomous vehicles: A media-based perception and adoption model. Transportation Research Part F Traffic Psychology and Behaviour 73: 80–91.