ChatGPT Exhibits Bias Toward Developed Countries Over Developing Ones,as Indicated by a Sentiment Analysis Approach

Large language models (LLMs) have transformed the landscape of Natural Language Processing (NLP). The capability of LLMs like ChatGPT in producing high-quality text closely mimicking human language is well documented (Adeshola & Adepoju, 2023; Chukwuere, 2024). Nevertheless, LLMs inadvertently mirror and perpetuate biases present in their training data (Caliskan et al., 2017). While content filtering techniques have been employed to mitigate harmful outputs (Markov et al., 2023), biases can persist within the model itself (Ray, 2023). Deploying biased models in real-world applications can have detrimental consequences, as demonstrated by incidents such as those involving artificial intelligence (AI) healthcare predictions (Obermeyer et al., 2019).

LLMs demonstrate various biases associated with gender (Gross, 2023), language (Georgiou, 2024), religion (Abid et al., 2021), politics (Rozado, 2023), and nationality (Venkit et al., 2023) among others. Zhu et al. (2024) investigated the nationality bias of ChatGPT using a sample of 195 countries, with descriptions provided in both English and Chinese. The authors evaluated the output using vocabulary richness, sentiment, and offensiveness metrics. Evaluations of language have also been conducted by humans and ChatGPT. The findings indicated that although the generated content was largely positive, ChatGPT produced negative content when given prompts with negative connotations. Although the model viewed its output as neutral, it consistently demonstrated self-awareness of nationality bias when evaluated using the same pairwise comparison annotation method employed by human annotators.

The examination of country bias in AI-generated language has received minimal scientific attention. In a relevant study, Boussidan et al. (2024) explored the biases of ChatGPT regarding various countries around the globe. The authors followed a sentiment analysis approach, prompting the model to assign a positivity score to each country. Prompts were provided in four different languages, namely, French, English, Russian, and Arabic. The findings revealed that North American and European countries received higher scores, whereas African countries received the lowest. South American and Asian countries typically fell in the middle range. The results also denoted variations across languages. When prompted in French, African countries, particularly those colonized by France, tended to receive more negative scores. In contrast, when prompted in English, the model assigned positive scores to Commonwealth nations like India and Australia. However, the sentiment analysis in the above study relied on the scores assigned to these countries by ChatGPT, using a scale developed by the authors. The conclusion drawn is that ChatGPT is biased toward specific countries. For instance, Salinas et al. (2023) reported that when prompted to select 20 nationalities, the model chose exclusively from Western countries, omitting African nations.

This study aims to fill a research gap by investigating the sentiments found in ChatGPT-generated language regarding developed versus developing countries. We uniquely employ a sentiment analysis framework, which indicates the sentiment scores of the generated texts about the countries under investigation. These scores were derived from an embedded online dictionary, which assessed the positivity of each word in the generated language. The comparison between developed and developing countries is performed through statistical modeling. Since the LLM is sensitive to biases based on previous research, we hypothesize that developed countries will exhibit higher sentiment scores than developing countries. By analyzing sentiment scores in AI-generated texts, the study aims to reveal potential biases, promoting fairer and more accurate representations. This effort supports ethical AI development, enhances trustworthiness in AI systems, and ensures informed decision-making.

Methodology

Results

The sentiment analysis indicated positive average sentiment scores (i.e., >0) for all countries under investigation. However, according to the descriptive statistics, the language related to the high HDI group had more positive sentiments compared to the low HDI group. Figure 1 shows the sentiment scores for both high and low HDI countries together with their SDs. The scores ranged between −0.29 and 0.57 for high HDI and −0.5 and 0.5 for low HDI. Figure 2 illustrates the sentiment scores and the SDs for each country with the high and low HDI.

OPEN IN VIEWER

Figure 1.

Sentiment scores for countries with high and low HDI.

OPEN IN VIEWER

Figure 2.

Sentiment scores for each country with high and low HDI.

We utilized a Bayesian regression model to assess whether ChatGPT exhibited sentiment differences between countries with high HDI and those with low HDI. According to the analysis, the credible interval (CI) for high HDI suggests that there is a 95% probability that the true value of the sentiment score lies between 0.13 and 0.20. As the values do not cross zero, there is strong evidence that the true value of high HDI was greater than zero, indicating in this case positive sentiments for these countries. Similarly, the CI for low HDI indicates a 95% probability that the true value of the parameter lies between 0.04 and 0.10. This provides strong evidence that the low HDI is significantly greater than zero, implying that these countries are associated with positive sentiments. Subsequent hypothesis testing exhibited strong evidence (ER = 3,900, PP = 1.00) that the high HDI countries exhibited higher sentiment scores than the low HDI countries. Table 1 shows the results of the Bayesian analysis and hypothesis testing.

OPEN IN VIEWER

Table 1.

Results of the Bayesian Analysis and Hypothesis Testing.

Main analysis
	Estimate	Est. error	l—95% CI	u—95% CI	Rhat	Bulk ESS
SD (intercept)	0.02	0.01	0.00	0.04	1.00	2286
HDIhigh	0.16	0.02	0.13	0.20	1.00	5268
HDIlow	0.07	0.02	0.04	0.10	1.00	5056
Sigma	0.16	0.01	0.14	0.17	1.00	5652

Hypothesis testing
Hypothesis	Estimate	Est. error	l—95% CI	u—95% CI	ER	PP
HDIhigh > HDIlow	0.10	0.02	0.06	0.13	3999	1.00

Note. Est. error = estimated error; CI = credible interval; l = lower limit; u = upper limit; Bulk ESS = Bulk Effective Sample Size; ER = evidence ratio; PP = Posterior Probabilities; HDI = Human Development Index.

Discussion

The study examined the language used by ChatGPT to describe developed and developing countries by employing a sentiment analysis framework. We utilized prompts to direct ChatGPT in generating discourses pertaining to selected developed and developing countries; these countries were divided based on their HDI scores. We subsequently elicited the sentiment scores for these texts using sentiment analysis in R. Comparisons between high HDI and low HDI countries were conducted using a Bayesian regression model.

The results demonstrated positive sentiments on average for both high and low HDI countries and each of the 20 countries added to the analysis. This is consistent with the findings of Zhu et al. (2024), who reported the generation of positive content by ChatGPT for various nationalities around the world. Thus, the model avoids using negative language for the description of these countries. However, the Bayesian regression analysis confirmed our initial hypothesis, since the language used by ChatGPT for the description of each country encompassed more positive sentiments for countries with high HDI than countries with low HDI. The former group included mostly European nations, while the latter group mainly included African countries. These results corroborate earlier findings. For instance, Boussidan et al. (2024) found that ChatGPT attributed higher positivity ratings to North American and European countries, while African countries received lower ratings.

Overall, ChatGPT presents with biases by distinguishing between developed and developing countries as seen in the sentiment analysis. This can significantly amplify racial and ethnic biases and stereotypes (Choudhary, 2024). By consistently using more positive language to describe developed countries and less positive language to describe developing ones, ChatGPT may perpetuate perceptions of superiority or inferiority based on national economic status. This could lead to the reinforcement of existing inequalities between more developed and less developed countries, influencing societal attitudes and potentially impacting policy decisions and resource allocation.

There is further evidence in the literature for social injustice stemming from AI algorithms. For example, the Optum algorithm in the health system of the U.S. labeled Black patients as less ill compared to White patients, even in cases where people from both racial groups were equally ill (Obermeyer et al., 2019). Such inequities be caused by algorithmic structure bias, insufficient data collected from particular groups, the use of algorithms that reflect or replicate biased human decision-making, and the reluctance of powerful institutions holding large datasets to reduce inequities, among others (Moore, 2022). Another source of inequities may be the lack of diversity among the computer scientists involved in developing AI systems (Hussain et al., 2015). More specifically, individuals from low socioeconomic status minority groups are significantly underrepresented in the high-tech workforce. A more diverse workforce could enhance AI products, as individuals from various social groups may better recognize how algorithms may or may not accurately represent their own communities. In addition, computer scientists and engineers often lack training in comprehending the various aspects of human data and their connections to systems of inequality (Joyce et al., 2018). Such training would be beneficial by fostering a more holistic approach to technology development, ensuring that algorithms and systems are designed with an awareness of societal impacts, biases, and disparities. This, in turn, can lead to more ethical and equitable outcomes, as well as technologies that better serve diverse populations. It is worth noting that the responsibility for the behavior or output of an AI system should not solely rest with the programmers who create it. Instead, society at large plays a crucial role in shaping the data and context from which the AI learns. This advocates for a broader understanding of accountability in AI development, highlighting the need for societal responsibility in shaping the data that informs AI systems to ensure they operate ethically (Ferrara, 2024).

The results of this study are relevant to the intersection of linguistics and psychology because they demonstrate how LLMs, such as ChatGPT, can reflect and potentially reinforce biases in the way countries are described based on their developmental status. From a linguistics perspective, the study sheds light on the language choices and sentiment patterns embedded in the model's descriptions, which may align with global power dynamics or stereotypes about developed and developing nations. This relates to how language can shape perceptions and reinforce societal hierarchies. This process involved tokenizing sentences into individual words and assigning sentiment scores using predefined dictionaries. Each word was evaluated based on its polarity (positive or negative) and intensity (the strength of the sentiment within those categories), providing a detailed sentiment analysis. From a psychological perspective, the findings touch on how exposure to biased language through AI can influence people's attitudes, beliefs, and perceptions about countries and their populations. This suggests that models like ChatGPT may unintentionally contribute to shaping biases or implicit attitudes through the way they frame information, which is a core concern in psychology. The present study highlights the need to address biases not only in the technical development of models but also in how they impact psychological constructs like perception, stereotyping, and social judgment.

Conclusions

A significant differentiation between developed and developing countries was observed in the language of ChatGPT on the basis of a sentiment analysis. These tentative findings could be important for AI developers who may consider adjusting the algorithm accordingly to reduce socially biased language in LLMs like ChatGPT. Future research can include a larger pool of countries and use additional metrics to investigate the language of the model. Furthermore, future work can examine the sociocultural impacts of biased AI-generated content on global perceptions and interactions.