Abstract
Obtaining fact-checking skills is becoming crucial as information disorder turns into a major issue for societies. The aim of this study is to evaluate the extent to which students benefited from the fact-checking module within the news literacy (NL) training and what kind of fact-checking behavior they exhibited. The research was conducted with two different groups of university students for two years using a quasi-experimental design method. Following the training, the students conducted their fact-checking process by making more queries, mainly using news titles, and mostly clicking on news platforms. Although many of the students did not show high performance after the training, it was seen that training created a certain awareness about the verification platforms/tools. The study highlighted the importance of information literacy skills in NL training provided at the university level and the characteristics that distinguish information behaviors during fact-checking from those in other information search processes.
Keywords
Introduction
Today, humanity faces the problem of dis/misinformation in politics, science, health, and many other fields. In this “post-truth” era, the fact that people evaluate and share information that is suitable for their own feelings, thoughts and beliefs as the “truth” has been one of the important problems that contribute to the spread of dis/misinformation (Cooke, 2018). Fake news, which plays a major role in the popularization of this concept, can deceive many people with its fast-spreading feature. A healthy skeptical approach has become more crucial than ever. However, the current information environment has made it necessary for individuals to acquire some new knowledge and skills in this regard. In this context, one of the issues that have been frequently highlighted recently is news literacy (NL) (Brown-Hulme, 2018; Vraga and Tully, 2021; Yorio, 2020). It is emphasized that evaluating the accuracy of news and information as a student, consumer or citizen is an important skill in today's world because of the effects of fake news on individuals and societies (Freeman, 2015).
On the other hand, the developing technology, the changing information environment, and being constantly exposed to intense amount of information during the day have left people facing information overload (Bawden et al., 1999). Algorithms present the content suitable for user profiles in a ready-made way and cause the emergence of an information environment in which the content flows continuously and rapidly on screens (Liu, 2020; Menczer and Hills, 2020). Thus, even if individuals do not actively seek it, they encounter a great deal of information, especially through social media.
The research area, defined as “information behavior” is studied to improve both systems and user skills in the process of individuals’ accessing information effectively and correctly. It is also naturally affected by this changing information environment (Bates, 2018; Case, 2007; Vakkari, 2008). While the studies in this field were more system-oriented in the first place, the user-centered approach has been adopted since the 1980s (Wilson, 2000). Information behavior is an important field of research that has existed in the information science literature for many years. With these studies, individuals’ information behaviors have been revealed according to occupational groups and disciplines (academicians, journalists, lawyers, etc.), roles (consumers, patients, students, etc.), demographic characteristics (age, gender, etc.), and the use of information sources. In addition, some theories and models (Dervin's Sense-Making Theory, Kuhlthau's Information Search Process Model, etc.) that help better understand information behaviors have been developed (Case, 2006, 2007; Fisher and Julien, 2009). Thanks to all these studies, it is possible to understand the relationships of individuals with information and to develop more effective information systems and services (Wilson, 2022). Therefore, studies to understand the information behaviors of the users help to improve both the systems and the user experience.
The change in information behaviors in daily life has become remarkable, especially with the advent of Web 2.0. On the other hand, everyday life information seeking (Savolainen, 1995), which refers to the acquisition of cognitive and expression-based elements that individuals use to improve themselves in daily life or to solve problems that are not directly related to their career, maintains its importance for people. Problems that individuals may encounter in daily life may be related to issues such as consumer habits, crime and safety, education, employment, transportation, health, recreation, discrimination, financial issues and legal problems (Savolainen, 2015; Warner et al., 1973). These issues also constitute the subject of the news that individuals obtain from different environments in daily life. Understanding the behaviors of individuals as news consumers in this new information environment where their relations with information have changed is also important in the fight against fake news and the phenomenon expressed as “information disorders” (Wardle, 2020).
There have been debates about the need to abandon the use of the term “fake news” (The Digital, Culture, 2018: 8; Christopoulou, 2018: 5–6; Wardle and Derakhsan, 2017: 5). The main reason behind the controversial nature of this concept is that it has started to lose its meaning all over the world in recent years due to political polarization and polemics. It has been argued that the term “fake news” should be avoided because it fails to define the complex nature of information pollution and has been used by politicians to describe news organizations that they find unpleasant and contrary to their views (Wardle and Derakhsan, 2017: 5). Instead, it is preferred to use the term “information disorders”, which refers to different types of information that can be spread intentionally or unintentionally, focuses on whether there is intent to harm others, and encompasses the concepts of disinformation, misinformation, and malinformation (Wardle, 2020).
Being “literate” in the fight against dis/misinformation
When it comes to choosing reliable information among the constant flow or evaluating the appropriate resources for the information need, it is important to have information literacy (IL) skills. IL is defined as the ability to understand when information is needed and to obtain it, to think critically about an information that is accessed and used, to make balanced decisions, and to use information effectively (American Library Association, 1989, 2000; Information Literacy Group, 2018). In other words, IL plays a key role in learning and implementing information seeking and usage skills (Limberg and Sundin, 2006). Due to these characteristics, IL is also closely related to the topic of information behavior.
Individuals with IL skills, which has become increasingly important due to developing technology and emerging of different information sources (American Library Association, 2000), are expected to be able to create appropriate search strategies to find the information accurately. Creating a search strategy consists of several steps such as determining the most appropriate keywords, using Boole operators and truncation appropriately for research (Landøy et al., 2020). It is possible to say that these steps are especially useful for academic databases. However, in Google, searches are mostly conducted in natural language (Kutz, 2022). Therefore, individuals generally do not use the search strategies that they are expected to create within the scope of IL skills in daily life. At this point, questions such as what kind of process individuals follow and how they decide on keywords come to mind in daily life where they are exposed to intense amount of information, for example, during fact-checking. Hence, to investigate different information-related elements in the fact-checking process, it is important to create various search queries and cross-checks from various sources (Silverman and Tsubaki, n.d.).
Reasons such as the widespread use of social media platforms, the activation of user-generated content, and the inadequate implementation of journalistic practices have made intentional or unintentional types of information disorders more visible. Accordingly, to reach correct information, it is no longer sufficient to evaluate only the accuracy and reliability of information sources, but it is necessary to verify the information items in different sources and environments. Besides IL skills, individuals need to learn the types of dis/misinformation, in what ways and where they can encounter them, not to believe in every content and information they encounter in different media. NL aims to provide individuals with knowledge, awareness and skills on these issues (Center for News Literacy, 2016a). One of the important NL skills is to investigate and verify the accuracy of a suspicious content in the sources used to get news in both daily and business life. In such cases, it is important to have some basic fact-checking skills.
Elements that can be used to verify any information item such as image, video and text perceived as news, or suspicious content we encounter as user-generated are listed as follows; provenance to verify the authenticity of the content, source to verify the uploader of the content, date to verify the event or content date, and location to verify the scene or content location (Silverman and Tsubaki, n.d.). These elements can guide individuals on where to focus and where to start questioning to fact-check suspicious news. Since the nature of each news is different, fact-checking process may also vary. However, it is extremely important not to adhere to a single element or source in fact-checking process and this applies to every news. It is necessary to question different sources and the information they provide and to cross-check with reliable sources (Buttry, 2014). Research on various elements can be done using verification tools like Google Images, Yandex Images, TinEye, Google Translate, and Google Maps (Silverman, n.d.).
Several initiatives are made to design and deliver NL training to help people acquire these abilities. One of the first initiatives in this direction was the founding of the Center for News Literacy at Stony Brook University's Journalism School in 2007 and the introduction of NL courses (Center for News Literacy, 2016b). Among others, News Literacy Project Checkology (2022), Poynter News University (2020), First Draft (2022), and Google News Initiative (n.d.) offer a variety of courses and reference materials. Although most of the efforts are student-oriented, contents are available to everyone. Creating training contents suitable for different age groups, especially starting from kindergarten to university level, and adapting them to the curricula are essential in terms of integrating NL skills into individuals’ daily lives. However, designing and delivering an efficient NL training is not an easy process. Because it is quite difficult to teach critical thinking skills that form the basis of both IL and NL trainings (Willingham, 2007, 2019). It has been stated in the literature that it is difficult to enable people to make these evaluations not only by providing a list of criteria for evaluating information, but also by internalizing the critical perspective (Julien, 2016).
Literature review
As the dis/misinformation problem becomes more apparent, it has become important to acquire and use fact-checking skills, which is an NL competency, among what can be done individually. Since the concept of fact-checking has entered our daily life relatively recently, it is possible to say that there are not many studies on the fact-checking behaviors exhibited in this process. Rather, there are some studies on the ways individuals perceive and evaluate the news. The results of these studies summarized below generally contribute to answering the question of what kind of NL training content should be created.
In a two-step study on Indian and US users (Sharma and Kapoor, 2022), information sharing and verification behaviors related to COVID-19 on social media were evaluated. For this purpose, it was examined how the messages with positive and negative polarity and the message type, which can be defined as news or rumors, affect the spread of information and verification behavior related to COVID-19. As a result, it was seen that messages with positive polarity encouraged Indian users to share, messages with negative polarity increased verification behavior. In contrast, US users took the opposite approach. Messages with negative polarity led these users to share more and messages with positive polarity led to higher verification behavior. Also, importance of the message for both Indian and US users mediates the relationship between message type and message sharing behavior.
The process followed by journalism students to collect online news was investigated in a different study (Tylor, 2015), in order to understand their judgments on the reliability of the content. In this context, the keystrokes of the students in the news collection process were examined, and then semi-structured interviews were conducted. The results showed that journalism students had a certain awareness of the importance of verification. In addition, most students remained on the first page of search engine results, and many of them even clicked on the first result. It was understood that students had a deep trust in search engines (especially Google) and relied on the credibility of search engine results.
Another study (Leeder, 2019) emphasized the importance of examining the pages in the search results by investigating the relationship between the information behaviors of university students and their ability to correctly evaluate the credibility and reliability of the news on the internet and their desire to share them on social media. Each of the 63 undergraduate students evaluated 12 stories with both real and fake content with the questionnaire presented. Students performed slightly better in detecting fake news compared to detecting true news stories. It has also been determined that there is a relationship between the use of certain critical evaluation behaviors such as examining the rest of the web page and devoting more time to evaluation to assess the reliability of the information and the correct identification of fake and real news. According to the results, information behaviors for effective evaluation of online news stories were determined as examining the rest of the web page to evaluate the reliability of the information, using more than one source to answer a question, and spending sufficient time to evaluate the online news found. The results also reveal the importance of verification strategies such as searching for other sources for fact-checking, using fact-checking platforms like Snopes to find information about a news content.
University students from different faculties were analyzed in a study (Powers and Koliska, 2016) in terms of how opinion leaders, including friends, family, teachers, and the social environment, shape their perceptions of finding news, assessing the reliability of news sources, and forming an opinion about the news media. The findings revealed that students relied on long-standing resources at the heart of the American news ecosystem, such as newspapers. In addition, it was determined that the students perceived the national magazines and television channels that they were familiar with from their homes as credible sources of news. It was also understood that this familiarity guided the opinions of the students. It is interesting to observe from the findings that students tend to rely on “popular” news sources. As a result, it was once again stated that one of the roles of NL training is to teach students to not trust blindly even the news coming from the prestigious journalists and to have the ability to think critically and distinguish disinformation as a news consumer.
In another study (Evanson and Sponsel, 2019), which highlighted the trust in sources accepted as authorities, a questionnaire on news consumption habits was applied to 511 first-year college students to determine how they relate to digital news. Students were asked to evaluate the news stories. The findings of the study revealed that the story with the fake URL was the one that students said they would share the easiest. However, most of those who reported that they would share the story also stated that they were not entirely sure about the accuracy of it. Students often quoted NBC, one of the well-established broadcasting organizations in the USA, as an indicator of reliability, and this showed that students valued authority. Although most of the students did not analyze the inconsistency between the title and the content of the story, it was also observed that they tended to focus on the textual clues in the main body. The study stated the importance of a critical approach to how authority is built, how malicious powers can exploit authority and educational content should be developed to emphasize them.
The findings of another study (Herrero-Diz et al., 2019) showed that news credibility can differ according to gender and graduation program. The results of a total of 188 university students studying communication and education who evaluated online news using the CRAAP (Currency, Relevance, Authority, Accuracy and Purpose) test were analyzed to reach this conclusion. Although both departments offer content that aims to strengthen students’ media literacy in their programs, it is understood that communication students make more accurate evaluations than education students. The study revealed that students generally had difficulty in distinguishing the accuracy of sources and that fake news received higher scores than the news based on facts.
Another study (Nee, 2019) explores the use of social media for news by teenagers and young adults in the Middle East and the United States and their actions to verify online news stories. Data collected through surveys and interviews indicated that teenagers tend to rely on more visual platforms, such as Instagram and private messaging apps, for news consumption. Findings also showed that young adults who use Facebook and Twitter are more likely to engage in fact-checking behaviors. The focus group interviews revealed that the most common fact-checking activities among teenagers include Googling the title of a news story and reading the comments on a post to assess the feedback received.
A study conducted with high school students in Australia (Johnston, 2020) showed that when it comes to the news presented in social media posts, students trust television news and online newspapers rather than social media. They were often able to distinguish legitimate news content from opinion but failed to notice the possible bias if the source of news was, for example, a political institution. Students also failed to evaluate accuracy or authority apart from social media posts, continued to rely on “evidence” even if it was false, often did not recognize the blue tick on social media accounts, and often did not realize that photos and videos could have been edited. The study concluded that training should draw attention to the social and political biases that may be at the source of information and emphasize the importance of verifying evidence from more than one source, that is, thinking more critically.
Another group of studies focused on improving university students’ fact-checking strategies with lateral reading (Brodsky, 2022; Brodsky et al., 2021a, 2021b). In one of these, the effect of a lateral reading curriculum on university students’ fact-checking skills was examined by conducting pre and post-tests. Students who followed the curriculum were more likely to correctly evaluate the reliability of lateral reading and online content compared to the control group (Brodsky, 2022). In the other study, a total of 221 college students in an online civics course were taught how to use lateral reading strategies to verify online information. It was found that students’ use of lateral reading to check information improved from pre-test to post-test (Brodsky et al., 2021a). In a further study, 136 students were exposed to the Digital Polarization Initiative (DPI) curriculum, while 94 of them received regular civics courses. Meanwhile, DPI is defined as a national effort to teach lateral reading strategies used by expert fact-checkers to verify online information. At post-test, students in the DPI sections were found to be more likely to use lateral reading to fact-check and accurately assess the credibility of information compared to the other group (Brodsky et al., 2021b).
In the literature, especially in studies conducted with university students (Herrero-Diz et al., 2019; Leeder, 2019; Nee, 2019; Tylor, 2015), it is seen that the credibility of online news and the verification behaviors exhibited in the processes of evaluating these news are emphasized. In another group of studies conducted with university students, the use of lateral reading in the development of fact-checking strategies is discussed (Brodsky, 2022; Brodsky et al., 2021a, 2021b).
This study investigates the fact-checking behaviors of individuals, which emerged as a necessity following this new information ecosystem and can be considered as a type of information behavior. Within the scope of a research that constitutes the basis of this study, a training program was implemented to provide NL skills to university students (Şencan, 2020). Teaching fact-checking steps was also a part of this training. This study aimed to evaluate to what extent the students benefited from the module of this training related to fact-checking and whether they could reflect their learning to the fact-checking process. For this purpose, whether these students could investigate the fact-checking elements as expected, and the information behaviors they exhibited in the meantime were examined by seeking the answers to the following research questions (RQ):
The answers will give an idea about the ways individuals follow during fact-checking and the points that distinguish the fact-checking process from other information seeking processes. It is thought that the study can produce results that can help to reveal the effect of changing information environments on individuals’ information behaviors from a different perspective. Thus conducting more effective awareness studies and creating more efficient NL training contents could be possible.
Research design
This study was carried out with part of the data collected during the quasi-experimental evaluation of an NL training program given to university students. The training in question was applied to Hacettepe University Department of Information Management first-year students in two consecutive years (2018: NL1 and 2019: NL2). Experimental group consists of the students who wanted to participate in the research voluntarily (NL1 Experimental group: 20, NL2 Experimental group: 16 students) and the rest of them formed the control group (NL1 Control group: 12, NL2 Control group: 22 students). In this study, the data obtained for the “fact-checking” module of the training were used and only students in the group receiving training were evaluated regarding their fact-checking behaviors. Therefore, the term “NL1/NL2 students” mentioned in the rest of the study refers to the experimental group students in two different years.
In both years, a pre-test which contains verifying two news samples was applied to the students to understand the pre-training fact-checking skill levels. Then, within the scope of the training, some theoretical and practical information regarding fact-checking procedures, resources, and platforms was provided to NL1 and NL2 students. Based on the findings and feedback from the students obtained from the NL1 training, the number of practices and course duration in the NL2 training content were increased. Accordingly, the training hours were increased from 3 h in NL1 to 6 h in NL2.
After the training, a post-test which contains verifying two news was applied to both student groups. The aim was to examine the fact-checking behaviors of the students, to evaluate whether there has been any progress in fact-checking skills after the training, and to determine the aspects of the training that need improvement. Due to the changes in the training applied to NL2, the results of the two trainings were evaluated separately. In addition, NL1 students received IL training before NL training and NL2 students received IL training as part of the undergraduate program in parallel with NL training. This situation also offered the opportunity to observe the effects of IL training on NL training.
Data collection and analysis
Pre- and post-tests used in the evaluation of training were administered to the students online. To evaluate the students’ behaviors in the fact-checking process, their screens were recorded with Camtasia software during searches, and then these recordings were watched to reveal the steps that the students followed for verification. Thus, a data set was constructed in which elements providing insight into the students’ information behaviors during the fact-checking process, such as the frequency of search terms, their content, and the types of websites visited, were systematically categorized.
The search terms expected to be used by the students were defined and scored before the tests to be applied, with reference to the previously mentioned verification elements which are provenance, source, date and location (Wardle, n.d.; Silvermen and Tsubaki, n.d.) (see Table 1). Thus, it was possible to evaluate students’ search performances for fact-checking elements and to compare the performances of two separate groups. Accordingly, it is sufficient to examine the “provenance” and “source” elements depending on the content of the news samples given in the tests. Only one news sample (Pre-test-News1) has specific time information. For this reason, it was expected that the “date” element would also be examined while conducting research on that news sample.
Scoring of fact-checking samples (pre- and post-test).
Since most of the news samples given in the tests and queries made during the fact-checking process are in native language of the students, the expected search terms are not translated into English.
Since the number of students in both groups was less than 30, non-parametric tests were used to compare the performances, regardless of other assumptions. Wilcoxon Signed Rank Test was applied to compare the pre- and post-test within each group. To determine if there were any differences between the groups of the two years, the pre- and post-test scores of NL1 and NL2 students were compared with the Mann-Whitney U Test. SPSS software was used for data analysis.
Results and discussion
Performances regarding fact-checking steps
In order to determine if there was an improvement in fact-checking skills associated with the training, the students’ steps to verify the news samples in the pre- and post-tests were examined and scored as a reference Table 1. In both years, the maximum total score that a student could receive in both the pre- and post-test was 10. Figure 1 presents the distribution of students’ scores.
Distribution of the students’ total scores from the pre- and post-tests.
The status of the student groups in both years before receiving NL training was compared according to their pre-test scores and a statistically significant difference was found between the median scores (NL1 = 6; NL2 = 3.5), (NL1-NL2: U = 57,000; z = −3.317; p = 0.001). From this, it was understood that NL1 and NL2 students differed from each other in terms of their ability to apply fact-checking steps before receiving training for fact-checking, and the difference was in favor of NL1 students. This difference suggests that the IL training that students receive as part of their undergraduate education may be effective in gaining a perspective on fact-checking. In other words, it is thought that NL1 students may have acquired some knowledge and skills regarding information verification and searching thanks to the “Information Literacy” course they completed in the previous semester within the undergraduate program they studied. On the other hand, it is estimated that NL2 students’ skill levels are more limited, since they have just started taking the course in question during the test phase. However, it was found that there was no statistically significant difference between the median pre- and post-test scores of NL1 students (pre-test = 6; post-test = 6.5) (z = −0.395; p = 0.693), while the median pre- and post-test scores of NL2 students (pre-test = 3.5; post-test = 6.5) showed a statistically significant difference (z = −3.216; p = 0.001).
The results show that the NL training did not significantly improve the fact-checking skills of NL1 students. However, these students -probably because of the IL course they have already completed- performed relatively well before NL training. On the other hand, the pre-training performances of NL2 students, who have just started taking the IL course, were statistically significantly lower than those of NL1 students. However, after the training, both groups exhibited similar performances (NL1&NL2 post-test: 6.5) in the post-test (U = 156,000; z = −0.129; p = 0.898). A comparison of the pre- and post-test results within each group suggests that NL2 students may have benefited more from the training. It can be thought that a more remarkable progress in terms of fact-checking skills in NL2 students may have been affected by the fact that these students received the updated training after the improvements in the training content. It is also worth noting that no student who received training was able to get a full score from the post-test.
These findings show that although the training given in the second year is relatively more effective, students cannot fully apply the fact-checking steps. To evaluate which aspects of training can be improved, the fact-checking behaviors of the students for each news sample were examined more closely.
Number of queries created for fact-checking and the search terms used
The number of queries made by the students was examined to evaluate whether they exhibited the tracking behavior as part of the fact-checking process. From Table 2, it is understood that the number of queries made by the students increased after NL training. This situation should be evaluated differently from the information behaviors exhibited when searching for information in daily life. For example, in terms of human-computer interaction and user experience research areas, which can be related to information behavior studies, the user who interacts with any information system for a daily information need or a research is expected to find what he/she is looking for with minimum clicks and in the shortest time for a positive user experience (ISO 9241–210:2019, 2019). Yet, when it comes to fact-checking, the search process may differ. Because, in this process, cross-checking, in other words, referring to different sources and searching from different elements is expected (Silverman and Tsubaki, n.d.). This may require spending more time and making more queries.
The number of unique queries made in pre- and post-tests.
When the screen records were examined, it was noticed that some students did not make any search for the news. Non-searchers are not included in this table.
The number of queries made by NL2 students increased in the post-test compared to the pre-test (see Table 2). This suggests that these students, who have not yet taken IL courses, may have benefited more from NL training than NL1 students. It is important to note that NL2 students may have learned some topics together with NL training, whereas NL1 students already had some knowledge from taking the IL course prior to the NL training. This difference in the post-test can also be considered as the effect of the undergraduate course in question.
Well-selected search terms help to define the existing information need and to achieve more accurate results that will meet it. Based on this, the categories and contents of the search terms used by the students to investigate the accuracy of the news samples in the pre- and post-test were examined (see Table 3).
Search terms used in the fact-checking process.
*Examination of the screen records revealed that some students did not conduct any searches for news. These non-searchers were not included in the data presented in the table.
** Search frequency refers to the total number of searches for the relevant category (for example, a student may have made more than one search in the same category).
Table 3 reveals that students made different attempts regarding the same search category for all news samples. For example, if the news content includes the name of the person, it is seen that different queries are derived in accordance with the news content, such as searching for the name directly, searching for the name of the person with the title, if any, and combining the names of the person and the institution in the same query. Considering that it is important to track different elements and to feed from different sources as much as possible in the fact-checking process, it is also important for students to be able to conduct alternative queries. On the other hand, it was seen that the students did not use the quotation mark at all in their searches during the fact-checking process. Only one student (NL1) searched for the news title in Pre-test-News1 and one student (NL2) searched for the journal title in quotes in Post-test-News1. Although it is important to know how different operators work and to use them appropriately in creating a search strategy for a research (Landøy et al., 2020), it is understood that during the fact-checking process, students generally resort to Google (Kutz, 2022), which is used in natural language, and do not use relatively complex search strategies. Even if we define the queries that students make by combining more than one category as a relatively complex one, it has also been observed that after the training they tend to produce a “less complex”, in other words, a single category query. This tendency may also be an indication that they decide more easily what to search for after they receive training.
For example, when searching information from a database for an academic purpose, it may be particularly important to create detailed search strategies, to know how to use truncation or Boolean operators correctly. However, it is also possible to complete the fact-checking process with simpler searches (see Table 1). Whether or not it requires establishing complex search strategies, the important thing in this process is to be able to decide both the piece of information to be traced and which of the results are more accurate in making a correct judgment about the news. At this point, it was thought that the query contents could give different clues about students’ tracking behaviors.
The search terms used indicate that if the person's name is mentioned in the news content, it also stands out in the queries. For example, in Pre-test-News1 (see Table 3), since the content of the text is based on the recommendations of a physician, it is noteworthy that the name of the physician and the institution to which he/she is affiliated are searched together. Regarding other news samples that contained person names (see Table 3, Pre-test-News2, Post-test-News1), the students tended to search both the news title and the person's name. This became more prominent in both groups. Another remarkable point is that in the news samples in the post-test, students also resorted to verification platforms (see Table 3, Post-test-News1&2). From this perspective, it is possible to say that training creates a certain awareness among students about verification platforms.
Another prominent point in Pre-test-News1 is that, unlike NL1, students in NL2 did not directly search the name of the institution mentioned in the news. To verify whether the person named in the relevant news works in the specified institution, it is necessary to search the person's name and institution. The number of students performing this query was higher in NL1 (68%; NL2: 22%). This suggests that students in NL1 are searching more consciously even though they have not yet received NL training and may have gained these skills from the IL training they have previously received. The majority of NL2 students (see Table 3, Pre-test-News1&2), on the other hand, instead of investigating different elements such as “the name of the organization”, “the name of the person, and the organization” included in the pre-test news samples, searched more in the news title category, which probably attracted their attention first and they thought defined the content of the news.
When the fact-checking queries used by the students in both groups are examined in general, it is also evident in the post-tests that they mostly refer to the news title in searches (see Table 3). For example, in Post-test-News1, the news title was used 39 times by 16 students (80%) in the NL1 and 28 times in total by 11 students (79%) in NL2. However, this news sample has a content that will make it possible to reach a more accurate judgment about the news when the journal title and the name of the person are used together. Instead, the fact that they search the news title suggests that students tend to be hasty in deciding on the piece of information to be traced, and that they find it more practical to track from the news title, which is the “easiest” clue about the news. In other words, it is understood that students do not make a conscious evaluation on the pieces of information that are most suitable to be used for tracking. This evaluation can actually be associated with critical thinking skills, and it can be thought that students have not benefited much from this aspect of the training. As a matter of fact, the literature also mentions the difficulty of providing people with such a critical assessment skill (Julien, 2016).
It is also possible that the training may have triggered the reflex of checking whether the news was published elsewhere, given that the tendency to check news titles increased slightly more, especially in the post-test. However, it would not be wise to begin the fact-checking process by relying solely on the news title, considering that there may be titles that do not accurately reflect the news content, especially online. The results of another study (Leeder, 2019) also revealed that the use of critical evaluation behaviors, such as examining the rest of the web page, using multiple sources to answer a question, and spending enough time evaluating the search results, is associated with the ability to make accurate assessments of news credibility and trustworthiness.
It was determined that neither NL1 nor NL2 students searched all the search term categories that were expected to be examined in the post-tests (see Table 1 and Table 3). According to this perspective, despite receiving NL training, students do not always think analytically about the news. Instead, they frequently start with the most obvious clues they can find (such as the news title), ignoring the requirements of fact-checking steps.
Types of pages visited for fact-checking
In addition to the search terms used by the students in the fact-checking process, distribution of the types of pages they visited was also examined (see Table 4). Accordingly, the students in both groups mostly visited the websites of online news platforms and/or newspapers in both pre- and post-tests. Although it is expected to check how a news encountered in the fact-checking process is handled in different sources (provenance and source research), it is estimated that students did not do this consciously at least during the pre-test phase. In fact, when it is considered that students search the title of the news (Table 3), the probability of reaching the same news content in different sources increases. At this point, it is worth mentioning the news coverage habits of news platforms in Turkey. It should also be considered that such platforms typically publish the news that certain agencies have released with minimal modification and that it may be possible for various news organizations to publish “bespoke” news with similar titles on some critical issues (e.g., political, social issues, etc.). Thus, even if it is to be examined how the news published in different sources during the fact-checking process, it is important to be aware of the possibility that the alternative sources examined may be biased and that malicious powers can manipulate the source or contents accepted as authority (Evanson and Sponsel, 2019). Therefore, it is useful to verify the news from independent media outlets as much as possible in the fact-checking process. The situation of the students, which seems to be far from this kind of sensitivity, suggests that students examined the first web pages they encountered as a result of the first query that came to their mind (such as the news title).
Pages visited during fact-checking process.
*“Institutional websites” are those with extensions of “.org”, “.gov”, or “.edu”. Other sites, with extensions such as “.com” and “.net”, are referred to as “independent websites”.
**Examination of the screen records revealed that some students did not conduct any searches for news. These non-searchers were not included in the data presented in the table.
***The number of students visiting the relevant page type is shown in the “n” column.
The news in the pre- and post-tests are qualified to allow searching from different sources in the fact-checking process, and students are expected to visit different sources such as institutional websites, fact-checking platforms, scientific publications. For example, Pre-test-News1, as mentioned before, requires the physician's name whose recommendations are included in the news to be searched together with the alleged institution he is affiliated with. In contrast, for this news, students mostly visited online news platforms (NL1; 84%; NL2: 78%) and blogs (NL1: 58%; NL2: 67%) (see Table 4). On the other hand, it is understood that the tendency to look at the website of the relevant institution to check whether the mentioned physician works in the institution specified in the news is relatively lower (NL1: 42%; NL2: 11%). There were 3 students (16%) in the NL1 who had previously received IL training, who also benefited from visual verification tools and platforms. These students searched for the name of the physician, switched to Google Images from the results and proceeded through the results here.
Although online news platforms (NL1: 85%; NL2: 86%) were visited in the first place by most of the students in the Post-test-News1, which is expected to verify a news story about health, it was observed that some students turned to scientific resources (NL1: 50%; NL2: 36%) and the institutional websites (NL1: 55%; NL2: 36%) expected to be consulted in this query. On the other hand, in Post-test-News1, it is noteworthy that some students (NL1: 25%; NL2: 1 student, 7%) also have Google Translate, which is in the dictionary category, among the sites they visit. Since the scientific journal mentioned in the news content and the articles in this journal are in English, students with different native languages need to better comprehend the content for verification. It is seen that students who struggle with language proficiency need to consult the dictionary for this reason. It can be concluded from this case that inadequacies in language skills are also reflected in verification behavior.
Pre- and Post-test-News2, on the other hand, especially highlights NL1 students in terms of the pages they choose to visit. The pages that these students visited in both tests were found to be more accurate sources to visit during the fact-checking process (see Table 4). In Pre-test-News2, a significant number of NL1 students (68%) visited institutional websites, whereas NL2 students mostly preferred online news platforms (73%). In Post-test-News2, while almost all the NL1 students (95%) preferred image verification tools more to verify the news, it was observed that the majority of NL2 students preferred to visit online news platforms (77%) and social media (69%), and then image verification tools (38%). It is also noteworthy that social media platforms are also dominant (69%) among the types of pages that NL2 students refer for this news. From the screen recordings obtained, it is understood that the students accessed the social media accounts where the image in the news or a similar one was shared and examined them. In addition to presenting the theoretical foundations in the training content given to the students, various fact-checking practices were applied on the computer. In particular, the findings obtained from Post-test-News2 give clues that students can use the information they have learned by experimenting more effectively. On the other hand, the fact that NL1 students stand out more than NL2 students in both the pre- and post-test in this sense can also be considered as the effect of the IL course they have taken before. The NL1 and NL2 students’ most visited web page types for Post-test-News2 reveal that NL2 students struggle to implement the information they obtained from the fact-checking training on image verification.
Conclusion
Especially with Web 2.0, it has become possible for users to produce their own content and share them in different environments, notably on social media. However, political propaganda activities; people basing their decisions on their own feelings, opinions, and beliefs instead of the facts; filtered content offered by platforms; and the algorithms they use have all contributed to the rapid dissemination of dis/misinformation.
Before this new information ecosystem was formed by user interaction, IL played a significant role in providing skills such as recognizing the need for information, effectively conducting information search processes, and appropriately evaluating and using the received information. Today, having IL skills is not sufficient on its own, although it remains important. Even if individuals do not always actively seek information in their daily or professional lives, they find a large amount of information readily (Liu, 2020; Menczer and Hills, 2020). Accordingly, critical assessment of exposed or accessed information has become more important than ever today. This is because it is often impossible to judge the content by evaluating it merely according to a limited set of criteria when this information is “news”. In addition to the news source, it is needed to trace bits of information and perform multidimensional fact-checking, including who, when, and where the content was created and/or shared. When doing this, it is crucial to develop various search queries and cross-check them from different sources (Silverman and Tsubaki, n.d.).
NL plays an important role in combating dis/misinformation and gaining critical thinking skills (Brown-Hulme, 2018; Vraga and Tully, 2021; Yorio, 2020; Center for News Literacy, 2016a). Fact-checking, which helps individuals track content and access accurate information, is one of the most important skills gained through NL training. The fact-checking process also requires the individuals to be suspicious about the information they encounter and to have the motivation to take action to verify it (Tully, 2021). However, this need does not emerge easily. Although tracking and searching skills are required for fact-checking, it should not be ignored that people may not always be motivated to verify the accessed content to meet a daily information need. Furthermore, the difficulty of gaining critical thinking skills for individuals (Willingham, 2007, 2019) also exists in developing NL training, which mainly focuses on this skill.
Considering all of these, this study aimed to evaluate the effect of a developed NL training on university students in terms of acquiring fact-checking skills. For this purpose, students were asked to verify a total of four news in the pre- and post-tests applied before and after the training and their transactions were recorded.
The findings of the study showed that the behaviors exhibited in the fact-checking process should be associated with the subject of “information behavior” and therefore with IL. In other words, the fact-checking process requires having IL skills as well as NL skills. Because making the right judgments requires keeping track of fact-checking elements, having effective search skills, and evaluating the information obtained. For example, to verify whether the person named in the Pre-test-News1 works in the institution specified in the news, it is necessary to make queries about the person's name and institution. The fact that more NL1 students were able to do this suggested that they searched more consciously even though they had not yet received NL training and gained these skills from the IL training they had previously received.
In the fact-checking process, we encounter a different scenario than searching for daily or scientific information. While an effective information search process aims to reach the resources that will meet the exact information need by making the fastest and most effortless search, the fact-checking process may require more time for repetitive searches. In addition, it is very important and sometimes complicated to create the most effective search strategies, especially in scientific information search processes. Daily information needs are met by Google-like search engines and usually by using natural language (Kutz, 2022). In the fact-checking process, it is necessary to make the right decision on the piece of information to be traced to reach a correct judgment about the news. This affects many variables, from the type of search to be performed to the resources to be consulted. The findings revealed that despite a news sample containing a reference to a publication that can be traced for verification, students preferred the “convenience” of conducting Google searches for the news title and person name. Similarly, the easiest piece of information to trace for almost all news samples was seen as the news title. This requires the questioning of the above-mentioned “motivation”. The questions of whether these students conducted these searches in the most practical way possible because they were in a testing environment and it was expected of them or whether their information behavior was a result of their generation's “practical” approach could not be answered within the scope of this study. However, it is thought that a more in-depth examination of these questions with different research may yield interesting results. Hence, Williams and Rowlands (2007), in their review of different studies on the characteristics of the Google Generation, reveal that young people in this generation have difficulty in developing effective search strategies as they do not understand their information needs sufficiently. Accordingly, it is understood that they prefer to express their information needs in natural language instead of evaluating which keywords can be more effective.
Students were seen to exhibit more accurate verification behaviors when news content had more distinct starting points. For example, for news content that requires visual search, most students who have received IL training as well as NL in particular have managed to use a visual verification tool, as expected. In contrast, for instance, in a case of news given health advice, the number of students who prefer the journal in which the research was published and the name of the researcher as their starting point is relatively low. When there are news samples that they need to decide which piece of information to search, it is seen that students prefer to search the news title, which is the most practical way. This situation suggested that the students were not at a sufficient level in terms of critical assessment skills despite the training they received. In fact, this result raises the difficulty of providing individuals with critical thinking skills. It is understood that more effort should be put into how to permanently teach this subject and that various strategies should be researched when designing IL or NL training.
Understanding the nature of news, the boundaries of freedom of thought, and their effects on the news is required in order to be able to correctly evaluate the sources discovered through searches and to avoid difficulties in choosing the news elements to be traced during the fact-checking process. It is also important to be aware of how country-specific political and social climates may affect news sources. It is understood from the findings that students mostly visit online news sites and/or newspaper web pages in pre- and post-tests. Although it is more likely that they have found the same news content across several platforms, given that students typically search for the title of the news sample. Having information about the influence of media owners and policymakers on news content and the functioning of the news media environment with background computer algorithms and profit-driven mechanisms (Ashley, 2020) will help to assess the quality of the news accessed during the fact-checking process. For this reason, it seems extremely important to emphasize these issues in a NL training to be developed.
Although the students in the study received undergraduate education in the field of information management and completed their IL and NL training which will help them meet the requirements mentioned so far, it cannot be said that they have achieved a full success in the fact-checking process. Students’ attempts to advance from more practical search terms, such as the title of the news, without analytical thinking when it comes to news, show that they can ignore the requirements of fact-checking steps. As a matter of fact, it was discovered that students most frequently perform the news title search even when its content covers a journal title, with searches for the journal title coming in second. A similar situation applies to pages visited during the same fact-checking process. Students mostly visited online news platforms for this news. This situation can be considered as the right approach to see whether the news published in different sources and, if so, in what context. On the other hand, what needs to be done is to check whether the scientist and the journal title mentioned in the news content actually exist, and whether a published study of the person in the journal in question is actually available. The fact that fewer students than those visiting online news platforms visit institutional websites, scientific databases, or publications that can be consulted for this purpose indicates that most students do not adequately reflect what is taught during training. This suggests that such training might be too late at the university level.
One of the most important features that distinguishes fact-checking behaviors from other information behaviors is the need for a relatively complex accuracy check based on factors such as source and provenance, rather than making a linear assessment of the suitability and reliability of one or a group of information sources. In doing so, the reliability of the evidence accessed for each element should also be evaluated. The fact that these elements and the methods to be used can vary according to the news content is also a part of this multidimensional process. This also requires making a judgment about which piece of information in the content can be traced more effectively and having sufficient motivation for all of these. The findings of this study also showed that this process may require having foreign language skills. For all of these reasons, it does not seem feasible to apply this process, which may be challenging for individuals, constantly in daily life, where there is lots of information. At this point, it can be thought that more effective results can be obtained if NL training stands out in creating conscious information consumers rather than gaining basic skills for fact-checking. On the other hand, it should be taken into consideration that it is not easy for individuals to acquire critical and analytical thinking skills, which are the basic condition of being a conscious consumer of information, and to turn them into a habit. The findings of this study may be a clue to the importance of integrating NL skills into different phases of education so that individuals can benefit more from such training.
Being able to think critically, comprehend the nature of news, and be aware of the degree to which people have an impact on how any piece of information is disseminated and its effects can be considered sufficient in everyday life. But it is also important that individuals with information-oriented specializations, such as journalists and information professionals, have competencies in fact-checking. Therefore, it is recommended to design the most effective training content possible on this subject and to make these areas of expertise part of their training. In this study, only information management students were studied. A similar study can be conducted on students from different disciplines to examine how fact-checking behaviors vary by field. Additionally, such comparative studies might guide the development of tailored educational content for different target audiences.
Considering all this, it is recommended to focus on the following while designing an NL training program for university students:
NL training will not be fully effective without IL skills. If the trainees do not have competencies in this regard, it may be more useful to consider the content of the training in this context. The fact-checking process requires users to have good information searching skills. The most crucial factors in this process are the ability to think critically, create motivation to start the fact-checking process, make the correct decision regarding the information pieces to be tracked down, and understand and evaluate the nature of the news and the information environment. Emphasizing the importance of learning fact-checking steps, including more exercises that let students actively utilize the fact-checking tools and procedures given in NL training, as well as adding more case studies to the training content, can assist in improving the trainees’ fact-checking behaviors.
Creating content for acquiring NL skills can be challenging. It’s important to assess the effectiveness of the training, particularly in raising awareness and changing behavior. Evaluations can provide insight into the audience’s knowledge levels and fact-checking behaviors, while providing guidance on how they should be instructed.
Footnotes
Acknowledgements
This paper is based on İpek Şencan's PhD dissertation submitted to Hacettepe University in 2020.
