Facing the era of generative artificial intelligence: Strategies of information and digital literacy in Chinese studies

Abstract

Since 30 November 2022, when OpenAI launched ChatGPT, generative artificial intelligence has emerged as a subject of growing prominence. Various academic divisions, including academic librarianship, are closely monitoring the potential impact of generative artificial intelligence on their respective futures. Esteemed librarian associations such as the Council on East Asian Libraries have actively utilized industry-wide conventions to foster scholarly conversations, discussions, and debates regarding the influence of generative artificial intelligence on librarianship. This project draws inspiration from such discourse and is conducted by four Chinese and East Asian studies librarians from four prestigious American universities. Based on extensive first-hand experiments, they conclude that amidst the opportunities and challenges presented by generative artificial intelligence, the specialized expertise and knowledge possessed by area studies specialists is increasingly significant in shaping its implications for their field.

Keywords

East Asia Asia Oceania artificial intelligence information literacy and instruction

Introduction

Since 30 November 2022, when OpenAI launched ChatGPT, artificial intelligence (AI)—particularly generative artificial intelligence (GenAI) that is designed to generate content—has emerged as a subject of growing prominence. Various academic divisions, including academic librarianship, are closely monitoring the potential impact of AI and GenAI on their respective futures. Esteemed library and librarian associations, such as the Council on East Asian Libraries and the International Federation of Library Associations and Institutions, have actively utilized industry-wide conventions to foster scholarly conversations, discussions, and debates regarding the influence of AI and GenAI on librarianship. Our project draws inspiration from such discourse and is conducted by four Chinese and East Asian studies librarians from four prestigious universities in the USA. Based on extensive first-hand experiments, we conclude that the specialized expertise of area studies specialists is increasingly crucial in shaping the impact of GenAI on our field.

This article uses case studies to demonstrate the first efforts in developing information literacy strategies, underscoring the pressing need for innovative approaches that integrate GenAI as a teaching and learning tool in educational environments. Information literacy strategies—which involve helping patrons identify information needs, formulating search strategies, evaluating sources, analyzing information and using it ethically, and promoting critical reading and thinking—are increasingly facing challenges in the age of GenAI (Mastering the Art of Information Literacy, 2023). Our study represents the first endeavor in East Asian librarianship to advocate for the advancement of information and digital literacy tailored to the GenAI era. We acknowledge the essential role of librarians in equipping patrons with the ability to critically evaluate GenAI-generated content, thereby fostering an understanding of both the potential and limitations of GenAI.

In this article, we evaluate ChatGPT's performance in generating content through four different types of tasks: accessing online information; summarizing provided materials; reorganizing content; and analyzing visual materials based on textual and visual inputs. These aspects are contextualized within the disciplines of Chinese languages, literature, history, and art history, respectively.

Our findings indicate that while ChatGPT is sophisticated and efficient in certain aspects, it could yield distorted, incomplete, or missing information, and exhibit difficulties in interpreting visual content. Overall, our project demonstrates the initial endeavor to develop information and digital literacy strategies in the field of Chinese studies in the era of GenAI. It also aims to pave the way for a more informed, ethical, responsible, and effective deployment of GenAI across various educational settings. While our study primarily focuses on experiments in Chinese and East Asian studies, the findings and insights presented here can offer guidance and inspiration for librarians across various disciplines on instructing users to effectively and ethically leverage AI and GenAI in their research tasks.

Literature review

Even before the advent of ChatGPT, academic libraries were incorporating AI and GenAI technologies into their operations and services. Nevertheless, scholarly discourse and research on the application of AI and GenAI in academic libraries was limited. An environmental survey conducted by two academic librarians in 2019 indicated a significant need for awareness among academic institutions about the potential of AI. This research found that none of the surveyed universities or their libraries had integrated AI into their strategic plans, and fewer than 20% had initiated any AI-related programming (Wheatley and Hervieux, 2020: 350, 352). Additionally, the need for more research focused on AI in librarianship spurred ongoing discussions regarding the need for academic libraries to provide AI education for both patrons and staff. Moreover, there was a noticeable scarcity of scholarly investigation into AI-related technologies within libraries, and concerns were additionally raised over the possibility of AI replacing the roles of librarians (Wheatley and Hervieux, 2020: 348–349). In late summer 2019, when the term “AI technology” encompassed popular search engines and platforms such as Google, Siri, Alexa, and Cortana, a follow-up survey by Hervieux and Wheatley (2021) was distributed among academic librarians across Canada and the USA. Despite only 12% of the respondents recognizing the presence of AI technologies in their libraries, a promising 43% were optimistic about the future possible integration of AI technologies in librarianship (Hervieux and Wheatley, 2021: 5, 8).

The introduction of ChatGPT in late 2022 marked a significant shift in this narrative. Utilizing large language model technology, ChatGPT has substantially changed the AI landscape. This has engendered a mix of uncertainty and inquiry among librarians who previously sought clarity on the fundamentals of AI and its functionalities (Hervieux and Wheatley, 2021). The advent of ChatGPT underscored the urgent need for librarians to expand their understanding and adapt to rapid technological advancements.

Since the launch of ChatGPT, scholarly engagement with AI in the context of librarianship has notably increased. Discussions frequently address the definition of AI and its practical applications within libraries (Cox and Mazumdar, 2024). A scoping review of studies on ChatGPT showed a nearly fivefold increase in publications between 2020 and 2023 with a focus on AI users and the ethical use of AI (Ali et al., 2023: 7). Recent research has explored effective AI utilization in libraries and its positive outcomes (Chen, 2023; Cox and Tzoc, 2023), strategies for knowledge discovery (Cox, 2023), and the assessment of AI's capability for thematic analysis and research (Hitch, 2023). The academic community has started to understand students’ use of and attitudes toward AI (Amaro et al., 2024), and explore AI's possible roles in educational settings, including its potential as a mentor, teammate, tutor, coach, tool, and student (Mollick and Mollick, 2023). Moreover, issues of governance, policymaking, and the ethics of AI usage in higher education and librarianship have been the subject of increasing scrutiny (Chan, 2023; Cox, 2022; Michalak, 2023). Although the prevailing sentiment in these studies is optimistic about AI's roles in academia, scholars have expressed lingering concerns that AI might undermine users’ ability to critically assess its outputs in the future (Blechinger, 2023).

Despite these advances, research on AI and GenAI in the context of East Asian studies librarianship, particularly with non-Roman materials and multilingual contexts, remains largely unexplored. Our project seeks to pioneer this investigation by emphasizing the crucial expertise, skills, and insights of subject librarians as part of the ethical implementation of AI in academic settings. This endeavor is critical for promoting the integration of AI in educational contexts and supporting the academic success of our patrons.

Methodology

This project is rooted in essential concerns regarding the reliability of GenAI in academic settings. Can East Asian studies librarians, alongside other academic stakeholders, place their trust in the outputs generated by GenAI systems? If the response is contingent on a case-by-case basis, to what extent can GenAI be effectively utilized as an auxiliary tool in research, teaching, and learning? Additionally, what preventive measures and steps are necessary when leveraging GenAI to meet the informational needs of patrons?

The current academic discussion predominantly emphasizes the critical necessity of examining AI applications in fields such as health, medicine, and science (Hopkins et al., 2023; Leopold, 2024; Walker et al., 2023). This focus has led to a noticeable lack of practical case studies and methodologies for evaluating AI and GenAI in the humanities, particularly within East Asian librarianship. Addressing this gap, our research methodology mirrors the everyday work activities of East Asian studies librarians in academic libraries across the USA. The study focuses on four principal disciplines within East Asian studies—language, literature, history, and art history—with a particular concentration on China to maintain a consistent framework.

In Chinese language education, courses are designed not only to teach the language but also to immerse students in its cultural context. Students are expected to develop a comprehensive understanding of the language through reading, writing, speaking, and listening, while also discerning the cultural nuances of terms and words.

In Chinese literature, researchers are required to have not only language proficiency but also a comprehensive understanding of the historical and sociopolitical contexts that shaped the creation of works. This comprehensive understanding is essential for interpreting texts and appreciating their significance within the broader cultural and historical narratives.

Chinese history requires researchers to acquire the necessary knowledge and skills to interpret historical materials across diverse formats. The complexity of this discipline is heightened by the inherently ambiguous nature of many historical texts, which permits multiple interpretations by historians who are often constrained by their own temporal contexts and by varying cultural, social, political, and ideological factors. Moreover, the dissemination of historical materials may result in information loss, necessitating researchers to identify and decipher primary sources through critical and objective lenses.

Art history is a discipline that examines the myriad meanings, contexts, and life cycles of visual objects, including their creation, circulation, collection, exhibition, adaptation, concealment, and even destruction. As a scholarly field, Chinese art history is rich in content and deep-rooted traditions. It demands profound understanding and specialized knowledge to discern the unique attributes of Chinese art and its interconnections with other cultural traditions.

The four disciplines central to this project highlight the specialized knowledge that is essential for conducting pertinent research. By focusing on these disciplines, we underscore the importance of evaluating both the effectiveness and reliability of GenAI tools that assist in research, teaching, and learning across these and many other fields. This evaluation necessitates specific expertise and a comprehensive understanding of the subject matter—areas in which our subject librarians possess specialized knowledge and skills. In other words, our expertise is crucial in navigating the complexities of GenAI applications in research, teaching, and learning, ensuring that such tools are leveraged effectively and responsibly.

In this project, we utilize GPT-4, a highly acclaimed and broadly utilized AI tool, to address diverse research inquiries in both the English and the Chinese language.¹ Although GPT-3.5 is available without cost, potentially attracting a higher volume of users, our selection of GPT-4 is strategic. This most advanced version offers enhanced functionalities, including the capability to access the Internet, upload essential supporting documents, and integrate various plug-ins. Our choice of GPT-4 facilitates a more comprehensive evaluation of the effectiveness and dependability of GenAI technologies in researching these specialized fields, contributing significant insights into the ethical employment of GenAI tools in Chinese studies.

Project description and findings

This section elucidates the deployment of GPT-4 in addressing research questions in the fields of Chinese language, literature, history, and art history. The specific prompts given to the GenAI and its subsequent responses are presented as figures and appendices. All of the documented interactions were exported between April and May 2024, with the exception of one export in August 2024 as per revision suggestions. The experimental framework closely resembles the research and reference questions encountered by East Asian studies librarians, encompassing a wide range of patrons in different situations.

This project utilizes these four disciplines as case studies to test and evaluate GPT-4's information-generation capabilities from several perspectives. For Chinese language, the experiment emphasizes GPT-4's ability to source information online and its proficiency in translating between English and non-English. In Chinese literature, the focus is on GPT-4's capacity to generate and summarize information from provided texts. For Chinese history, the project examines GPT-4's ability to reorganize information from given texts. In Chinese art history, GPT-4 was tested on its ability to analyze visual content based on provided textual and visual inputs.

As an advanced large language model, GPT-4 has demonstrated remarkable proficiency in comprehending requirements, analyzing texts, and synthesizing and restructuring information. However, it lacks the ability to investigate, access, and comprehend the cultural and historical contexts related to given topics. Of greater concern is the system's capacity to produce responses that seem legitimate, accurate, and pertinent, but only individuals with deep knowledge in these fields can critically assess the accuracy and relevance of the generated content. This limitation underscores the necessity for subject knowledge and the application of new information and digital literacy strategies to critically evaluate GenAI-generated contents. By promoting these practices, librarians and information professionals can help ensure the successful and dependable implementation of AI and GenAI within academic environments.

Another finding of this research concerns the integration of a persona into prompt design. While many recommendations for prompt engineering suggest incorporating a persona into the prompt provided to the system (OpenAI, 2024), our findings suggest that integrating a persona does not significantly improve the quality of results, as evidenced in the response included in the Chinese language and Chinese literature sections and illustrated in Appendices 1, 3, and 4. The specialized and non-Euro-American nature of our investigations may be a contributing factor, necessitating further discussion.

Chinese language

This experiment is designed to evaluate GPT-4's ability to generate information from online sources and its proficiency in translating between English and Chinese. It explores the potential of GenAI in fulfilling the primary objectives of language learning, which focuses on equipping students with the skills necessary to write concisely in the target language and leverages GenAI's advanced capabilities in translation and language processing.

The experiment mimics an intermediate-level writing assignment in which students are asked to compose a 150-word paragraph about the Lunar New Year in Chinese. The objectives of the assignment are to encourage students to apply newly learned vocabulary and grammar, refine their Chinese writing skills, and enhance their understanding of Chinese language and culture.

The task begins by instructing the system to generate a paragraph in English about the Lunar New Year, followed by translating this passage into Chinese (Figure 1). To align with the assignment's requirements, the system is then prompted to condense the passage and simplify the vocabulary to meet intermediate-level Chinese proficiency (Figure 2). GPT-4 demonstrates its prowess in language processing and translation, yielding impressive results with minimal input. Additionally, when tasked with incorporating specific words and grammar phrases into the Chinese passage, GPT-4 rapidly produces commendable results (Figure 3).

Several concerns arise, however, upon a critical examination of the results provided by GPT-4. The case study uses the same prompts to examine GPT-4's performance twice, once in April 2024 (Appendix 1) and again in May 2024 (Appendix 2). Since GPT-4 focuses on generating language rather than offering accurate information (Saeidnia et al., 2024), it is understandable that the response generated in April 2024 differs from that of May 2024 (Figure 4). While neither response is entirely incorrect, both fail to provide a comprehensive overview of the Lunar New Year. The first response correctly highlights the significance of the Lunar New Year across various East and South Asian countries, whereas the second response inaccurately suggests that it is solely a Chinese celebration. Additionally, the second response lacks details of the cultural events and traditional foods associated with the Lunar New Year, unlike the first response. However, the cultural events mentioned in the first response overlook the diversity of Lunar New Year celebrations in China, such as variations between the northern and southern regions and among different ethnic groups.

Further issues arise concerning the use of GenAI in education, particularly regarding students’ privacy and academic integrity. For example, another common writing task in language courses involves crafting a self-introduction paragraph in the target language. If GenAI tools are permitted, instructors should consider cautioning students to safeguard their privacy by not inputting their personal information into the systems. This case study also highlights the potential negative impact of using GenAI tools without proper guidelines, such as those surrounding plagiarism and the delivery of misleading information, which can hinder genuine learning.

From this evaluation, it is evident that while GPT-4 can access online information effectively and answer general questions, including providing language translations, it also produces outputs that might contain misleading elements or perpetuate stereotypes. Furthermore, the translation capabilities present challenges in foreign language education by undermining deep engagement with the language and culture. Therefore, integrating GenAI tools into educational settings responsibly requires a collaborative effort among librarians and instructors to ensure these technologies enhance rather than detract from teaching and learning. It involves setting clear guidelines, fostering critical engagement with GenAI-generated content, and developing new pedagogies in language instruction that uphold academic integrity and enhance the learning experience.

Chinese literature

This task evaluates GPT-4's capabilities in reading, summarizing, and analyzing Chinese texts, which is essential for students to gain foundational knowledge and familiarity with the historical and cultural contexts of Chinese Sinophone authors and their works. Based on the test evaluation, concerns have emerged regarding the system's ability to interpret and represent the information from such texts accurately.

The initial phase of the test involves uploading an autobiographical preface by Lu Xun (鲁迅, 1881–1936), a seminal figure in modern Chinese literature, from his 1923 short story collection A Call to Arms (Nahan, 呐喊; Lu, 2017). After uploading the original Chinese preface, the system is tasked with summarizing Lu Xun's life and historical background based on this autobiography (Appendix 3). A refined prompt, incorporating the directive to refrain from consulting or citing external sources, is offered to GPT-4 with the intention of guiding the system more effectively toward reliance solely on the provided text (Appendix 4).² The responses demonstrate that GPT-4 is capable, to some extent, of understanding and processing texts in Chinese, and of providing a reasonable overview of the author's life as depicted in the autobiography. Impressively, even though the title A Call to Arms was not provided to GPT-4 in the first test (Appendix 3), the system was able to associate the preface with the book and offer a concise overview.

However, the test results highlight two significant issues. First, much of the system's overview of the autobiography follows online sources, such as the entry in Baidu Baike (2024), a Chinese-language collaborative online encyclopedia, rather than solely the text provided. Even though the second prompt emphasizes that the system should not reference any online resources, the response from GPT-4 includes information beyond the scope of the provided preface. This includes Lu Xun's original name and hometown, as well as other information that is not mentioned in the provided text (Figure 5).

Second, GPT-4 misrepresents significant details from the text and generates distorted and misleading information. One event from the original text, significant to Lu Xun's life and in the history of Chinese literature, came from a time when Lu Xun was a student in Japan. He was deeply shocked after watching a slideshow in class depicting a Chinese man with bound hands who was “spying on the Japanese military for the Russians” and was about to be decapitated as a public example (Lu, 2017: 3).³ GPT-4's response to the first prompt states that the Chinese man was not a spy waiting to be decapitated but rather was “being used for human experiments by Russian” (Figure 6).⁴ In the response to the second prompt, the system also returned with incorrect information that “[a] Chinese man was about to be executed after the Japanese army used him as a military spy” (Figure 5).⁵ These misrepresentations indicate the critical deficiencies of GenAI tools in understanding content.

The findings suggest that GPT-4 constantly accesses online information to generate its outputs. Furthermore, a significant drawback of large language models such as GPT-4 is their tendency to blend factual information with fabricated content, posing challenges in verifying the accuracy of the results. This underscores the critical roles of librarians in guiding patrons when using GPT-4 and other GenAI tools for educational, scholarly, and research purposes, ensuring they are aware of these constraints. Moreover, recognizing these limitations necessitates extensive testing and a profound comprehension of GenAI, highlighting the crucial expertise that librarians bring to this context.

Chinese history

This task is designed to test GPT-4's ability to comprehend historical texts and reorganize information into requested formats. Organizing materials into tables is a common research method that enhances the understanding and retention of original data, thereby facilitating more effective analysis. In the context of Chinese history, the evolution of languages and writing styles over centuries presents challenges for contemporary scholars attempting to comprehend ancient texts. Therefore, mastery of Classical Chinese is essential for understanding these pre-modern histories. This test uses the historical text “Annals of Qin” (“Qin benji,” 秦本紀) to assess GPT-4's capabilities in understanding Classical Chinese text, detecting and extracting specific names and terms from extensive texts, and identifying, categorizing, and reorganizing relevant information.

“Annals of Qin” is a chapter from Records of the Grand Historian (Shiji, 史記), one of the most important texts in Chinese history that was composed around 91 bce. The chapter details the history of the Qin State (770–221 bce) and concludes with a concise account of the Qin Dynasty (221–206 bce) founded by Qin Shi Huang (秦始皇, 259–210 bce), the first emperor to unify China.

The task commences with offering GPT-4 the original text sourced from the Chinese Text Project (Sturgeon, 2011). One objective is to assess the system's capability in generating a table delineating the genealogy of the ruling family of Qin based on its comprehension of the text. Another aim is to evaluate GPT-4's capacity in summarizing the pivotal historical events during the Qin era as per the provided text.

Multiple attempts were made to identify optimal strategies for processing large amounts of text with the system. The authors observe that uploading a text file to GPT-4 is more efficient than copying and pasting the entire text directly into the system. The upload method facilitates more comprehensive information extraction and generates the most accurate summaries of the historical turning points.

Overall, the results suggest that the system is capable of processing large volumes of pre-modern text. As detailed in Appendix 5, which showcases the best result from all the attempts, the system effectively detected and extracted the correct personal names and relevant details from the text. Moreover, it demonstrates proficiency in formatting the genealogy table and in translating and summarizing the information. This capability indicates that the system could serve as a valuable tool at the outset of the research process, providing a solid foundation for further scholarly analysis.

The analysis of the system's performance also reveals several notable deficiencies. First, the criteria for selecting names to be included in the results remain unclear. One finding is that both frequently and infrequently mentioned figures in the text could be omitted. For example, both the often-referenced Duke Mu of Qin (秦缪公, 659–621 bce) and the seldom-addressed Gongbo (公伯, 893–845 bce) disappear in GPT-4's results in half of the attempts. Another observation is that discrepancies exist between the summarization of individual figures’ accomplishments and historical occurrences. For instance, the achievements of Duke Mu of Qin, renowned for advancing military expansion and establishing regional power, and Duke Xiao of Qin (秦孝公, r. 361–338 bce), credited for adopting Legalism (法家), were sometimes erroneously associated with other figures.

Second, GPT-4 made consistent errors in representations of family relationships. For example, Feizi (非子, ?–858 bce), the founder of the Qin State, was granted the title Qin Ying (秦嬴) on being ennobled. The system, however, occasionally treated Feizi and Qin Ying as different people, and incorrectly positioned them within the relevant lineage.

Lastly, despite being given instructions to generate summaries solely of the provided text, the system incorporated information from online sources. This is evident in summaries about Qin Shi Huang, who unified China and started the Qin Dynasty. All attempts referenced his harsh rules and the construction of the Great Wall, which can be easily found online (Pandaist, 2024), even though these details were not mentioned in the original text provided to the system.

In general, the results suggest that GenAI tools can be beneficial for researchers by assisting with text mining primary sources and reorganizing information, thereby enhancing understanding and retention. The test demonstrates, however, that due to technological limitations, GPT-4 cannot fully understand the concept of historical significance from a given text and may misread the content and provide misinformation. Therefore, it is imperative for researchers to conduct thorough fact-checking and critical analysis. Relying solely on GenAI tools for text mining primary sources without proper verification can lead to the accumulation and dissemination of misinformation. It is essential for librarians to educate researchers about both the advantages and limitations of GenAI, as well as guide them in recognizing the importance of meticulous scrutiny to ensure precision and reliability in scholarly research.

Chinese art history

This task examines the effectiveness of GPT-4 in interpreting images after exposure to relevant text-based information. The results suggest that despite being provided with contextual and historical knowledge, the system is unable to comprehend images. Of concern is the tendency of the system to rely on textual information it can access, leading to the generation of inaccurate, misleading, and irrelevant content related to the images.

The focus of the project revolves around tasking the system with producing a formal analysis of an object. Formal analysis is recognized as a foundational methodology in the disciplines of art history, critique, and appreciation (Adams, 2003; Barnet, 2013; D’Alleva, 2005). The term “formal” relates to the attributes that shape the visual appearance of an artwork, thereby highlighting its aesthetic and structural dimensions.

The prompt administered to the GPT-4 chatbot comprises three interconnected components (Appendix 6). Initially, the system is tasked with retrieving the web page featuring the designated object and its pertinent information. Subsequently, the chatbot is directed to summarize an uploaded chapter from A Short Guide to Writing about Art by Barnet (2013), which explicates the principles of formal analysis within art historical discourse, and provides definitions and examples. The final component requires the chatbot to compose a formal analysis based on its newfound understanding.

The selected object for this analysis is the “Brush pot with episode from life on Sima Guang,” which is housed at the Cleveland Museum of Art (n.d.). The piece is a cylindrical brush pot adorned with three sequential panels that narrate an episode from the childhood of the renowned scholar-official Sima Guang (司马光, 1019–1086). As the museum's website indicates, this tale highlights his early displays of exceptional talent and bravery. While other children hesitated in fear, the young Sima Guang took decisive action to rescue a friend who had fallen into a large water-filled jar by breaking it.

The response of GPT-4 to the first step showcases the system's proficiency in accessing, comprehending, and paraphrasing information from the museum's web page dedicated to this object, which provides essential details about the artifact and succinctly recounts the depicted story. GPT-4’s response encompasses the type of object, historical context, materials used, and narrative it illustrates.

The second step aims to provide the system with an understanding of the principles of visual analysis in art history. After accessing Barnet's (2013) dedicated chapter on this methodology, the system concisely articulates that formal analysis examines a work's visual components, explores the interplay among these elements, and assesses how they collectively articulate the meaning of the artwork. It shows that the system is able to process and distill the core concepts from the uploaded document.

The third step leverages the insights gained from the previous tasks, guiding the chatbot to compose a formal analysis essay concerning the brush pot, conforming to the principles outlined by Barnet. As shown in the illustrations (Figures 7–9), the decoration on the brush pot effectively unfolds the narrative across its panels. The first panel (Figure 7) portrays two figures engaged in a dynamic interaction, with one figure extending an arm toward the left, guiding the viewer’s gaze in that direction. Rotating the pot reveals the subsequent panel, where the critical moment of the narrative is captured (Figure 8). This scene centers on a large jar brimming with water, with only a small foot protruding from it, signaling the imminent peril of drowning. Surrounding the jar, a depiction of five children illustrates their panic as they flee, but another child is depicted near the jar, poised with a stone, ready to break the jar and effectuate the rescue. The final panel (Figure 9) shows two children joyfully flying a kite, symbolizing the peaceful conclusion to this worrying event.

The response of GPT-4 underscores the key concerns that motivated this project: AI could generate content that appears accurate but is, in fact, incorrect, misleading, and irrelevant. In the introductory section of the response, it asserts that the analysis will explore how the elaborate decorations reveal “the period's cultural and artistic endeavors”—a statement that lacks relevance and logic. As previously mentioned, while the formal characteristics of an artifact could reflect its meanings, it is improbable that the attributes of a singular object could signify the cultural and artistic aspirations of an entire era—not to mention that this object did not, and does not, possess the historical significance to warrant such conclusions.

In the section of the response titled “Subject Matter and Narrative Development,” the system indicates that “[e]ach element, from the figures’ expressions to the broken vat's shards, contributes to the story's dramatic tension and its resolution.” However, there are critical inaccuracies in this analysis, especially since there is no representation of the “broken vat's shards” across the three panels. Second, given the size of the object and the modest capabilities of its creator(s), the facial expressions of the figures are not rendered with sufficient detail for close examination. Notably, in the panel depicting the climax of the story (Figure 8), one can even arguably state that the two children on the right-hand side of the scene are depicted as smiling (Figure 10). Considering the story's context, their portrayal should convey fear and urgency, and this discrepancy is likely due to limitations in the creator’s (or creators’) skill set.

In summary, the response indicates a significant limitation of the tool in understanding visual information. Despite being proficient in language-based materials and having real-time Internet access, the system tends to produce responses that appear authentic but are fundamentally flawed. Due to GPT-4's limited capacity for understanding visual materials, the specialized knowledge of librarians in visual information literacy—the ability to comprehend, evaluate, and utilize visual information—is essential.

Conclusion and further discussion

This research evaluates GPT-4's capabilities in generating information within the context of higher education, aiming to foster further actions, policymaking, new information literacy strategies, and collaboration among librarians, instructors, and academic partners. Our project tested the system in key areas across four Chinese studies disciplines. For Chinese language, the evaluation focused on results derived from online sources and GPT-4's capability for translation. In the disciplines of Chinese literature, Chinese history, and Chinese art history, the system was provided with original texts and visual materials before performing tasks. These tests aimed to assess GPT-4's ability to summarize text, reorganize content, and analyze visual information.

The findings indicate that GPT-4 exhibits exceptional proficiency in utilizing vocabulary, comprehending grammar, translation, and text mining. However, concerns remain regarding the system's potential and capability to generate misinformation and its inability to critically evaluate information accessed from the Internet. This study underscores the importance of the expertise of East Asian studies librarians in effectively navigating the opportunities and challenges presented by GenAI technologies in academic settings. The results highlight the need for a critical approach to GenAI integration, ensuring that experts in specific fields guide and verify GenAI's output to enhance its utility while safeguarding academic integrity.

The tests demonstrated in the previous section, particularly those related to the Chinese language and history, highlight how GenAI tools have established paradigms and norms for accessing and selecting information, potentially limiting opportunities for exploring and generating new knowledge. This poses challenges to existing information literacy strategies, necessitating creative methods for evaluating information sources, fact-checking, and conducting effective searches. Consequently, librarians and instructors across all disciplines should collaborate to tailor course policies and develop new information literacy methods that address specific needs and requirements. Additionally, subject and area studies librarians should actively participate in broader institution-wide discussions on GenAI usage guidelines. Their specialized knowledge, especially in non-western and non-English languages, can contribute to more thoughtful and comprehensive considerations regarding the deployment and use of GenAI technologies in academia.

Meanwhile, academia should recognize the potential benefits of utilizing GenAI technologies in educational settings. GenAI has shown great promise in facilitating interactive learning, such as helping students create tools to aid memory, including tables and flash cards. The newly released GPT-4o, with its ability to interact with users through voice in real time, further underscores its potential as a tutoring tool for diverse learning scenarios. This capability enhances the dynamic nature of learning, allowing for more personalized and responsive educational experiences that can adapt to individual student needs and learning styles.

Footnotes

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Bing Wang

Shuqi Ye

Notes

Author biographies

Bing Wang is the Chinese Studies Librarian and an Assistant Professor at the University of Illinois Urbana-Champaign. Her research interests center around Chinese art and visual culture.

Cecilia Zhang is the Chinese Studies Librarian at the University of Kansas. Her research interests include diversity, equity, inclusion and belonging in library collections and classifications; AI and East Asian librarianship; East Asian culture; social movements; and politics.

Khamo is the China Studies Librarian at George Washington University. She manages the collection development of all Chinese-language materials and offers research support for Chinese studies at the university.

Shuqi Ye is an early career East Asian Studies Librarian at the University of Minnesota. Her research interests include East Asian studies librarianship, East Asian history, and rare East Asian materials.

Appendix 1

Appendix 2

Appendix 3

Appendix 4

Appendix 5

Appendix 6

References

Adams

(2003) Looking at Art. Upper Saddle River, NJ: Prentice Hall.

Ali

Murray

Momin

, et al. (2023) The knowledge and innovation challenges of ChatGPT: A scoping review. Technology in Society 75: Article 102402.

Amaro

Barra

Della Greca

, et al. (2024) Believe in artificial intelligence? A user study on the ChatGPT’s fake information impact. IEEE Transactions on Computational Social Systems 11(7): 5168–5177.

Baidu Baike (2024) Nahan zixu. Available at: https://baike.baidu.com/item/%E3%80%8A%E5%91%90%E5%96%8A%E3%80%8B%E8%87%AA%E5%BA%8F/8508885 (accessed 28 May 2024).

Barnet

(2013) A Short Guide to Writing about Art. Boston, MA: Pearson.

Blechinger

(2023) Reflections on information literacy in the ChatGPT era. Pathfinder 4(1): 163–172.

Chan

CKY

(2023) A comprehensive AI policy education framework for university teaching and learning. International Journal of Educational Technology in Higher Education 20: Article 38.

Chen

(2023) ChatGPT and its possible impact on library reference services. Internet Reference Services Quarterly 27(2): 121–129.

Cleveland Museum of Art (n.d.) Brush pot with episode from life on Sima Guang. Accession number 1964.179. Available at: https://www.clevelandart.org/art/1964.179 (accessed 19 April 2024).

10.

Cox

(2022) The ethics of AI for information professionals: Eight scenarios. Journal of the Australian Library and Information Association 71(3): 201–214.

11.

Cox

(2023) How artificial intelligence might change academic library work: Applying the competencies of literature and the theory of the professions. Journal of the Association for Information Science and Technology 74(3): 367–380.

12.

Cox

Mazumdar

(2024) Defining artificial intelligence for librarians. Journal of Librarianship and Information Science 56(2): 330–340.

13.

Cox

Tzoc

(2023) ChatGPT: Implications for academic libraries. College and Research Libraries News 84(3): 99–102.

14.

D’Alleva

(2005) Look! Again: Art History and Critical Theory. Upper Saddle River, NJ: Prentice Hall.

15.

Hervieux

Wheatley

(2021) Perceptions of artificial intelligence: A survey of academic librarians in Canada and the United States. Journal of Academic Librarianship 47(1): Article 102270.

16.

Hitch

(2023) Artificial intelligence augmented qualitative analysis: The way of the future? Qualitative Health Research 34(7): 595–606.

17.

Hopkins

Logan

Kichenadasse

, et al. (2023) Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift. JNCI Cancer Spectrum 7(2): pkad010.

18.

Leopold

(2024) Editor’s spotlight/take 5: How does ChatGPT use source information compared with Google? A text network analysis of online health information. Clinical Orthopaedics and Related Research 482(4): 574–577.

19.

Lu X (2017) Nahan. Beijing: Tai hai chu ban she.

20.

Mastering the Art of Information Literacy (2023) Chapter 2: Seven habits and skills. Available at: https://fod.infobase.com/PortalPlaylists.aspx?wID=104680&xtid=291371 (accessed 11 August 2024).

21.

Michalak

(2023) From ethics to execution: The role of academic librarians in artificial intelligence (AI) policy-making at colleges and universities. Journal of Library Administration 63(7): 928–938.

22.

Mollick

(2023) Assigning AI: Seven approaches for students, with prompts. The Wharton School Research Paper. Available at SSRN: https://ssrn.com/abstract=4475995 or http://dx.doi.org/10.2139/ssrn.4475995.

23.

OpenAI (2024) Prompt engineering. Available at: https://platform.openai.com/docs/guides/prompt-engineering (accessed 21 May 2024).

24.

PANDAIST (2024) Chinese Dynasty: Qin Dynasty (221 to 206 BCE). Available at: https://pandaist.com/blog/en/chinese-dynasty-qin-dynasty-221-to-206-bce (accessed 23 May 2024).

25.

Saeidnia

Kozak

Lund

, et al. (2024) Evaluation of ChatGPT’s responses to information needs and information seeking of dementia patients. Scientific Reports 14: Article 10273.

26.

Sturgeon

(2011) Chinese text project: Qin benji. Available at: https://ctext.org/shiji/qin-ben-ji (accessed 21 May 2024).

27.

Walker

Ghani

Kuemmerli

, et al. (2023) Reliability of medical information provided by ChatGPT: Assessment against clinical guidelines and patient information quality instrument. Journal of Medical Internet Research 25: Article e47479.

28.

Wheatley

Hervieux

(2020) Artificial intelligence in academic libraries: An environmental scan. Information Services and Use 39(4): 347–356.