ChatGPT is a concise,comprehensive,and accurate alternative to Google for answering common hand therapy questions

Introduction

Artificial intelligence (AI) programs like Chat Generative Pretrained Transformer (ChatGPT) are digital technologies that allow computers to replicate human intelligence. A user of an AI application like ChatGPT can provide the software with a question to, within seconds, receive a humanlike answer. ChatGPT version 3.5 (ChatGPTv3.5) is free to use, and the initial answer can be modified by engaging in a ‘text dialogue’ during which the user prompts the software to adjust its output. This technology has been instrumental in enhancing the business industry, the field of law, and the education system.^1–3 ChatGPT writes code, translates text, and summarizes documents, along with many other functions. The technology is also used for more routine tasks applicable to individual users, like creating recipes or planning trips to new destinations. With so many potential uses, it is no surprise that ChatGPT exploded onto the market to become one of the fastest growing consumer applications in history.^3,4 The role of AI software like ChatGPT in the healthcare field is more complicated given safety concerns associated with the potential for dissemination of incorrect information.

Rising healthcare costs and physician shortages combined with commonplace access to internet has lead many people to seek medical information online.^5,6 There are a handful of websites crafted by academic institutions that provide evidence based medical advice, but the majority of information online is of questionable quality and has the potential to undermine the patient-provider relationship and plan of treatment or therapy.^7–9 ChatGPT is similarly flawed, being known to ‘hallucinate’ and confidently present a user with false information. ¹⁰ With so many people using ChatGPT, there are likely a large number of patients asking the software medical questions. Researchers have examined ChatGPT’s role as an avenue for providers to look up information in clinic.^11,12 Research has also been conducted to assess ChatGPT’s ability to answer patient questions relative to medical websites, and its ability to pass medical board exams.^13–16

No studies have assessed ChatGPT’s proficiency at answering common questions patients have about hand therapy. We sought to determine whether ChatGPT or the first five tabs appearing for a Google search, which account for the majority of web traffic, can better answer common questions about hand therapy.

Methods

This study was a ‘cross-sectional, rater-based comparison study’. We systematically compared responses from Google and ChatGPTv3.5 based on predefined criteria without manipulating variables. ¹⁷

Selecting questions and downloading answers

Ten certified hand therapists at a single health care institution each individually noted the most common questions patients ask them. Four questions were listed most frequently and were selected, after a group discussion and agreement, to represent common things their patients ask them and potentially search on the internet (Table 1). These questions were then answered using Google and ChatGPT version 3.5 (ChatGPTv3.5) (OpenAI, San Francisco, CA, USA) in January of 2024. To remove bias and influence from previous searches, the browsing history and cache were cleared for both Google and ChatGPTv3.5 prior to conducting the search. No additional information beyond the question was given to either Google or ChatGPT during the search. Location services were also turned off. All searches were performed on the same device on the same network. The answer outputs from ChatGPTv3.5 and the content from the first five webpages on Google, excluding advertisements, for each search were copied and anonymized. Anonymization included removal of any content that would indicate the source of the information so that blinded grading of the answers could be performed. Non-text content from Google, including images and links, was not included. All responses were reformatted to a uniform font, size, and spacing prior to grading.

OPEN IN VIEWER

Table 1.

Four questions about hand therapy selected for inclusion in this study from a poll and vote among all certified hand therapists at one institution.

Question#	Question
Q1	When can I resume activities after carpal tunnel surgery?
Q2	When will the swelling go down after carpal tunnel surgery?
Q3	Should I squeeze a ball after carpal tunnel surgery?
Q4	When will my finger stop hurting after a finger fracture?

Data collection

Three certified hand therapists (KC, SG, MP) blindly and individually graded the answers from ChatGPTv3.5 and Google using Likert scales (Table 2). The three participating therapists were Certified Hand Therapists with clinical experience ranging from early-career to more than 30 years in hand therapy. Accuracy was assessed using a six-point Likert scale. ¹² Completeness was assessed using a three-point Likert scale. ¹² A simple, three-point Likert scale was used to assess the conciseness of answers ¹⁷ (Table 2). The hand therapists also indicated whether they thought the answer they were grading was generated by ChatGPTv3.5 or Google. All raters participated in a brief orientation session to explain the study design and randomization method to ensure consistent application of the scales with minimized bias.

OPEN IN VIEWER

Table 2.

Likert scales used to grade answers for accuracy, completeness, and conciseness.

What is being graded	Likert scale
Accuracy	1- Completely incorrect
	2- More incorrect than correct
	3- Approximately equal correct and incorrect
	4- More correct than incorrect
	5- Nearly all correct
	6- Correct
Completeness	1- Incomplete (addresses some aspects of the question, but significant parts are missing or incomplete)
	2- Adequate (addresses all aspects of the question and provides the minimum amount of information required to be considered complete)
	3- Comprehensive (addresses all aspects of the question and provides additional information or context beyond what was expected)
Conciseness	1- Did not answer the question
	2- Answered the question, but not concisely
	3- Answered the question concisely

Overall, for each question the therapists graded six answers. Five of these came from Google, and one was from ChatGPTv3.5. The order of answers provided to the therapists for grading for each question always followed the order that the Google websites appeared in after the search, although the therapists were not aware of this. A six-sided dice was used to randomize which spot in the order ChatGPTv3.5 would be presented in (see Table S1).

Results

Descriptive statistics for accuracy, completeness, and conciseness scores for ChatGPT and Google responses are presented in Figure 1. Across all graded responses, ChatGPT consistently outperformed Google in all three metrics.

OPEN IN VIEWER

Figure 1.

Mean scores for ChatGPT and Google on Likert scales for accuracy (a), completeness (b), and conciseness (c). Scores from the three graders are averaged for question 1 (Q1), question 2 (Q2), question 3 (Q3), and question 4 (Q4). The comparison between ChatGPT and Google is also represented by averaging scores across Q1-Q4, represented by ‘overall’.

Discussion

The internet is an easily accessible source of information for healthcare patients with previous reports demonstrating that greater than 50% of patients use the internet to investigate health concerns.^18,19 Given the popularity of AI platforms, such as ChatGPT, recent literature regarding the use of AI in the field of orthopedics has garnered increasing interest.^13,20 However, to date, no studies have investigated the use of AI in the field of hand therapy.

We report that ChatGPT answers to common hand therapy questions were significantly more accurate, complete, and concise than Google answers when evaluated by certified hand therapists at our institution. These findings are similar to those previously reported in a study of ChatGPT responses to questions about common hand surgeries which demonstrated that ChatGPT provided high quality answers. ¹³ However, that study reported that ChatGPT-generated answers required a college reading level to comprehend and often failed to reference the source of their material calling into question its reliability. ¹³ While our paper did not assess the readability of ChatGPT and Google answers, we demonstrated that ChatGPT responses were significantly more concise than Google answers.

Figure 1 shows the average conciseness scores for Google and ChatGPT were 1.60 and 3.00, respectively. An average conciseness score less than 2.00, according to the scale shown in Table 2, suggests that Google answers sometimes did not even answer the question at all. ChatGPT routinely answered questions concisely, achieving a perfect score on the conciseness scale as would be expected for AI software with its programming. This is because ChatGPT is designed to provide direct, well-organized answers only to what is being asked, without unnecessary details. Google provides, from its collection of webpages, the best matches to what is searched. Google responses therefore vary significantly in length and sometimes fail to directly answer the queried question. Regarding accuracy and comprehensiveness, ChatGPT scored consistently higher because Google is unable to perfectly answer the question being asked. Google provides the user with webpages based on what key words are used in the search, but Google itself is not responsible for the content of the webpages. Therefore, questions are often only partially answered leaving the reader with only some of the information they need whereas ChatGPT responses are tailored to the exact question being asked. Moreover, many Google pages are biased in that they aim to sell the user something, whereas ChatGPT is not designed to advertise products, although it may occasionally generate content that resembles promotional language. One potential approach to optimize information gathering is to use ChatGPT to obtain an initial, synthesized overview of a topic, followed by verification through traditional search engines such as Google, which provide access to multiple primary sources and allow users to evaluate information across different websites.

Previous literature has questioned the accuracy of AI generated responses. ²¹ While the current study demonstrates high quality responses that were deemed both more accurate and complete than answers generated by Google, these results do not negate the possibility for ChatGPT to mislead a patient with inaccurate information. The term ‘hallucination’ has been coined to refer to situations where ChatGPT confidently presents false information. The AI software has even been known to also fabricate citations. ¹⁰ Google is not without flaws as well, being known to frequently present websites that contain incorrect medical information. ²² Therefore, just as a provider should be hesitant to instruct a patient to ‘Google it’, unless of course they direct patients to trustworthy academic cites, we believe patients should not be told to ‘ChatGPT it’. ChatGPT will continue to be used by patients to seek information, including answering medical questions. Our results are helpful for context, as it is important for providers to understand the resources frequently accessed by their patients in order to ensure that their patients refer to accurate sources. As mentioned previously, ChatGPT is a great tool for obtaining an overview of a topic that can be followed up with a Google search to access more in-depth information from reputable sites.

Accurate sources include websites constructed by academic institutions that are regularly updated using evidence-based information. ChatGPT version 3.5, which was used in this study, does not have access to real-time data and can only source information up to the most recent update which was 2 years prior to conducting this study. The new version of ChatGPT, version 4.0, is frequently updated but requires a paid subscription for use and is therefore less likely to be used by patients. Previous studies have shown that ChatGPT version 3.5 is worse at answering orthopedic hand questions compared to an academic website. ¹³ The present study compared ChatGPT with the first five Google tabs to more accurately reflect patient searching practices. An individual can select whichever website they want on Google, but research shows that the first result on Google captures 28.5% of users, and the top five results account for 70% of clicks. ²³ Therefore, we sought to compare ChatGPT with the websites that patients often navigate to.

This study has several limitations. First, despite blinding efforts, the hand therapists correctly identified which answers were produced by ChatGPT and which came from Google which introduces a bias. Second, ChatGPT is a novel tool and with further development, like updated sourcing of information for the free version, could become a more reliable resource for patients. This study did not compare readability of information between ChatGPT and Google which also affects the way people interpret information. The conciseness scale was developed specifically for a prior study for our group and lacks validation, thus limiting the interpretation of its results. Google results were presented in the order of which they appeared on the Google search page, introducing possible order effects. As all of our graders were certified hand therapists with experience in the field, our results cannot necessarily be extrapolated to people who are unfamiliar with medical jargon. Users can instruct ChatGPT to provide answers at a lower readability level, but not all users are familiar with this function. When selecting questions, hand therapists likely had recall bias. Moreover, patients at our institution may ask different questions than patients at other locations. Importantly, patients may ask ChatGPT questions that they do not feel comfortable asking their hand therapist. Finally, our study obtained responses from Google and ChatGPT using a cleared browser with location services turned off. It is well known that Google and ChatGPT both use an individual’s search history on the respective platform to modify the output. Google also incorporates a users’ location to provide more relevant answers. It is therefore likely that users would obtain different answers than we tested in this study which could affect the comparison. For example, the Google output for a student in school to become a certified therapist would differ from someone who works as an artist. Moreover, despite clearing history and disabling locations services the Google search return is still influenced by algorithmic variability which could be mitigated in future studies by using multiple devices or locations. Clinical experts were selected to evaluate the accuracy and appropriateness of responses; however, future work including individuals with lived experience may provide additional insight into how patients interpret and apply this information. Furthermore, the clinical experts all came from one institution possibly limiting the extrapolation of the findings. Finally, large language models have also been shown to reproduce societal biases present in training data, including gender and racial biases. Such biases may influence the perceived trustworthiness or usability of AI-generated medical information among individuals from equity-deserving groups and represent an important consideration when evaluating the role of AI in patient education.

Conclusion

ChatGPT version 3.5 is more accurate, comprehensive, and concise compared to a Google search when answering common questions about hand therapy. Nevertheless, version 3.5 in its current form is not qualified to replace websites built by medical societies or academic institutions and therefore should not be recommended by providers to patients as a source of medical information.

Supplemental material