Artificial Intelligence for Urology Research: The Holy Grail of Data Science or Pandora's Box of Misinformation?

Introduction

Recent advances in artificial intelligence (AI) models have prompted unprecedented medical research interest. ¹ Free public access to advanced large language models (LLMs) such as ChatGPT and Bard has helped significantly to lower the barrier to entry to utilize AI and discover potential use cases for it. LLMs are deep-learning algorithms that utilize data sets to analyze large amounts of information to predict and produce plausible language. ² Integration of LLMs into the practice of medicine is an area of active investigation across several medical specialties. ³ Indeed, several prominent medical journals, such as the New England Journal of Medicine, have even started new series or opened new submission categories dedicated to AI. ^{4 –6} One exciting potential domain yet to be fully explored is the utility of AI, and specifically the novel publicly available LLMs, for urologic epidemiology.

Arguably the most common disease treated by endourologists, and urologists in general, is kidney stone disease. Accordingly, understanding the epidemiology of kidney stone disease and factors such as cost, prevalence, and incidence is of great importance and was a key component of the Urologic Diseases in America Project in the 2000s. ⁷ Data on kidney stone epidemiology are often derived from pooled analyses of large national health data sets such as the National Health and Nutrition Examination Survey (NHANES), National Inpatient Sample, and Medicare claims. Aggregating and interpreting these data sets can involve advanced training in data science, putting such studies out of reach for the average investigator.

However, if the data sets could be accessed and analyzed by an LLM, the ability to study urologic epidemiology could suddenly be limited only by an investigator's creativity and curiosity, rather than by advanced training in data science. One could study local urologic epidemiology at the national, regional, statewide, and potentially even local level.

To this end, as a starting point, we aimed to investigate whether easily accessible LLM-based AI interfaces such as Bard and ChatGPT would be capable of performing a large-scale calculation of the incidence and prevalence of kidney stone disease in the United States over various periods of time and localities.

Materials and Methods

Results

When asked to access and perform calculations on the NHANES database, ChatGPT denied the ability to do so. “I can't access external databases, including NHANES, or perform real-time data analysis.” This question was repeated with different wording several times with no change in response.

Bard intermittently claimed the ability to access the NHANES database and perform calculations (Fig. 1). When prompted to perform the requested prevalence and incidence calculations on the NHANES database, Bard was able to produce both accurate and inaccurate or irrelevant results without a clear pattern. Bard's calculations for the incidence of kidney stones from 2015 to 2018 were 2.1% (95% CI 1.5–2.7), 1.75% (95% CI 1.6–1.9), and 0.8% (95% CI 0.7–0.9), while the published number was 2.1% (95% CI 1.5–2.7). A similar pattern of sporadically accurate values was obtained when calculating prevalence.

OPEN IN VIEWER

FIG. 1.

Bard claiming ability to access and perform calculations on the National Health and Nutrition Examination Survey database.

Bard was able to detail the statistical steps that it claimed to have used in the epidemiologic calculations, and even produced tables broken down by year with sample sizes (Fig. 2). Only with additional prompting did Bard acknowledge that the final product was obtained from online sources rather than from a unique calculation and even provided a link to our reference article from Scales et al. (Fig. 3).

OPEN IN VIEWER

FIG. 2.

Kidney stone prevalence table generated by Bard.

OPEN IN VIEWER

FIG. 3.

Bard providing detailed steps of a prevalence calculation and later rescinding its computational claims.

ChatGPT and Bard were both able to produce a unique code (Python) to perform the intended prevalence and incidence calculations on the downloaded data sets (Fig. 4a, b), however, these could not be executed without consistent errors. Troubleshooting the code through additional prompts with the AI interface was attempted, however, the errors persisted.

OPEN IN VIEWER

FIG. 4.

(a) Nonfunctional custom-built Python code generated by Bard to calculate the prevalence of kidney stones. (b) Nonfunctional custom-built Python code generated by ChatGPT to calculate the prevalence of kidney stones.

The following are example quotes from Bard. In addition, we have included four representative chat logs in the Supplementary Data to show the exact wording used when interacting with the interface.

We found that both Bard and ChatGPT were not able to effectively complete the task. Irrespective of the wording of the prompts, the AI interface showed a lack of ability to integrate external data through direct access or uploading of materials and a lack of ability to generate the functional code. As the AI-generated calculations were not newly analyzed and as such not considered valid, we did not perform the planned comparative statistical analysis to our reference standard.

Discussion

To our knowledge, this was the first study of its kind to assess the ability of LLMs such as ChatGPT and Bard to determine the prevalence and incidence of a urologic pathology via independent analysis of a large publicly available health care database. Our results showed that in their current state, ChatGPT and Bard are incapable of performing such an analysis. ChatGPT could not access the intended publicly available database and both ChatGPT and Bard generated a nonfunctional Python code that would not effectively execute on a downloaded data set. Bard frequently claimed to have access to the NHANES database and that it had the ability to perform calculations.

The results of the requested prevalence and incidence calculations were inconsistent and often included results that had been taken from other studies without acknowledgment. The most concerning finding as shown in Figure 3 was that Bard often gave detailed descriptions of their calculation such as downloading an XPT file from the database, filtering the data, performing the statistical analysis, and providing a result that matched the comparative standard. It was not until inconsistencies in the responses from Bard led us to pose the question, “did you actually calculate those numbers… or did you pull them from existing sources,” at which point Bard apologized and admitted to taking the numbers from other sources. Overall, both LLMs were ineffective, and there were no real data to perform the intended comparison with the reference data we chose.

The introduction of free, easily accessible LLM-based AI interfaces such as ChatGPT in November 2022 and Bard in March 2023 has coincided with a substantial increase in the number of AI-related publications in the medical field. ¹ LLMs are complex algorithms capable of countless tasks, such as text generation, coding, predictive modeling, and analysis among others. ² In the realm of medicine, the ease of access to these tools may lead to advancements in research and the way we care for patients, although the extent of their capabilities is not yet clear. Instant and accurate epidemiologic calculations would allow clinicians to better understand their population and provide more nuanced care.

How to best use AI to assist in epidemiologic projects, however, is a difficult question. The development of useful models for processing epidemiologic data is a complex process that is fraught with many challenges such as estimating appropriate sample sizes, identifying clinically relevant features within large data sets, and dealing with missing data. ¹⁴ Reasoning and deduction in particular are known challenges for LLMs as they need to both comprehend information and deduce de novo answers rather than pull from existing sources. A recent large survey by Chang et al. performed a comprehensive review of the success and failure of LLMs in various tasks. In mathematical reasoning, the models have shown mixed results and can give unpredictable answers depending on the complexity of the problem. Most LLMs are capable of addition, subtraction, and some multiplication; however, division, exponentiation, trigonometry functions, logarithm functions, and other complex mathematical problems are still challenges.

Despite these hurdles, successive iterations of these models (e.g., GPT 3, GPT 3.5, and GPT 4) show continued improvement in each reasoning domain and handling new data compared with their predecessors. ¹⁵ Effectively navigating the reasoning challenges of LLMs and developing reliable models will take continued work; however, future versions will likely continue to improve the ability to perform complex calculations.

With the promise that AI in health care offers, come also significant pitfalls and ethical concerns. Our findings raised an interesting dilemma of whether we can ever truly trust the results of AI calculations, particularly when it might relate to patient care where the stakes are high. LLMs have been shown to generate misinformation through confabulations, falsify data, and contribute to plagiarism. ^2,16 AI systems are prone to bias and, through reliance on algorithmic predictions and overrepresented populations in data sets, can perpetuate inequities in disadvantaged populations. ¹⁷ Furthermore, as AI interfaces gain access to more patient data, it has the potential to not only compromise the quality of medical research but in its worst form potentially affect the care and privacy of patients. The questions of authorship and who is to be held accountable when the said data are inaccurate or misleading represent real challenges. One solution we propose to protect readers is for publications to include explicit disclaimers for all material generated by AI. We also suggest the adoption of standardized methodologies and checklists for appropriate incorporation of AI-generated research, similar to the way the PRISMA checklist is used for systematic reviews and the CONSORT statement is used for randomized control trials. ^18,19 In addition, we expect that as LLMs are further developed they will, similar to growth in humans, develop a greater ability to recognize their own limitations in knowledge without overstating the confidence of their findings. We believe the important concerns raised by our findings can signal to endourologist researchers to proceed with significant caution when utilizing AI in the design of their studies.

The ideal model for a standardized methodology to incorporate LLMs in research would ensure complete transparency and reproducibility of methodology, provide external validity of AI capabilities, and ensure data privacy, which are features not present in common LLMs such as Bard or ChatGPT. The Translational Evaluation of Health care AI framework, created in 2021, is one such framework that aims to address the issue of trust and unrecognized poor performance of LLMs within health care. The structure assesses various aspects of AI systems regarding relevance, safety, ethical considerations, and efficiency. ¹⁶ In rigorously assessing LLM tools and advocating for a governance approach to LLMs, it aims to keep LLMs used in medicine in check. Frameworks such as these, in addition to official regulatory requirements, are important in developing safety measures for any form of AI that may influence patient care.

Studies from Scales et al. and Hill et al. were large studies that performed calculations of the prevalence and incidence of kidney stone disease and provided us with our reference standards to which we could compare results produced from the LLMs. ^12,13 In scrutinizing the calculation results from Bard, we were able to eventually recognize that some findings appeared remarkably similar to these published findings, and with further questioning uncovered that results were taken from other sources and were not de novo calculations. This is a dangerous finding as without a high degree of suspicion, users could be convinced of the process and results. Thankfully, most LLM chat interfaces are currently constrained in their handling of medical information. ²⁰ However, limited integration of this technology within medicine is slowly being seen such as the partnership announced between the electronic medical records company Epic and OpenAI, the owner of ChatGPT. ²¹ Which companies and developers will continue down this path and be willing to assume the regulatory and validation process necessary to bridge the gap into the medical field is unclear.

There are several limitations to our study. ChatGPT and Bard were chosen in our experiment as these are easy to understand and free; however, they only represent one small fraction of AI, and they may not be the most appropriate tool for the task of an incidence and prevalence calculation. Likely, the ideal tool would be an AI interface that is designed specifically for the subject of epidemiology and has been rigorously tested and validated to be accurate and reliable. As discussed above, newer versions of these common LLMs will likely continue to improve the capability to perform such tasks. In addition, the descriptive nature of the study does not allow for any hard conclusions to be drawn from its findings. ChatGPT and Bard did not produce a valid output for the requested calculations to be able to statistically compare the results with known data.

However, we still believe our findings to be important, as they demonstrate important limitations in the power of LLMs and serve as an important cautionary tale and counterpoint to the recent surplus of studies boasting the power of LLMs such as ChatGPT and Bard.

Conclusions

ChatGPT and Bard are LLMs that are currently incapable of performing large-scale epidemiologic calculations. In their current state, these tools are fraught with inconsistencies and lack of transparency. Application of these tools within medical research should be with caution and limited until they have been validated for an intended purpose. As AI progresses, robust testing and regulation will be needed before a non-epidemiologist can reliably obtain the desired results.