Evaluating DeepSeek-R1 for gynecological oncology disease consultation in telemedicine: A comparative study with human doctors

Introduction

Women’s health has always been a major global health concern.^1,2 Gynecological oncology diseases are known for being diverse, with different levels of benignity and malignancy, and various presentations. ³ This means that early intervention and treatment, as well as effective care in the late stages of the disease, are essential. For example, ovarian cancer is a malignant gynecological tumor that has always posed a significant threat to women’s lives and health. ⁴ It has placed a substantial burden on people’s lives and health in various countries and remains one of the most challenging malignancies for medical practitioners to overcome.^5–7 Nevertheless, due to atypical early symptoms and rapid disease progression, patients are often diagnosed in advanced stages with a poor prognosis. ⁴ Therefore, early screening and timely, effective medical advice for suspected cases are essential. However, due to uneven regional economic development, inadequate social infrastructure, and the unbalanced distribution of medical resources, many women in developing countries around the world do not have access to quality healthcare services or timely medical advice. ⁸ Fortunately, the continuous development of internet technology and the emergence of telemedicine have greatly alleviated this problem.

Telemedicine refers to the medical form in which healthcare providers use an online platform to offer medical consultations, diagnostic recommendations, and therapeutic advice to patients in need, either free of charge or for a fee. ⁹ Telemedicine enables quality medical resources to reach a wider area, ¹⁰ and it is convenient for patients and their families. It also helps medical workers conduct more in-depth disease screening, thus increasing the early detection rate and reducing the death rate. Additionally, it provides valuable data for doctors and epidemiologists to use in epidemiological disease investigations. Studies have demonstrated the importance of telemedicine in chronic disease management.^11–16 For example, in the management of atrial fibrillation, telemedicine can connect experts with village doctors. ¹⁶ Through this connection, experts can provide consultations and guidance for difficult cases. The platform can also provide continuous education for village doctors, who can conduct in-depth follow-up visits to patients’ homes. ¹⁶ The results of the study showed that patient compliance increased and complex cardiovascular events decreased with this format. This program is both sustainable and replicable, representing a new solution to the global problem of rural chronic disease management, as represented by atrial fibrillation. ¹⁶ Furthermore, another study has demonstrated that it is useful to utilize mobile CT scanners with remote artificial intelligence (AI) assistance for telemedicine-enhanced lung cancer screening in underserved communities. It not only can reduce barriers to healthcare access for these communities but also promote the early diagnosis and treatment of lung cancer. Furthermore, leveraging AI can significantly enhance the efficiency and quality of processing screening results. ¹⁷ Other studies have also revealed the significant value of telemedicine in managing diseases for elderly patients with type 2 diabetes, as well as in providing care services for older adults who are housebound.^18,19

In China, “Ding Xiang Doctor” and “Good Doctor Online” are two well-known telemedicine platforms. Through these platforms, patients can access high-quality medical care. Under the platforms’ supervision, most of the doctors’ opinions are valuable. With personal information withheld, the platforms select a series of typical consultations for people’s reference, which are also of high quality. However, accessing medical consultations on both platforms still has a number of limitations. Firstly, telemedicine services on these platforms are costly, and the number of Q&A sessions is limited. Second, doctors are still required to respond online, often at the expense of their rest time. This increases their workload and makes it difficult for them to respond to medical consultations. To solve these problems, people are turning to artificial intelligence, and large language models (LLMs) are playing an increasingly significant role in telemedicine.

Among all kinds of LLMs, Chat Generative Pre-trained Transformer (ChatGPT) has been attracting a lot of attention, and there have been numerous researches on the use of ChatGPT in telemedicine. In 2024, some researchers analyzed the effectiveness of using ChatGPT in telemedicine for orthopedic-related diseases and found that ChatGPT was more effective than human doctors for popularizing medical knowledge. ²⁰ The scholar also found that ChatGPT was not significantly different from human doctors in other aspects, ultimately concluding that ChatGPT could assist doctors in telemedicine. ²⁰ Recently, China has also made significant progress in developing LLMs. One notable model is DeepSeek, which has received global attention due to its excellent performance and relatively low API price.^21,22 The DeepSeek-V3 model is reported to outperform other advanced LLMs, such as Claude-3.5 and GPT-4, in a number of domains, while DeepSeek-R1, which was released on January 20, 2025, is claimed to outperform OpenAI-o1 in a variety of tasks. Studies have explored DeepSeek’s performance in the Chinese National Medical Licensing Examination ²³ and in the gastroenterology board examination. ²⁴ However, compared to ChatGPT, the application of DeepSeek-R1 in telemedicine, especially for gynecological oncology diseases, has not been studied in depth.

This study aims to evaluate the performance of DeepSeek-R1 in gynecological oncology telemedicine by comparing its responses with those of human doctors on the platforms “Ding Xiang Doctor” and “Good Doctor Online.”

Methods

Results

We selected four common gynecological oncology diseases from “Ding Xiang Doctor” and “Good Doctor Online,” totaling 600 cases, including ovarian cancer, endometrial cancer, cervical cancer and vulvar cancer. Many of these cases included voice responses, while others contained images or required image analysis. We used these two criteria to screen out 489 cases at the outset. We then assessed the completeness of related information, such as personal basic information, pathology reports, and laboratory reports. We removed 29 cases with incomplete information, leaving 82 cases that met the requirements. After anonymizing the sources, we provided the doctors’ responses and the corresponding DeepSeek responses for the cases that met the criteria to three professional obstetricians and gynecologists for evaluation. Figure 2 illustrates this process completely. Figure 3 presents all scoring results in the form of a heatmap. We then conducted a statistical analysis to derive a series of results.OPEN IN VIEWER

OPEN IN VIEWER

Figure 3.

Heatmap of the all scoring results.

It is found that DeepSeek’s responses scores were higher than that of human doctors across all seven perspectives. For MA, DeepSeek’s mean score is 8.276 with a standard deviation of 0.3695, while human doctors’ mean score is 7.654 with a standard deviation of 0.5252. And the difference is statistically significant (p < 0.0001). For CA, DeepSeek’s mean score is 8.439 (SD = 0.3845), compared to 7.650 (SD = 0.4623) for human doctors, with p < 0.0001. For CE, DeepSeek’s mean score is 8.406 (SD = 0.3745), whereas human doctors score 7.159 (SD = 0.7848), p < 0.0001. For SAC, DeepSeek’s mean score is 8.407 (SD = 0.3553), versus 7.317 (SD = 0.6016) for human doctors, p < 0.0001. For PST, DeepSeek scores a mean of 8.114 (SD = 0.3997), while human doctors score 6.695 (SD = 0.8610), p < 0.0001. For HC, DeepSeek’s mean score is 8.447 (SD = 0.3680), compared to 6.748 (SD = 0.7743) for human doctors, p < 0.0001. For OS, DeepSeek achieves a mean score of 8.348 (SD = 0.2884), while human doctors score 7.204 (SD = 0.6023), p < 0.0001.

These results indicate that among the seven metrics, the lowest scores for both human doctors and DeepSeek occur in the PST perspective. Human doctors perform best in the MA perspective, whereas DeepSeek attains its highest score in the HC perspective. Figure 4 presents the score values for MA, CA, CE, SAC, HC, PST and OS. Figure 4 also includes a radar chart summarizing the scores of human doctors and DeepSeek across all seven perspectives comprehensively. Table S1 and Table S2 show the results of the Wilcoxon matched-pairs signed rank test and ICC respectively. The results of the Wilcoxon matched-pairs signed rank test and ICC demonstrate the reliability of the study conclusions.OPEN IN VIEWER

Discussion

Our study demonstrates that DeepSeek-R1 performs excellently in online remote consultations for obstetric and gynecological oncology diseases, surpassing human doctors across all evaluated metrics. We consider that DeepSeek’s high scores are closely related to its response style. Its answers are characterized by templated, modular, and formulaic structures, which produce standardized and highly logical responses. This standardization facilitates evaluators in tracing the logical flow of the answers, thereby enabling more accurate scoring. Furthermore, DeepSeek’s responses tend to be highly detailed and require longer generation times, covering a broad and comprehensive range of content. Such thoroughness significantly reduces potential omissions and flaws, making higher scores more attainable. However, excessively lengthy responses substantially increase the cognitive burden on patients and their families during online consultations and may also result in important information being overlooked. Therefore, further development by relevant practitioners is warranted to produce generated texts that are more targeted and appropriately concise.

Besides, it is observed that, regardless of whether responses originate from human doctors or DeepSeek-R1, the scores for the PST perspective were the lowest among the seven evaluated dimensions. This indicates a notable deficiency in both DeepSeek and human doctors in translating specialized medical knowledge into language that is readily understood by the general public.This issue has also been highlighted in existing research. For instance, studies addressing AI-assisted cardiopulmonary resuscitation (CPR) education have noted that DeepSeek’s responses contain numerous technical terms. ²⁶ Several factors contribute to this phenomenon. For human doctors, daily clinical rounds, medical record documentation, and professional communications foster a habit of using dense professional terminology and concise logical structures. This habit may lead to an implicit assumption that remote consultation recipients possess a certain level of medical knowledge. Moreover, compared with online consultations, face-to-face interactions enable doctors to provide more detailed and vivid explanations through facial expressions, body language, and visual aids, which are constrained in virtual settings. Regarding DeepSeek, although its responses are logically coherent and detailed, they tend to be templated and formal. This templated style employs extensive academic expressions, which may cause comprehension difficulties for patients during online remote consultations. To address this issue, it is recommended that healthcare providers consciously explain essential terms or concepts before delivering responses in future online consultations. For DeepSeek, further development by relevant practitioners is anticipated, or alternatively the design of more precise prompts could enhance the translation of professional content into language understandable by the general public.

Additionally, human doctors scored highest in the MA compared to the other six perspectives, indicating that human doctors possessed a solid foundation of medical knowledge and extensive clinical experience, and can promptly incorporate the latest clinical guidelines into clinical practice. As a result, MA stands out relative to other indicators.

Moreover, among the seven evaluated metrics, DeepSeek was observed to achieve the highest score in HC. This outcome is evidently related to the explicit inclusion of “humanistic care” content in its responses. In contrast, human doctors’ provision of humanistic care is often implicitly integrated within their replies, making it less overt and more difficult to capture. In some cases, it may even be overly simplified, resulting in relatively lower scores. However, it should be noted that although DeepSeek’s explicit presentation of humanistic care contributes to its favorable scoring, the effectiveness and meaningfulness of such formalized expression remain questionable. Much of the humanistic care content generated by DeepSeek consists of generic statements lacking personalization tailored to the specific circumstances of the patients. Furthermore, the segregation of this content into a separate paragraph places excessive emphasis on politeness and formality, weakening its connection to the preceding medical information. This generates a sense of detachment and perfunctoriness, casting doubt on whether it truly provides the comfort and reassurance expected from authentic humanistic care for patients and their families. For future research, it is recommended that patients and their families participating in remote online consultations should be involved in the evaluation of the “humanistic care” metric to better assess its emotional soothing effect. Additionally, these findings highlight the need for human doctors to place greater emphasis on delivering emotional value, thereby providing more comprehensive physiological and psychological support.

Undoubtedly, this study has several limitations. First, all evaluating experts are drawn from a single institution, which may introduce bias into the assessment results. Therefore, multicenter studies will be pursued in future research. Second, the study is confined to four gynecological oncology diseases, leaving many prevalent conditions unexamined. Whether evaluation results differ among diseases within the same specialty warrants further investigation. Additionally, diseases from other medical specialties are not included. Given the considerable differences in severity, treatment complexity, and clinical approaches across disciplines, future studies should incorporate multiple specialties to provide a more comprehensive understanding. Furthermore, this study only included text-based teleconsultation cases and excluded those involving clinical imaging analysis, which is critical for gynecological oncology diagnosis and management. Therefore, exploring the multimodal analysis capability of the model that integrates imaging and text represents an important direction for future research.

Conclusions

In conclusion, this study systematically evaluated the performance of the DeepSeek-R1 in remote consultations for gynecological oncology diseases across seven dimensions: medical accuracy, clinical applicability, communication effectiveness, safety and compliance, humanistic care, popular science translatability, and overall satisfaction. The results demonstrate that DeepSeek-R1 outperforms human doctors on all assessed metrics. However, despite its comprehensive content, rigorous logic, and clear structure, the length of its responses may increase patients’ cognitive burden, and both the form and substance of its humanistic care require further refinement. Human doctors excelled in providing personalized consultations but are limited by workload constraints and still need to enhance their expression of humanistic care. This study highlights the promising potential of advanced large language models, exemplified by DeepSeek-R1, in remote gynecological oncology healthcare, particularly in medically underserved regions. Future research should pursue multi-center, multi-disciplinary comprehensive evaluations to further optimize the model’s response specificity, conciseness, clarity, and patient-centered humanistic care expression, thereby improving the quality of remote medical services and patient outcomes.

Supplemental material

Supplemental material

Supplemental material