Abstract
Large language models (LLMs) are quickly being adopted in educational contexts. In this editorial, we draw attention to three technically-oriented challenges educators may face using LLMs in educational settings when considering student-facing deployment: trust, privacy, and context dependence. Although these challenges are not as well-known as others, such as hallucination and interpretability, they pose significant concerns in educational settings if they are not understood and mitigated before implementation. We discuss practical challenges, risks, and research opportunities associated with each of these technical challenges.
Introduction
Generative artificial intelligence, exemplified by large language models (LLMs), is rapidly being integrated into educational settings. Early review studies are showing that LLMs may support learning (He & Cordie, 2024; Liu et al., 2025) and engagement (Heung & Chiu, 2025). Although LLMs clearly hold promise for providing individualized learning experiences at scale, they are also accompanied by a variety of challenges (Giannakos et al., 2025). Some of the widely known issues associated with LLMs are well documented, such as their ‘black box’ nature (Rai, 2020; Zapata-Rivera & Arslan, 2024) and tendency towards hallucination (Banerjee et al., 2025; Ho et al., 2024; Ji et al., 2023; Martino et al., 2023). Our purpose is not to discuss these well-documented challenges. Rather, we wish to draw educational researchers’ attention to three foundational challenges with LLMs that are infrequently discussed in educational contexts, whose existence necessitates mitigation by developers and institutions before their widespread, mainstream implementation in schools: Trust, Privacy, and the Context Dependent Nature of LLMs. We outline conceptual, high-level arguments and practical directions for research in relation to Privacy and Trust, while providing more narrow recommendations on how to embrace the Context Dependence of LLMs. We argue that these three conceptual areas, Trust, Privacy, and the Context Dependent Nature of LLMs, represent substantial future research opportunities for educational researchers.
Trust
Research is direly needed with regards to the trustworthiness of LLMs (broadly conceived) and how users build trust in LLMs in educational settings. Trust in artificial intelligence is a complex concept and recent conceptualizations of trust in automated systems have predominately viewed trust from a relational lens (Chiou & Lee, 2023; Goldshtein et al., 2025). Relevant areas of research include truthfulness, safety, fairness, robustness, privacy, machine ethics, transparency, and accountability (Huang et al., 2024).
Educators need to understand that a user’s trust in LLMs can be influenced by how the model was trained and how it is prompted. For example, it is common for models to show sycophantic tendencies, which is when they endorse or take on a user’s perspective even when the user’s perspective is factually inaccurate or ethically inappropriate (Cheng et al., 2025; Fanous et al., 2025; Khan et al., 2024; Malmqvist, 2025; Sun & Wang, 2026). Sycophancy is not limited to factual information, as LLMs can also have sycophantic tendencies around subjects that are open-ended, or where there is no ground truth (e.g., a user asks for advice), and could “reinforce harmful implicit assumptions, beliefs, or actions” (Cheng et al., 2025, p. 1). The potential harms of sycophantic behavior should thus be clear to any educator, particularly educators who work with students who may be in vulnerable mental states, such as experiencing anxiety or depression. Unfortunately, sycophancy is not yet being broadly discussed in educational contexts.
Recent research is showing that sycophancy, along with other aspects of LLMs such as friendliness, can influence trust (Sun & Wang, 2026). As researchers move forward with LLMs in classroom settings, it will be important to develop a deeper understanding of how learners, teachers, parents, and administrators develop trust in LLMs and in what contexts this is the case (Goldshtein et al., 2025). Moreover, it will be important for researchers to continue to investigate how to eliminate, or at minimum mitigate, traits like sycophancy that can reiterate harmful beliefs and actions. Until sycophancy is well understood and can be consistently mitigated, we encourage researchers and educators to deploy LLMs cautiously, increase students’ AI literacy (Long & Magerko, 2020; Ng et al., 2021a, 2021b; Zhang et al., 2025) to increase students’ critical thinking when interacting with LLMs, and adopt LLMs with appropriate guardrails (see Liu, Zhao, et al., 2025; Liu, Zhao, et al., 2025).
Privacy
Some may envision future schools where each student will work with their own personalized LLM-based learning partner (Sharma et al., 2025). Although conceptually intriguing, data privacy, among other ethical, pedagogical, and logistical challenges, becomes a serious concern. Although logged data from LLM-based interactions are often deidentified for storage, deidentified data has been reidentified in the past (Yacobson et al., 2021), representing a serious risk.
We support a privacy by default approach to LLM deployment in schools where student interaction data with LLMs never leaves their personal device, with options for zero data retention. We believe this is one of the strongest approaches currently available to protecting student LLM-related data privacy in schools.
We recognize that some may view privacy by default as too restrictive to research, particularly for those who want to study how students interact with LLMs. When the collection and aggregation of student-LLM chat data may be necessary for research (e.g., collecting a corpus of dialogue for fine-tuning or analysis), we encourage researchers to be mindful of the end-to-end data pipeline. For example, many research studies leverage proprietary LLMs served via API by external corporations or entities. This means that the data are generally processed on infrastructure owned by those outside of the research team. Are there guarantees said data are secure, with zero data retention in the areas of the data pipeline outside of the researchers’ direct control? A preferable alternative from our perspective are learning systems that leverage small, specifically-trained, open-weight models that run on the user’s computer when they interact with web pages. This technology already exists (Chen et al., 2025; Ruan et al., 2024). Should this not be feasible, it is quite possible to run open weight LLMs on either school district or university infrastructure where the data pipeline can be controlled end-to-end by the research team. Given researchers’ increased focus on small, efficient, privacy-focused models, it seems reasonable that these are near-term, if not immediately plausible solutions to help mitigate the data privacy problem and represents a substantial research opportunity for those interested in furthering the development or training of such models. The benefits of small language models for education have been argued elsewhere (Allison et al., 2025; Wei et al., 2025), but are seen by many as the future of private artificial intelligence.
Privacy by default and the use of small, open weight LLMs are only the tip of the proverbial iceberg for how one could protect student privacy when chatting with LLMs. There are multiple potential solutions to these privacy-related challenges, but the vast majority require additional research in educational settings. We encourage researchers to not only explore the use of open weight models and datasets for educational use cases, but also to develop and release those models for educational use cases.
Context Dependence of LLMs
It is well documented that LLMs can be very context dependent and sensitive to the prompt (B. Cao et al., 2024; C. Cao et al., 2025; Mannekote et al., 2025). As such, it may be inadvisable to develop generic, one-size-fits-all prompts. However, there are various ways researchers can leverage this context dependence to provide educators with evidence-based tools they can use in their classrooms. These can include fine-tuning open models, publishing open data sets for training models, and publishing educationally relevant benchmarks for testing LLMs. We briefly explore each of these ideas below.
Fine-Tuning Open Models
One should understand that all LLMs are inherently biased by the data and policies they were trained on. This in itself is potentially problematic, as those training the model can position their models towards narratives they prefer by steering the training in that direction. Educational researchers can perform some types of LLM training with relative ease. For example, fine-tuning is a post-training strategy that can help an LLM refine its understanding of specific domains by optimizing its parameters (Wu et al., 2025). Fine-tuning can curve the aforementioned types of biases in training data towards the fine-tuning dataset (or the researchers’ preferred bias based on the data used) to guide model performance (Scarlatos et al., 2025); however, there are no guarantees that this completely alleviates this bias. For example, we can bias an LLM towards using motivational language or providing scaffolding rather than explicit answers, but there is no guarantee that it will be able to replicate the behavior(s) consistently throughout a long conversation (Schroeder et al., 2026).
A challenge educational institutions currently face is that there are a limited number of LLMs that are focused on educational tasks 1 . We are unaware of any models that have been pretrained specifically on curriculum-relevant materials 2 . This problem is more complex than it sounds at face value, and research is needed to develop educationally-grounded and theory-guided LLMs. We have decades of evidence about the effects and important features of intelligent tutoring systems (Kulik & Fletcher, 2016; Ma et al., 2014; Steenbergen-Hu & Cooper, 2014; VanLehn, 2011) and educational scaffolding (Belland et al., 2017; Kim et al., 2018) that can help guide the types of feedback, scaffolding, and other pedagogical communications an LLM could use.
These challenges present substantial opportunities for researchers interested in the development of effective LLMs for educational use cases. While pretraining LLMs on the scale of large corporations (e.g., OpenAI, Anthropic) can be prohibitively expensive, fine-tuning existing models is inexpensive, well understood, and well documented (Liu, Zhao, et al., 2025). Fine-tuning LLMs may allow the user to align the LLM with what they feel are more desirable outputs (Anisuzzaman et al., 2025; Zhang et al., 2024), but may not be the perfect solution in all learning contexts (Liu, Zhao, et al., 2025). For example, potential lines of research include aligning LLMs to follow the theory-driven lessons learned from intelligent tutoring research.
Publishing Open Data Sets and Educationally-Relevant Benchmarks
A critical barrier to successful fine-tuning of LLMs for educational use cases is the lack of representative open educational data, especially the interaction data between students and LLMs. While a lack of representative, open educational datasets is a long-standing problem for educational technologists, this is a two-fold problem in relation to LLMs.
First, fine-tuning LLMs requires datasets that are essentially examples that the LLM can learn from. In educational use cases, this could be an example of student questions and appropriate responses. The development of authentic data sets for LLM fine-tuning should be a priority for the field going forward, as synthetic (LLM-created) datasets are unlikely to perfectly mirror evidence-based pedagogy. A necessary question then becomes, how do we create such a dataset in a way that protects the users’ privacy? This is not a new problem, spans many domains, and can be influenced by regional factors and regulations. We encourage researchers to review not only relevant regulations, but also the extensive literature in the area (e.g., Alter & Gonzalez, 2018; Carvalho et al., 2025; Meyer, 2018; Morehouse et al., 2025; Ursin et al., 2019) and techniques researchers can use with their data before sharing, such as differential privacy (Dwork, 2025; Kabir et al., 2025).
Second, it is impossible to empirically document the effectiveness of a fine-tune without some sort of metric. This is where the development of educationally-relevant LLM benchmarks with education-appropriate usability metrics are direly needed. Again, the development of appropriate benchmarks will require the publication of open data sets of relevant data.
Creating educationally-relevant benchmarks is not a trivial task and represents an important research opportunity. Based on our on-going work in the area, we offer two considerations for those interested in creating educationally-relevant benchmarks: they should be designed with sufficient complexity to prevent perfect scores from the “best” current models, and they should account for a variety of factors (e.g., pedagogical alignment, sycophancy) throughout realistic, multi-turn, long-context conversations (Schroeder et al., 2026).
First, we argue that single-turn conversational evaluations/benchmarks may not be very useful in an educational context if the goal is measuring how well a model can follow system-prompted formatting (e.g., well-defined, theoretical alignment) because many open models can follow basic system-prompted instructions for the first response. A critical question becomes, will it retain its behavior over a multi-turn, longer-context conversation? This is particularly relevant due to what is colloquially known as “context rot”, where LLM response accuracy tends to degrade as the context length increases (Du et al., 2025; Li et al., 2024). Researchers are beginning to find ways to help mitigate this (Du et al., 2025) but we are unaware of any research around this in educational contexts.
Second, an effective educational benchmark should consider how teachers provide feedback to students in classrooms. Rather than using a single specific, rigid, evidence-based framework for responding, teachers may often choose how to respond based on various forms of evidence-based pedagogy (choosing the best approach for the situation at hand), their understanding of the students’ question(s), and the individual student. We encourage researchers to consider how to approach creating benchmarks that evaluate how well LLMs can communicate in a more natural way, mimicking the intricacies of a human teacher-student conversation over multiple turns. An important component of such a benchmark would be a consideration of sycophancy or hallucination if a student pushes back against the model’s opinion, especially in longer context conversations.
Although these research challenges are not easily solved, they are approachable with current technologies and represent much-needed areas of research. The key to solving these challenges lies in the availability of representative, open data sets and open models. As more educationally-relevant data are published, researchers can leverage them to help create precise, realistic evaluation metrics that enable us to create better models for educational use cases.
Conclusion
There are substantial internal and external pressures for K-12 educational institutions to integrate LLMs into students’ learning. Although these pushes are understandable given the promises of LLMs, we feel it is important that educators use these tools with their “eyes wide open” to the advantages and disadvantages of current LLMs. Promises are not evidence, and much evidence in the peer-reviewed literature could be considered preliminary and not broadly generalizable, given that sample sizes may be small, experiments not well-controlled for confounds, and LLMs have only been broadly implemented for less than three years. We must keep in mind that LLMs are not unlike the breakthrough educational technologies of the past. For example, massive open online courses (MOOCs) held promise to democratize education for all (Dillahunt et al., 2014). However, over time, research has shown that while MOOCs provide specific benefits, they are not without their own pitfalls. While many educational researchers have been focused on pedagogically-oriented topics with LLMs, such as how LLMs can more effectively provide feedback and scaffolding, we encourage researchers not to ignore the more technical challenges accompanying LLMs in the hopes that they can be resolved or, at a minimum, improved and mitigated, before widespread deployment in schools.
Footnotes
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is based upon work supported by the National Science Foundation and the Institute of Education Sciences under Grant DRL-2229612. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the U.S. Department of Education.
Declaration of Conflicting Interests
The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Noah L. Schroeder serves on the editorial board of the Journal of Educational Computing Research. He played no role in the peer review process.