Abstract
From late spring of 2020 to the present, the U.S. educational conversation about learning in K-12 public schools during the pandemic has been about the fact that students did not learn as much, widening the achievement gap due to digital learning. Some K-12 public schools and school districts have reported no learning loss. This exploratory study examined elements shared by teachers, administrators, and parents that accounted for potential reasons why these individual K-12 public school districts did not experience the learning loss during digital learning that has been at the center stage of the narrative about educational experiences during the pandemic. Analysis of the results showed four elements that were central to these school districts’ experiences: having an instructional framework; determining a clear pedagogical function; connecting technology to the pedagogical function; capitalizing on formative assessments. Data also revealed that the participants tended to believe that a focus on pedagogy before technology integration was crucial.
Introduction
In March of 2020, U.S. states began to implement shutdowns and stay-at-home orders to prevent the spread of COVID-19 (CDC, 2023). Almost overnight, K-12 schools were challenged to reinvent how schooling was done, from face-to-face instruction to full-blown digital learning. Parents and educators alike were challenged to ensure their students were connected digitally and had appropriate materials and software to learn (Turner, 2022). Over the next few months, educators began speculating about the educational experiences that K-12 students had while learning digitally and that achievement would be impacted negatively. Prior empirical study suggests that digital learning is effective in supporting student achievement (Colvin et al., 2014; Means et al., 2010). However, literature on the effects of the pandemic shows that the educators were often correct: educational achievement of most students during the pandemic decreased, resulting in learning loss for students (see Kuhfeld et al., 2020).
Learning loss refers to the general loss of knowledge or skills, often due to a gap or discontinuity in students’ educational experience (Ingram, 2022; Stoltzfus, 2021). From late spring of 2020 to the present, the U.S. conversation about K-12 public school learning during the pandemic has been about how students did not achieve as much, due to digital learning (Sparks, 2023). Empirical study has shown that student learning loss in core subjects as measured by standardized tests places students anywhere from three months to a year behind (Dorn et al., 2020; Kuhfeld et al., 2022; Levenson, 2020; NAEP, 2023; Patrinos, 2023; Sparks, 2023; Turner, 2022).
However, the most prominent study by Fahle et al. (2022), indicates there were K-12 public school districts that did not experience learning loss due to digital learning. While the pandemic affected all students, it did not affect all students equally, especially given the fact that digital learning has been shown to be effective in supporting student achievement (Beck, 2022; Colvin et al., 2014; Means et al., 2010). Specifically, Kane states that the “pandemic was like a band of tornadoes that swept across the country. Some communities were left relatively untouched, while neighboring schools were devastated” (Spector, 2022, p. 1).
This exploratory study focuses on U.S. K-12 public school districts in the Midwest that still managed to academically achieve and avoid or mitigate learning loss despite the digital learning format during the pandemic. When discovering that these districts’ data in relationship to previous cohorts were equivalent and, in some cases, higher to what these districts experienced prior to the pandemic, it becomes imperative to find out what these K-12 public school districts did with digital learning that they believe accounts for their achievement (Fahle et al., 2022). This study therefore focuses on the elements that educators identified in their individual public K-12 schools and/or districts that allowed them to avoid the learning loss that has been at the center stage of the narrative about educational experiences during the pandemic. As the following literature review will describe, to date, there is limited research on the elements that individuals within these K-12 public schools believe helped their students avoid or reduce learning loss.
Literature Review
This section begins with a discussion of possible definitions of learning loss. Next, the research on learning loss and how some districts experienced it while others did not, even though all students were digitally learning, is presented. This follows with relevant literature on the effectiveness of digital learning and ends with gaps in the literature while describing next steps to help address the identified gaps. This review was conducted by searching education research databases such as Academic Search Complete, ERIC, Education Full Text, EBSCO, and various journals, with combinations of key words such as learning loss, digital learning, and/or pandemic.
Definition of Learning Loss
As Chen and Krieger (2023) state, “learning loss” has become the new buzzword in education following COVID-19. There is no single definition that is common in the literature or even in general usage (Stoltzfus, 2021). In the most general sense, learning loss refers to the urgency to make up for what was lost or a yearning for missed opportunities (Asadullah et al., 2023). This could be as simple as the fifth-grade class did not experience the annual field trip to the first-grade team that missed an entire unit on CVC words. Learning loss in this qualitative sense is a longing for something that is typically experienced (Turner, 2022).
In a more quantitative sense, learning loss can refer to the decline in K-12 public school academic achievement as measured by achievement tests (Fordham Institute, 2023). Specifically, this refers to “declines in expected rates of academic progress” often expressed as standard deviations below a performance on an assessment in a given year (Fuchs, 2023, p. 278). This type of learning loss can also venture into areas related to student growth. The Great Schools Partnership indicates that lower quality teaching can lead to slower academic achievement which produces learning losses in relation to where students are expected to be at a specific point in time, for example, as calculated by a value added metric (Chetty et al., 2014; Great Schools Partnership, 2013; Kane et al., 2013; Koedel et al., 2015). Studies performed by SAS analytics found that highly effective teachers can teach students up to a year and a half (or more) of content in a single year, while others may teach students only a half year of content (Rivers & Sanders, 1996; SAS Institute, 2021). If students receive poor quality teaching over multiple years, learning loss compounds, decreasing the students’ chances of catching up with their peers or completing school (SAS Institute, 2022).
This more quantitative definition of learning loss often raises concerns from educators as the learning loss is based on standardized tests (Cortes-Albornoz, et al., 2023; Ingram, 2022; Zhao, 2022). Specifically, educators question whether standardized tests are reliable, their results valid, and/or ultimately whether such tests can measure learning that is not necessarily quantifiable (Harris et al., 2011; Popham, 2017; Zhao, 2022). Specific to the pandemic, Chen and Krieger (2023) state because the experiences in which students’ learning unfolded were different and students simply learned different things that parents, teachers, educators, etc., might not be accustomed to them learning or could even be measured by a test, the interpretation of achievement data from the pandemic could be circumspect.
For the purposes of this study, learning loss refers to the general sense of loss of knowledge or skills, often due to a gap or discontinuity in students’ educational experience (such as that caused by a pandemic), that is confirmed by some evidence-based measure. These evidence-measures can be any formal assessment that an educator uses to judge progress on student achievement and/or growth. The evidence-based measure is included in this definition for two reasons. First, these metrics are a part of our national educational story and often regulated by law, despite concerns with their usage (ESSA, 2015). Second, and perhaps more important, is the fact that experts believe educators need both qualitative and quantitative data to truly determine what is happening in schools (Chappuis et al., 2021; Knips et al., 2022; Sharratt & Fullan, 2022).
Studies on Learning Loss
This section is organized into three parts. First, smaller studies that indicate that learning loss exists as well as studies that suggest that something must be done about perceived learning loss in U.S. K-12 public schools are discussed. Next, the first comprehensive empirical analysis showing learning loss in the general student population in U.S. K-12 public schools is examined (Kuhfeld et al., 2022). Finally, the critical study by Fahle et al. (2022) and subsequent commentary by Riordan and Kane (2023) that showed that in some K-12 public schools students experienced learning loss, while in others there was none, is addressed.
During 2020 and 2022, state departments of education published twenty-eight studies that indicated that learning loss existed based on summative test results for mathematics and language arts. Participation rates on those assessments range from 55% to 98%. Drops in mathematics scores across all states range from 1% to 20% while drops in language arts were anywhere from 0% to 17%. For example, the Tennessee Department of Education (2021) reports that English Language Arts scores dropped in overall proficiency five points and that mathematics dropped 12 points. Another study in Ohio (Kogan & Lavertu, 2021) indicated that achievement declined .23 standard deviations, proficiency was down 9%, and learning loss was 50% larger for black students compared to white students. There was no conclusivity to explain why the scores dropped.
Similar studies were also completed by curriculum/assessment publishers. For example, in Renaissance’s (2020) study on 3.27 million reading students and 2.02 million students in mathematics across all fifty states, they concluded that in reading, students could be as far as 7 weeks behind and as far as 12 weeks or more in mathematics. Curriculum Associates (2020) in their study on 1.1 million U.S. K-12 students in reading and 1.2 million in mathematics indicated that the percentage drops ranged from 10% to 16% in grades 1 to 5; they indicated there were fewer students on grade level as compared to prior years. There was no common reason for what accounted for learning loss based on these studies. Ideas ranged from digital learning, to lack of access to the internet or technology, to differences in socioeconomic status, to even a lack of professional learning for teachers.
In searching for research on learning loss, many studies and articles suggested that learning loss needed to be addressed; these studies and articles did not empirically conclude if learning loss was even present. For example, in a study by researchers Chinna and Sunkesula (2023, p. 1), ways to address learning loss in mathematics were based on what the researchers noticed: “the majority of young learners lost their basic literacy and numeracy skills.” Hanushek (2023) called for the importance of recruiting the best teachers to deal with the pandemic learning loss but did not explain where learning loss had emanated from nor how learning loss was determined to exist and Royan et al. (2022), used test-to-stay programming, a type of COVID-19 monitoring, as a potential way to minimize learning loss.
Realizing that many of these aforementioned studies did not rigorously quantify information, Kuhfeld et al. (2022) created a national study that was the first to conclusively determine that learning loss existed across the nation. Kuhfeld et al. (2022) compared reading and mathematics performance in the fall of 2019, 2020, and 2021 using the Measure of Academic Progress Growth (MAP) test. Across the sample of 5.4 million U.S. students, mathematics achievement in the fall of 2020 was −0.12 to 0.18 standard deviations lower than the fall of 2019. The drop from 2021 to 2019 was −.21 to −.27 standard deviations. In reading, although performance at almost all grades increased from 2019 to 2020 (ranging from a decrease of −0.02 to increases of +0.05), declines of −0.09 to −0.18 SDs were observed by fall 2021. In disaggregating the data, Kuhfeld et al. found that achievement gaps between students in high-versus low-poverty schools had widened by fall of 2021. Using test scores, the team also concluded that the drop in numbers was alarming, but that the adverse findings should not be a reflection on teacher quality or effort, but rather the conditions of learning, that is, digital learning.
A final seminal study on learning loss, referred to as the Education Recovery Scorecard and spearheaded by Harvard and Stanford Universities, determined that the average U.S. public school student lost the equivalent of a half of a year of mathematics and a quarter year in reading, but that there are districts, between 3% and 6%, that saw mathematics achievement rising (Fahle et al., 2022). The study further confirmed that the poverty rate is very predictive of how much students lost and that the pandemic widened disparities between high and low poverty schools. Lastly, and most importantly, the study indicated that in some school districts students fell behind by as much as a grade level, but that in other districts there was no decline or students outperformed pre-pandemic levels. The authors did not examine the ways in which digital learning was implemented to explain why some districts experienced loss and others did not. The authors also note there are specific regions (e.g., the Midwest) with the least fluctuation and that more research should be done to explain how digital learning was implemented in the schools within these regions.
In summary, it appears there was some learning loss in some K-12 public school districts as measured by standardized tests or curriculum-based assessments, but in others there was no learning loss. Several of these studies identify that digital learning may be responsible. The Education Recovery Scorecard study indicates that it is important to find out the ways in which digital learning was implemented to explain why some districts experienced loss and others did not. Accordingly, since these studies reference digital learning, studies on the effectiveness of digital learning were explored next.
Studies on Digital Learning
In examining the literature on digital learning in K-12 U.S. public schools, two relevant areas were examined. The first area discusses empirical studies that suggest that digital learning is effective. The second area addresses theoretical pieces that examine strategies that educators should use to implement digital learning. In the first area, effectiveness of digital learning, approximately twenty-five studies were found. These studies ultimately concluded that digital learning can be effective. In a meta-analysis by Means et al. (2010), it was determined that “Students in online conditions performed modestly better, on average, than those learning the same material through traditional face-to-face instruction” (p. xiv). Specifically, they point out that students can learn up to five times more material online than in a face-to-face course (Means et al., 2010). A second study by MIT showed that online learning was as effective as a traditional course regardless of how much preparation and knowledge students start with (Colvin et al., 2014). A more recent study showed that students enrolled in a virtual learning environment prior to the pandemic outperformed students experiencing remote learning for the first time during the pandemic (Beck, 2022).
In addition to these empirical studies, there are many theoretical pieces that discuss strategies that educators should use to implement digital learning. For example, Hughes and Roblyer (2023) concluded that being a connected educator through technologies like X, formerly known as Twitter, help teachers collaborate and build relationships that ultimately help students. Drost (2016) identified that using digital tools such as interactive presentation software through formative processes helps to motivate students and increases learning. Miller (2020) discussed games, video, and audio to improve learning.
In summary, both empirical and theoretical research suggests that digital learning is effective. Yet, from the prior section in this literature review, some U.S. K-12 public school districts experienced learning loss and others did not. Fahle et al. (2022) indicate that it is important to find out the ways in which digital learning was implemented to explain why some districts experienced loss and others did not.
Gap in the Literature
In searching education research databases such as Academic Search Complete, ERIC, Education Full Text, EBSCO, and various journals, no empirical studies on schools and school districts that did not experience learning loss and the ways in which they implemented digital learning were found. This exploratory study was designed in response to this lack of empirical research. It examines elements shared by teachers, administrators, and parents that they believe accounted for why their individual K-12 public school districts did not experience the learning loss that has been at the center stage of the narrative about student educational experiences during the pandemic.
The Study
This exploratory study attempts to determine what teachers, administrators, and parents believe accounted for a lack of learning loss in their context. It sought to answer one research question: What elements account for a school district’s lack of learning loss during digital learning in the pandemic?
Methodology
This study used an exploratory methodology to guide inquiry. This “broad-ranging, purposive, systematic” methodology maximized “the discovery of generalizations leading to description and understanding” (Stebbins, 2011, p. 3). Exploratory research is the opposite of confirmatory research; its purpose is to gain some initial understanding into the research question so additional studies can be later developed to build a more conclusive theory. In research contexts where there is little understanding and when there has been no systematic empirical inquiry, the exploratory study design is preferred since it focuses on “emergent generalizations” (pp. 6–7).
Due to the nature of exploratory research, the generalizations that result will not be transferable, nor is that the desired outcome (Stebbins, 2011). As Stebbins articulates, exploratory research is not concerned with all possible situations; it instead focuses on learning what it can from the given constraints, especially in situations where there is limited research. As such, there are no randomized controlled trials.
Participants included in this study were teachers, parents, and administrators in U.S. K-12 public school districts. Each participant taught, oversaw, or interacted with either language arts or mathematics as these are the two areas that are the focus of the learning loss narrative in the U.S. In addition, all participants needed to be from the Midwest as this is the area of the country that experienced the least fluctuation in achievement data (see Fahle et al., 2022).
Examples of Questionnaire and Semi-Structured Interview Questions.
Eighty-eight original questionnaires were completed, but nineteen were not included in the data set (ten had a substantial number of non-responses, two were from a non-U.S. institution, six were from private institutions, and one was from a higher education institution). Following an initial analysis, an additional request was sent to willing participants (those that had completed the questionnaire and agreed to be contacted) to see if they could help provide additional contacts within their sphere. For example, in a case where a principal responded, this principal was asked to see if they had a teacher and/or parent who would be willing to participate. This brought the total number of questionnaires completed to one-hundred forty-eight. All participants that were included must have indicated on one of the questionnaire questions that their data metrics were the same or higher from before the pandemic.
The final data set included thirty-one public school districts with ninety-three participants in schools throughout the Midwest region of the United States. As is typical with research, perfect sampling is impossible, and specifically in exploratory research, perfect sampling is irrelevant. As Stebbins (2011) indicates, the goal of exploratory research is to build understanding into a research question rather than empirical certainty, so it is not an issue that not every type of school district, grade, participant, or content area was present. Twenty-five of the school districts had at least one parent, one administrator, and one teacher who completed a questionnaire; four school districts had two of each; two school districts had two teachers, one parent, and one administrator. There were a total of 30 parents, 34 teachers, and 29 administrators (e.g., principal, assistant principal, curriculum director, etc.). Of the teachers, sixteen worked with mathematics, thirteen worked with language arts, and two teachers worked in both content areas; all administrators oversaw language arts and math and parents indicated they were familiar with both. Teachers and administrators were evenly distributed between grade bands with seven or eight of the participants representing grades K to 2, 3 to 5, 6 to 8, 9 to 12. Teachers and administrators represented various years of experience, from 2 years to 33 years. Parents represented each of the grade bands, but in some cases, a parent could represent multiple grade bands as they had several children.
Sixty-two percent of the participants were female, 34% of the participants were male, and 4% chose not to identify. Seventy-five percent of the participants were Caucasian, 10% were African-American, 3% were Asian, 4% were Hispanic and the remainder chose not to identify. Thirty-nine of the school districts self-identified as suburban school districts, 36% urban, and 25% rural. Ultimately, this distribution of participants is very similar to what is found with the National Center for Education Statistics (National Center for Education Statistics). One commonality that appeared from the demographic section is that the participants had minimal technology setup issues at the start of the pandemic (e.g., students were already one-to-one or connectivity was high). This is similar to what Riordan and Kane (2023) found.
In addition to questionnaires, semi-structured interviews were also conducted with fifty-eight participants to learn more about their questionnaire responses (Adams, 2015; Stebbins, 2011). Thirty-three participants were female, twenty-three were male, and two chose not to identify. Forty-one participants were Caucasian, six were African-American, three were Asian, three were Hispanic, and five chose not to self-identify. Twenty-one participants were teachers, eighteen were administrators, and nineteen were parents, roughly representing 1/3 of the distribution. Of the twenty-one teacher participants, nine represented mathematics, ten represented language arts, and two taught both. There were between thirteen and sixteen participants representing each of the grade level bands, K to 2, 3 to 5, 6 to 8, 7 to 12.
These semi-structured interviews asked 10 to 12 questions and were aligned to the questionnaire sections. A flexible design (Rubin & Rubin, 2012) was used to engage with the participants in a way that helped them share their experience regarding the digital learning their students received. The participants chosen were selected for two reasons: (1) they expressed interest in sharing additional details; and (2) the researchers wished to probe into the various statements that the participants said or wrote to fully gain insight into the research question. Their responses provided a deeper understanding of the elements that the participants believed limited their school/district’s learning loss. Table 1 provides sample questions from the semi-structured interview questions for reference.
Examples of Data Coding.
In terms of the positionality of the research team, one member of the research team is a university professor and former K-6 teacher who teaches classes in pedagogy with more than twenty years’ experience. The other is a central office administrator, former K-12 teacher, and university professor with 20 plus years of experience. Both researchers have worked in suburban, rural, and urban contexts in the U.S. at a variety of grade levels. Following Holmes's (2020) approach to positionality, the researchers are in-betweeners, having explicit knowledge of what happened in K-12 schools during the pandemic, but did not teach in a K-12 public institution; both were teaching at the university during the pandemic. No personal data from the research team’s context was included in this study, but in one case a participant was known to one of the researchers. Lastly, this research team has successfully used exploratory methodology in two large scale studies (see Drost & Levine, 2015, 2023).
The research team worked to minimize bias where they could, knowing that all research is ultimately biased. This was accomplished by clearly documenting and describing the research process, ensuring equitable distributions of participants, having the results triangulated by asking the participants about the results, and comparing the results with known research.
Results
This study was conducted through a particular interpretation of exploratory study (Stebbins, 2011); data analysis proceeded through recursive stages of data gathering, open coding, memo writing, and analysis. The results are presented in response to the research question and organized around four data labels that were determined to be key elements that our participants shared.
What Elements Account for a School District’s Lack of Learning Loss During Digital Learning in the Pandemic?
This exploratory study examined elements shared by teachers, administrators, and parents that account for potential reasons why these individual K-12 school districts did not experience the learning loss that has been at the center stage of the narrative about educational experiences during the pandemic. The elements identified are aligned to four major data labels: (1) having an instructional framework; (2) determining a clear pedagogical function; (3) connecting technology to the pedagogical function; and (4) using the formative assessment cycle. It is important to note that there was no large differential (2%–4%) on the responses between language arts and math.
This first element, having an instructional framework, is about the methodology a teacher uses to plan instruction aligned to common units of instruction. Specifically, it means that a teacher plans lessons in a particular manner and with identified strategies that the teacher and students use to achieve common learning targets (Toth, 2022). It is about ensuring that every teacher knows what an effective lesson looks like in a school or district to ensure a guaranteed and viable curriculum where common units of instruction are articulated across a grade, course, etc. (DuFour & Marzano, 2011). It does not mean that the art of teaching is lost or that teachers follow scripts. Some typical frameworks are the 5E model (NASA, 2020), Understanding by Design (UBD) (Wiggins & McTighe, 2005), or even Madeline Hunter’s Model (Hunter, 1982).
The second element, determining a clear pedagogical function, is about being explicit about the way a teacher wants students to go about learning the material during a particular part of the lesson. This element is not about the procedures or activities that the teachers use, but rather the ways in which students and teachers work with the content as they are developing the lesson to meet the learning goal. Examples of pedagogical function might be to explain concepts or to show students how to do something, to brainstorm, or to have students practice concepts.
The third, connecting technology to the pedagogical function, is about thinking about the pedagogical function first and then choosing the right digital learning tool to support it and not the other way around. It refers to teachers using the most appropriate digital tool for the pedagogical function at hand.
The final element is about utilizing formative assessment. Formative assessment is any activity that a teacher uses to make instructional adjustments and to provide feedback to students in relationship to a learning target (Drost, 2014). While students are engaging in the learning process, teachers make moves (Duckor & Holmberg, 2017) to effectively drive a lesson so that all students understand.
While these labels are not comprehensive in expressing every nuance, they do portray the core features of the data set. The labels were determined by finding repetitive key statements by the participants and concepts expressed within the data; they were checked with the participants for accuracy. Refer to Table 2 for examples of the labels.
Element 1: Instructional Framework
As shared earlier in this section, an instructional framework is a tool to help teachers organize lesson design in a particular manner and helps identify strategies that the teacher and students use to achieve common learning targets. It is adaptable enough to meet varying teaching styles and student needs and ensures that the curriculum can be guaranteed and viable.
This element was identified in approximately 70% of participants’ responses as a key element that they believe accounted for the lack of learning loss in their building or district. This element was coded in approximately 93% of the administrators’ responses, 70% of the teachers’, and 35% of the parents'. As an administrator described, “Using the 5E framework in all subjects really helped [with learning loss] as it provided consistency and certainty that all teachers were providing the most effective instruction.” In discussing this with the teacher from the same district, the teacher further clarified and said, “without an instructional framework, teachers would be left to figure things out on their own. 5E which we’ve used for decades gave us the consistency in how to plan and just like face-to-face, ensuring student learning as we knew what to do.”
Another teacher from a different school echoed the same thoughts: “As team leader, the first thing I did in March was talk to my grade-level peers and [I] asked them how we should plan our lessons. We continued using UBD as we knew it was effective. The UBD framework allowed us to talk about how we could immediately improve our lessons that first few months when it seemed like we were scrambling.”
What was common from about half of the participants is that the instructional model did not need to be the same across the entire school, just that one was present. As one principal described, “We don’t expect every teacher to use the same format, but we do require a researched-based framework aligned to common units so that we know what an effective lesson looks like and can have conversations about how to improve. This was identical to what we do face-to-face and I think helped our students achieve.” This was similar to two responses. One teacher explained that “We as a team used the I-do, we-do approach and that helped keep us focused on what we needed to teach but also helped our kids stay focused” while another teacher said that there was no learning loss because “even though our model was different than sixth grade’s, instructional frameworks in high poverty cultures help create relationships with students, maximize the learning, and plan for student and teacher thinking with common expectations.”
In addition to the fact that the model did not have to be the same across the school building, six participants (both teachers and administrators) mentioned that they did change models during digital learning. As one teacher explained, “as we learned to adapt online, we also changed our instructional framework from I-do, We-do, to UBD as we found that it was more robust and helped us teach more effectively as our data showed.” One administrator mentioned that his school implemented a framework during the first few months, something they never had done before, to ensure that “everyone got the key parts of the standards met and to help lesson [sic] the workload.”
The responses from parents in relation to this element were varied. Approximately half of the parents specifically mentioned that an instructional framework was helpful in reducing learning loss. While they were not necessarily familiar with this term, they were able to describe it as illustrated by this quotation: “I think my kids were able to learn more effectively digitally because all of the lesson[s] were set up the same way with an explore, engage section. It allowed them to focus on learning [,] not tech…” Some parents noticed that as they were watching classes, pedagogical frameworks seemed to jump out at them but were unsure if this was something that had always been done in their students’ schools. For example, this parent stated that “I would talk to my brother and describe what I was seeing in Blackboard with these sections labeled Explore, Elaborate, Evaluate … He had no idea what I was talking about. When I talked to the teacher, I learned that this was a way of helping all students learn at higher levels. It seemed like it was really effective…” Another parent shared similar thoughts but added, “while the same format was there, it wasn’t robotic. I could see teachers being creative as they were doing things, making the learning come alive and helping the learning stick for my son and daughter.”
In summary, the importance of this element is crucial because it sets the stage for teachers to create lessons that connect to proven practices that help students meet or exceed grade-level expectations as well as have a way to discuss the practice of designing lessons. As one principal described, “learning loss didn’t exist for us because we had a researched-based instructional framework with a common scope and sequence. It helped us keep pace and reduce gaps. It helped us focus on what we can control, discuss how to make improvements and ultimately do what we do best, teach!”
Element 2: Pedagogical Function
Determining a clear pedagogical function is about the way a teacher wants students to go about learning the material during a particular part of the lesson. It is specifically about the ways in which students and teachers work with the content as they are developing the lesson to meet the learning goal. For example, one pedagogical function could be to introduce a concept: students could watch a video, read a textbook, or examine a picture. While the pedagogical function is still the same, to introduce, the activities vary. The teacher’s clarity on the pedagogical function is central to this element and what participants described as crucial to reducing learning loss.
This element was identified in 55% of participants' responses as a key element they believed accounted for the lack of learning loss in their building or district. This element was coded in approximately 66% administrators’ responses, 83% of teachers, and 27% of parents. Teachers were particularly cognizant of this: “Because the format of learning changed, I had to rethink like a first-year teacher. What was the function that I was trying to accomplish? Issues were happening when I wasn’t clear on the function, such as assessment or modeling.” Another teacher in a separate district further expanded on this: “In digital learning, I kept getting reminded by my principal about why I want[ed] them to do that activity. Once I focused on the function of the learning, I saw huge improvements in students learning during digital learning.”
A principal explained the importance of this element in this way: “Having a clear function is about the why you’re doing what you’re doing in the classroom. They teach us that in teacher prep school. I noticed though at first that teachers forgot this when we went remote. So, I reminded team leaders and asked people to start sharing them. This made a huge difference as we were now explicitly instructing.” An administrator in another school shared this, “When teachers were explicit about the method that they were using, I could see kids learn much more effectively when I would pop in on their Zoom.”
Only 27% of parents noted pedagogical functions in their responses. While this ultimately makes sense as parents might not be familiar with this in terms of instruction, one parent who informally shared that she had a background in education but never taught, stated the following about why she felt learning loss was not prevalent in her students or school: “I noticed that when some teachers were explicit about the manner they were having the kids do things, like reviewing or discussing, kids were more focused and attending to what they needed to learn.”
Element Three: Connecting Technology to Pedagogical Function
Data analysis revealed that a third element is about putting pedagogy before technology. This element suggests that educators first think about the pedagogical function first (element 2) and then find a technology tool that supports this pedagogical function. A concrete example would be to first identify that students need to review. An appropriate tool that supports that function would be to use Quizziz or Kahoot, digital review games. Using the digital tool Padlet, a digital bulletin board, may not be effective in this case.
This element was identified in 50% of participants’ responses as a key element they believe accounted for the lack of learning loss in their building or district. This element was coded in approximately 43% administrators’ responses, 90% of teachers, and 18% of parents. As one teacher shared, this element was crucial to not only student achievement but teacher success with digital learning: “It was a game changer when I realized that I needed to find an app that matched the learning rather than just trying anything that I had access to.” Another teacher confirmed this with “I thought about the way I needed to teach and then determined what technology to use. For example, I needed students to brainstorm so I used Padlet rather than Google Classroom. This really solidified our learning rather than our technology.” A third teacher further illustrated this: “I was taught to think pedagogy then technology. This helped me keep the purpose of what we were doing in mind and keep things manageable. Because we all knew what the learning was to be, the tech just flowed.”
Administrators also noticed this element. One curriculum director indicated that “I noticed our teachers really focusing on their pedagogy before thinking about the technology. I think this made all the difference because teachers were trained to teach.” A second administrator gave an example, “For my teachers that were struggling at first, they would say things like ‘I need to use Flipgrid.’ I noticed where we were succeeding were teachers who were saying ‘I need to get students to converse, and as a result should use Flipgrid. This was a huge ah-hah!” As this principal quote shows, once the pedagogical function of conversing was established, the choice of which tool to use became clear. The technology simply became the vehicle to get to the pedagogical function.
From the parent perspective, data were clear that pedagogy before technology was crucial in minimizing learning loss. One particularly illuminating comment from a participant was: “I remember hearing the science teacher say that everything was Edpuzzle. It just didn’t seem to always fit, and I don’t think my son learned. In another class where he really learned, the teacher was always clear and would say something like we’re going to collaborate and discuss, so we need to post and respond to questions on [C]lassroom.” In another example, a parent who had students in two different buildings said that “I think my kids learned because the teachers were worried or maybe focused on the goal rather than the app.” An additional parent also confirmed this with their comment “It was clear that we were focusing on learning social studies, rather than learning various apps.”
In summary, the importance of this element is crucial because it sets the stage for teachers to focus on ensuring that the tools they are using are supporting their goals tied to the overall framework of the lesson. As one administrator shared, “We thrived because my teachers identified [a] pedagogical function and then determined technology tools to support that function and not the other way around.”
Element 4: Using the Formative Assessment Cycle
Formative assessment is a teacher’s use of a wide range of activities to make instructional adjustments while providing feedback to students about their learning. This element suggests that teachers were using information about student learning they were gathering to help plan instruction and close gaps in learning. This element was identified in 42% of participants' responses as a key element they believe accounted for the lack of learning loss in their building or district. Thirty-three percent of administrators, 81% of teachers and 12% of parents mentioned assessment for learning or a type of formative assessment, such as a quick check, in their response about why they did not experience learning loss in their district/school.
Several administrators shared that they believed formative assessment practices were key to all learning, so it was critical to continue during digital learning. For example, “Students can’t master content without teachers constantly checking on their learning. Since this is so crucial, I directed teachers to strategically collect something each day to help plan next steps.” Another participant shared, “One of our bedrock principles is using assessment for learning. I asked teachers weekly how they were continuing to do that. I think this made all the difference as our common assessment results overall didn’t shift from prior years and our state tests were right where we expected them to be, if not a bit higher.”
Parents also highlighted formative assessment in their responses as well. One parent wrote, “I really appreciated the teacher’s quick checks because it helped my kid and me know what to focus on next. The feedback was super helpful…my kid even said that it made learning easier, and I think it helped him remember things…” Another parent also shared that “teacher feedback really helped because my daughter knew what to study or pay attention to or ask for clarification from the teacher.”
Most interesting were the teachers’ responses. Several teachers shared that they capitalized on formative assessment processes and believed this reduced learning loss. For example, one participant stated, “Because everything had to be turned in generally, I could use everything to figure out exactly what each kid needed next and then use some differentiated instruction.” Another said, “I really had to directly plan rigorous formatives as I couldn’t always see their faces and could close learning gaps.”
On the other hand, four teacher participants mentioned that although they valued the formative assessment process, it became more challenging during digital learning: “Because I wasn’t seeing them do the work, I wasn’t sure if it was actually their work, so I had to change the way I did formatives.” In probing into this result, one participant said they had to adjust their formative moves: “Because I wasn’t sure if they were cheating, or just googling everything, I had to ask better questions that helped me identify what they were learning and where there were gaps in learning.”
Discussion
This exploratory study found four emerging elements that potentially account for why certain schools did not experience learning loss. These areas were identified in some capacity by all three participant groups: administrators, parents, and teachers. It serves as the major finding of this study and suggests the idea that pedagogy must come before technology.
Pedagogy coming before technology is not a new idea, but it is one that is likened to the age-old chicken and the egg conversation. Given the data, pedagogy should come first because the focus of instruction becomes the academic learning objective rather than learning the technology tool. As the TPACK (Mishra & Koehler, 2006) model of technology integration describes, technological pedagogical content knowledge is crucial. This means that a teacher knows the content they teach and the instructional strategies that will be effective in teaching that content. Then, the teacher determines the technology that needs to be integrated to be effective, not before.
In checking with our participants to determine the accuracy of this finding, nearly 90% agreed that this was primary and ultimately what reduced learning loss for their students. This perspective was exemplified by one participant’s quote, “When I focused on technology, learning suffered. When I focused on pedagogy, learning soared.” Empirical and theoretical research also suggests that pedagogy needs to come before technology (Hughes & Roblyer, 2023). As Kilbane and Milman (2023, p. 21) state, “the goals of instruction … should drive the design of learning experiences and technology’s integration within it.” This data also anecdotally has matched our own individual experiences as teachers/professors. When we lose sight of pedagogy because we are too busy focusing on technology, our instruction suffers. This is similar to the new teacher who thinks students have learned because they had fun in a lesson (Drost & Levine, 2015). When teachers are clear on their pedagogy, they make learning clear for students (Archer & Hughes, 2011; Good & Brophy, 2003). When clear learning is coupled with authentic formative assessment processes, the literature is clear that learning increases (Black & Wiliam, 1998; Drost, 2014; Duckor & Holmberg, 2017; Popham, 2017; Ruiz-Primo & Furtak, 2006).
In addition to this major finding, two additional findings were identified. The literature is filled with support for the idea that instructional leadership is critical, but oftentimes the reality of being an educator, particularly an administrator, reduces the amount of time that can be spent on this (Davis, 1998; Fullan, 2005; Glatthorn, 1998; Squires, 2009). Murphy (2006) explicitly stated that the distinguishing factor for effective schools is a school principal who exhibits strong curriculum-instructional leadership. Lunenburg and Ornstein (2022) conclude that achievement is a function of instructional leadership. In our data set, we saw several responses that support this idea: “As team leader, the first thing I did in March was talk to my grade-level peers and [I] asked them how we should plan our lessons.” And, “I asked teachers weekly how they were continuing to do that [during digital learning]. I think this made all the difference as our results overall didn’t shift.”
A corollary to this finding is that an instructional framework is crucial to ensuring a guaranteed and viable curriculum that helps support student achievement as measured by assessments. In looking at our participants’ responses, overwhelmingly, no matter what perspective—teacher, parent, or administrator, 70% of participants' responses identified common units of instruction arranged around a framework as a key element that they believe accounted for the lack of learning loss. As Learning Focused (2021) described, having an instructional framework with common units of instruction provides consistency, organization, and learning because the work is grounded in a collective vision of effective instruction. Specifically, teachers are not left to figure out things their own way. Such an approach, as DuFour and Marzano (2011) describe, ultimately improves student achievement, which is exactly what happened in this study related to digital learning and learning loss in these school districts and buildings.
A second finding was that a combination of the four elements were needed to effectively work at reducing learning loss. Specifically, as one American History/Language Arts teacher said in a follow-up interview to discuss the data, “these are not necessarily linear, but you really need them all.” An example was provided to show how they all interact. In Figure 1, this teacher uses an I-do, We-do, You-do approach that follows her district’s framework. In each of these sections, there is clear evidence of pedagogical function. She indicated that in the I-do section, her function was to draw on prior knowledge. In the we-do section, there was a discussion function. In the you-do, the teacher assesses to determine what students know about a particular topic. The tools that were chosen were clearly aligned to the function: for example, when the teacher wanted to draw on prior knowledge, she used an Edpuzzle video. When the teacher wanted to discuss, she used Canvas’s discussion board feature. Lastly, we see formative assessment process all throughout: group discussions allowed the teacher to figure out what students were processing, checks for understanding were completed with quizzes, and a culminating activity was used to determine what students learned during that lesson and to make adjustments for future lessons. One teacher's digital lesson plan showing the interplay of the four elements.
Limitations
Certain limitations are present that must be considered when interpreting and discussing the data results. A frequently cited limitation of any exploratory research study is related to transferability (Stebbins, 2011). Because of the relatively small sample size and the limited geographic context, these data and discussion may only be relevant to these school districts. As Stebbins points out, additional studies to create concatenated exploration may be warranted.
Time and technology were also a limitation in this study; people are generally busy. At various times during interviews, Zoom crashed and some people had to leave early. We as a research team sometimes had to run from meeting to meeting as well. This may have resulted in missed opportunities for comments related to the research study.
Another possible limitation is related to observation. There was no observation of how these four elements were actively implemented within classrooms. Given the nature of this study where participants are asked to determine what makes the difference in learning loss in their school district’s context, this would not have been feasible. However, future exploration during similar situations may prove insightful.
Lastly, the data reported is subjective as it is based on participant experience. This may mean that certain insights may not have been fully expressed yet or needed more time to develop. It would be interesting to review with these participants several years out to see if they would come to similar conclusions. Nonetheless, as with any study that aligns with the exploratory research viewpoint (Stebbins, 2011), the emerging generalizations are based on what the participants said at the time.
Recommendations for Future Research
One potential line of inquiry may be to study the effectiveness of various instructional frameworks that the participants used to see if there are any differences in reducing learning loss. For example, while the 5E model was mentioned by several participants, others mentioned that they had a home-grown approach or used UBD. Does the type of framework affect learning loss?
A second area for further study relates to the actual digital applications that teachers used, once they connected to pedagogy. Are certain applications more effective than others? Are they more aligned to standards and/or assessments than others? Does one app over the other produce more learning?
A third area is to determine what elements of the actual digital learning lesson are most effective as well as to determine whether all four elements are needed. As Figure 1 shows there are other techniques present in the lesson (e.g., provocative prompts, a technique connected to the Visible Learning Project, 2023). Are there certain techniques that may be more effective in digital learning?
Another suggestion is to look beyond the K-12 public schools within the Midwest, expanding to other geographic areas to learn how learning loss and digital learning are connected. Will the same elements be discovered or will something else be uncovered? In addition, comparative studies on other countries can also be completed to see if the same elements are evident across all systems of education.
Lastly, at the time of this study, Generative AI was not available for usage in classrooms. Given this new technology, it may be prudent to determine how AI might influence our understanding of learning loss, especially given AI’s paradoxical nature. Could the tool be used to accelerate learning or even help to close gaps? As Lim et al. (2023) explain, even though COVID-19 forced education into digital learning, some educators “still rely on basic technologies … to replicate physical into virtual lessons” rather than transforming the classroom and ultimately perhaps accelerating learning as the application can promote learning agency for students (p. 2). Given that Generative AI applications can serve as a method of transformation, it may be worthwhile to empirically study the tool in relation to digital learning.
Conclusion
Literature indicates that digital learning can be effective and was effective for some schools/districts during the pandemic. This exploratory study yielded key findings that may be of use to educators everywhere in leveraging the power of digital learning. What was common to those K-12 public school districts who were being successful, and ultimately common in this research, is that teachers did what they did best: focus on the teaching. Specifically, if they first understood the appropriate pedagogy, then they were able to make effective choices about which technologies to employ. In our own experiences as educators, we have seen the power a pedagogy-first approach to digital learning can have in terms of improving student success. While some may hope that educators should never need this information again, we suggest differently. By putting these ideas into place in our classrooms now and connecting with appropriate technology tools, our students' abilities to learn at high levels can and should increase exponentially.
Footnotes
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
