Examining the assessment literacy required for interpreting score reports: A focus on educators of K–12 English learners

Literature review

Methods

Participants

In Phase 1 of the study, 1437 EL educators from Grades K–12 completed an online survey. Participants were recruited via convenience sampling by distributing the survey using an existing WIDA listserv for K–12 EL educators. During the time of data collection, approximately 37 states and territories were part of the WIDA Consortium. Participants were from 35 US states and territories. Among the participants, 60% taught students in Grades K–5, and the remaining 40% taught students in Grades 6–12. Educators had varying years of teaching experience: 10% had 0–2 years, 20% had 3–5 years, 25% had 6–10 years, and 45% had more than 10 years of teaching experience.

In Phase 2, 18 educators participated in interviews: 12 were in-service EL teachers and six were EL coordinators, all recruited during a national conference attended by K–12 educators. This conference was selected due to its popularity with the large numbers of educators who were the specific focus of this study. All educators taught or supported ELs ranging from pre-Kindergarten to Grade 12. While EL teachers interacted with ELs on a daily basis, EL coordinators had experience as EL teachers and were supervising EL teachers at the district level at the time of the interview. As summarized in Table 1, educators’ backgrounds varied: they were from 13 US states and had on average 12.6 years of experience in the field. All but two had at least a master’s degree; two had doctorates.

OPEN IN VIEWER

Table 1.

EL educators.

ID	State	EL position	Highest level of education	Years of experience	Grades taught/supported
AM	Tennessee	Teacher	Master’s	6	5–8
CK	Pennsylvania	Teacher	Master’s	4	6
CS	Minnesota	Teacher	Master’s	12	K–5
DL	Colorado	Coordinator	Doctorate	40	PK–12
EG	Illinois	Teacher	Bachelor’s	3	6–8
IT	North Dakota	Teacher	Master’s	12	9–12
JE	Georgia	Coordinator	Master’s	14	PK–12
JP	Pennsylvania	Teacher	Bachelor’s	12	K–5
LG1	Pennsylvania	Teacher	Master’s	4	K–12
LG2	Oklahoma	Coordinator	Master’s	16	PK–12
LL	Wyoming	Teacher	Master’s	15	K–6
LS1	Pennsylvania	Teacher	Master’s	16	K–12
LS2	Nevada	Teacher	Master’s	13	K–8
MC	New Jersey	Coordinator	Master’s	17	5–8
NW	Minnesota	Teacher	Master’s	14	6–11
SC	Delaware	Coordinator	Master’s	14	K–12
TJ	Alabama	Coordinator	Doctorate	5	K–12
VU	Pennsylvania	Teacher	Master’s	10	K–4

Findings

Study findings are organized in relation to the three research questions. To address these questions, relevant findings from the survey and interview results are presented below.

K–12 EL educators’ assessment literacy was examined by first identifying the degree of educators’ understanding and interpretation of ACCESS score report information. This analysis not only involved examining what educators understand from the score report, but also exploring the score report information that they perceived to be helpful or unclear. Specifically, findings from Survey Item 10 and Interview Questions 10–12 were used to respond to this research question, as shown below:

OPEN IN VIEWER

Survey Item 10: 10. To what extent is the following information, as reported in the new Individual Student Report, helpful in understanding students’ performance on ACESS for ELLs? (Likert scale with four options for each item: not helpful at all, somewhat helpful, helpful, very helpful) (1) Summary graph of students’ scores (2) Language domain scores (listening, reading, writing, speaking) (3) Composite scores (overall, literacy, comprehension, oral language) (4) Description of English language proficiency levels Interview Questions 10–12 10. Do you understand each section of the score report? 11. Which section(s) provides the most helpful information to understand ELs’ language proficiency? Why? 12. Is there any information, word, or graph in this score report that is unclear? What? Why?

Findings from Survey Item 10 indicate sections of the score report that educators found helpful and meaningful. These sections indicate the assessment literacy required for interpreting ACCESS scores, as educators can likely find the graphs, scores, and descriptions helpful only if they have an understanding of the student information presented. As seen in Figure 1, approximately half of the participants rated the description of English language proficiency levels to be “very helpful,” suggesting it to be the most helpful.

OPEN IN VIEWER

Figure 1.

Helpfulness of the new Individual Student Report for understanding students’ performance (n = 1424).

The description of English language proficiency levels provided information on what students can do at each of their proficiency levels. Survey findings show that it was not only important to identify the numeric scores that students received, but also to understand what skills students can actually demonstrate. The description directly connects to how ELs may be able to perform in classroom settings and provides instructional support to educators. This finding suggests that educators can easily digest score report information that is connected to concrete language skills of ELs. It also reveals what educators value, that is, verbal descriptions that connect numeric scores to actual student demonstrations and performances.

Moreover, interview findings added details regarding the degree of assessment literacy educators demonstrated in interpreting ACCESS score reports. The survey findings indicated the importance of understanding language domain scores for listening, reading, speaking, and writing, together with composite scores. The interview results specifically indicated the type of scores, such as proficiency level, scale score, and confidence band, that were particularly helpful or challenging when interpreting ACCESS results. As detailed below, although the majority of educators were able to interpret proficiency level scores, some struggled to understand scale score or confidence band information.

According to findings from Interview Question 11, five participants (AM, LS2, IT, EG, and VU) found that the proficiency level information (on the four language domains and the combined domains) was necessary and useful when interpreting the score report. For example, EG stated, “You can see what modality your students are going to perform better in, so you can visually see if your students are better at one [language domain] or another.”

In addition, four participants (Participants CS, EG, JP, and NW) indicated that the proficiency level was useful for interpreting further the students’ growth in their English language ability. They used proficiency levels to check for improvements in ELs’ scores. For example, CS compared the proficiency level index in the current report with that from the previous year’s report; districts often provided previous year’s score report information on their online database systems. She believed that “if there’s a growth … a child should … at least grow 0.5 [in their proficiency level].”

Together with the proficiency level information, findings indicate that the description of English language proficiency levels, located at the bottom of the score report, helped EL teachers understand what their students can do with their current English proficiency levels. This assistance also applies to cases in which EL teachers share student information with mainstream classroom teachers or content subject teachers in the same school who have ELs in their classes, as EG describes below.

I really like the box at the bottom with what students can do, especially when trying to explain to [other mainstream or content] teachers, when the teacher is trying to figure out “well, how do I deal with this student, I don’t know what to try,” so I think it’s really nice to have concrete things that [a] student at or around [that] level generally can do. (EG)

On account of content teachers’ lack of training in interpreting ACCESS data, they are often uncertain about what ELs in their classrooms can do using English. EL teachers can function as advisers for content teachers by helping them better understand the score report and what they should expect from ELs in their classrooms.

Findings on Interview Question 12 revealed that several EL educators (AM, CK, CS, EG, LG1, and LS2) do not fully comprehend the technical terms in the score report, for example scale score or confidence band. These educators struggled to understand scale scores. AM specially addresses his uncertainty below.

seeing the number 367 [on the scale score], I don’t know what I’m supposed to do with that number versus 320. All I know is that 368 is higher than 320. And I don’t know what to do with those. (AM)

He added, “[t]his is lack of my background knowledge about looking at scale scores … I just don’t know.” As suggested by AM, this knowledge gap could be partly attributable to the lack of training that EL educators receive or not having the time or access to score report resources. Attributing issues identified in terms of score report interpretation is generally challenging as there may be multiple reasons underlying the problems, ranging from the assessment literacy of the educators, to the design of the score report, to the complexity of the assessment.

Overall, findings offer a general picture of K–12 EL educators’ assessment literacy in relation to ACCESS scores. They have the ability to interpret the proficiency levels and connect the numeric levels with descriptions of what students can do using their English language. The description of English language proficiency levels are meaningful as they provide concrete instructional implications. However, educators still lack knowledge of technical terms, such as scale scores and confidence bands.

Research Question 2 was addressed by examining the materials and resources that K–12 EL educators used to support their interpretations of ACCESS scores. To this end, findings from Survey Item 9 and Interview Question 19 were analyzed.

OPEN IN VIEWER

Survey Item 9 9. How often do you refer to the WIDA ACCESS for ELLs Interpretive Guide when interpreting data reported in the Teacher Report? Interview Question 19 19. Do you refer to other resources to understand the score report? (e.g., Interpretive Guide) If yes, how and to what extent?

The Interpretive Guide for Score Reports is available to the public on the WIDA website. This guide is designed to be the main resource for supporting educators’ interpretation of ACCESS score reports. We used response data from Survey Item 9 to examine how often participants referred to the Interpretive Guide to understand the score report information. Results showed they accessed it with varying degrees (Table 2). Approximately 60% of the participants referred to the guide (i.e., sometimes, often, or all the time), whereas 40% of them rarely or never used it.

OPEN IN VIEWER

Table 2.

Frequency of use of the Interpretive Guide.

Use of the Interpretive Guide	Responses	%
Never	187	13
Rarely	388	27
Sometimes	595	42
Often	220	15
All the time	43	3
Total	1433	100

In response to Interview Question 19, several participants indicated that they were aware of the Interpretive Guide. Two educators (LG2, VU) used it as a reference, while three others (CK, EG, LL) rarely used it. The reasons for teachers’ limited use of the Interpretive Guide vary, but as CS said, the main reason was a lack of time.

No, no, I think it’s not hard to comprehend, but what I miss is that … having someone – explain the nitty-gritty, the highlight, rather than me going through it [using the Interpretive Guide], and not being an expert in data, and reading through it, it would take me a lot longer, I understand it. But if we have a person [with expertise in data, such as an EL coordinator,] that can just say, “okay this is what you need.” And [teachers] go through it, and they, it clicks for them, and they transfer it in the student … and but no, it’s not difficult. (CS)

CS acknowledged that she is not “an expert in data” and her lack of expertise means she must devote a considerable amount of time to interpreting fully the score report. This response showed that she prefers to have someone explain the scores to her because the technical vocabulary used in the Interpretive Guide makes the guide less user-friendly. In addition, lack of time is an issue for educators who need to interpret and understand their students’ score reports. When planning revisions to the Interpretive Guide, WIDA should consider the fact that educators (and especially EL educators) have a limited amount of time to read the guide.

Six educators (AM, CS, EG, JP, LS1, and VU), who share score reports with content teachers, reported that content teachers, because of their lack of training and limited knowledge about how to interpret language-related data, rely on EL teachers to interpret the reports. Because EL teachers reported that content teachers depend on them for interpretation, it is critical to have a clear Interpretive Guide that first considers the needs of EL teachers.

Moreover, four educators (JE, JP, LS2, and NW) referred to the WIDA Can Do Descriptors (WIDA, 2016), a separate document from the score report, which provide detailed examples of what students at various levels of proficiency can do with their English language in school settings. JE responded as follows: “Teachers look specifically at these language domains and correlate them to the Can Do Descriptors to get some ideas for differentiation.” She used the proficiency level index in conjunction with the Can Do Descriptors.

These results indicate that educators referred to the Interpretive Guide for Score Reports and the WIDA Can Do Descriptors to interpret score reports. However, they had limited time to refer to these resources, suggesting the need for more concise and user-friendly materials.

Findings from Survey Items 14 and 15, and Interview Questions 14 and 20 were examined to respond to Research Question 3:

OPEN IN VIEWER

Survey Items 14–15: 14. What other information should be included in the Individual Student Report? (Check all that apply) (1) Previous year’s scores (2) Description of all proficiency levels (3) Explanation of scale scores and confidence bands (4) More details of students’ ability in each language domain (5) Suggestions for using the score report (6) What teachers/parents could do to improve students’ language ability (7) Other: ______________ 15. Do you have any additional comments regarding the Individual Student Report? Interview Questions 14 and 20: 14. Is there any additional information that you think should be included? What? Why? 20. Do you have any additional comments about either version of the score report or about how you use the information from the current score report?

In response to Survey Item 14, the majority of participants (91%) wanted to know about students’ scores from previous years (Table 3). According to individual comments, participants were very much in favor of having previous scores included in the score report. They wanted information on students’ growth (from the previous year), not only regarding their overall language ability, but also in individual language domains of listening, reading, speaking, and writing. Comments (individual notes written in the “other” option) also showed the need for an additional score index that indicates students’ performance in comparison to other EL students. This information could allow teachers to understand how students are performing compared to the norm.

OPEN IN VIEWER

Table 3.

Suggestions for the new score report.

Suggestions for the new score report	Responses	%
Previous year’s scores	1302	91
Description of all proficiency levels	629	44
Explanation of scale scores and confidence bands	579	40
More details of students’ ability in each language domain	825	57
Suggestions for using the score report	589	41
What teachers could do to improve students’ language ability	931	65
Other	84	6

Note: Participants were allowed to select multiple options; therefore the total does not add up to 100% and is not reported.

In addition, findings from Survey Item 14 suggest including actionable information to improve students’ language ability (65%; Table 3). For instance, individuals suggested creating enhanced online videos and documents to guide/train teachers how to interpret and utilize score reports to inform instruction. Findings from the same item indicated the need for more details regarding students’ ability in each language domain (57%). Some respondents requested more fine-grained details, such as students’ language performance and strength/weakness in various language standards, including the languages of mathematics and science. Other individual comments indicated the need for exit criteria information in each domain.

In addition, the response to the open-ended Survey Item 15 revealed the need to provide score reports soon after test administration. Currently, it can take up to 4 months to receive the score report. Since children make progress quickly in their language development, a report received months after the test administration may not be very informative. At the latest, score reports need to arrive before the new school year to grant teachers the time to plan for next year. Other individual comments indicated the need for a section to mark students’ placement in language instruction educational programs based on their test scores, and score reports in students’ English language. These comments included having a separate translated version or having the translated version on the back page of a double-sided report.

Results from Interview Questions 14 and 20 generally confirmed the findings from the survey items. Although the new score report is meant to serve as an enhanced version that replaced the old score report, participants offered various suggestions for improvement. First, several participants (CK, CS, LG1, LS2) suggested that WIDA include test-takers’ scores from the previous year. As demonstrated by the response of CS, having the previous year’s scores in the score report would make it more convenient for educators easily to interpret student growth: “I think it’s just a dream of mine, but it would be nice to have … the previous year, maybe … Then we don’t have to do a lot of digging … so it’s just convenient for us [and] would be nice.” According to LG1, parents and EL teachers can benefit from seeing at first glance how much their students’ English language proficiency has grown over the year. CS specifically suggested that using different colors for the past vs. current year’s scores may be helpful.

Educators also indicated that they wanted to see the full description of English proficiency levels to have a clearer idea of what students can demonstrate in their proficiency levels. The descriptions on the current score reports are limited to the level of the students’ performance in the four language domains (listening, reading, speaking, and writing). In addition, NW wanted “more information in … for reading and writing tasks, like some … like more types of things that students can and cannot do … I think that would be helpful for classroom teachers.”

Educators (CS, LG1, LS1, MC, and SC) also wanted more information to support their instruction, especially for teaching content areas. For example, LS1 wanted information on the specific lessons/tasks that she could have her students work on to improve their weaknesses. However, it may not be appropriate to include such detailed amounts of instructional materials in a score report. Also, WIDA does not necessarily recommend that educators use ACCESS scores to inform their daily instruction as ACCESS is not formative in nature. Rather, due to the summative nature of ACCESS, WIDA recommends that its stakeholders use ACCESS scores for making placement and programming decisions.

Other suggestions included the following: (1) adding students’ home language information; (2) adding information regarding student’s test completion rate; (3) enlarging the font of essential information, such as composite scores; and (4) delivering the reports more quickly. Although ACCESS score reports may be shipped to districts and schools in advance, educators may not have access to them. SC, commenting on a need to have score reports available more quickly, put it this way:

Part of the challenge, if I could also add, is that we get this report for us probably a week and half before we start school. And so by then, you know, class lists are already made, things are already set up, and it’s very difficult, especially [at] the middle school, high school level, we try to put all of our ELs on one team in each grade level, so they can be better supported. And we end up having to move kids around, last minute … So, we were all hoping with the … newer tests starting earlier and being more computer-based, we might get things earlier so that a lot more prep could be handled, conversations could take place to get better use of the data earlier rather than once the kids are back. (SC)

Overall, educators’ suggestions provide guidance in further improving the quality and delivery of the score reports, which could ultimately enhance educators’ assessment literacy to interpret the reports.

Discussion and conclusion

In this study, we investigated the assessment literacy that K–12 EL educators need for interpreting score reports and identified the resources they use to interpret those reports. As this study was part of a larger project at WIDA that aimed to enhance the quality of ACCESS score reports, findings provided guidance for refining the score reports and enhancing score report interpretive guides. Results show that educators find interpreting technical terms, such as scale scores or confidence bands, to be challenging, which suggests the need for describing these terms with more clarity and ease.

These study findings provide theoretical implications by confirming previous research which indicates that K–12 educators often lack assessment literacy and experience difficulty understanding certain terms in score reports (Lukin et al., 2004; Zapata-Rivera et al., 2010; Zwick et al., 2014). The current study found that K–12 EL educators struggle with conceptually understanding scale scores and confidence bands. This could be partly because educators frequently refer to proficiency level scores, which are used to make high-stakes decisions such as EL identification and reclassification, as required by federal and state policies. It may also be partly because proficiency level scores are easily connected to the Standards framework in which the assessment is situated, providing straightforward score interpretation for educators. However, scale scores are less intuitive for educators and are more challenging to connect to the Standards framework, whereas confidence bands are seen as overly technical and confusing by many educators, making it challenging to comprehend the terms.

In addition, the findings help fill a gap in the literature regarding K–12 EL educators’ score report interpretation and use of resources. During the interviews, educators suggested including more information in the score reports, such as information about student growth. This suggests that educators are not only interested in measures of student performance at a single point in time, but whether the students are making progress over time. Regarding the use of score report resources, due to time pressures, educators did not have sufficient time to utilize resources made available to support score interpretation. In addition, the existing materials may have been too lengthy and complex for educators. This reveals a dilemma in promoting score report interpretation: educators need more resources for score report interpretation, but they also lack the time to read the materials. These findings suggest that the resources offered to educators should be easily accessible and digestible.

Results also provide practical implications for further supporting K–12 EL educators’ assessment literacy by indicating the areas in which educators need more support. Based on the feedback from the educators, the following suggestions need to be considered for future English language proficiency assessment score reports:

One limitation of the study was not having detailed information regarding the educators, including the type of EL instruction they provide, such as pull-out or push-in instruction, or the type and amount of training participants received in relation to score report interpretation (e.g., online webinars and professional learning opportunities). Although most participants had at least a master’s degree in education or a related field, not all master’s programs offer courses in language assessment. Therefore, it is unclear how much previous assessment literacy educators had and to what extent their training influenced their test score interpretation.

Another limitation was the sampling method to recruit survey participants. Owing to the difficulty of recruiting a large number of respondents for the survey study in Phase 1, we used a convenience sampling method by distributing the online survey link to educators on the WIDA listserv. Although this method led to the collection of more than 1400 samples, it weakens the generalizability of the findings. Using a stratified sampling method may have led to better data collection.

Study findings provide implications for future research. Research team members specifically examined the assessment literacy required by K–12 EL educators for interpreting score reports from ACCESS, which is an annual standardized English language proficiency assessment. Although it is crucial to interpret accurately large-scale language assessment scores in order to examine EL students’ English language proficiency and their language growth, other types of assessment literacy may be needed. Educators engage in various assessment-related activities in K–12 EL settings, such as creating classroom-based assessments to measure and interpret students’ in-class performance. These specific assessment literacy skills are crucial in everyday class settings, yet educators may not receive sufficient guidelines or support in order to develop them. Therefore, future studies need to investigate further the various assessment literacy skills (necessary for large-scale and classroom-based assessments) that are relevant to K–12 EL settings. In addition, the field may benefit from future research on educators’ assessment literacy, initiated by independent researchers or state education agencies. Moreover, future studies should examine how educators’ assessment literacy allows them to use the score report information. Although this study reports findings on the degree of educators’ assessment literacy in relation to interpreting score report information, educators do much more with score reports, such as making various decisions for EL students. Helping educators to interpret more effectively the technical data on their students will lead to better decisions for those students and, ultimately, better educational outcomes.