Administration and Results of a State Alternate Assessment Based on Alternate Academic Standards in Science for Students Who Are Blind and Have Low Vision

Registration

In 2017, approximately 2.0% of students statewide participated in the AA-AAS-science exam. Students were eligible for the tactile version of AA-AAS-science if recommended by their IEP team and classified as having a visual impairment or received services from a teacher of students with visual impairments. During the 2016–2017 academic school year, 34 (14.7%) of the 231 students eligible completed the tactile AA-AAS-science. It was unclear why the remaining 197 students who were eligible did not choose the tactile version of the exam. This was an IEP team or teacher decision.

Demographic Information

Twenty (58.8%) students completed the grade 4 exam, 7 (20.6%) students completed the grade 8 exam, and 7 (20.6%) students completed the grade 11 exam (age range: 9.8–19.0 years). Eighteen (52.9%) students were male, and 16 (47.1%) were female. Eighteen (52.9%) students were white, 7 (20.6%) were Black, 6 (17.7%) were Asian, and 3 (8.8%) were Latinx. Students’ primary disability labels were as follows: 21 (61.8%) visual impairment; 10 (29.4%) multiple disabilities; 2 (5.9%) intellectual disability; 1 (2.9%) deaf-blind. Secondary and tertiary disabilities were not reported. Exactly half of the students 17 (50.0%) attended a school for the blind. Thirteen (38.24%) students attended their local day school. Three (8.82%) students attended a specialized school for individuals with intellectual disabilities. One (2.94%) student attended a specialized school for individuals with autism. No information was available regarding the students’ visual diagnoses or clinical and functional vision. Demographic information for test administrators was not available.

The AA-AAS-Science Tool

The tactile AA-AAS-science consisted of 35 questions for grades 4 and 8 and 30 questions for grade 11. The AA-AAS-science contained questions from four categories: nature of science, biological sciences, physical sciences, and Earth and space science. Questions were multiple-choice with one correct answer and two distractors (see Figures 1 and 2). The tactile AA-AAS-science was typically administered by teachers of students with visual impairments. Other administrators were classroom teachers, school counselors, or related service providers.

OPEN IN VIEWER

Figure 1.

Sample tactile alternate assessment based on alternate academic standards (AA-AAS)-science question, Tier 1.

OPEN IN VIEWER

Figure 2.

Sample tactile alternate assessment based on alternate academic standards (AA-AAS)-science question, Tier 2.

Test Administrator Training

All test administrators completed online training modules covering five topics: accessing the assessment, recording student responses, camera positioning, test administration, and test security. The training was required before administrators could access test materials. On average, administrators completed training in approximately 60 min. The administration manual included additional instructions regarding student accommodations. This included cutting the response array pictures from the test book, presenting materials on a black background, and encouraging the use of a divided work tray to create an organized working environment with clear boundaries for students interacting with the materials tactually. The manual explicitly stated administrators were permitted and encouraged to substitute items from the tactile kit with items used to teach the same concepts to the student.

Test Administration

All items on the tactile test were delivered orally or orally in conjunction with sign language. Administrators followed a script, which presented an initial prompt and three answer choices orally and in the form of pictures. A scripted second prompt was included for students who did not answer correctly after the initial item prompt. Test administrators were instructed to follow the scripts but were permitted to augment questions with verbal descriptions. Test administrators could terminate testing for students who failed to engage with the assessment, demonstrated inappropriate or violent behavior, or had a medical issue during testing.

Test administrators had three options for assessment administration: nondigital, partially digital, and fully digital. All formats required students’ responses to be submitted digitally through a secure website. Nondigital administration used paper materials for the administrator and student. Partially digital administration used paper materials for the student and digital materials for the administrator. Full-digital administration did not use any paper materials. No administrators of the tactile AA-AAS-science opted for fully-digital administration. All administrations were video recorded and uploaded to the AA-AAS-science website.

Student Scores

Student responses were scored using a 3-parameter Item Response Theory (IRT) model to account for student guessing (Hambleton & Swaminathan, 1985). A group of Special Education teachers from across the state performed a standard setting using the Bookmark method. The standard setting resulted in cut scores that were used to place students into performance categories (emerging, novice, proficient, advanced) based on their IRT score (Cizek & Bunch, 2007). Test scores for different years were equated using an IRT-based equating method so interpretations of performance categories across years remained comparable (Kolen & Brennan, 2004).

Students unable to complete any test items were not given a score (e.g., student behavior interrupted the first item of the test). Students whose testing was discontinued because they showed no meaningful interaction with the first three items received a score of emerging for attempted participation. Students who were discontinued were scored on their performance for the portion of the assessment completed.

Coding Testing Accommodation and Strategies

Test videos were viewed by a primary external reviewer to ensure the accuracy of students’ responses entered by test administrators and to code common accommodations, including prompting procedures used to assist students in locating the tactile pictures or objects to be explored. These accommodations were in line with the accommodations cited by Zebehazy et al. (2012). A second reviewer separately viewed videos of 11 (32.4%) students for reliability. The reviewers scored the administration on 11 variables: use of the tactile kit, use of braille, use of slant board, hand-over-hand prompting, hand-under-hand prompting, presentation of materials on a black background, presentation of materials on a work tray, difficulty of materials, student engagement, and whether prompts gave away answers. The reviewers agreed in 121 (96.7%) of 125 instances. In instances of disagreement, the primary coder's score was used.

Student Performance

Of the 20 students who took the grade 4 exam, one student took the Tier 2 exam and scored Advanced. Of the 19 students who took the Tier 1 exam, 2 (10.5%) students scored Proficient, 3 (15.8%) scored Novice, and 13 (68.4%) scored Emerging. One (5.0%) student did not receive a score.

Of the seven students who took the grade 8 exam, one student took the Tier 2 exam and scored Advanced. Of the six students who took the Tier 1 exam, 1 (16.7%) scored Proficient, 2 (33.3%) scored Emerging, and 1 (16.7%) scored Novice. Two (33.3%) students did not receive scores.

Of the seven students who took the grade 11 exam, two students took the Tier 2 exam. One scored Advanced; the other scored Novice. The remaining five students took the Tier 1 exam. Three (60.0%) students scored Novice, and 2 (40.0%) scored Emerging. The results of a Shapiro-Wilk test of normality by grade and tier demonstrated that student scores were not normally distributed in grades 4 (.72, df = 20, p < .000) and 8 (.72, df = 7, p = .006) but were normally distributed in grade 11 (.88, df = 7, p = .224). This indicates that students’ scores in grades 4 and 8 did not follow a bell curve with most students’ scores falling in the middle and only a few students scoring lower or higher.

Across grade levels, four students took the Tier 2 exam, and 30 students took the Tier 1 exam. There was a significant positive correlation between tier placement and score (r(32) = .689, p < .000) indicating most students were appropriately placed within the tiers. In 17 (50.0%) of the 34 administrations, testing was discontinued because the student did not actively participate in the exam.

The number of correctly and independently answered questions ranged from 0.6 to 6.5 depending on grade and tier across the four test categories (Figure 3). There was a significant positive correlation between tier placement and completion of the exam (r(32) = .37, p = .04). A significant correlation between tier placement and overall score (r(32) = .90, p < .000) and student score by category: nature of science (r(32) = .71, p < .000), biological sciences (r(32) = .48, p = .004), physical sciences (r(32) = .40, p = .018), and Earth and space science (r(32) = .34, p = .049) were found. There was also a significant correlation between tier placement and performance on the AA-AAS-English Language Arts—r(32) = .63, p < .000—and the AA-AAS-Math: r(32) = .52, p = .002. Similarly, there was a significant correlation between overall exam score and score by category: nature of science, r(32) = .89, p < .000; biological sciences, r(32) = .84, p < .000; physical sciences, r(32) = .78, p < .000; and Earth and space science, r(32) = .81, p < .000. There was a significant correlation between the overall score on the AA-AAS-science and AA-AAS-English Language Arts—r(32) = .55, p = .001—and no significant correlation between overall scores on the AA-AAS-science and AA-AAS-Math, r(32) = .35, p = .055.

OPEN IN VIEWER

Figure 3.

Student performance by grade level, tier, and question category.

Administration of the Assessment

Videos of each test administration were coded for accommodations. The most common accommodation was the presentation of test materials on a black background (N = 13; 38.2%). This was highly, positively correlated—r(32) = .562, p = .001—with the accommodation of a divided work tray. The divided work tray used in 8 (23.5%) administrations all appeared to be black work trays by APH. Two (5.9%) administrators provided some assessment in braille. Three (8.8%) administrators used a slant board. Of the 34 test administrators, 12 (35.3%) used hand-over-hand to facilitate student engagement with materials, 4 (11.8%) used hand-under-hand to facilitate student engagement with materials, and 18 (52.9%) did not use physical prompting procedures (see Figure 4).

OPEN IN VIEWER

Figure 4.

Percentages of students receiving popular accommodations.

The tactile AA-AAS-science included a kit with relevant tactile materials. Test administrators were permitted to use classroom materials for continuity between instruction and examination and facilitate communication for students who were nonverbal. Most administrators used materials from the tactile kit. Nineteen (55.9%) administrators used only the kit, 4 (11.7%) used the kit in conjunction with teacher-made materials, 1 (2.9%) used primarily teacher-made materials, and 10 (29.4%) did not use tactile accommodations. Students without tactile accommodations relied on the scripted oral delivery of all items or oral delivery in conjunction with sign language which was provided for all test takers.

No testing accommodations were significantly correlated with assessment scores. The only testing accommodation significantly correlated with student engagement was a slant board, which was negatively correlated with student engagement, r(32) = −.378, p = .039. Student engagement was closely correlated with the difficulty of the content—r(32) = −.701, p < .000—and use of a work tray positively correlated with attending a specialized school for the blind, r(32) = .416, p = .014. Student grade level negatively correlated with the use of a work tray—r(32) = −.437, p = .010—and presentation of materials on a black background, r(32) = −.620, p < .000.

Appropriateness of the Assessment for the Student

The two reviewers found test administrators reliably recorded student responses in most cases. The camera angle made it difficult-to-impossible to discern which answers were chosen by a few students who responded nonverbally. In three instances, external reviewers could not distinguish whether the student intentionally responded or if the student made an uncontrolled movement. One student used eye gaze to select answers, so external reviewers could not reliably determine student selections. In these cases, reviewers indicated that it could not be determined if the teacher had reliability recorded the students’ answers.

Reviewers were asked to determine if the materials were too difficult for the student. In 29 instances, external reviewers offered an opinion, finding the version of the AA-AAS-science exam administered was too difficult for 17 (50.0%) of the students. Student engagement was closely correlated with working memory requirements—r(32) = −.610, p < .000—and difficulty of assessment content, r(32) = −.701, p < .000.

Accommodations and Student Demographics

The variety of accommodations used reflected the diversity of students taking the exam. The most common accommodation, the presentation of materials on a black background, was not surprising as many students with visual impairments and significant intellectual disabilities have cortical visual impairments and need support to focus their visual attention. The frequency with which the APH black divider tray was seen as an accommodation, and its general availability throughout the United States, suggest that truly standardized methods of accommodation can be implemented across a diverse student population. Future research is warranted into the correlation between accommodations and age; there might be a developmental connection between younger students being more likely to receive test materials in a tray or on a black background.

It is noteworthy that attendance at a school for the blind was negatively, significantly correlated with having a primary disability label of visually impaired, r(32) = −.59, p < .000. Also, there was a significant positive correlation between placement at a school for the blind and visual presentation of items along with a presentation that used other senses, r(32) = .35, p = .040. Further exploration is warranted into student demographics as they relate to school placement and disability label selection as well as the encouragement of students’ sensory skills development through the expanded core curriculum area of sensory efficiency.

Distribution of Test Scores

One important finding is the lack of normally distributed data for the fourth and eighth grade test results. This lack of normally distributed scores limited the statistical methods which could be applied to the analysis of these data since many statistical tests are based upon the assumption that data are normally distributed. The lack of normally distributed scores, however, does not indicate the tactile AA-AAS-science test was not well-designed. Statistically, the small sample size made the risk of a Type II error, or a false negative, difficult to avoid. The lack of a normal distribution could simply be a result of not having enough students participate in the exam to see a true, statistical representation of the distribution of scores.

There are a variety of factors that could explain why the distribution was not normal. The lack of a normal distribution could relate to assessment difficulty. The level of accommodations students received during instruction may also contribute to the lack of normally distributed scores since half of the students attended a local school as opposed to a school for the blind. Teachers in inclusive settings report having little to no training in how to work with students with severe disabilities (Attwood et al., 2019; Cook, 2001; Knight et al., 2019). If students with severe intellectual disabilities and visual impairments are not receiving adequate instruction, the distribution of scores would not be normal even if the test adequately measured student learning.

Limited access to testing materials almost certainly affected score distribution in grades 4 and 8, since many students were unable to fully access the test. Test items may have been more difficult when they are presented nonvisually. Only one administrator made extensive use of materials with which the student seemed to be familiar. Conversely, four administrators did not use any materials with which students seemed to be familiar. Nineteen test administrators used materials from the tactile kit without personalization to the students. One test administrator became confused about a question during administration and said, “This is what happens when you don’t download the test until the day of the test.” This teacher's lack of preparation echoes attitudes established in the literature (Attwood et al., 2019; Cook, 2001; Knight et al., 2019).

Conclusions

It is concerning that approximately one-third of administrators did not provide students with any tangible accommodations for the tactile AA-AAS-science. It is perhaps more concerning, however, that of the 24 teachers who did provide some tactile access, only one extensively used materials familiar and meaningful to the student. Novel tactile materials can be meaningless to someone who is not skilled in the techniques used to access these materials (Picard & Lebaz, 2012). The test taker's ability to understand questions and communicate an answer is greatly compromised and comparable to testing them in a foreign language. Many students did not independently hold the tactile materials, explore the materials in a systematic way, or use more than one hand. Yet, the ability to explore and obtain information from a new tactile picture or object is much improved when both hands and multiple fingers are used (Picard & Lebaz, 2012). Twelve administrators used hand-over-hand prompting to assist the student in contacting the tactile materials, but this was typically a brief swipe of the hand over the object rather than a systematic exploration. Overall, two-thirds of administrators provided either no or insufficient access and exploration of meaningful tactile materials to students, making it highly unlikely they would find success and understand the questions being asked.

Implications for Practice

The purpose of this study was to understand the administration of and performance of students with visual impairments and intellectual disabilities on the AA-AAS-science, a state-wide summative multiple-choice assessment. Tests were administered, in most cases, by a teacher of students with visual impairments. Many test administrators provided little or no tactile accommodations. Questions can be more difficult without visuals, so it is crucial that students have full access. To provide full access, test administrators need to be prepared with meaningful tactile accommodations, preferably materials that have been used during instruction, and allow students time for meaningful interaction with tactile accommodations.

Limitations

There are several limitations to this study of the tactile AA-AAS-science. To protect student confidentiality, students’ visual diagnoses, clinical and functional vision, and learning media were not available to researchers. The accommodation of adjusted visual complexity of the question and answer array, a potentially important accommodation, was not able to be coded due to video camera angles. Likewise, it was difficult to know exactly what students were seeing or touching since the camera captured the test administration, not the student's perspective. Analyses were post-hoc, and researchers could not add or ask any questions beyond the scope of the standard exam administration. Lastly, the level of difficulty was assessed through independent coding by the authors rather than the use of a standardized reference or tool.

Future Directions

Future studies should assess the consistency of tactile materials and communication strategies between instruction and assessment. If accommodations differ between instruction and assessment, it would be helpful to know if this is a thoughtful decision or poor preparation. In line with prior research for students on alternative assessments (Browder et al., 2003), it would be interesting to compare scores on the AA-AAS-science with their IEP goals and access to the grade-level curriculum. To further understand the attitudes and preparedness of administrators, future studies should incorporate test administrators’ demographics including role within the school, formal training to work with students with significant intellectual disabilities and visual impairments, experience working with students with intellectual disabilities and visual impairments, and attitudes about testing procedures and appropriateness of the tactile AA-AAS-science. It would also be wise to explore different types of assessments for students with significant intellectual disabilities and visual impairments. The test was deemed too difficult for 17 (50%) of students. This could indicate the assessment is not appropriate for tactile learners regardless of accommodations because the assessment was, ultimately, not designed for tactile learners.