Bridging the Gap of Graphical Information Accessibility in Education With Multimodal Touchscreens Among Students With Blindness and Low Vision

Abstract

Introduction: Informational graphics and data representations (e.g., charts and figures) are critical for accessing educational content. Novel technologies, such as the multimodal touchscreen which displays audio, haptic, and visual information, are promising for being platforms of diverse means to access digital content. This work evaluated educational graphics rendered on a touchscreen compared to the current standard for accessing graphical content. Method: Three bar charts and geometry figures were evaluated on student (N = 20) ability to orient to and extract information from the touchscreen and print. Participants explored the graphics and then were administered a set of questions (11–12 depending on graphic group). In addition, participants’ attitudes using the mediums were assessed. Results: Participants performed statistically significantly better on questions assessing information orientation using the touchscreen than print for both bar chart and geometry figures. No statistically significant difference in information extraction ability was found between mediums on either graphic type. Participants responded significantly more favorably to the touchscreen than the print graphics, indicating them as more helpful, interesting, fun, and less confusing. Discussion: Accessing and orienting to information was highly successful by participants using the touchscreen, and was the preferred means of accessing graphical information when compared to the print image for both geometry figures and bar charts. This study highlights challenges in presenting graphics both on touchscreens and in print. Implications for Practitioners: This study offers preliminary support for the use of multimodal, touchscreen tablets as educational tools. Student ability using touchscreen-based graphics seems to be comparable to traditional types of graphics (large print and embossed, tactile graphics), although further investigation may be necessary for tactile graphic users. In summary, educators of students with blindness and visual impairments should consider ways to utilize new technologies, such as touchscreens, to provide more diverse access to graphical information.

Keywords

haptics touchscreens graphics STEM learning

A recent inventory of graphics found in mathematics textbooks for grades 5–8 (Emerson & Anderson, 2018) noted graphics on nearly every page. One estimate puts the general number of figures per textbook page at around 1.3 (Roth et al., 1999). As education continues to shift toward the very visual, digital landscape, encounters of graphical representations and summaries are likely to increase.

Unfortunately, while graphically representing information can be a powerful approach in education, not all students have equal access. For students with visual impairments (ie, those who are blind or have low vision), it is a very different, oftentimes frustrating process to access, extract, and interpret the same information from an image as someone with vision (Hullman et al., 2011). The impact of unequal access to graphical information can be found in all academic areas, but particularly in the science, technology, engineering, and mathematics (STEM) content areas. This graphical access challenge contributes to fewer professionals with visual impairments in STEM disciplines as well as higher unemployment rates (44%) overall (McDonnall & Sui, 2019).

☑Earn CEs Online

by answering questions on this article.

For more information, visit:

https://www.aerbvi.org/

Nonvisual Access to Graphical Information

As more and more students with visual impairments use assistive technology in classrooms, there are some mechanisms to help mitigate the challenge of graphical accessibility (Kelly, 2009; Zhou et al., 2011). Graphics are currently accessed without vision in two primary ways: tactually (by touch), aurally (by audio), or a combination of the two. Images can be rendered tactually in a number of ways, most commonly with the use of embossers (e.g., Tiger embosser (ViewPlus, 2018), heat-sensitive paper (e.g., American Thermoform, 2018), and material medleys such as Wikki-Stix, “puff paint,” pipe cleaners and rubber stickers. Their use, however, is limited and prohibitive. For instance, tangible, tactile graphics require significant time to produce, often require a professional to create, and can be expensive (Gorlewicz et al., 2018). Additionally, these techniques are incapable of quick, dynamic changes which occur often within the modern classroom.

In addition to tactile graphics, there has also been further development of text-based audio rendering of graphics, which have become much more popular in recent years due to the growth of mobile devices, such as smartphones and tablets, and personal computers. Screen readers such as VoiceOver (Apple), TalkBack (Google), and JAWS (Freedom Scientific) aurally present textual information in the digital space. In addition to text-based information, screen readers are capable of relaying some graphical information, but this is contingent on the appropriate use of image labels (e.g., “alt text”). The relationship among programmatical usage of alt text, screen reader users, and description is complex. Although labels are easy to create from both an author and a developer standpoint, the information and its delivery are rarely sufficient for complex images. These labels should contain pertinent information about a digital image, be it an illustration, photograph, or data representation, with the necessary level of detail for the context in which the image is presented. Ideally, the description of an image conveys equivalent information to the nonvisual user to which the visual user has access. Unfortunately, image descriptions are not usually populated by individuals who are well-versed in accessible description; therefore, the quality and usefulness of the descriptions can vary widely, if they are present at all. This variability directly affects learners who must use these graphics to perform successfully in their classes (Singleton & Neuber, 2020). This challenge is further magnified in STEM courses where data representation and figure illustrations play an important, meaningful role in understanding textual content.

Recently, an alternative to tactile graphics and text-to-speech audio outputs has emerged in the form of the multimodal touchscreen. Touchscreens, such as those found on tablets and smartphones, are popular devices capable of touch, audio, and visual feedback. The promise of the multimodal touchscreen has been demonstrated as an alternative means of accessing visual content (e.g., Giudice et al., 2012; Gorlewicz et al., 2018, 2020; Klatzky et al., 2014; Palani et al., 2020). A set of perceptually motivated and scientifically evaluated guidelines for rendering accessible, digital graphics on a touchscreen have been derived from previous work in the multimodal touchscreen space (Gorlewicz et al., 2020; Palani et al., 2020). The current work elaborates on this established precedent, evaluating touchscreen-based graphic renderings in educational contexts and compares it to the current state-of-the-art for accessing graphics using embossed, tactile printouts.

Creating Graphics on Multimodal Touchscreens

Multimodal graphics take advantage of the visual, auditory, and haptic outputs of a touchscreen to create diverse means with which to explore and understand an image. The process of creating graphics on multimodal touchscreens accessible to individuals with visual impairments has been informed by a number of key works (e.g., Gorlewicz et al., 2020; Palani et al., 2020). In short, to make a digital graphic accessible, the following must be taken into account: rendering of key graphical elements, assigning feedback to graphical elements, promoting good user search strategies, and making hardware adaptations (when necessary) (Gorlewicz et al., 2020). The graphics used in this work follow these guidelines.

For bar charts, the title, bar labels, and axis labels are announced when users engage with them by tapping or sliding their finger across the screen. When encountered, bars announce their data group and vibrate with different patterns corresponding to the data groups. To avoid fatigue due to audio repetition, bars only announce their labels the first time a user engages with a bar with their finger. If a user wishes to hear the label again, they can either enter and exit another bar or double-tap on the bar. For our bar charts, the chart's y-axis grid lines announce their value aurally, removing the necessity for users to trace them back to the y-axis to obtain the value at that point. Double-tapping anywhere on the bar will announce the value of the bar at that point. See Figure 1 (left) for a visual example of a bar chart rendered for this work.

For geometry figures, line segments vibrate to support the following of the outline around the shape and obtain an overall understanding of the shape. Similar to the bars of the bar chart, segment labels (e.g., XY), measurements (e.g., 23 cm), and directions (e.g., horizontal, vertical, and diagonal) are also aurally announced the first time they are encountered. The vertex point at which two line segments meet is differentiated by an audio effect (“ding”) unless the vertex point has a measurement itself in which case it's value is announced (“90 degrees”). Visually, right angles are denoted with a red square to indicate their value. Finally, to denote the inside of the shape from the outside, a clicking noise is played when a user is moving their finger around on the inside. The clicking noise signifies that, although there is nothing in the space that is being explored, it is still important to the understanding of the shape. See Figure 1 (right) for a visual example of a geometry figure rendered for this work.

Contributions

Access to graphical information presents a unique challenge for students with visual impairments participating in STEM where success is thoroughly interwoven with the understanding and use of graphics. This work demonstrates how multimodal touchscreens, which utilize auditory, touch, and visual modalities, can be used to facilitate the understanding of digital, STEM graphics. The objective of this work is to assess the application of previously developed guidelines in the context of two categories of mathematics graphics: bar charts and geometry figures. The process of creating multimodal, STEM graphics, is shared and findings are presented from a study done with school-aged students currently enrolled in STEM classes. The findings from this work contribute to the assessment and value estimation of using multimodal, touchscreen devices to support graphical learning for students with visual impairments.

Methods

This study explored the affordances of multimodal touchscreen graphics as compared with traditional tactile embossed and large-print graphics. Three bar charts and three geometry figures were rendered on the touchscreen and in print (large print and embossed). For both digital and physical mediums, students were evaluated for 1) information orientation and extraction ability and 2) attitudes towards both mediums.

Information extraction is defined as the ability to obtain information from the bar charts and geometry figures (i.e., labels, measurements, and titles). Information orientation is defined as the ability to understand where information can be located (i.e., the bars of a bar chart). Information orientation helps inform if participants are able to recognize key aspects of the graphics whereas information extraction helps inform participant ability to use the information to answer questions.

The following hypotheses were formulated:

Students will perform similarly on information orientation tasks using graphics presented with a touchscreen and standard print mediums.

Students will perform similarly on information extraction tasks using graphics presented with a touchscreen and standard print mediums.

Students will have similar attitudes towards graphics presented with a touchscreen and standard print mediums.

Experimental Design

This experiment utilized a between-subjects design to investigate two Render conditions (touchscreen vs. print) of two graphic types (bar charts and geometry figures). Participants were randomly assigned to be presented with either the bar charts or the geometry figures to minimize fatigue effects in each condition and account for time constraints. Graphic types were presented using two mediums: touchscreen and print (depending on the needs of the participant). The order in which the mediums were presented to the participant was randomized to help mitigate possible learning effects. This study was approved by the presiding university's Institutional Review Board.

Materials

Six (3 bar charts, 3 geometry figures) graphics were created for this study (Figure 2). One bar chart and one geometry figure were used for participant training.

Samsung Galaxy Tab S3 tablets were used to present the material in the touchscreen condition. All touchscreen graphics were made according to previously established guidelines (Gorlewicz et al., 2020; Tennison et al., 2020) regarding the creation and rendering of multimodal graphics. Touchscreen-based graphics were outfitted with visual, audio, and touch feedback (see Figure 1 for an example of how feedback was assigned).

A ViewPlus Tiger EmPrint embosser was used to create the tactile embossed materials. Embossed graphics were created by following the guidelines for graphics in educational contexts of the Braille Authority of North America (BANA, 2010). Large-print graphics were similarly made to educational standards. Print graphics were of similar size to those on the touchscreen to facilitate mapping between mediums and to decrease the cognitive cost of working between mediums. To this end, large-print and touchscreen graphics also utilized the same colors. A demographics questionnaire was administered verbally to each participant and included age, sex, diagnosis, and age of onset, as well as educational experience (i.e., courses taken, tactile graphics experience).

A set of questions was created to assess participant ability to orient to and extract information from the graphics (see Tables 1 and 2). Questions were developed through an iterative process with experts in the field of STEM education for students with visual impairments. Questions were criterion-referenced and related to the graphics that are typical of a classroom-based assessment of graphical literacy.

Table 1.

An Example of Information Orientation and Extraction Questions Asked to Participants in the Bar Chart Group by Medium.

Question type	Touchscreen	Print
Orientation	How many bars are on the page? What is the title of the graph? What is the x-axis labeled as? How many bars on the page? What is the title of the graph? What is the x-axis labeled as? What group selected more milk products?	How many bars are on the page? What is the title of the graph? What is the y-axis labeled as? How many bars are on the page? What is the title of the graph? What is the y-axis labeled as?
Extraction	What percentage of voters selected “yes,” that they planned to vote in 2020? What percentage of voters selected “no,” that they did not plan to vote in 2020? How many boys selected grain products? How many girls selected milk products?	What percentage of voters selected “yes,” that they planned to vote in 2016? What percentage of voters selected “no,” that they did not plan to vote in 2016? What percentage of voters selected “yes,” that they planned to vote in 2016? What percentage of voters selected “no,” that they did not plan to vote in 2016?

Table 2.

An Example of Information Orientation Questions Asked to Participants in the Geometry Group by Medium.

Question type	Touchscreen	Print
Orientation	How many sides does the figure have? Are any of the sides parallel? How many sides does the figure have? Are there any right angles? What shape is this? (If triangle response) What kind of triangle is this?	How many sides does the figure have? Are any of the sides parallel? How many sides does the figure have? Are there any right angles? What shape is this? (If triangle response) What kind of triangle is this?
Extraction	How do you know those sides are parallel? What is the length of side WX? What is the length of side XY? What is the measure of angle WZY? (If right angles) What is the length of the side opposite the right angle? What is the measure of the left angle of the shape?	How do you know those sides are parallel? What is the length of side WZ? What is the length of side XY? What is the measure of angle ZYX? (If right angles) What is the length of the side opposite the right angle? What is the measure of the left angle of the shape?

Eleven questions were asked about the bar charts and 12 questions were asked about the geometry figures (some geometry questions were dependent on correctly answering a previous question and thus participants could have received additional questions). Out of the 11 questions related to bar charts, 4 questions measured information extraction and 7 measured information orientation skills.

Participants received a questionnaire after using each medium, assessing participant attitudes towards graphics presented on the touchscreen and in print. This questionnaire consisted of 7 items rated on a Likert scale of 1–7, where 1 indicated the statement was absolutely not true and 7 indicated the statement was absolutely true. The items are as follows:

1–4. The graphics were helpful, interesting, pleasant, and easy to understand.

5. It was fun to use the graphics.

6. It was boring to use the graphics.

7. It was confusing to use the graphics.

Participants

Twenty participants, who ranged in age from 13 to 21 years, were recruited from two residential schools in the Western and Southern regions of the United States that serve students with visual impairments (see Table 3). All participants had some form of visual impairment. A majority (N = 16) of the participants were congenitally visually impaired. However, four participants acquired visual impairment 2 to 3 years prior to the start of the study. A majority of the participants were students with low vision who preferred to use large-print materials during the study rather than embossed materials. All participants were enrolled in high school and had taken or were currently taking courses in mathematics (including geometry, algebra, trigonometry, and pre-calculus) and science (including biology, environmental physics, chemistry, and anatomy). All participants received a $50 gift card for taking part in the study.

Table 3.

Participant Demographics.

Number	Sex	Age in years	Diagnosis	Print preference
1	M	18	Retinal dystrophy	Large print
2	F	16	Congenital	Large print
3	F	21	No corneas	Large print
4	M	15	Coloboma	Large print
5	M	15	Cortical visual impairment	Large print
6	F	15	Unknown	Large print
7	M	15	Retinopathy of prematurity	Large print
8	M	15	Unknown	Large print
9	M	15	Leber's congenital amaurosis	Large print
10	M	13	Optical nerve hyperplasia	Large print
11	F	18	Tuberculosis	Tactile Embossed
12	M	17	Optic nerve hypoplasia	Large print
13	F	18	Optic nerve inflammation	Tactile embossed
14	M	18	Peters anomaly	Tactile embossed
15	M	18	Rieger syndrome	Large print
16	M	18	Optical nerve hyperplasia	Large print
17	M	18	Albinism and nystagmus	Large print
18	F	18	Retinitis pigmentosa	Large print
19	F	18	Aniridia and cataracts	Large print
20	F	17	Retinopathy of prematurity	Tactile embossed

Procedure

Four researchers collected data at two participating residential schools. The researchers participated in an 1-h training session with the lead experimenter, in which a written protocol was established. Experimenters conducted a mock study with the lead experimenter to establish adherence to the protocol. Experimenters followed this protocol for each participant session.

Two researchers were stationed at each school. Consent and assent forms were sent to the student's parents for their approval of their children's participation in the study. Children with cognitive and intellectual difficulties were excluded from the study. Out of the two researchers in each school, one presented the bar charts and the other presented geometry figures to participants.

The study began with verbal confirmation of consent from the participants. Then, the demographics questionnaire was administered, and the experiment commenced. Study sessions took approximately 1 h.

At the start of exposure to the touchscreen and print graphics, participants were trained to use the medium. Training consisted of one bar chart or one geometry figure and four to five questions about the graphic. These were criterion questions that were to be answered correctly before proceeding to the experimental session, ensuring that participants understood how to use the medium and the task requirements. During the training time period, the participants were encouraged to ask questions and seek help, since no help would be given during the experimental session to reduce experimenter interference bias.

Experimental conditions included two bar charts or two geometry figures, depending on the participant's graphic type assignment. During the experimental condition, participants were given as much time as they needed to initially explore each graphic (typically under 5 min) before being prompted by the researcher to answer five to six questions about the graphic, which were designed to measure participants’ skills in orienting themselves to and extracting information from the graphic. Participants were allowed to consult with the represented figures during and between questions throughout the study. After each medium, participants were administered the attitudes questionnaire. The researchers documented participant answers and the sessions were video recorded for data analysis. At the close of the 60-min study session, participants were compensated for their time.

Results

Both descriptive statistics (means, standard deviations, and histograms) and inferential statistics (paired-sample t-test) were used to test the three hypotheses. To test the first hypothesis, which states that students will perform similarly on information-orientation tasks using graphics presented with a touchscreen and standard print graphic mediums, data from participants’ responses to questions that indicated their ability to orient themselves to the bar charts and geometry figures were used for the analysis (see Table 4).

Table 4.

Descriptive Statistics for Data Orientation.

Variables	Bars		Geometry
Variables	Touchscreen bars	Embossed/ large print	Touchscreen	Embossed/ large print
Correct responses [CR(SD)] *	6.6(.51)	5.7(.67)	5.5(1.08)	5(1.24)
Maximum	7	6	6	6
N	10	10	10	10

CR(SD) = Mean correct response (standard deviation).

Using the touchscreen, participants were able to orient to information on 6.6 (SD = .51) out of 7 bar chart items and 5.5 (SD = 1.08) out of 6 geometry items on average. Five participants used tactile images and 15 used large-print images in the print graphic condition. For participants who used tactile graphics, their average performance on the bar charts for the orientation questions was 6 (SD = .00) out of 7 questions, and on the geometry figures was 3.6 (SD = 1.15) out of 6 questions. Additionally, for participants who used large print, their average performance on the bar charts for the orientation questions was 5.1 (SD = .64) out of 7 questions, and for the geometric figures was 4.5 (SD = 1.27) out of 6 questions. Since participants’ performance on the tactile and large-print graphics were similar, they were combined for future analysis.

These data were screened for normality and skewness. Observing the histogram indicated that participants’ performance was normally distributed for the touchscreen bar charts and geometry figures as well as the embossed geometry figures. Skewness was observed in the performance of the embossed bar chart condition, where participants performed slightly higher than average, however skewness and kurtosis tests indicated that this difference was acceptable. Skewness for all variables was between −2 and +2, and kurtosis was between −7 and +7 (Byrne, 2010; George & Mallery, 2010; Hair et al., 2010). Therefore, a paired sample t-test was conducted to test whether there were significant differences in performance between the touchscreen and embossed condition for the orientation questions.

The paired sample t-test suggested that participants performed significantly better in orienting themselves to information on the touchscreen for bar charts compared to the print medium, t(9) = 3.85, p < .05, 95% CI (.37,2.03), d = .73. However, no significant difference was found for geometry figures in either condition, t(9) = 2.23, p = .052, 95% CI (-.01,1.39), d = .70.

Participants’ responses to bar charts and geometry figures information extraction questions were analyzed to test the second hypothesis that students will perform similarly on information extraction tasks using graphics presented with a touchscreen and standard print graphics mediums. Five participants from the bar chart print condition had incomplete response sets and were removed from the analysis. Only 15 responses (5 using tactile images and 10 using large print) were analyzed to investigate the information extraction hypothesis. In the touchscreen condition, the average performance on the bar charts was .80 (SD = .83) out of 6 questions and for the geometric figures was 4.7 (SD = 1.25) out of 6 questions. For participants who used tactile images for the bar charts their performance was 0 (SD = .0) out of 6 questions, and for the geometric figures was 3.6 (SD = 1.52) out of 6 questions. For participants who preferred large print, the average performance for the bar charts was 0 (SD = .0) out of 6 questions and for geometric figures was 4.7 (SD = 1.11) out of 6 questions. Since participants’ performance for tactile images and large print were similar, they were combined for future analysis.

For the bar charts, the deviation range of participant responses from the correct responses was between −10 to 69 when using the touchscreen medium and −12 to 23 when using the print medium. Deviation from the correct response was determined by subtracting participant responses from the correct answer. If the participant responded “54” to a question with a correct answer of “57,” the deviation score would be −3.

Participants were also evaluated on their ability to extract the correct information from the bars of the bar charts within a range determined by the average finger pad width centered on the correct answer (Dandekar et al., 2003). For within-a-range correct responses, participants answered an average of 3.5 (SD = .52) bar chart items correctly using the touchscreen and 2.8 (SD = 1.6) items correctly using the print mediums (Table 5).

Table 5.

Descriptive Statistics for Data Extraction (Bars).

Variables	Touchscreen exact (E*)	Embossed/ large exact (E*)	Touchscreenrange (R*)	Embossed/ range(R*)
Correct responses [CR(SD)] *	.90(.74)	.00 (.00)	3.5(.52)	2.8(1.64)
Maximum	4	4	4	4
N	5	5	5	5

CR(SD) = Mean number of responses that were correct (standard deviation).

E = Mean exact correct response.

R = Mean within the range correct responses.

To test the third hypothesis that students will have similar attitudes towards graphics presented with a touchscreen and standard print graphic mediums, differences in attitudes between the mediums were analyzed. Participants had a significant positive attitude towards the touchscreen graphics as compared to the print graphics, t(19) = 2.67, p < .001, 95% CI (.11–1.06), d = .94. All participants (N = 20) felt that the touchscreen was significantly more helpful, interesting, pleasant, easy to understand, fun, and less confusing than the large-print or tactile graphics.

Discussion

The results of this experiment are promising for the application of multimodal touchscreens as a means of accessing graphical information in STEM educational settings. Extracting and orienting to information was highly successful by participants using the touchscreen and was the preferred means of accessing graphical information when compared to the static, print image for both geometry figures and bar charts.

Bar Charts

Extracting exact values of the bars was difficult, but not impossible for students when using the touchscreen graphics. Although no student using the print graphics was able to extract the exact bar values for the bar charts, touchscreen users were able to extract almost 1 correct value on average. This finding highlights a particular challenge for touchscreen and print graphics, but the latter in particular. With print graphics, if the axis resolution is not fine enough, it is impossible to do more than guess at what value the top of the bar reaches. However, touchscreens have a slight advantage in that obtaining the maximum height of the bar can be programmed in a way that is accessible by a gesture performed at the top of the bar. In the touchscreen condition, the limiting factor is a student being able to tell where the bar stops, which is why the bars were outfitted to produce haptic feedback. A student knows when they have reached the top of the bar when the tablet stops vibrating.

When correct response was widened from precise to a range based on the average finger pad width, participants using the touchscreen were still more successful in interpreting the bar value, utilizing the touchscreen feedback to obtain the correct bar value on average for 3.5 (N = 4) of responses compared to 2.8 (N = 4) in the print condition.

Geometry Figures

Geometry can be challenging for many students with visual impairments. Since it was uncertain whether the study's participants had prior experience with geometry, no questions were created for which participants would have to derive information through calculation. Therefore, participants’ ability to orient themselves to the graphic and its components was of particular interest when they were evaluating the geometry figures. The multimodality of the geometry figures helped participants quickly understand the layout of the geometry figures. For instance, although a majority of participants leveraged their residual vision for most tasks, having both sound and vibration was beneficial for anchoring to important information, such as right angles and vertex points. However, participants spent far less time interacting with the geometry figures than the bar chart figures, likely because of the redundancy of information (most information was presented both visually and aurally) on the graphic for the low vision user.

Participant Feedback

Participants remarked that the touchscreen felt more interactive and responsive and they were appreciative of the variety of means with which the graphical information could be accessed and interacted. When using the touchscreen, participants could use vision, touch, and sound to obtain information about the graphic. The variety of output and increased information processing channels strengthens learning and interpretation. Participants expressed that, even if they did not directly benefit from all types of feedback, they “liked having it as an option” and “I will still use [those modalities].” The methods participants had at their disposal to interact with the touchscreen graphics met expectations. The gestures and responses were reminiscent of their experiences using their personal mobile devices, leading to quick information extraction times after they became oriented to the device.

During observation, it was readily apparent that participants were less engaged when using the print graphics. Most participants left the print graphic on the table, answered questions about it quickly and without much interactivity, opting to close the distance between themselves and the print only when necessary. With the print medium, it was easy to get a quick, overall understanding of the graphic being presented, but it offered none of the additional benefits of the touchscreen, which was reflected in the students’ responses on the attitudes questionnaire. The touchscreen offers students dynamic and diverse interaction with the material which, as stated previously, increases the channels with which students are able to obtain information.

In contrast, although for many participants touch and audio was unnecessary due to the magnified visuals, participants were more involved exploring the graphic on the touchscreen. Participants often remarked to the experimenter that, although they did not need to explore the touchscreen beyond sight, they wanted to experience the audio and touch effects. Additionally, a couple of participants in the geometry figure group remarked that the sound and touch feedback combined with the visuals made for a more memorable experience.

Limitations

Due to the sample size, a combination of both inferential and descriptive statistics were necessary to garner meaningful takeaways from these data. The research team acknowledges that these takeaways, therefore, cannot be generalized to all students with visual impairments but determined that interpreting these data as a case study still leads to meaningful insight into the state of using multimodal touchscreens to facilitate the understanding nonvisual access of STEM graphics. Although caution should be used in interpreting these data due to the small sample size, it is encouraging to find that those students who have used the multimodal graphics in general did so successfully and expressed a preference towards using them.

Although efforts were made to mitigate possible learning effects (e.g., randomization of condition given first), we acknowledge the presence of learning effects as a threat to the validity of our study, especially as each participant received the same graphics in both print and touchscreen mediums. Participants were not told that they could experience the same graphic more than once and were asked different questions of the graphic each of the two times it appeared to them, encouraging participants to explore and interpret the graphic as one, which was a totally new approach for the students. We acknowledge that some perceptive participants may have recognized that they were the same graphic.

In the experiment, a majority of participants have low vision despite recruitment efforts to include a more diverse sample of high school–aged students with visual impairments. Therefore, our findings cannot be broadly generalized to those with different needs in the visual impairment community. However, the graphics created for the touchscreen were created according to the standards in each modality, so that students could leverage the modalities with which they felt most comfortable using and found most helpful. Although most of the students with low vision could see the touchscreen in some capacity, the students still enjoyed using the touchscreen features that offered an alternative means of accessing information than just through a visual interface. It was noted that the audio information was helpful for quick reminders of important information, such as which bar they were on or the value of a line segment, without having to pick up the tablet from the table. Students also found the touch feedback enjoyable, remarking that it made the graphic feel more “real” and interactive when compared to the static print image.

Although students could read the labels of the bars by tapping when using the touchscreen, a bar chart legend information was missing from the static, large-print graphics, prompting participants to either try to ask what the bars represented (information the experimenter could unfortunately not provide to avoid biasing the experiment) or make educated guesses as to what the bars could be labeled according to context clues. Most students were able to appropriately guess the labels, but some were not, which led them to incorrectly respond to questions when they might have otherwise been able to garner that information had the legend been available.

Implications for Practitioners

This study demonstrates the ability of students with visual impairments to extract pertinent information from touchscreen-based graphics with a level of effectiveness that is comparable to traditional types of graphics (large-print and embossed, tactile graphics). The students were more engaged when they were working with the tablet, likely because the use of graphics on the touchscreen was novel, but also because it provided multimodal information about the graphic in ways that were both exploratorily familiar and offered broader means of access. The field of visual impairment should consider ways to utilize novel technologies, like touchscreens, to provide more diverse access to graphical information for students with visual impairments.

Practitioners should also recognize that the education field as a whole is shifting to a more digitized paradigm for providing educational materials (Presley & D’Andrea, 2008). Teachers of students with visual impairments must work diligently to not let their students be “left behind” by mainstream educational platforms. Practitioners must ensure that students are provided access and training on using appropriate assistive technology devices and particular focus should be on those that are more universally designed, such as touchscreen tablets (Presley & D’Andrea, 2008).

Future Directions

This work informs the need to create a robust system capable of providing all types of graphics dynamically to students with visual impairments and demonstrates that multimodal touchscreens are a promising platform for graphical content (e.g., Gorlewicz et al., 2018, 2020; Palani et al., 2020; Tennison et al., 2020). However, additional data are necessary to demonstrate that students are able to “read” and understand graphics presented multimodally on touchscreen tablets at a rate that is on par with traditional, tactile graphics and description. Additionally, a better understanding of the presentation of more complex graphics on multimodal touchscreens is necessary to meet the needs of students in advanced courses. The importance of providing diverse access to educational, graphical information via multimodal touchscreen platforms must also be demonstrated through more experimentation with larger samples and more diverse stimuli. The educational field needs to emphasize to developers of smartphones and tablets that there is a need for robust audio capabilities and strong, customizable vibro-tactile motors (Wild et al., 2022). This work presents preliminary data on the promise of multimodal, touchscreen graphics and demonstrates that the structure with which the graphics were created were helpful, however this study is only the beginning of the investigation into what is possible for the platform.

Footnotes

Authors' Note

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Science Foundation, (grant numbers 1644471, 1644476, 1644491, 1644538, 1822800, 1845490, 2048356, 2048394, 2048428).

References

American Thermoform. (2018). Swell touch paper. http://www.americanthermoform.com/product/swell-touch-paper

Braille Authority of North America (BANA). (2010). Guidelines and standards for tactile graphics. http://www.brailleauthority.org/tg/index.html

Byrne

B. M.

(2010). Structural equation modeling with AMOS: Basic concepts, applications, and programming. Taylor and Francis Group Publication.

Dandekar

Raju

B. I.

Srinivasan

M. A.

(2003). 3-D finite-element models of human and monkey fingertips to investigate the mechanics of tactile sense. Journal of Biomechanical Engineering, 125(5), 682–691. https://doi.org/10.1115/1.1613673

Emerson

R. W.

Anderson

(2018). What mathematical images are in a typical mathematics textbook? Implications for students with visual impairments. Journal of Visual Impairment & Blindness, 112(1), 20–32. https://doi.org/10.1177/0145482X1811200103

George

Mallery

(2010). SPSS for Windows step by step: A simple guide and reference 18.0 update. Taylor & Francis.

Giudice

N. A.

Palani

Brenner

Kramer

K. M.

(2012). Learning non-visual graphical information using a touch-based vibro-audio interface. Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility (Assets’12) (pp. 103–110), ACM New York, NY, USA.

Gorlewicz

J. L.

Tennison

J. L.

Palani

H. P.

Giudice

N. A.

(2018). The graphical access challenge for people with visual impairments: Positions and pathways forward. In Interactive Multimedia-Multimedia Production and Digital Storytelling. IntechOpen. https://www.intechopen.com/chapters/64669

Gorlewicz

J. L.

Tennison

J. L.

Uesbeck

P. M.

Richard

M. E.

Palani

H. P.

Stefik

Smith

D. W.

Giudice

N. A.

(2020). Design guidelines and recommendations for multimodal, touchscreen-based graphics. ACM Transactions on Accessible Computing (TACCESS), 13(3), 1–30. https://doi.org/10.1145/3403933

10.

Hair

J. F.

Black

W. C.

Babin

B. J.

Anderson

R. E.

(2010). Multivariate data analysis: A global perspective. Prentice Hall.

11.

Hullman

Adar

Shah

(2011). Benefitting InfoVis with visual difficulties. IEEE Transactions on Visualization and Computer Graphics, 17(12), 2213–2222. https://doi.org/10.1109/TVCG.2011.175

12.

Kelly

S. M.

(2009). Use of assistive technology by students with visual impairments: Findings from a national survey. Journal of Visual Impairment & Blindness, 103(8), 470–480. https://doi.org/10.1177/0145482X0910300805

13.

Klatzky

R. L.

Giudice

N. A.

Bennett

C. R.

Loomis

J. M.

(2014). Touch-screen technology for the dynamic display of 2D spatial information without vision: Promise and progress. Multisensory Research, 27(5-6), 359–378. https://doi.org/10.1163/22134808-00002447

14.

McDonnall

M. C.

Sui

(2019). Employment and unemployment rates of people who are blind or visually impaired: Estimates from multiple sources. Journal of Visual Impairment & Blindness, 113(6), 481–492. https://doi.org/10.1177/0145482X19887620

15.

Palani

H. P.

Fink

P. D.

Giudice

N. A.

(2020). Design guidelines for schematizing and rendering haptically perceivable graphical elements on touchscreen devices. International Journal of Human-Computer Interaction, 36(15), 1393–1414. https://doi.org/10.1080/10447318.2020.1752464

16.

Presley

D’Andrea

F. M.

(2008). Assistive technology for students who are blind or visually impaired: A guide to assessment. AFB Press.

17.

Roth

W. M.

Bowen

G. M.

McGinn

M. K.

(1999). Differences in graph-related practices between high school biology textbooks and scientific ecology journals. Journal of Research in Science Teaching: The Official Journal of the National Association for Research in Science Teaching, 36(9), 977–1019. https://doi.org/10.1002/(SICI)1098-2736(199911)36:9<977::AID-TEA3>3.0.CO;2-V

18.

Singleton

K. J.

Neuber

K. S.

(2020). Examining how students with visual impairments navigate accessible documents. Journal of Visual Impairment & Blindness, 114(5), 393–405. https://doi.org/10.1177/0145482X20953268

19.

Tennison

J. L.

Uesbeck

P. M.

Giudice

N. A.

Stefik

Smith

D. W.

Gorlewicz

J. L.

(2020). Establishing vibration-based tactile line profiles for use in multimodal graphics. ACM Transactions on Applied Perception (TAP), 17(2), 1–14. https://doi.org/10.1145/3383457

20.

ViewPlus. (2018). Home. http://www.viewplus.com

21.

Wild

Smith

D. W.

Kelly

Fast

(2022). The National Research Agenda for Science, Technology, Engineering and Mathematics (STEM) for Students with Visual Impairments. The Ohio State University. https://u.osu.edu/nationalresearchstemvisualimpairment/the-national-research-agenda-document

22.

Zhou

Parker

A. T.

Smith

D. W.

Griffin-Shirley

(2011). Assistive technology for students with visual impairments: Challenges and needs in teachers’ preparation programs and practice. Journal of Visual Impairment & Blindness, 105(4), 197–210. https://doi.org/10.1177/0145482X1110500402