Abstract
Feedback-based iterative refinement is important in the development of any human-computer interface. The present work aims to evaluate and iteratively refine an immersive learning environment called Scale Worlds (SW), delivered via a head-mounted display (HMD). SW is a virtual learning environment encompassing scientific entities of a wide range of sizes that enables students an embodied experience while learning size and scale. Five usability experts performed think aloud while carrying out four interactive tasks in SW and compared three different design options during A/B testing. Improvement features based on the feedback from an earlier SW usability evaluation as well as HMD-specific features were examined. Usability experts completed the post-study system usability questionnaire, the NASA task load index, and a bipolar laddering survey that collected subjective perception of specific SW features. Results show that the progress panel (an improvement feature) was informative while the instructions (another improvement feature) caused clutter. The experts indicated clear usability preferences during A/B testing, which helped resolve three sets of theory-usability conflicts. The overall assessment of SW paved a path for theory-usability balance and provided valuable insights for designing and evaluating usability in immersive virtual learning environments.
Keywords
Introduction
Virtual reality (VR) is a computer generated stereoscopic visual environment that affords embodied learning experiences, where learners may physically engage in learning activities (Dalgarno & Lee, 2010; Johnson-Glenberg et al., 2014) and visualize metaphors of intangible abstract STEM concepts such as forces (Altmeyer et al., 2020; Chen et al., 2020). Of the various integral STEM concepts, the Next Generation Science Standards propose “scale, proportion, and quantity” as a crosscutting concept that can aid students in making connections across scientific disciplines (National Research Council, 2022). Research has shown that learners hold inaccurate ideas about the size of scientifically relevant entities (Delgado, 2013; Magana et al., 2012; Tretter et al., 2006). Traditional textbook-based education can be limiting: it uses static visual information, emphasizes facts over process, and fails to guide students to a more integrated understanding of science concepts (Duncan et al., 2011). Moreover, embodied cognition theory emphasizes the coupling between the use of body motions and cognitive activities (Arroyo et al., 2017; Skulmowski & Rey, 2018), which is not an experience directly available in textbooks. It is important to devise a learning environment that supports the conceptualization of size and scale through embodied cognition.
Our research team has developed Scale Worlds (SW), which is a virtual learning environment encompassing scientific entities of various sizes. The development of SW was guided by theories of visual representations (Peterson et al., 2021) and scale cognition (Delgado, 2013; Magana et al., 2012). In SW, scientific entities—such as a right whale and a water molecule—are distributed among distinct environments, each of which corresponds to an exponent in scientific notation.
The first iteration of SW was implemented in an immersive Cave Automatic Virtual Environment (CAVE) because a CAVE affords users seeing their own bodies and supports embodied cognition as users compare their bodies against the virtual scientific entities. The usability evaluation of SW-CAVE (Wu et al., 2022) generated feedback for iterative design (Savage, 1996). A user-centered design process (Abras et al., 2004) was followed to gather and integrate user feedback and requirements during usability testing, which then informed the refinement of SW. Presently, a head-mounted display (HMD) version of SW has been developed for regular classroom use. The team has followed a design cycle (Kimbell et al., 1991) to develop HMD-appropriate features.
The present work iteratively refines and integrates improvement features of SW, evaluates its usability, balances theory-usability design decisions, and recommends subsequent iterative refinement strategies for future immersive virtual learning environments.
Related work
Here we provide an overview of SW and the usability evaluation of SW-CAVE as part of the related work (Wu et al., 2022). The SW-CAVE usability evaluation examined usability factors such as likeability of various interactions through A/B testing, and to help resolve any contradictions between theory-driven design features of SW.
Description of Scale Worlds
SW is a virtual learning environment encompassing scientific entities of a wide range of sizes. In SW, users can apparently shrink or grow as they travel to different “scale worlds,” each with a scientific entity at a size removed from others in tenfold increments. In each “world,” users can simultaneously view five entities of various sizes (Figure 1). A user interface (UI) in the form of a panel with numeric information is placed in front of the entities (Figure 2).
A zoomed-out view of Scale Worlds. Users can simultaneously see the following five entities when they are in “Right Whale World”: robin, human, Right Whale, football field, Brooklyn Bridge.
Instructions on the controller and navigation panel. A user can point at the exponent on the numeric panel using the ray cast from the wand, then click a specific button to enter another scale world. A user can also move between worlds by pointing at the decimal of the standard notation and clicking a corresponding button. The instructions would gradually disappear after a user has scaled using the interactive elements three times.
Major takeaways from the SW-CAVE usability evaluation
While using SW-CAVE, usability experts expressed the need for increased functionality to support learning scale, including instructions and a navigational map. Three design conflicts between usability and theory arose during A/B testing (Wu et al., 2022). First, there was disagreement in terms of the number of armatures used to provide measurement and suggest size. Specifically, usability experts in the SW-CAVE evaluation were split between the “forest” style posts (a configuration that had ruler-like columns extending in all directions like a forest to increase perspectival cues) and “path” style posts (a less-distracting double row of the same columns). Second, as for the interaction mode, usability experts were also split. Some preferred pressing the buttons on the wand because this was an expected and easier input method, while others preferred the swipe gesture interaction because it was interesting and aligned with scaling up/down (“swiping” up/down) to the other scale worlds. Third, as for the type of numeric animation, two usability experts preferred a longer staggered animation because it enabled them to see the details during scaling, while three experts preferred a simultaneous animation because it was smoother and faster. Given the inconclusiveness of the results, the present work will further examine these design elements.
Methods
Participants
Five usability experts (3 males, 2 females) of mean age 33.6 years (range 26-46) were recruited. The recruitment of five usability experts followed usability literature (Faulkner, 2003) as five was considered a favorable cost-benefit ratio. Informed consent was obtained from all participants. Eligible participants had a graduate degree with relevant experience in human factors or have at least two years of experience in user interface design and evaluation and meet the inclusion criteria. All participants reported experience using some type of VR technology.
Equipment
SW was rendered using Unity 3D (version 2019.4.30f1) and displayed using an HMD (VIVE Pro, HTC Corporation, Taiwan). The HMD has a resolution of 1440 × 1600 pixels per eye and has built-in inertial measurement units that monitor a user’s head orientation to deliver the appropriate visual perspectives. Participants interacted with virtual objects and UI with a controller. Participant think aloud was audio recorded using a SONY IC recorder.
Iterative design of Scale Worlds for HMD
In addition to the original features present in SW, interaction instructions shown in Figure 2 (e.g., “Aim at exponent, hold trigger, swipe up or down”) and a progress panel shown in Figure 3 (a number line that tracks the “worlds” that a user has visited) have been added based on the feedback from the SW-CAVE evaluation. Furthermore, teleportation ability and a “home button” were added to SW-HMD to match the common HMD application elements. The teleportation ability allowed users to travel to a place in SW without physically walking (Figure 4). Teleportation was restricted to four points to elicit consistent scale observations (e.g., to prevent a user from assuming a change in location resulted in a change in size), and to reinforce the logarithmic scale of the environment (e.g., allow users to see five entities with tenfold differences in size from a single vantage point). The home button permitted a user to immediately return to the home position (in front of the navigation panel) if they had teleported too far.
Progress panel with a list of icons representing each world. The small icons would only appear once that world has been visited, and then a user could “jump” to that world. Hover state (icon turns red) shown at the Human icon when the ray cast out from the controller hit it.
Four teleport points (red rings) in the environment and were indicated by human-shape red outlines. User could teleport by aiming a pointing ray at a red ring and pulling the trigger button on the controller.
Furthermore, an alternate version of the SW-HMD (SW-HMD-A) was designed and developed based on the team’s informal experiences with first-time users as well as continual reflection on scale cognition theory (Figure 5). This was an attempt to holistically and systematically address the issues, inspired by the team’s experience observing various first-time users of SW-HMD in informal sessions. The SW-HMD-A has a limited floating platform instead of an infinite floor, which emphasizes the size of the entities rather than the environment itself. Free teleportation was implemented to give users more freedom, with the prescribed space of the platform serving as a subtler limit on user movement than the prescribed teleportation points. A pink “home” location serves as a starting point, and a button on the controller allows users to teleport back to it. A “scenic view” location is visible to the side of the entities, roughly corresponding with a previous teleportation point, suggesting a possible vantage point without forced teleportation. Color contrast was adjusted in response to the “void” that is visible around the platform. The posts were reduced to only two, as freer user movement permitted multiple sightlines, reducing the need for the perspectival cues provided by repeated posts.
SW-HMD-A. The HMD alternative version with a limited floating platform. A pink “home” location serves as a starting point.
Experimental procedure and usability evaluation tasks
Four primary tasks
The experimenter introduced SW and briefed the participants on the purpose of the usability evaluation and the associated evaluation tasks. There were four primary evaluation tasks: (1) general exploration of SW, (2) evaluation of UI elements, (3) examination of scale posts, and (4) interactions with scaling elements and animations. The participants were encouraged to think aloud and verbalize feedback whenever they performed actions. Upon completing the four tasks, the Post-Study System Usability Questionnaire (PSSUQ; Lewis, 2002) and the NASA Task Load Index (NASA-TLX; Hart, 1988) were administered.
A/B testing
Following the four primary evaluation tasks, A/B testing was performed to gather user preferences on (1) mode of scaling interactions, (2) armature posts, and (3) duration of scaling animations. Bipolar laddering (Fonseca, 2015), a participatory subjective exploration method on user experience, was administered to understand user perception and preferences for the alternatives.
Improvement features and HMD-specific elements evaluation
Participants then explored the instructions and progress panel, while having the ability to teleport and return “home” using the home button. Their scaling and teleportation preferences and feedback were noted.
SW-HMD-A evaluation
Participants were then transferred to experience SW-HMD-A. Open-ended questions regarding the design features and noticeable differences impacting user experience were verbally asked to guide participants to explore the environment and gather feedback regarding the possible improvements and deficits compared to SW-HMD.
Variables and analysis
PSSUQ and NASA-TLX
The PSSUQ evaluated the usefulness, information quality, and interface quality of SW on a 7-point Likert scale (1=strongly agree, 7=strongly disagree). Any PSSUQ item that received a 4 (neutral) or greater than 4 (disagree) is reported in the Results, as they suggest indifference or dissatisfaction. The NASA-TLX was administered to assess the workload of using SW.
Themes from think aloud
Qualitative, usability comments from think aloud were extracted and then grouped into themes.
Bipolar laddering on preference
Scores and explanatory feedback were collected based on the participants’ experience and likeability of the three features of SW during A/B testing. Participants ranked their sentiment toward an element of SW (positive or negative) and then scored their satisfaction or dissatisfaction on a scale of 0 to 10 (the score could be 0 if the participants felt neutral toward the element), which yielded scores ranged from -10 to 10. The reason for the scores was also explained. Preference rate (percent of participants preferred that the alternative) and average score (average score obtained by five participants) were calculated for each alternative.
Results
PSSUQ
Three items regarding “information quality” and five items regarding “interface quality” were rated with a 4 or higher by more than two participants. Three participants were neutral or disagreed with “It was easy to find the information I needed.” Two participants reported neutral or disagree with “I liked using the interface of this system.” Participants stated that there was too much information on the navigation panel with the instructions appearing in front of the numeric notations, which might lead to the unpleasantness of the user interface. Three participants indicated that the system did not have all the functions and capabilities they expected. Specifically, they wanted to use the controller to interact with the entities, for example, grabbing one entity and moving it around.
NASA-TLX
Participants reported relatively higher levels of mental demand and frustration (Figure 6).
Mean NASA-TLX score for subscales in the order from 1 through 6: Mental demand (mean=16.07, sd=9.44), Physical demand (mean=3.00, sd=6.71), Temporal demand (mean=7.87, sd=4.27), Performance (mean=3.73, sd=2.40), Effort (mean=4.00, sd=1.24), and Frustration (mean=12.80, sd=10.86).
Think aloud
Recurring themes from think aloud during the four tasks were extracted (Table 1). Three participants stated that the instructions on the navigation panel were difficult to read, due to “there is too much text” or “the starburst is distracting.” Two participants were confused about the red outlines of the human-shape paper doll for teleportation. Participants found the teleport points useful by providing them more freedom of exploring SW and different perspectives to view entities and the virtual environment. While they agreed that a home button would be useful, it was indicated that the home button would not be necessary if the home position was set as a teleport point. Three participants were confused about what was meant by a “world,” and three participants reported confusion about the “your current height” text.
Usability issues from think aloud regarding the user interface. Count is the number of participants who has mentioned the same issue.
Bipolar laddering
The preference rate and average score are presented in Table 2. Four participants preferred the button clicks because “it’s simple” while the gesture interaction “requires moving hand all the time.” For posts, one participant preferred the forest post because “it’s good for comparing size.” Three participants preferred the path armature because “like having more space” and “helpful for size comparison.” For scaling animations, all participants preferred the simultaneous animation because “it seems more responsive” and the staggered animation “seems delay.”
Bipolar laddering preference rate and average score for each A/B testing alternative. Note: one participant gave same score for Forest and Plain.
SW-HMD-A evaluation
Four usability experts preferred the SW-HMD-A over the SW-HMD because the SW-HMD-A was “more polished and slicker” and provided the ability to move around more freely. One expert, in SW-HMD, stated repeatedly that it would be “interesting” to be able to move under entities, and appreciated seeing entity details up close in SW-HMD-A. One expert described free teleportation as “being able to chuck myself where I want to be,” and reinforced the benefit of inspecting entities at close proximity. Another expert expressed that at certain distances from entities in SW-HMD, the posts became a “distraction,” and another stated that, in SW-HMD-A, “I think getting rid of the pillars was an improvement.” One expert connected the platform design to the scaling animation: “Something about the treatment of the periphery makes it feel larger … I have a definite feeling of my world shifting more in this one.”
Discussion
The PSSUQ helped identify two major information quality and interface quality problems associated the navigation panel, and their reasons were revealed through think aloud. Specifically, participants frequently commented on the unpleasantness of the dense instructions superimposed on the navigation panel. Three possible reasons were: (1) the red text and starburst made the navigation panel hard to read, (2) a slowly moving human entity behind the text made it hard to read, and (3) instructions disappearing after several actions was somehow confusing. An actionable solution is to provide a tutorial session before the formal experience of SW, with instructions spatially separated from the navigation panel and user height-setting functionality. Participant also commented on the inability to grab and move entities because they wanted to directly manipulate the entities. This feedback will be addressed by adding a grabbing functionality in the next version of SW. In positive feedback, the newly added controller instructions interfered less with general visual information and thus this feature was favored.
The usability challenges in fact affected the participants’ workload. Workload while operating SW-HMD varied across the subscale, with mental workload and frustration being the highest of the six subscales. The findings from the present study shared the same pattern as the SW-CAVE evaluation results (Wu et al., 2022), where participants also reported higher levels of mental demand and frustration. Since SW was created as a learning environment for scale, we anticipated a higher score in the mental workload subscale. It is suspected that the frustration was a result of confusion from excessive instructions, which was in line with the think aloud feedback. To alleviate frustration, a tutorial session instead of the instructions on the navigation panel will be implemented. Another cause for the mental demand and frustration might be the requirement for aiming at the exponent or decimal for scaling. A simplified interaction mode will be implemented to ease frustration from the usability perspective.
The A/B testing was especially practical and useful in determining the user-preferred alternatives. In particular, participants perceived the SW-HMD as a VR product while our previous SW-CAVE was perceived as an educational prototype, which is an interesting observation. Therefore, participants expressed the tendency to evaluate SW-HMD from the usability perspective over embodied cognition theory, and thereby expressed stronger preferences for specific alternatives that focused on usability. Four of the five experts had clear preference for button clicks over gesture interaction because “it’s simple,” which solidified the design decision to implement button clicks for future versions to benefit usability. For posts, path was the most preferred while only one participant preferred the forest. Participants found the forest “overwhelming and distracting” and no longer helped with immersion as in SW-CAVE. For future versions of SW-HMD, the “path” will be implemented for depth perception and mitigating distraction from the “forest.” The staggered animation was designed to support learning by conveying cause and effect. However, participants perceived it to be a “delay” and did not express any concern about “losing information due to animations happen simultaneously.” All five experts expressed a preference for the simultaneous animation, and it will be implemented in the future versions for user satisfaction and usability. Presenting design alternatives to the usability experts effectively finalized the design decisions.
Improvement features implemented based on the feedback from SW-CAVE helped address usability problems and the evaluation provided insights for iterative refinement of SW. For the progress panel, all participants reported that they “liked it” and thought it was useful to track the worlds they explored and currently located. It was a good example of design to benefit “information quality” and “interface quality.” For the instructions, participants preferred the instructions on the controller to those on the navigation panel. The feedback indicated that there should not be too much information overlapping on the UI which might lead to user dissatisfaction.
Four of the usability experts preferred the SW-HMD-A over the SW-HMD, with the remaining expert nevertheless stating that “this scale thing makes way more sense now,” thus suggesting that despite preference, SW-HMD-A is better aligned with the scale cognition agenda. Experts gave positive feedback for free teleportation for the benefit of user freedom and inspecting entities at close proximity. The evaluation about posts in SW-HMD-A was in line with the results in SW-HMD that “posts became a distraction at certain distances” and it does not help with immersion in HMD. In addition, experts’ comments connecting the platform design to the scaling animation revealed that SW-HMD-A was created, in part, to revisit the features holistically, and the usability expert comments reflect that features interact with each other and must be accounted together. Overall, the evaluation results confirmed the improvements and advantages of SW-HMD-A compared to SW-HMD, and SW-HMD-A provided valuable insights for the next iteration of SW-HMD.
Limitations
The research enlisted the aid of usability experts, whose perspectives may differ from that of Scale Worlds’ intended users, who are students. As a result, general issues concerning the system’s usability were pinpointed, and a follow-up study is scheduled to investigate perspectives of the target audience.
Conclusion
The HMD version of Scale Worlds is a virtual environment where users can interact with a 3D user interface to scale themselves up or down by powers of ten. The use of SW in VR has shown potential for enhancing scale cognition and learning. The iterative evaluation of SW examined the usability and different design options of SW, as well as the improvement features and HMD-specific features. The results will be utilized to enhance the SW experience and in the refinement of the next iteration of SW.
