Validity and reliability of the mobile virtual reality subjective visual vertical measurement system: Comparison with conventional bucket test

Abstract

Background

Subjective visual vertical (SVV) test is a key functional assessment tool that provides insights into vestibular imbalance. Mobile virtual reality SVV measurement system (MVR-SVV) has the potential to facilitate simple, low-cost, and reliable measurements.

Objective

This study aimed to verify the reliability and validity of MVR-SVV by comparing its data with the previously established bucket test (bucket-SVV).

Methods

Thirty-eight healthy adults completed both bucket-SVV and MVR-SVV tests. The reliability and validity of MVR-SVV were examined using intraclass correlation coefficients (ICCs), Pearson’s correlation, Bland-Altman plots (BAP), and minimum detectable change (MDC).

Results

BAP results indicated that the limits of agreement for the SVV angles were 1.61 to −1.24°. No fixed errors were identified (p = 0.13), although a small proportional error was detected (y = −0.59x + 0.157, p < 0.001). Pearson’s correlation coefficient between bucket-SVV and MVR-SVV was 0.716 (p < 0.001). Within-day reliability was poor for bucket-SVV, with ICC = 0.33–0.38, but moderate for MVR-SVV, with ICC = 0.70–0.71. Between-day reliability was poor for both methods, with ICC = 0.38 for MVR-SVV and ICC = 0.28 for bucket-SVV. MDC was 1.78° for bucket-SVV and 2.67° for MVR-SVV.

Conclusions

Our findings suggest that MVR-SVV can be used for assessing SVV. Its portability, availability, and reliability make it a valuable tool for clinicians in clinical settings.

Keywords

subjective visual vertical smartphone application vestibular disorders telemedicine digital device

Introduction

The subjective visual vertical (SVV) test is a key functional assessment tool that provides insights into vestibular imbalance by quantifying a patient’s perception of verticality.¹ SVV tilt has been identified in various diseases and can indicate signs of abnormal tilt, leading to dizziness and impaired postural control.^2,3,4 The standard methods for evaluating SVV include the hemispheric dome method⁵ and the light bar-in-the-dark method,⁶ both of which rely on specialized equipment. Although these standard methods yield valuable diagnostic information, they are often expensive, difficult to set up, and cumbersome to transport. Consequently, this form of SVV measurement is often not feasible in the clinical setting. Zwergal et al. introduced the bucket test to measure SVV.⁵ The bucket test can be used as a screening tool for utricular lesions due to its simplicity and reliability. Some studies developed a “virtual SVV system” employing virtual reality (VR) technology to obtain more compact and reliable measurements.^7,8,9,10 Although these virtual SVV systems are effective, they cannot completely address the issues of simplicity and high costs because they require specialized VR equipment and computers. Therefore, SVV measurements using smartphones have been reported to be less expensive and more convenient.¹¹ However, this method has room for improvement, as the smartphone screen frame can serve as a reference for vertical positioning. To overcome these limitations, we previously proposed a simple and affordable measurement method that combines commercially available VR goggles and a smartphone.^12,13 This method is characterized by its simplicity, affordability, and accessibility, as it eliminates the need for specialized equipment and enables assessments in various settings. However, despite its demonstrated potential in evaluating older adults and patients with vestibular disorders, its validity relative to conventional SVV tests has yet to be rigorously investigated. Additionally, healthcare providers who use the conventional bucket test as an outpatient screening tool are seeking information on whether MVR-SVV measurements using technology provide comparable results. This study aimed to evaluate the reliability and validity of the mobile virtual reality SVV measurement system (MVR-SVV), which combines VR goggles and smartphones, by comparing its data with those from the previously established bucket test (bucket-SVV). Future directions for the use of MVR-SVV will also be discussed.

Materials and methods

Participants

Thirty-eight young individuals (age = 20.6 ± 0.6 years), free from injury, participated in the study (sex = 21 male, 17 female). None of the participants reported any cognitive conditions, history of neurological disease, or use of medications that interfered with their activities of daily activities. All study procedures adhered to the principles of the Declaration of Helsinki and its subsequent amendments and were approved by the ethics committee of Nagano University of Health and Medicine, Japan (approval number: 2021-1). Written informed consent was obtained from all participants.

Procedures

All participant measurements were completed for both the bucket-SVV and MVR-SVV, with each test consisting of 10 trials. This number was based on previous research showing that averaging approximately 10 repetitions in the bucket method yields a stable estimate of subjective visual vertical.⁵ While that study focused on the bucket method, we applied the same number of trials to the MVR-SVV to ensure reliability and enable direct comparison. Each test lasted approximately 5 min, followed by a 10–20 minute break between assessments to reduce potential fatigue effects. The entire testing procedure was completed within 30 min. To evaluate between-day reliability, retests of both the bucket-SVV and MVR-SVV were conducted between 2 and 4 weeks after the initial assessment. The order of all tests was randomized using a computer.

The bucket-SVV setup is depicted in Figure 1. The device was assembled using commercially available opaque buckets.⁵ A line of diameter was affixed at the bottom of the bucket using phosphorescent tape (Figure 2(a)). The vertical line of the inner diameter coincides with the 0° point of the protractor affixed to the bottom of the bucket. A weighted string, attached to the 0° mark on the outer bottom of the bucket, swung freely and intersected the protractor, quantifying bucket deviation from the vertical direction in degrees. Participants sat upright with their heads secured using a chin rest to maintain a neutral head position. To ensure reliable SVV measurements, head tilt was minimized throughout the test. They looked into a non-translucent plastic bucket, ensuring their entire field of vision was within the bucket. The bucket was rotated by the researcher at various angles. Participants were then verbally instructed by the researcher to adjust the bucket until they perceived the internal line as straight vertically. Following this adjustment, SVV measurements, expressed in degrees, were recorded from outside the bucket based on the position of the string on the protractor.

Figure 1.

Configuration for the bucket test of subjective visual vertical.

Figure 2.

View of the subjective visual vertical test. (a) Diametric line in glow-in-the-dark tape on the inside bottom of the bucket. (b) Measurement screen of mobile virtual reality subjective visual vertical measurement system.

The MVR-SVV comprised a commercially available iPhone, smartphone application “SVV” (Developer: T.K., H.F.), VR goggles for smartphones, and a Bluetooth remote control. After operating the smartphone and entering the SVV app measurement screen, it was placed in the VR goggles. Similar to the bucket-SVV setup, participants sat upright, and their heads were secured by a chin rest (Figure 3). When the participant wore the VR goggles, a luminous bar was displayed, as shown in Figure 2(b), and the participant could rotate and vertically adjust the issuing bar using the left and right buttons on the remote control. The center button on the remote control allowed the participant to proceed to the next trial and repeat the same process to measure 10 SVV values. In this study, the iPhone screen was mirrored by Wi-Fi and monitored on a sub monitor to ensure that the participants operated the device properly. If the goggles or the smartphone built into the goggles are tilted, there may be a difference between the visually perceived vertical position and the measured value. The SVV application recognized device tilt from the built-in sensor of the smartphone and corrected the SVV value by subtracting the deviation. The results displayed the raw data of the angle, tilt angle of the sensor, and the corrected SVV value. In this study, the corrected SVV values were adopted after setting and measuring the head and smartphone as vertically as possible.

Figure 3.

Configuration for the mobile virtual reality subjective visual vertical measurement system.

Statistical analysis

The initial step involved assessing agreement between the two devices by generating a Bland–Altman plot for the SVV angle of each testing protocol. Differences in SVV measurements between the two methods were plotted against the mean results. A one-sample t-test was employed to compare differences between the two methods. Linear regression was utilized to test the significance of potential proportional bias between the mean difference in SVV angles for bucket-SVV and MVR-SVV and the mean of these two measurements.¹⁴ Pearson’s correlation was employed to test the strength of agreement for validity between bucket-SVV and MVR-SVV. The intraclass correlation coefficient (ICC) was calculated to determine within-day reliability (ICC across 10 trials on the same day) and between-day reliability (ICC between mean values of Day 1 and Day 2). A model with two-way random effects and single-measure intraclass correlation coefficients (ICC^6,14) was employed to evaluate reliability. Point estimates of the ICCs were interpreted as follows: excellent (>0.9), good (0.75–0.9), moderate (0.5–0.75), and poor (<0.5).¹⁵ The standard error of measurement (SEM) and minimum detectable change (MDC) were calculated alongside the ICC values to assess measurement reliability and detect meaningful changes in SVV values. Specifically, SEM agreement was computed using the between-day variance obtained from ICC estimation, incorporating both systematic errors and residual variance to reflect measurement reliability (SEM agreement = √σ between measurement + σ residual).^16,17 Variance components were obtained using a multilevel mixed-effects model (restricted maximum likelihood estimates).^16,17 This calculation follows the methodology described by de Vet et al.¹⁶ The MDC, indicative of the minimum change in individuals detectable beyond measurement error with a confidence of 95%, was determined as 1.96√2SEM.¹⁸ Statistical significance was set at p < 0.05. All statistical analyses were performed using SPSS software (version 25.0; IBM SPSS Inc., Armonk, NY, USA).

Results

Table 1 presents SVV angles for bucket-SVV and MVR-SVV from the initial test (Day 1) and retest (Day 2). Notably, both bucket-SVV and MVR-SVV exhibited mean SVV close to 0°, accompanied by a standard deviation of <2°. Bland–Altman plots for the SVV tests are depicted in Figure 4, revealing limits of agreement for SVV angles ranging from 1.61° to −1.24°. No obvious fixed errors were observed (p = 0.13). A small proportional error is detected (y = −0.59x + 0.157, p < 0.001, with a 95% confidence interval of the slope between −0.85 and −0.34). Regarding concurrent validity, the correlation coefficient between the two tests was 0.716 (p < 0.001). The ICC values for within-day reliability were 0.33 for Day 1 and 0.38 for Day 2 in the bucket-SVV group, indicating poor reliability. In the MVR-SVV group, the ICCs were 0.70 for Day 1 and 0.71 for Day 2, showing moderate reliability. The between-day reliability was low for both tests, with bucket-SVV being lower than MVR-SVV (ICC bucket-SVV = 0.28, ICC MVR-SVV = 0.38). The SEM for bucket-SVV was 0.64° and that for MVR-SVV was 0.96°. The MDC for bucket-SVV was 1.78°, and the SEM for MVR-SVV was 2.67°.

Table 1.

Reliability and concurrent validity analysis of subjective visual vertical angle in bucket test and mobile virtual reality SVV measurement system.

	Bucket-SVV (SD)	MVR-SVV(SD)	Mean Diff (95% CI)	Pearson coefficient
Day 1 SVV angle	0.11 (0.78)	−0.15 (1.25)	0.26 (−0.03, 0.55)	0.713
Day 1 ICC (within-day reliability, 95% CI)	0.33 (0.22, 0.47)	0.70 (0.60, 0.80)
Day 2 SVV angle	−0.01 (0.74)	−0.12 (1.22)	0.11 (−0.20, 0.42)	0.634
Day 2 ICC (within-day reliability, 95% CI)	0.38 (0.26, 0.54)	0.71 (0.61, 0.81)
Day 1 and 2 Mean	0.05 (0.61)	−0.14 (1.02)	0.18 (−0.06, 0.42)	0.716
Day 1 and 2 Diff (95% CI)	0.12 (−0.18, 0.42)	−0.03 (−0.48, 0.42)
ICC (between-day reliability, 95% CI)	0.28 (−0.04, 0.55)	0.38 (0.07, 0.62)
SEM	0.64	0.96
MDC	1.78	2.67

SVV: subjective visual vertical; Bucket: bucket test; MVR: mobile virtual reality; SD: standard deviation; Mean Diff: mean difference; CI: confidence interval; ICC: intraclass correlation coefficient; SEM: standard error of the measurement; MDC: minimum detectable change.

Figure 4.

Bland–Altman plots for the subjective visual vertical tests.

Discussion

This study compared two clinically available methods for assessing the SVV: bucket-SVV and MVR-SVV. The Bland-Altman analysis revealed proportional errors between the two methods but no fixed errors, with a correlation coefficient of 0.716. MVR-SVV showed higher within-day reliability compared with bucket-SVV, although between-day reliability was low for both methods. However, all the reliability results were higher for MVR-SVV than for bucket-SVV. Importantly, the MVR-SVV demonstrated concurrent validity with the bucket-SVV, suggesting its potential as a portable, cost-effective, and valid tool for assessing SVV.

One of the key strengths of this study is its focus on developing a cost-effective and widely accessible assessment tool. Previous studies have utilized VR systems for SVV measurement, highlighting their potential for accurate assessments of vestibular function.^7–10 However, these systems often require costly VR equipment and computers, limiting their practical application in clinical settings. In contrast, the MVR-SVV represents a significant advancement in terms of affordability and portability. Smartphones are ubiquitous, and VR goggles are increasingly affordable, making this system highly practical for routine clinical use. By leveraging commonly available technology, this approach democratizes access to vestibular assessment tools, enabling their use in a broader range of clinical and community settings. Although studies on smartphone and VR goggle combinations for SVV measurement exist, including two by our team, none have rigorously validated this method against traditional SVV tests.^12,13,19 This study addresses this gap, establishing the criterion-related validity of MVR-SVV against the bucket-SVV.

The MVR-SVV demonstrates criterion-related validity with bucket-SVV, establishing its value as a tool for assessing SVV. Bucket-SVV, a clinically useful method, has been shown to be comparable to the hemispheric dome method, the standard for SVV assessment.⁵ Our findings revealed a significant correlation between MVR-SVV and bucket-SVV, with no fixed errors in the Bland-Altman plot. Proportional errors were identified, indicating that MVR-SVV values tend to overestimate as they diverge from zero. This overestimation may correspond to an underestimation by bucket-SVV due to cognitive bias, as assessors may unconsciously align results closer to zero when evaluating healthy individuals. Despite these proportional errors, they remained within the 95% limits of agreement, supporting the equivalence of MVR-SVV and bucket-SVV.

In addition to its simplicity and affordability, the MVR-SVV benefits from its ability to calculate tilt angles using the iPhone accelerometer. This feature enhances its applicability for assessing head tilt SVV²⁰ and for monitoring patients with postural control abnormalities, making it particularly valuable in clinical populations with vestibular dysfunction or stroke. By addressing these additional clinical needs, the MVR-SVV extends beyond the capabilities of traditional SVV assessment tools.

MVR-SVV showed higher within-day reliability than bucket-SVV, but both exhibited low between-day reliability. These findings are consistent with those of previous studies. Wengier et al. found that the reliability of SVV between five trials was higher for smartphone tests than for bucket tests.¹¹ In our study, among 10 trials, MVR-SVV exhibited an ICC of >0.7, whereas bucket-SVV showed an ICC of <0.4. The results of the within-day reliability study support the findings of previous studies. Regarding between-day reliability, Michelson et al. reported that the bucket-SVV had an ICC of 0.15.⁸ In our study, bucket-SVV had an ICC of 0.28, and MVR-SVV had an ICC of 0.38. The previous study was retested on the same day with a 30-minute time gap,⁸ whereas our study was retested over a period of 2 weeks. The SVV in healthy individuals may fluctuate in both positive and negative directions around 0°, even over short periods, indicating sensitivity to the effect of time. However, the higher reliability of MVR-SVV compared with bucket-SVV, despite the influence of the time period, may suggest the superior accuracy of MVR-SVV. Additionally, this study included only healthy participants, which may have contributed to lower ICC values. As ICC is influenced by variance among individuals, the low variability in SVV measurements within this group may have resulted in artificially lower reliability estimates, despite consistent measurements. Previous studies have reported greater variability in SVV measurements among patients with vestibular disorders or stroke,^21,22 which may improve reliability estimates. Considering this, the between-day reliability values of 0.28 for bucket-SVV and 0.38 for MVR-SVV should not be interpreted as inherently poor but rather as a reflection of the limited variability among healthy participants.

The MDC results also revealed that changes of ≤1.78° for bucket-SVV and ≤2.67° for MVR-SVV fall within measurement error, making small SVV variations in healthy individuals statistically insignificant. However, both methods are effective for identifying larger SVV deviations in patients with vestibular disorders or stroke. Gómez et al. reported SVVs of 8.4° at initial testing, decreasing to 3.2° after 2 weeks and 1.4° after 4 weeks in patients with acute vestibular disorders.²² Similarly, studies on patients with stroke or dizziness reported significantly larger SVV deviations, far exceeding MDC thresholds.^4,21,23 These findings highlight the clinical utility of both methods for populations with abnormal SVV.

This study has several limitations. First, the use of the bucket test as a reference, although validated and clinically utilized, is not considered the gold standard. Secondly, individual differences exist in terms of determining the visual vertical, with some participants meticulously adjusting their verticals before deciding, whereas others make quick decisions. Despite this variability, MVR-SVV demonstrated higher reliability, emphasizing its potential as a robust and accessible tool for SVV assessment.

Conclusion

The MVR-SVV demonstrated higher within-day and between-day reliability compared with bucket-SVV. Unlike bucket-SVV, MVR-SVV does not require manufacturing and can be measured using readily available VR goggles, remote controllers, and smartphones. It offers cost-effectiveness and space efficiency compared with the gold standard dome test. Further research should investigate its diagnostic and prognostic accuracy across various populations, including older adults, vestibularly impaired, and patients with stroke. As smartphone-based medical applications become more prevalent, particularly in telemedicine initiatives,²⁴ the adoption of this system among patient populations could contribute to the advancement of evidence-based medicine and telemedicine.

Footnotes

Acknowledgements

The authors would like to thank the study participants for their time and dedication.

ORCID iDs

Hirofumi Ogihara

Tomohiko Kamo

Yasuyuki Kurasawa

Shota Hayashi

Tatsuaki Kuroda

Ryozo Tanaka

Reiko Tsunoda

Hiroaki Fushiki

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by a Grant-in Aid for Early-Career Scientists from JSPS KAKENHI in Japan [grant numbers JP22K17641].

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

Gnanasegaram

Parkes

Cushing

, et al. Stimulation from cochlear implant electrodes assists with recovery from asymmetric perceptual tilt: evidence from the subjective visual vertical test. Front Integr Neurosci 2016; 10: 32.

Čakrt

Slabý

Kučerová

, et al. Subjective visual vertical and postural control in patients following cochlear implantation. J Vestib Res 2023; 33: 403–409.

Pereira

Kanashiro

Maia

, et al. Correlation of impaired subjective visual vertical and postural instability in Parkinson’s disease. J Neurol Sci 2014; 346: 60–65.

Sakagami

Wada

Shiozaki

, et al. Results of subjective visual vertical tests in patients with vertigo/dizziness. Auris Nasus Larynx 2022; 49: 342–346.

Zwergal

Rettinger

Frenzel

, et al. A bucket of static vestibular function. Neurology 2009; 72: 1689–1692.

Böhmer

Mast

. Assessing otolith function by the subjective visual vertical. Ann N Y Acad Sci 1999; 871: 221–231.

Lee

Hong

, et al. Implementation of a subjective visual vertical and horizontal testing system using virtual reality. Comput Mater Continua (CMC) 2021; 67: 3669–3679.

Michelson

McCaslin

Jacobson

, et al. Assessment of subjective visual vertical (SVV) using the “bucket test” and the virtual SVV system. Am J Audiol 2018; 27: 249–259.

Ulozienė

Totilienė

Balnytė

, et al. Subjective visual vertical and visual dependency in patients with multiple sclerosis. Mult Scler Relat Disord 2020; 44: 102255.

10.

Wang

Winnick

, et al. Test-retest reliability of subjective visual vertical measurements with lateral head tilt in virtual reality goggles. Tzu Chi Med J 2021; 33: 294–300.

11.

Wengier

Ungar

Handzel

, et al. Subjective visual vertical evaluation by a smartphone-based test-taking the phone out of the bucket. Otol Neurotol 2021; 42: 455–460.

12.

Hayashi

Kamo

Ogihara

, et al. Test-retest reliability of dynamic subjective visual vertical and visual dependency in older adults using virtual reality methods. Percept Mot Skills 2024; 131: 2069–2084.

13.

Umibe

Fushiki

Tsunoda

, et al. Development of a subjective visual vertical test system using a smartphone with virtual reality goggles for screening of otolithic dysfunction: observational study. JMIR Form Res 2024; 8: e53642.

14.

Bland

Altman

. Measuring agreement in method comparison studies. Stat Methods Med Res 1999; 8: 135–160.

15.

Koo

. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 2016; 15: 155–163.

16.

de Vet

Terwee

Knol

, et al. When to use agreement versus reliability measures. J Clin Epidemiol 2006; 59: 1033–1039.

17.

Heidemann

Godballe

Kjeldsen

, et al. The Otitis Media-6 questionnaire: psychometric properties with emphasis on factor structure and interpretability. Health Qual Life Outcome 2013; 11: 201.

18.

Terwee

Bot

de Boer

, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007; 60: 34–42.

19.

Totilienė

Uloza

Lesauskaitė

, et al. Impaired subjective visual vertical and increased visual dependence in older adults with falls. Front Aging Neurosci 2021; 13: 667608.

20.

Yagi

Morita

Kitazawa

, et al. Head roll-tilt subjective visual vertical test in the diagnosis of persistent postural-perceptual dizziness. Otol Neurotol 2021; 42: e1618–e1624.

21.

Bonan

Guettard

Leman

, et al. Subjective visual vertical perception relates to balance in acute stroke. Arch Phys Med Rehabil 2006; 87: 642–646.

22.

Gómez García

Jáuregui-Renaud

. Subjective assessment of visual verticality in follow-up of patients with acute vestibular disease. Ear Nose Throat J 2003; 82: 442–444.

23.

Mori

Nakamura

Hashimoto

, et al. Novel characterization of subjective visual vertical in patients with unilateral spatial neglect. Neurosci Res 2021; 163: 18–25.

24.

Noda

Kuroda

Nomura

, et al. Smartphone-assisted medical care for vestibular dysfunction as a telehealth strategy for digital therapy beyond COVID-19: scoping review. JMIR Mhealth Uhealth 2023; 11: e48638.