This study demonstrates that the use of augmented reality interfaces, designed based on a theoretically driven human-centered framework, can significantly improve task performance. The domain of unmanned ground vehicle landmine/unexploded ordnance detection and disposal was selected because it is a real-world example of a complex human-robot interaction task. Participants in this study were assigned to one of seven experimental conditions. A new method (the augmented multisensory interface design, or AMID, method) was used to identify two unisensory and one multisensory performance-enhancing augmented reality (AR) interface solutions, along with three less appropriate interface solutions. All participants were trained in the operation of the AR devices they were assigned and also in telerobotic landmine detection operations. After completing the training, participants were given the task of clearing a minefield. Results show that participants in the AMID-compliant solution groups significantly outperformed the control group and their non-AMID-compliant group counterparts in terms of the numbers of landmines correctly identified. The AMID-compliant multisensory condition was found to result in significantly increased landmine detection, increased sensitivity, and decreased task time.
References
1.
AkamatsuM.MacKenzieI. S.HasboucT.(1995).A comparison of tactile, auditory, and visual feedback in a pointing task using a mouse-type deviceErgonomics, 38, 816–827
2.
AzumaR. T.(1997).A survey of augmented realityTeleoperators and Virtual Environments, 6, 355–385
3.
AzumaR.BaillotY.BehringerR.FeinerS.JulierS.MacIntyreB.(2001).Recent advances in augmented realityIEEE Computer Graphics and Applications, 21(3), 6–8
4.
BenderD. B.(1987).Comparison of the effects of superior colliculus and pulvinar lesions on visual search and tachistoscopic pattern discrimination in monkeysExperimental Brain Research, 69, 140–145
5.
BoffK.KaufmanL.ThomasJ. P.(1986).Handbook of perception and human performance (Vols. 1 and 2). New YorkWiley-Interscience
6.
BoudA. C.HaniffD. J.BaberC.SteinerS. J.(1999).Virtual reality and augmented reality as a training tool for assembly tasks In Proceedings of the IEEE International Conference on Information Visualization (pp. 32–36). Los Alamitos, CA: IEEE Computer Society
7.
BregmanA. S.(1996).Perceptual interpretation and the neurobiology of perception In LlinàsR.ChurchlandP. S. (Eds.), The mind-brain continuum: Sensory processes (pp. 203–217). Cambridge, MA: MIT Press
8.
BristowH.BaberC.CrossJ.WoolleyS.(2002).Evaluating contextual information for wearable computing In Proceedings of the 6th International Symposium on Wearable Computing (pp. 179–185). Los Alamitos, CA: IEEE Computer Society
9.
BrownD. G.StriplingR.CoyneJ. T.(2006).Augmented reality for urban skills training In Proceedings of the IEEE Virtual Reality Conference (VR′06, pp. 249–252). Los Alamitos, CA: IEEE Computer Society
10.
ChuC.MoJ.GadhR.(2002).A quantitative analysis on virtual reality-based computer aided design system interfacesComputing and Information Science in Engineering, 2, 216–223
11.
DeutschD.(1981).The octave illusion and auditory perceptual integration In TobiasJ. V.SchubertE. D. (Eds.), Hearing research and theory (Vol. 1, pp. 99–142). New York: Academic Press
12.
DrascicD.MilgramP.(1996).Perceptual issues in augmented reality In Proceedings of the SPIE Conference on Stereoscopic Displays and Virtual Reality Systems III (Vol. 2653, pp. 123–134). Bellingham, WA: SPIE
13.
14.
FooteJ.(1999).An overview of audio information retrievalMultimedia Systems, 7(1), 2–10
15.
FurmanskiC.AzumaR.DailyM.(2002).Augmented-reality visualizations guided by cognition: Perceptual heuristics for combining visible and obscured information In Proceedings of the International Symposium on Mixed and Augmented Reality (pp. 215–225). Los Alamitos, CA: IEEE Computer Society
16.
GaverW. W.(1991).Technology affordances In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Reaching Through Technology (pp. 79–84). New York: ACM Press
17.
GrauwinkelK.FagelS.(2006).Crossmodal integration and McGurk-effect in synthetic audiovisual speech. In Proceedings of the SPECOM-2006 Conference (pp. 489–492). Retrieved September 28, 2009, from http://www.eurasip.org/Proceedings/Ext/SPECOM2006/papers/091.pdf
18.
HaasM.BeyerS.DennisL.BrickmanB.HettingerL.RoeM.NelsonW.SnyderD.DixonA.ShawR.(1997).An evaluation of advanced multisensory display concepts for use in future tactical aircraft (Rep. No. AL/CF-TR-1997–0049). Wright-Patterson Air Force Base, OHArmstrong Laboratory
19.
HaniffD. J.BaberC.(2003).User evaluation of augmented reality systems In Proceedings, Seventh International Conference on Information Visualization (pp. 505–511). Los Alamitos, CA: IEEE Computer Society
20.
HarrisL.JenkinM.ZikovitzD. C.(1999).Vestibular cues and virtual environments: Choosing the magnitude of vestibular cue In Proceedings of the IEEE International Conference on Virtual Reality (pp. 229–236). Piscataway, NJ: IEEE
21.
HettingerL. J.BrickmanB. J.RoeM. M.NelsonW. T.HassM. W.(1996).Effects of virtually-augmented fighter cockpit displays on pilot performance, workload, and situation awareness In Proceedings of the Human Factors and Ergonomics Society 40th Annual Meeting (pp. 30–33). Santa Monica, CA: SAGE Publications
22.
HirshI. J.WatsonC. S.(1996).Auditory psychophysics and perceptionAnnual Review of Psychology, 47, 461–484
23.
HoC.SpenceC.TanH. Z.(2005).Warning signals go multisensory In Proceedings of the 11th Annual Conference on Human-Computer Interaction (Vol. 9, pp. 1–10). Mahwah, NJ: Erlbaum
24.
KnudsenE.LacS.EsterlyS.(1987).Computational maps in the brainAnnual Review of Neuroscience, 10(1), 41–65
25.
26.
LantzG.MichelC. M.Pascual-MarquiR. D.SpinelliL.SeeckM.SeriS.LandisT.RosenI.(1997).Extracranial localization of intracranial interictal epileptiform activity using LORETA (low resolution electromagnetic tomography)Electroencephalography and Clinical Neurophysiology, 102, 414–422
27.
LeeJ. D.HoffmanJ. D.HayesE.(2004).Collision warning design to mitigate driver distraction In Proceedings of SIGCHI 2004 Conference on Human Factors in Computing Systems (pp. 65–72). New York: Association for Computing Machinery
28.
LevineM. W.ShefnerT.(2000).Fundamentals of sensation and perception (2nd ed.). New YorkOxford University Press
29.
MaidaJ. C.BowenC. K.PaceJ.(2007).Improving robotic operator performance using augmented reality In Proceedings of the Human Factors and Ergonomics Society 51st Annual Meeting (pp. 1635–1639). Santa Monica, CA: SAGE Publications
30.
MassieT. H.SalisburyJ. K.(1994).The PHANTOM haptic interface: A device for probing virtual objects In Proceedings of the ASME Winter Annual Meeting, Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (Vol. 1, pp. 295–300). New York: American Society of Mechanical Engineers
31.
32.
MunhallK. G.GribbleP.SaccoL.WardM.(1996).Temporal constraints on the McGurk effectPerception & Psychophysics, 58, 351–362
33.
NavonD.MillerJ.(2002).Queuing or sharing? A critical evaluation of the single-bottleneck notionCognitive Psychology, 44(1), 193–251
34.
OviattS.(2006).Human-centered design meets cognitive load theory: Designing interfaces that help people think In Proceedings of the 14th Annual ACM International Conference on Multimedia (pp. 871–880). New York: ACM
35.
PolingG. L.WeisenbergerJ. M.KerwinT.(2003).The role of multisensory feedback in haptic surface perception In Proceedings of the 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS′03, pp. 187–194). Los Alamitos, CA: IEEE Computer Society
SandersonP.WatsonM.(2005).From information content to auditory display with ecological interface design: Prospects and challenges In Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting (pp. 259–263). Santa Monica, CA: SAGE Publications
38.
SchaabB.DresselD.(2002).Training for adaptability and transfer on digital systems (Research Report 1782). Alexandria, VAU.S. Army Research Institute for the Behavioral and Social Sciences
39.
SniderR. S.StowellA.(1944).Receiving areas of the tactile, auditory, and visual systems in the cerebellumJournal of Neurophysiology, 7, 331–357
40.
SohmerH.KinartiR.(1984).Survey of attempts to use auditory evoked potentials to obtain an audiogramBritish Journal of Audiology, 18, 237–244
41.
StanislawH.TodorovN.(1999).Calculation of signal detection theory measuresBehavior Research Methods, Instruments, & Computers, 31, 137–149
42.
StanneyK.SammanS.ReevesL.HaleK.BuffW.BowersC.GoldiezB.NicholsonD.LackeyS.(2004).A paradigm shift in interactive computing: Deriving multimodal design principles from behavioral and neurological foundationsInternational Journal of Human-Computer Interaction, 17, 229–257
43.
StoneR. T.(2008).Augmented multisensory interface design: Performance enhancement capabilities and training potential. Doctoral dissertation, State University of New York at Buffalo, Amherst, NY
44.
TanZ.GrayR.YoungJ.TraylorR.(2003).A haptic back display for attentional and directional cueingHaptics-e, 3(1), 1–20
45.
TerrenceP. I.BrillJ. C.GilsonR. D.(2005).Body orientation and the perception of spatial auditory and tactile cues In Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting (pp. 1663–1667). Santa Monica, CA: SAGE Publications
46.
TombuM.JolicoeurP.(2003).A central capacity sharing model of dual-task performanceJournal of Experimental Psychology: Human Perception and Performance, 29, 3–18
47.
TsagarakisN.GrayJ.ZannoniC.PetroneM.TestiD.VicecontiM.(2006).A haptic-enabled multimodal interface and framework for preoperative planning of total hip arthroplasyIEEE Multimedia, 13(3), 40–48
48.
VitenseH. S.JackJ. A.EmeryV. K.(2003).Multimodal feedback: An assessment of performance and mental workloadErgonomics, 46, 68–87
49.
VoraJ.NairS.GramopadhyeA. K.DuchowskiA. T.MelloyB. J.KankiB.(2002).Using virtual reality technology for aircraft visual inspection training: Presence and comparison studiesApplied Ergonomics, 33, 559–570
WelchR.WarrenD. H.(1986).Intersensory interactions In BoffK.KaufmanL.ThomasJ. P. (Eds.), Handbook of perception and human performance (Vol. 1, pp. 25.21–25.36). New York: Wiley
52.
WickensC. D.(2002).Multiple resources and performance predictionTheoretical Issues in Ergonomics Science, 3, 159–177
53.
XieB.SalvendyG.(2000).Prediction of mental workload in single and multiple task environmentsInternational Journal of Cognitive Ergonomics, 4, 213–242
54.
ZwislockiJ.(1960).Theory of temporal auditory summationJournal of the Acoustical Society of America, 32, 1046–1060
