Subjective Measurement of Trust: Is It on the Level?

Abstract

Psychometrics are increasingly being used to evaluate trust in the automation of safety-critical systems. There is no consensus on what the highest level of measurement is for psychometric trust. This is important as the level of measurement determines what mathematics and statistics can be meaningfully applied to ratings. In this work, we introduce a new method for determining what the maximum level of measurement is for psychometric ratings. We use this to assess the level of measurement of trust in automation using human ratings about the behavior of unmanned aerial systems performing search tasks. Results show that trust is best represented at an ordinal level and that it can be treated as interval in most situations. It is unlikely that trust in automation ratings are ratio. We discuss these results, their implications, and future research.

References

Annett

(2002). Subjective rating scales: Science or art? Ergonomics, 45(14), 966–987.

Bolton

M. L.

(2008). Modeling human perception: Could Stevens’ power law be an emergent feature? In Ieee international conference on systems, man and cybernetics (pp. 1073–1078).

Cohen

B. H.

(2013). Explaining psychological statistics. John Wiley & Sons.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.

Deming

W. E.

(1943). Statistical adjustment of data. Wiley.

Ellermeier

Faulhammer

(2000). Empirical evaluation of axioms fundamental to Stevens’s ratio-scaling approach: I. Loudness production. Attention, Perception, & Psychophysics, 62(8), 1505–1511.

Furr

R. M.

Bacharach

V. R.

(2013). Psychometrics: An introduction (2nd ed.). Los Angeles: Sage.

Guilford

J. P.

(1954). Psychometric methods. New York: McGraw-Hill.

Hoff

K. A.

Bashir

(2015). Trust in automation integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434.

10.

Kessler

T. T.

Larios

Walker

Yerdon

Hancock

(2017). A comparison of trust measures in human–robot interaction scenarios. In Advances in human factors in robots and unmanned systems (pp. 353–364). Springer.

11.

Laming

D. R. J.

(1997). The measurement of sensation. Oxford University.

12.

Lee

J. D.

Moray

(1992). Trust, control strategies and allocation of function in human-machine systems. Ergonomics, 35(10), 1243–1270.

13.

Lee

J. D.

See

K. A.

(2004). Trust in automation: Designing for appropriate reliance. Human factors, 46(1), 50–80.

14.

McGrath

P. A.

Seifert

C. E.

Speechley

K. N.

Booth

J. C.

Stitt

Gib-son

M. C.

(1996). A new analogue scale for assessing children’s pain: an initial validation study. Pain, 64(3), 435–443.

15.

Michell

(2008). Is psychometrics pathological science? Measurement, 6(1-2), 7–24.

16.

Mosteller

Tukey

J. W.

(1977). Data analysis and regression: a second course in statistics. Addison-Wesley.

17.

Muir

B. M.

(1987). Trust between humans and machines, and the design of decision aids. International Journal of Man-Machine Studies, 27(5-6), 527–539.

18.

Narens

(1996). A theory of ratio magnitude estimation. Journal of Mathematical Psychology, 40(2), 109–129.

19.

NCSS. (2016). Deming regression. In NCSS statistical software (pp. 303-1–303-33). NCSS, LLC.

20.

Rasmussen

Kingston

Humphrey

(2018). A brief introduction to unmanned systems autonomy services (UxAS). In 2018 international conference on unmanned aircraft systems (icuas) (pp. 257–268).

21.

Stevens

S. S.

(1946). On the theory of scales of measurement. Science, 103(2684), 677–680.

22.

Stevens

S. S.

(1956). The direct estimation of sensory magnitudes: Loudness. The American journal of psychology, 69(1), 1–25.

23.

Trendler

(2009). Measurement theory, psychology and the revolution that cannot happen. Theory & Psychology, 19(5), 579–599.

24.

Velleman

P. F.

Wilkinson

(1993). Nominal, ordinal, interval, and ratio typologies are misleading. The American Statistician, 47(1), 65–72.

25.

Zimmer

(2005). Examining the validity of numerical ratios in loudness fractionation. Perception & Psychophysics, 67(4), 569–579.