In this paper we describe a methodology for utilizing team mental models as a basis for evaluating the usability and utility of collaborative systems. We present a case study of the evaluation of team mental models within the NeoCITIES 3.1 simulation. Paired comparison ratings, which are one of the most popularly used methods of team mental model assessment, were used to capture the team members’ taskwork-related knowledge, which was then compared to ratings from subject matter experts. These analyses were the driving force behind several design modifications in the NeoCITIES interface. We discuss the limitations of the method and its implications within the scope of collaborative systems evaluation and the field of HCI.
References
1.
BoshernitsanM.GrahamS.HearstM. (2007). Aligning development tools with the way programmers think about code changes. Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing San Jose, California, USA.
2.
ConnorsE. S. (2006). Intelligent Group Interfaces: Envisioned Designs for Exploring team Cognition in Emergency Crisis Management. The Pennsylvania State University, State College, Pa. Unpublished Dissertation.
3.
CramtonC. (2002). Finding common ground in dispersed collaboration. Organizational Dynamics, 30(4), 356-367.
4.
DeChurchL.Mesmer-MagnusJ. (2010). Measuring shared team mental models: A meta-analysis. Group Dynamics, 14(1), 1-14.
5.
EllisA. (2006). System breakdown: The role of mental models and transactive memory in the relationship between acute stress and team performance. Academy of Management Journal, 49(3), 576-589.
6.
GreenbergS.BuxtonB. (2008). Usability evaluation considered harmful (some of the time). Paper presented at the Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems.
7.
HellarD. B. (2009). An Investigation of Data Overload in Team-Based Distributed Cognition Systems. The Pennsylvania State University, State College, Pa. Unpublished Dissertation.
8.
HsuJ. S. C.ChangJ. Y. T.KleinG.JiangJ. J. (2010). Exploring the impact of team mental models on information utilization and project performance in system development. International Journal of Project Management, 29(1), 1-12.
9.
HsuJ. S. C.ParoliaN.JiangJ. J.KleinG. (2007). The Impact of Team Mental Models on IS Project Teams’ Information Processing and Project Performance. Paper presented at the International Research Workshop on IT Project Management2007.
10.
JonesR. (2006). The development of an emergency crisis management simulation to assess the impact a fuzzy cognitive map decision-aid has on team cognition and team decision-making. Pennsylvania State University University Park, PA, USA.
11.
MarksM.SabellaM.BurkeC.ZaccaroS. (2002). The impact of cross-training on team effectiveness. Journal of Applied Psychology, 87(1), 3-13.
12.
MarksM.ZaccaroS.MathieuJ. (2000). Performance implications of leader briefings and team-interaction training for team adaptation to novel environments. Journal of Applied Psychology, 85(6), 971-986.
13.
MathieuJ. E.HeffnerT. S.GoodwinG. F.Cannon BowersJ. A.SalasE. (2005). Scaling the quality of teammates’ mental models: equifinality and normative comparisons. Journal of Organizational Behavior, 26(1), 37-56.
14.
MazurekM.ArsenaultJ.BreseeJ.GuptaN.IonI.JohnsC.. (2009). Access control for home data sharing: Attitudes, needs and practices. Paper presented at the Proceedings of the 28th international conference on Human factors in computing systems, Atlanta, GA.
15.
MohammedS.DumvilleB. (2001). Team mental models in a team knowledge framework: Expanding theory and measurement across disciplinary boundaries. Journal of Organizational Behavior, 22(2), 89-106.
16.
MohammedS.HamiltonK. (in press). Studying team cognition: The good, the bad and the practical. In HollingsheadA. B.PooleM. S. (Eds.), Research Methods for Studying Groups: A Behind-the-Scenes Guide: Taylor & Francis/Routledge.
17.
PfaffM. S. (2008). Effects of mood and stress on group communication and performance in a simulated task environment. The Pennsylvania State University, State College, Pa. Unpublished Dissertation.
18.
Smith-JentschK. A.CampbellG. E.MilanovichD. M.ReynoldsA. M. (2001). Measuring teamwork mental models to support training needs assessment, development, and evaluation: Two empirical studies. Journal of Organizational Behavior, 22(2), 179-194.
19.
Smith-JentschK. A.Cannon-BowersJ. A.TannenbaumS. I.SalasE. (2008). Guided team self-correction: Impacts on team mental models, processes, and effectiveness. Small Group Research, 39(3), 303.
20.
WallerM.GuptaN.GiambatistaR. (2004). Effects of adaptive behaviors and shared mental models on control crew performance. Management Science, 50(11), 1534-1544.
21.
WegnerD. (1987). Transactive memory: A contemporary analysis of the group mind. Theories of group behavior, 185-208.
22.
WellensA. R.ErgenerD. (1988). The cities game: A computer-based situation assessment task for studying distributed decision making. Simulation & Gaming, 19(3), 304-327.
