Abstract
Why publish a guidebook on formulating good questionnaires now? Haven’t we been writing questions for decades? Aren’t we are seemingly doing just fine with them? Moroney and Cameron deftly provide several cogent examples to illustrate how questionnaire development is more challenging than it first appears. Reading this book, I reflected back on examples from my own 40+ years of human factors and ergonomics (HF&E) research wherein I struggled, and often, but not always, succeeded in extracting usable user performance measures, and accompanying insightful user-opinion data from respondents grappling with my own forms of questionnaires, structured debriefing interviews, and the like.
Oh, how I recall administering the Baddley’s logical reasoning scale to hundreds of soldiers on an armor-filled field test training ground. One bright soldier kept getting low scores on daily repeated measures of that simple cognitive performance test. Dozens of one-line questions were posed on a sheet of paper, using similar, but catchy wording throughout, such as: “A does not precede B,” then presented the pair as “B-A, True or False?” [the answer of course is “true”]. After several days, seemingly spontaneously, that young man began getting very high scores. When asked why the improvement, he said simply that on the previous night his wife explained to him what the word “precede” meant. Who would have thought?
In another example, I offered a lengthy list of drug medications in a questionnaire inquiring of certified medical examiners doing exams of commercial truck drivers: “which of these drugs do you recommend for your drivers?” Long story short, the medical examiners did not recognize most of the medications listed, and thus we obtained a paucity of usable data.
For me, the multiple lessons were clear: “use care in ensuring the language used is understandable to the respondents”; ensure you know well your target audience and about their particular jobs; and then do “pretest dry runs” of your questions or test items. Do all of this to be sure respondents are providing answers consistent with the research goals, and so forth.
Among my success stories, my use of structured interview questionnaires in debriefings with commercial drivers helped gain numerous insights about how to redesign new high technology driving safety systems, which months of precise driving performance data, analyzed to the hilt, could not provide. User opinions often provide meaningful data about how and why experimental research data (eg, observed or measured performance) appear as they do. And yes, by the way, I witnessed firsthand how open-ended questionnaires with women versus men respondents can produce different results. P.S.: We got many more usable answers (data) from women drivers. Many more examples come to mind; can you think of your own such lessons learned?
This booklet is a part of the Human Factors and Ergonomics Society’s (HFES) series of User’s Guides to Human Factors and Ergonomics Methods. In the “afterword” to their book, Moroney and Cameron repeat an axiom that guided their thinking in preparing this gem of a guidebook: “Anyone can write a questionnaire, but not necessarily well.” Their central message is that good questionnaire design must follow a respondent-centered systems approach focusing on a specific well-defined research question or an applied concern. Six maxims form the core of their guidance for developing a quality questionnaire:
Interact with your respondents in a respectful manner [view a questionnaire as a conversation].
Use a respondent-centered systems approach [two full chapters help tailor the process to specific situations].
Identify the questions you need to answer [precisely defining what you need to know and why].
Use language with care and precision [guidelines for writing question stems and pitfalls to avoid]—this Chapter 5 set of guidelines is my personal favorite in the whole book.
Choose your response options with sensitivity [exhibits describe advantages and disadvantages of frequently used response formats; tradeoffs to make].
Realize that you are not the respondent [pretesting questionnaires is essential to ensure validity].
In this book the lessons to be learned, and the implications for researchers’ approaches to gaining user information stretch far beyond realms of designing human-centered equipment systems. It includes for example, coverage of the types of questions that might be asked during election voting polls, or in determining customer satisfaction with restaurant meals and service, and so on.
Of the many insightful lessons to be leaned in this wonderful book, in particular I enjoyed reading the authors’ treatise on identifying the ins and outs of trying four different methods of questionnaire administration, including: mailed out questionnaires and expected rates of returns; use of internet (web-based) surveys; conducting telephone interviews; and face-to-face (on-site) interviews. I found their Chapter 3 discussion of needed sample-size pointers for surveys based on probabilities, and margins of error to be very helpful; for example, for a population of 100,000 to achieve a 95% confidence interval with an error rate of plus or minus 5% requires a sample size of about 400. Web links to probability and sampling sizes for a number of popular survey forms are provided, for example, Survey Monkey and others.
Five excellent appendixes offer very useful information in the back of the book. Readers who favor the use of Likert-type scales will appreciate Appendix D, which lists wording options for 35 Likert-type scales from 5 to 7 points each. For readers who want more, a supplemental website provides additional material in four tabs: Questionnaire designer Frequently Asked Questions; pretest/posttest case study; questionnaire design forms and templates; and links to questionnaire checklists, sample size calculators, selected articles, and more.
No doubt as you read this book, you too will find yourself recritiquing questionnaires you have used previously. And if you are new to this business, you will undoubtedly benefit immensely from the many lessons learned conveyed by these two attentive authors.
Footnotes
Gerald P. Krueger, PhD, CPE, is the HFES overall editor for book reviews in Ergonomics in Design. He has an extensive background and experience in conducting laboratory and field tests involving evaluations of equipment operator performance with new and novel systems under development (both military and civilian applications). Thus, he has frequently employed user questionnaires, rating scales, structured interviews, and so on, in his work. He is a hearty supporter of gaining user/equipment operator insights into how and why they perform the way they do in tests and in research projects.