Contemporary Issues in the Analysis of Data: A Survey of 551 Psychologists

Abstract

We asked active psychological researchers to answer a survey regarding the following data-analytic issues: (a) the effect of reliability on Type I and Type II errors, (b) the interpretation of interaction, (c) contrast analysis, and (d) the role of power and effect size in successful replications. Our 551 participants (a 60% response rate) answered 59% of the questions correctly; 46%) accuracy would be expected according to participants' response preferences alone. Accuracy was higher for respondents with higher academic ranks and for questions with “no” as the right answer. It is suggested that although experienced researchers are able to answer difficult but basic data-analytic questions at better than chance levels, there is also a high degree of misunderstanding of some fundamental issues of data analysis.

Get full access to this article

View all access options for this article.

References

Amabile

T.M.

(1983). Brilliant but cruel: Perceptions of negative evaluators. Journal of Experimental Social Psychology, 19, 146–156.

Amabile

T.M.

Glazerbrook

A.H.

(1981). A negativity bias in interpersonal evaluation. Journal of Experimental Social Psychology, 18, 1–22.

Asch

(1955). Opinions and social pressures. Scientific American, 193, 31–35.

Cohen

(1962). The statistical power of abnormalsocial psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145–153.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.), Hillsdale, NJ: Erlbaum.

Cohen

(1990). Things I have learned (so far). American Psychologist, 45, 1304–1312.

Cohen

(1983). Applied multiple regression/correlation analysis for the behavioral sciences (2nd ed.), Hillsdale, NJ: Erlbaum.

Dawes

R.M.

(1969). “Interaction effects” in the presence of asymmetrical transfer. Psychological Bulletin, 71, 55–57.

Milgram

(1963). Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67, 371–378.

10.

Minturn

E.B.

Lansky

L.M.

Dember

W.N.

(1972, April). The interpretation of levels of significance by psychologists: A replication and extension. Paper presented at the meeting of the Eastern Psychological Association, Boston.

11.

Nelson

Rosenthal

Rosnow

R.L.

(1986). Interpretation of significance levels and effect sizes by psychological researchers. American Psychologist, 44, 1276–1284.

12.

Rosenthal

(1991). Cumulating psychology: An appreciation of Donald T. Campbell. Psychological Science, 2, 213, 217–221.

13.

Rosenthal

Gaito

(1963). The interpretation of level of significance by psychological researchers. Journal of Psychology, 55, 33–38.

14.

Rosenthal

Rosnow

R.L.

(1975). The volunteer subject New York: Wiley-Interscience.

15.

Rosenthal

Rosnow

R.L.

(1991). Essentials of behavioral research: Methods and data analysis New York: McGraw-Hill.

16.

Rosnow

R.L.

Rosenthal

(1989a). Definition and interpretation of interaction effects. Psychological Bulletin, 105, 143–146.

17.

Rosnow

R.L.

Rosenthal

(1989b). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276–1284.