Angoff and Nedelsky Standard Setting Procedures: Implications for the Validity of Proficiency Test Score Interpretation

Abstract

The Angoff and Nedelsky standard setting procedures are currently in use in many educational proficiency testing situations. This study has examined how local content specialists performed when applying these procedures to objective-referenced instruments in reading and mathematics. Research questions addressed several issues related to the validity of test score interpretation, such as the effects of judges' demographic characteristics on ratings, the consistency of cut scores for different groups of judges applying the same method, the cut score differences between the two methods, and the pattern of interrater correlations among item ratings. The results revealed several differences between the standard setting procedures in terms of both level and consistency of the cut scores generated. Also, a possible content-related interaction with the standard setting method was observed.

References

Andrew, B. J. and Hecht, J. T. A preliminary investigation of two procedures for setting examination standards. Educational and Psychological Measurement, 1976, 36, 35-50.

Angoff, W. H. Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational Measurement, Washington, D. C.: American Council on Education, 1971.

Behuniak, P. An investigation of standard setting methods for objective referenced proficiency tests. Unpublished Ph.D. dissertation, University of Connecticut, 1980.

Brennan, R. L. and Lockwood, R. E. A comparison of two cutting score procedures using generalizability theory. A paper presented to the annual meeting of the National Council on Measurement in Education, San Francisco, April 1979.

Ebel, R. L. Essentials of educational measurement. Englewood Cliffs, N. J.: Prentice-Hall, 1972.

Hambleton, R. K. , Powell, S. and Eignor, E. R. Issues and methods for standard setting. A paper presented to the annual meeting of National Council on Measurement on Education, San Francisco, April, 1979.

Jaeger, R. M. Measurement consequences of selected standard setting models. In M. A. Bunda and J. R. Saunders (Ed.), Practices and Problems in Competency-Based Measurement. National Council on Measurement in Education Monograph, Washington, D. C., 1979.

Koffler, S. L. A Comparison of Approaches for Setting Proficiency Standards . Journal of Educational Measurement. 1980, 17, 167-178.

Meskauskas, J. H. Evaluation models for criterion-referenced testing: Views regarding mastery and standard setting . Review of Educational Research, 1976, 46, 133-158.

10.

Nedelsky, L. Absolute grading standards for objective tests . Educational and Psychological Measurement, 1954, 14, 3-19.

11.

Skakun, E. N. and Kling, S. Comparability of Methods for Setting Standards. Journal of Educational Measurement. 1980, 17, 229-235.

12.

Zieky, M. J. and Livingston, S. A. Manual for setting standards on the basic skills assessment tests. Princeton, N. J.: Educational Testing Service, 1977.