The problem of mastery decisions and optimizing cutoff scores on criterion-referenced tests is con sidered. This problem can be formalized as an (em pirical) Bayes problem with decisions rules of a monotone shape. Next, the derivation of optimal cutoff scores for threshold, linear, and normal ogive loss functions is addressed, alternately using such psychometric models as the classical model, the beta-binomial, and the bivariate normal model. One important distinction made is between deci sions with an internal and an external criterion. A natural solution to the problem of reliability and validity analysis of mastery decisions is to analyze with a standardization of the Bayes risk (coefficient delta). It is indicated how this analysis proceeds and how, in a number of cases, it leads to coeffi cients already known from classical test theory. Fi nally, some new lines of research are suggested along with other aspects of criterion-referenced test ing that can be approached from a decision-the oretic point of view.
References
1.
Alf, E.F., & Dorfman, D.D.The classification of individuals into two criterion groups on the basis of a discontinuous pay-off function. Psychometrika, 1967,32,115-123.
2.
Besel, R.Using group performance to interpret individual responses to criterion-referenced tests. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA, February-March 1973. (EDRS No. ED 076 658.)
3.
Cox, R.C., & Graham, G.T.The development of a sequentially scaled achievement test. Journal of Educational Measurement, 1966, 3, 147-150.
4.
Davis, C.E., Hickman, J., & Novick, M.R.A primer on decision analysis for individually prescribed instruction (ACT Technical Bulletin No. 17. Iowa City, IA: The American College Testing Program, 1973.
5.
DeGroot, M.H.Optimal statistical decisions. New York: McGraw-Hill, 1970.
6.
de Gruijter, D.N.M.A criterion-referenced point bi-serial correlation coefficient. Tijdschrift voor Onderwijsresearch, 1978, 3, 257-261.
7.
Emrick, J.A.An evaluation model for mastery testing. Journal of Educational Measurement, 1971, 8, 321-326.
8.
Ferguson, T.S.Mathematical statistics: A decision theoretic approach. New York: Academic Press, 1967.
9.
Fhanér, S.Item sampling and decision-making in achievement testing. British Journal of Mathematical and Statistical Psychology, 1974, 27, 172-175.
10.
Glass, G.V.Standards and criteria. Journal of Educational Measurement , 1978, 15, 237-261.
11.
Hambleton, R.K., Hutten, L.R., & Swaminathan, H.A comparison of several methods for assessing student mastery in objectives-based instructional programs. Journal of Experimental Education, 1976, 45, 57-64.
12.
Hambleton, R.K., & Novick, M.R.Toward an integration of theory and method for criterion-referenced tests . Journal of Educational Measurement, 1973 ,10, 159-170.
Kalin, S., & Rubin, H.Distributions possessing a monotone likelihood ratio. Journal of the American Statistical Association, 1956,1, 637-643.
17.
Keats, J.A., & Lord, F.M.A theoretical distribution for mental test scores. Psychometrika , 1962, 27, 59-72.
18.
Koppelaar, H., van der Linden, W.J., & Mellenbergh, G.J.A computer program for classifications in dichotomous decisions based on dichotomously scored items. Tijdschrift voor Onderwijsresearch, 1977, 2, 32-37.
19.
Lewis, C., Wang, M., & Novick, M.R.Marginal distributions for the estimation of proportions in m groups. Psychometrika, 1975, 40, 63-75.
20.
Kriewall, T.E.Aspects and applications of criterion-referenced tests. Illinois School Research, 1972, 9, 5-18.
21.
Lindgren, B.W.Statistical theory (3rd ed.). New York: Macmillan, 1976.
22.
Livingston, S.A.Criterion-referenced applications of classical test theory. Journal of Educational Measurement, 1972, 9, 13-26.
Lord, F.M., & Novick, M.R.Statistical theories of mental test scores. Reading, MA : Addison-Wesley, 1968.
25.
Luce, R.D., & Raiffa, H.Games and decisions. New York: John Wiley & Sons, 1957.
26.
Macready, G.B., & Dayton, C.M.The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 1977,2, 99-120.
27.
Mellenbergh, G.J., Koppelaar, H., & van der Linden, W.J.Dichotomous decisions based on dichotomously scored items: A case study. Statistica Neerlandica, 1977, 31, 161-169.
28.
Mellenbergh, G.J., & van der Linden, W.J.The internal and external optimality of decisions based on tests. Applied Psychological Measurement , 1979, 3, 259-273.
29.
Millman, J.Passing scores and test lengths for domain-referenced measures. Review of Educational Research, 1973, 43, 205-216.
30.
Mokken, R.J.A theory and procedure of scale analysis. The Hague: Mouton, 1971.
31.
Novick, M.R., Lewis C., & Jackson, P.H.The estimation of proportions in m groups. Psychometrika, 1972, 38, 19-46.
32.
Novick, M.R., & Lindley, D.V.The use of more realistic utility functions in educational applications. Journal of Educational Measurement, 1978, 15, 181-191.
33.
Raiffa, H., & Schlaifer, R.Applied statistical decision theory. Boston: Division of Research, Harvard Business School, 1961.
34.
Robbins, H.An empirical Bayes approach to statistics. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability (Vol. 1). Berkeley: University of California Press, 1956.
35.
Robbins, H.The empirical Bayes approach to statistical decision problems. Annals of Mathematical Statistics, 1964, 35, 1-20.
36.
Swaminathan, H., Hambleton, R.K., & Algina, J.A Bayesian decision-theoretic procedure for use with criterion-referenced tests. Journal of Educational Measurement, 1975, 12, 87-98.
37.
van der Linden, W.J.Binomial test models and item difficulty. Applied Psychological Measurement, 1979, 3, 401-411.
38.
van der Linden, W.J.Using aptitude measurements for the optimal assignment of subjects to treatments with and without mastery score. Psychometrika, 1981, 45, in press.
39.
van der Linden, W.J.Estimating the parameters of Emrick's mastery testing model. Submitted for publication, 1980.
40.
van der Linden, W.J. & Mellenbergh, G.J.Optimal cutting scores using a linear loss function . Applied Psychological Measurement, 1977, 1, 593-599.
41.
van der Linden, W.J., & Mellenbergh, G.J.Coefficients for tests from a decision theoretic point of view. Applied Psychological Measurement, 1978, 2,119-134.
42.
Wilcox, R.R.A note on the length and passing score of a mastery test. Journal of Educational Statistics, 1976,1, 359-364.
43.
Wilcox, R.R.A note on decision-theoretic coefficients for tests. Applied Psychological Measurement, 1978, 2, 609-613.
