Previously proposed methods for calculating the kappa measure of nominal rating agreement among multiple raters are not applicable in many situations. This paper presents a more general computational method which can be used across a broader range of rating designs, including those in which raters vary with respect to their base rates and pairs of raters vary with respect to how many subjects they rate in common. A Monte Carlo method for determining the statistical significance of this generalized kappa coefficient is discussed.
References
1.
Cohen, J.A coefficient of agreement for nominal scales . Educational and Psychological Measurement, 1960, 20, 27-36.
2.
Conger, A. J.Integration and generalization of kappa for multiple raters . Psychological Bulletin, 1980, 88, 322-328.
3.
Davies, M. and Fleiss, J. L.Measuring agreement for multinomial data. Biometrics, in press.
4.
Fleiss, J. L.Measuring nominal scale agreement among many raters . Psychological Bulletin, 1971, 76, 378-382.
