Coefficient Lambda for Interrater Agreement Among Multiple Raters: Correction for Category Prevalence

Abstract

Fleiss’s Kappa is an extension of Cohen’s Kappa, developed to assess the degree of interrater agreement among multiple raters or methods classifying subjects using categorical scales. Like Cohen’s Kappa, it adjusts the observed proportion of agreement to account for agreement expected by chance. However, over time, several paradoxes and interpretative challenges have been identified, largely stemming from the assumption of random chance agreement and the sensitivity of the coefficient to the number of raters. Interpreting Fleiss’s Kappa can be particularly difficult due to its dependence on the distribution of categories and prevalence patterns. This paper argues that a portion of the observed agreement may be better explained by the interaction between category prevalence and inherent category characteristics, such as ambiguity, appeal, or social desirability, rather than by chance alone. By shifting away from the assumption of random rater assignment, the paper introduces a novel agreement coefficient that adjusts for the expected agreement by accounting for category prevalence, providing a more accurate measure of interrater reliability in the presence of imbalanced category distributions. It also examines the theoretical justification for this new measure, its interpretability, its standard error, and the robustness of its estimates in simulation and practical applications.

Keywords

interrater agreement Fleiss’s Kappa multiple raters prevalence-agreement effect coefficient Lambda

Introduction

Cohen (1960) introduced the Kappa coefficient to assess the degree of interrater agreement between two raters or methods evaluating the same subjects. Fleiss (1971) later extended this measure to accommodate multiple raters assigning categorical ratings on a nominal scale. Several Kappa-like agreement measures have been proposed in the literature, including Gwet’s AC1 (2001) and Krippendorff’s Alpha (2004). Interrater agreement reflects the extent to which raters’ judgments are consistent, reproducible, and reliable. Such raters may include clinical psychologists, teachers, social scientists, or professionals from various fields—such as medicine, psychology, sociology, or business—who evaluate subjects on a particular phenomenon or trait using categorical rating scales with any number of categories (Cousineau & Laurencelle, 2017; Dawson, 2017; Kraemer et al., 2002; Moons & Vandervieren, 2022).

Researchers studying aggressive behavior in children may observe and rate instances of aggression within a controlled play environment. Each episode is recorded, and the observers’ ratings are later compared to assess the level of interrater agreement concerning the presence and intensity of aggression. Similarly, when evaluating anxiety, three different methods—self-report questionnaires, clinical interviews, and psychological assessments—may be used to classify adult participants into four levels of anxiety. The goal is to determine the extent of interrater agreement among these methods in accurately classifying anxiety levels. In the context of test development, subject-matter experts evaluate items to assess content validity. They rate the extent to which each item measures the intended construct, and their ratings are compared to determine interrater agreement regarding item validity.

Fleiss’s Kappa offers greater flexibility than Cohen’s Kappa, as it accommodates any number of raters—a feature particularly useful in a variety of research contexts and applications. Both Cohen (1960) and Fleiss (1971) recognized that a certain level of agreement among raters can occur by chance. Cohen defined this chance agreement as the joint product of the marginal probabilities for each category, whereas Fleiss defined it as the square of the sum of the marginal probabilities for each category. Accordingly, both coefficients—Cohen’s Kappa and Fleiss’s Kappa—are calculated by subtracting the expected chance agreement from the observed proportion of agreement.

However, it is important to clarify that the chance agreement in these measures does not assume that raters are assigning categories randomly in the naive sense. Rather, it reflects the level of agreement that would be expected if each rater applied their judgments independently—whether those judgments stem from noise, differing interpretations, or consistent but unaligned understandings of the categories. This point has been emphasized in critiques (e.g., Brennan & Prediger, 1981), which argue that the statistical correction may not reflect real-world rating processes, where raters are typically trained or otherwise expected to apply consistent criteria.

Since the development of Cohen’s Kappa and Fleiss’s Kappa as chance-corrected measures of interrater agreement, several interpretive issues and paradoxes have been identified (Brennan & Prediger, 1981; Delgado & Tibau, 2019; Warrens, 2010; Zec et al., 2017). As noted by Feinstein and Cicchetti (1990a) and Mandrekar (2011), these challenges arise because Kappa statistics are influenced not only by the actual agreement between raters but also by confounding factors such as rater bias and the prevalence of categories. One key criticism is that the “chance agreement” in Kappa reflects the level of agreement expected under the assumption that raters assign categories independently, based on possibly differing interpretations—not necessarily that their assignments are random in a naive sense. Consequently, Kappa can yield deceptively low values even when observed agreement is high, or vary inconsistently across studies with similar rating behaviors.

Therefore, relying solely on Kappa values to assess agreement can be misleading. To enhance interpretation, researchers are encouraged to report additional metrics—such as prevalence indices and bias indices—alongside Kappa, and to clearly state any assumptions regarding the raters’ shared or divergent understandings of the rating categories. This provides a more transparent and nuanced account of the factors influencing interrater agreement.

Moreover, several key factors influence the behavior and interpretation of agreement measures: number of raters, number of categories, category prevalence, variability in prevalence patterns, category definition and interpretability, and rater bias and shared frameworks. For the effect of the number of raters, the effect of adding more raters on agreement is not uniform—it depends on the context. In many real-world settings, increasing the number of raters can lead to more disagreement, simply because each rater brings a unique interpretation, and more opinions create more room for divergence. However, in structured settings with shared training or clearly defined categories, additional raters may actually increase agreement by stabilizing assessments and averaging out individual variability.

When the observed proportion agreement is fixed when comparing interrater agreement among different number of raters, the true proportion agreement and interrater agreement coefficients is expected be smaller for more raters. This is because achieving the same level of observed agreement with more raters generally requires stronger agreement patterns, leading to a relative decrease in true agreement. A well-designed agreement measure should remain interpretable across different numbers of raters and avoid systematic bias (inflation or deflation) due to changes in rater count alone. This claim is examined in greater detail in a subsequent section.

As the number of categories increases, agreement typically becomes more difficult to achieve due to finer distinctions and greater opportunities for disagreement. A good agreement measure should reflect this increase in complexity, but not penalize raters disproportionately for using more nuanced rating schemes.

Moreover, agreement measures should account for the effect of category prevalence. In interpreting the effects of category prevalence on agreement, the quantity $\frac{1}{p_{e}}$ , where $p_{e} = \sum_{j} p_{j}^{2}$ , provides an intuitive and informative summary of balance in marginal category distributions. This measure, sometimes referred to as the reciprocal of Simpson’s index or the effective number of categories, reflects the degree to which category usage is evenly distributed across $J$ categories. It takes values between 1 (when all ratings fall into a single category, i.e., maximal imbalance) and $J$ (when ratings are evenly distributed, i.e., perfect balance). Conceptually, it can be interpreted as the number of categories over which the ratings are effectively spread. As such, $\frac{1}{p_{e}}$ complements observed agreement by capturing the structure of the marginal distributions.

When certain categories dominate, high agreement may arise due to base rates rather than shared understanding. Conversely, when categories are rarely used, observed agreement may be low even if raters are applying similar criteria. A conceptually sound agreement measure should adjust for these base rate effects in a way that distinguishes between structural prevalence and true concordance. For variability in prevalence patterns, when raters differ in how frequently they use categories—whether due to interpretation, bias, or training—agreement measures must consider this variability. Some indices (e.g., Cohen’s Kappa) are sensitive to asymmetric marginal distributions, which may or may not reflect disagreement. Measures that assume symmetrical or identical marginal distributions may misrepresent the true level of concordance.

The clarity or ambiguity of category definitions directly affects agreement. Clear, unambiguous categories foster higher agreement, while vague or overlapping categories increase interpretive divergence. An ideal agreement measure should not assume all categories are equally interpretable or equally likely to be used. Finally, agreement may be influenced by systematic rater biases, training differences, or varying conceptual frameworks. Agreement measures that assume all raters share the same understanding of categories may overstate true consensus. Conversely, assuming fully independent interpretations may understate meaningful agreement. A conceptually robust approach must acknowledge the spectrum between shared and divergent interpretations.

It is well-established that observed proportion agreement is influenced by the prevalence of rating categories. The way in which these categories are defined significantly affects how raters interpret and evaluate the subject matter, which in turn shapes the distribution of responses. Category prevalence directly impacts the calculation of Fleiss’s Kappa by altering the expected agreement under chance. Specifically, Fleiss’s Kappa estimates chance agreement based on the marginal distributions of the raters, which can reflect either independent judgments or systematic differences in category usage (Feinstein & Cicchetti, 1990b; Fleiss et al., 1969, 1979; Vach, 2004).

When category distributions are imbalanced—that is, when one category is substantially more prevalent than others—the expected chance agreement becomes skewed. This can result in inflation or deflation of the Kappa statistic, depending on which category dominates. In such situations, Kappa may fail to reflect the actual level of interrater agreement: high prevalence can artificially increase Kappa, overstating agreement, while low prevalence can suppress it, understating agreement (Byrt et al., 1993).

These distortions highlight the importance of explicitly addressing how prevalence and interpretive alignment among raters influence agreement metrics. This consideration underlies our motivation to explore a prevalence-corrected agreement measure that more accurately captures agreement independent of category imbalance.

To address the criticisms and paradoxes associated with category prevalence and rater bias, several researchers have proposed alternative agreement coefficients to Cohen’s Kappa and Fleiss’s Kappa. Notable examples include Gwet’s AC1 (2001) and Krippendorff’s alpha (2004). These measures aim to offer more stable estimates of agreement, particularly under conditions of prevalence imbalance or minimal bias. However, despite these improvements, all of these coefficients remain chance-adjusted and continue to rely—explicitly or implicitly—on the assumption of random category assignment by raters. As such, they are not immune to the foundational limitations that affect traditional chance-corrected agreement measures.

Taken together, these factors point to the need for agreement measures that are not only mathematically sound but also behaviorally and contextually grounded. Our proposed coefficient aims to better align with these conceptual expectations by adjusting for the systematic effects of category prevalence while preserving the interpretive value of observed agreement.

This paper introduces a novel agreement coefficient that moves beyond traditional chance-adjusted measures, aiming to address several issues and paradoxes associated with existing agreement statistics. It begins by examining the impact of category prevalence on the level of agreement among raters. The paper then defines and estimates the category prevalence-agreement effect, using this to adjust the observed proportion agreement. Building on this adjustment, a new coefficient of agreement for multiple raters or methods on categorical scales is proposed. In addition, the paper discusses the interpretation and significance of this new coefficient, along with its sampling stability, including the estimation of its standard error.

Coefficient Fleiss’s Kappa

Let $N$ denote the total number of subjects rated by $R$ raters, and let $J$ represent the number of mutually exclusive categories used for classification. Let $i$ index the subjects, where $i = 1, \dots, N$ ; $r$ index the raters, where $r = 1, \dots, R$ ; and $j$ index the categories, where $j = 1, \dots, J$ . The data can be organized in an $N \times$ R (Subjects × Raters) matrix $X$ , with $x_{ij} = j$ , the label of the category that rater $r$ assigned to subject i. Each rater assigns exactly one category to each subject, ensuring that the category assignments are mutually exclusive.

To facilitate the calculation of the agreement coefficient, we first count how many times each specific category is assigned by all raters for each subject. This count is defined as $y_{ij}$ , representing the number of assignments of category $j$ by the $R$ raters for subject $i$ . Therefore, $y_{ij}$ can take integer values from 0 up to $R$ . For each subject $i$ , the sum of these counts across all categories satisfies $\sum_{j} y_{ij} = R$ . An $N \times J$ (Subjects × Categories) matrix $y_{ij}$ can be created from $x_{ij}$ , with $y_{ij}$ = the number of raters that classify subject $i$ into category $j$ . Then define $y_{j}$ as the average count in column $j$ , and $P_{j} = \frac{y_{j}}{R}$ as the average prevalence of category $j$ .

In Fleiss’s Kappa, the overall observed proportion agreement is calculated as the average agreement among $R$ raters across all subjects $(i = 1, \dots, N)$ and categories $(j = 1, \dots, J)$ . For each subject $i$ , there are $(\begin{matrix} y_{ij} \\ 2 \end{matrix}) = y_{ij} (y_{ij} - 1) / 2$ unique agreement pairs of raters from the $y_{ij}$ raters who assign category $j$ . The maximum possible number of such rater agreement pairs assigning category $j$ , assuming all $R$ raters assign that category, is $(\begin{matrix} R \\ 2 \end{matrix}) = R (R - 1) / 2$ . That is,

P_{o} = \frac{1}{N} \sum_{i} \sum_{j} \frac{y_{ij} (y_{ij} - 1) / 2}{R (R - 1) / 2} = \frac{1}{N} \sum_{i} \sum_{j} \frac{y_{ij} (y_{ij} - 1)}{R (R - 1)} .

(1)

Hence, the overall observed proportion agreement is the proportion of agreeing pairs of assignments out of the total $R (R - 1)$ possible agreement pairs for each subject, summed over all categories. $P_{o}$ represents the observed proportion of agreement and takes any real value between 0 and 1. Fleiss’s Kappa assumes that a certain level of agreement occurs purely by chance (Fleiss, 1971). Under the assumption that raters assign categories randomly, the expected chance agreement proportion is calculated as follows:

P_{e} = \sum_{j} p_{j}^{2} = \sum_{j} {(\frac{y_{j}}{R})}^{2},

(2)

where $y_{j} = \frac{1}{N} \sum_{i} y_{ij}$ represents the average number of assignments to category $j$ by the $R$ raters across all subjects. Therefore, Fleiss’s Kappa is calculated as the observed proportion of agreement corrected for the amount of agreement expected by chance, expressed as the proportion of observed agreement beyond chance relative to the maximum possible agreement beyond chance.

κ = \frac{P_{o} - P_{e}}{1 - P_{e}} .

(3)

Landis and Koch (1977) provided interpretation guidelines for Fleiss’s Kappa and other Kappa-like agreement coefficients.

Gwet’s AC1 coefficient (2001) utilizes a computation of the observed agreement that is similar to that used in Fleiss’s Kappa. However, it differs in how it estimates the expected agreement by chance. Unlike Kappa, which assumes random assignment of categories based on marginal probabilities, Gwet’s AC1 adjusts for chance agreement using an alternative model that reduces sensitivity to category prevalence and marginal distribution imbalances. This results in a more stable and often more robust estimate of chance-corrected interrater agreement, particularly in cases of highly skewed or unbalanced data. The observed proportion agreement and the correction for chance agreement are defined by Gwet’s AC1 as

P_{o} = \frac{1}{N} \sum_{i} \sum_{j} \frac{y_{ij} (y_{ij} - 1)}{R (R - 1)} and P_{e} = \frac{1}{J - 1} \sum_{j} \sum_{j'} P_{j} (1 - P_{j}) .

(4)

Krippendorff’s α statistic (Krippendorff, 2004) is widely used in the field of interrater agreement, particularly in content analysis and related areas where subjective judgment is involved. It accommodates various data types—including nominal, ordinal, interval, and ratio scales—and allows for missing data, making it a versatile and robust measure for assessing the reliability of ratings across multiple coders. The proportion of observed agreement is defined differently as

P_{a} = (1 - \frac{1}{N \bar{t}}) P_{a 0} + \frac{1}{N \bar{t}},

(5)

where $P_{a 0} = \frac{1}{N} \sum_{i} \sum_{j} \frac{t_{ij} ({\bar{t}}_{ij +} - 1)}{\bar{t} (t_{i} - 1)}$ , $\bar{t} = \frac{1}{N} \sum_{i} t_{i}$ , and ${\bar{t}}_{ij +} = \sum_{j'} t_{ij'}$ . Here, $t_{i}$ is the number of raters who assigned a particular category $j$ to subject $i$ , and $\bar{t}$ is the average number of raters per subject. The proportion of chance-expected agreement is defined as

P_{e} = \sum_{j} \sum_{j'} P_{j} P_{j'},

(6)

where $P_{j} = \frac{1}{N} \sum_{i} t_{ij} \bar{t}$ and $P_{j} = \frac{1}{N} \sum_{i} t_{ij}, \bar{t}$ . Hence, Krippendorff’s αis

α = \frac{P_{a} - P_{e}}{1 - P_{e}} .

(7)

All of these interrater agreement statistics are designed to assess agreement for categorical data, and their values typically range from –1 to 1. A value of 1 denotes perfect agreement, a value of 0 reflects agreement no better than chance, and negative values indicate agreement worse than chance.

A New Coefficient of Agreement

The new agreement coefficient calculates observed proportion agreement similarly to Fleiss’s Kappa but introduces a fundamentally different correction mechanism. Whereas Fleiss’s Kappa and many traditional agreement coefficients estimate expected agreement under the assumption that raters assign categories independently and randomly (Fleiss, 1971), the new coefficient explicitly accounts for the systematic influence of category prevalence on observed agreement. Rather than attributing all agreement beyond chance to randomness, this method recognizes that variations in agreement may arise from how raters interpret and apply the categories—especially when those categories differ in clarity, salience, or conceptual distinctiveness.

Category prevalence can reflect more than just rater tendencies; it may also stem from the intrinsic features of the categories themselves, such as clarity of definition, social desirability, or relevance to the subject matter. For example, clearly defined categories are more likely to produce consistent interpretation and higher agreement, while vague or ambiguous categories often reduce agreement due to variability in how they are understood. In addition, the number of categories influences agreement: raters tend to agree more when using fewer categories (e.g., two vs. three or more). By correcting for the effect of category prevalence—not by assuming randomness, but by acknowledging the structured influence of interpretive clarity—the proposed coefficient aims to offer a more valid reflection of interrater agreement in real-world settings.

The quantity $y_{j} = \frac{1}{N} \sum_{i} y_{ij}$ represents the relative prevalence of category $j$ across all subjects. It takes a real value between 0 and $R$ , with larger values of $y_{j}$ indicating higher prevalence of that category. In essence, when a category is more common, individuals often find it easier to agree on its attributes—a phenomenon observed in domains such as social judgment and decision-making. This tendency, known as the prevalence-agreement effect, reflects the increased likelihood of agreement among raters when certain categories dominate. Such dominance may arise from properties like conceptual clarity, social desirability, intuitive appeal, or definitional simplicity. These characteristics—linked to the traits being measured—affect how categories are interpreted and thus influence both their prevalence and the resulting agreement levels.

Importantly, high prevalence and high agreement can co-occur not because raters are influenced by each other or by prior category usage (which is typically unknown to them), but because certain categories are more universally understood or more readily mapped onto shared conceptual frameworks. Agreement can be caused by category prevalence when raters appear to agree often, not because they truly interpret their rating the same way, but because one category is so common that they both happen to pick it frequently. This inflates or deflates the observed agreement and can mask the true level of interrater reliability. In this sense, the prevalence-agreement effect reflects a structured, non-random relationship between category properties and rater behavior. Recognizing this effect helps justify the need for agreement measures that adjust for prevalence in a way that acknowledges these systematic influences, rather than attributing them solely to chance.

To quantify this, the prevalence-agreement effect ( $P_{v}$ ) is defined as $P_{v}$ the difference between the observed proportion agreement and the true proportion agreement that would exist if all categories were equally prevalent: $P_{v} = P_{o} - P_{t}$ , where $P_{o}$ is the observed proportion agreement, and $P_{t}$ is the true proportion agreement after adjusting for the effects of category prevalence. $P_{t}$ represents what the agreement would likely be if all categories were equally represented and no prevalence bias existed.

When $P_{v} = 0$ , the categories have a neutral effect on rater agreement, meaning that the observed proportion agreement ( $P_{o}$ ) accurately reflects the true proportion agreement ( $P_{t}$ ). In this case, category prevalence does not distort the extent of agreement. If $P_{v} > 0$ , the categories are particularly appealing or easily recognizable to raters, leading to an inflation of the observed agreement. As a result, $P_{o}$ overestimates the true level of agreement ( $P_{t}$ ), since part of the consensus arises from category-driven alignment rather than genuine rater consistency. Conversely, if $P_{v} < 0$ , the categories may be vague, ambiguous, or otherwise unappealing to raters, making agreement more difficult. In this case, $P_{o}$ underestimates $P_{t}$ , as the low observed agreement is partially due to category-related confusion or disagreement, rather than an actual lack of consensus among raters.

Hence, it is expected that the extent of true agreement among multiple raters will be higher for categories with greater prevalence, while categories with lower prevalence are likely to produce less true agreement. For each category $j$ , the number of agreeing rater pairs attributable to that category—based purely on its prevalence—can be expressed as $(\begin{matrix} y_{j} \\ 2 \end{matrix}) = y_{j} (y_{j} - 1) / 2$ , representing the number of rater pairs that assigned the same category out of the $(\begin{matrix} R \\ 2 \end{matrix}) = R (R - 1) / 2$ possible pairs. This reflects the proportion of agreement that may arise due to the dominance of that category. Summing across all categories, the category-agreement effect can be defined as the total contribution of all categories to the observed agreement, based solely on their prevalence:

P_{v} = \sum_{j} \frac{y_{j} (y_{j} - 1)}{R (R - 1)} .

(8)

$P_{v}$ is the proportion agreement based on the average assignments by the multiple raters over all subjects. This value serves as the basis for estimating how much of the observed agreement could be attributed to uneven category distributions, and it plays a central role in adjusting the observed agreement to obtain a more accurate measure of true interrater agreement. This equation reveals several important properties of the prevalence-agreement effect, $P_{v}$ :

$P_{v}$ can take both negative and positive values when the number of categories $J > R$ , but only positive values when $J \leq R$ .

The lower limit of $P_{v}$ is given by $\frac{R - J}{J (R - 1)}$ , which occurs when all categories are equally prevalent—that is, when $y_{j} = \frac{R}{J}$ and the variance $σ^{2} (y_{j}) = .$ As the variance $σ^{2} (y_{j})$ increases (i.e., as the distribution of category prevalence becomes more uneven), $P_{v}$ increases accordingly.

The lower limit of $P_{v}$ decreases and asymptotically approaches $- \frac{1}{R - 1}$ as the number of categories $J$ increases.

Conversely, the lower limit of $P_{v}$ increases and approaches $\frac{1}{J}$ as the number of raters $R$ increases.

The upper limit of $P_{v}$ is the observed proportion agreement $P_{o}$ , which would occur if all agreement were attributable to prevalence.

When $P_{v} = 0$ , category prevalence has a neutral effect on rater agreement—that is, observed agreement reflects true agreement.

When $P_{v} > 0$ , category prevalence has a positive effect, indicating that the categories are appealing or easy to agree upon, thus inflating the observed agreement.

When $P_{v} < 0$ , category prevalence has a negative effect, suggesting that the categories are ambiguous or less attractive, thereby deflating the observed agreement.

The degree of agreement actually attained—free from the influence of category prevalence—is defined as the true proportion agreement, denoted by $P_{t}$ , and calculated as $P_{t} = P_{o} - P_{v}$ . This formula provides an estimate of the agreement that would be observed if category prevalence had no effect. If $P_{v} = 0$ , there is no prevalence-agreement effect, and $P_{t} = P_{o}$ . However, when $P_{v} \neq 0$ , the observed proportion agreement does not accurately reflect the true level of agreement, $P_{t} \neq P_{o}$ . More specifically, If $P_{v} > 0$ , indicating that category prevalence positively biases agreement, the true proportion agreement is less than the observed agreement ( $P_{t} < P_{o}$ ). In this case, $P_{o}$ overestimates the actual agreement. If $P_{v} < 0$ , indicating that category prevalence negatively biases agreement, the true proportion agreement is greater than the observed agreement ( $P_{t} > P_{o}$ ). Here, $P_{o}$ underestimates the actual agreement. $P_{t}$ is estimated by taking the difference between the observed proportion agreement calculated from the actual category assignments made by multiple raters for each subject and the expected agreement based on the average distribution of category assignments across all subjects. This adjustment ensures that the final agreement measure more accurately reflects true interrater consensus, independent of distortions introduced by unequal category prevalence.

Interrater agreement can then be estimated by normalizing the true proportion agreement $P_{v}$ using the maximum attainable agreement after removing the prevalence-agreement effect—that is, $1 - P_{v}$ . The quantity $1 - P_{v}$ represents the theoretical upper bound of agreement that could be achieved in the absence of any bias introduced by category prevalence. It defines the portion of the observed agreement space that is unaffected by the prevalence-agreement effect. So, the agreement coefficient for interrater agreement is

λ = \frac{P_{o} - P_{v}}{1 - P_{v}} .

(9)

The coefficient $λ$ represents the proportion of observed proportion agreement to the maximum attainable proportion agreement, after both have been corrected for the prevalence-agreement effect. In essence, $λ$ quantifies the extent to which true agreement—free from the influence of category prevalence—has been achieved among raters. This coefficient can also be interpreted as the proportion of joint ratings in which there is true agreement, excluding the portion of agreement that results from uneven or biased category prevalence. Unlike chance-corrected measures such as Fleiss’s Kappa, $λ$ makes no assumption of random assignment and instead adjusts for structural effects introduced by the prevalence and properties of the categories themselves.

When coefficient $λ$ , as defined in Equations (1, 8, and 9), is applied to the case of two raters or methods (R = 2), it yields values equal to those produced by the agreement coefficient proposed by Almehrizi (2025). Therefore, the coefficient $λ$ can be seen as a generalization of that two-rater agreement coefficient Lambda, extending its applicability to scenarios involving multiple raters. This extension allows for consistent interpretation of agreement across varying numbers of raters while maintaining the same foundational logic of adjusting for the prevalence-agreement effect.

Interpretation of Coefficient Lambda

For any number of raters and categories, coefficient $λ$ maintains a consistent scale and interpretation, making it suitable for comparing agreement across diverse settings. The scale of $λ$ ranges from 0 to 1, where the upper limit of $λ = 1$ occurs only when there is perfect agreement among all raters—that is, $P_{o} = 1$ . In this case, all subjects exhibit complete rater agreement. Notably, even under perfect agreement, the prevalence-agreement effect $P_{v}$ may take on any value less than 1, depending on the distribution of category prevalence. $λ = 1$ also indicates that the total agreement corrected for category prevalence is all due to true agreement among raters.

The lower limit of $λ = 0$ occurs only when there is complete disagreement among raters—that is, $P_{o} = 0$ and $P_{v} = 0 .$ This scenario reflects a total absence of rater consensus, with no subject showing agreement across any category, indicating the highest possible level of disagreement. $λ = 0$ also indicates that the total agreement has no true agreement among raters but instead due to complete disagreement. For interpretation purpose of coefficient Lambda, the following relationship is useful:

P_{o} - P_{v} = \frac{λ}{1 - λ} (1 - P_{o}),

(10)

where $P_{o} - P_{v}$ denotes the true proportion of interrater agreement and $1 - P_{o}$ denotes the proportion of disagreement among raters. In words, the true proportion of agreement is equal to $\frac{λ}{1 - λ}$ times the proportion of disagreement. This formulation provides a valuable interpretation of coefficient Lambda as a measure of interrater agreement coefficient. A value of λ = 0 indicates that the true proportion of agreement is null and all ratings reflect complete disagreement among raters. Conversely, λ = 1 indicates that the proportion of disagreement is null and all ratings reflect complete agreement among raters. A value of λ = 0.5 indicates that the true proportion of agreement equals the proportion of disagreement, meaning that raters show equal proportions of agreement and disagreement. For example, λ = 0.7 indicates that the true proportion of agreement exceeds by $\frac{λ}{1 - λ} = 2.333$ times the proportion of disagreement, reflecting more agreement than disagreement among raters. In contrast, λ = 0.4 indicates that the true proportion of agreement is smaller than the proportion of disagreement by $\frac{λ}{1 - λ} = 0.667$ times, reflecting more disagreement than agreement. Other values of λ can be interpreted in the same way.

The value $λ = 0.5$ serves as a special reference point and retains a consistent interpretation regardless of the number of raters or categories. It represents a midpoint on the agreement scale, indicating that half of the maximum attainable agreement (after accounting for the prevalence-agreement effect) has been achieved. This reference case occurs when $P_{o} - P_{v} = 1 - P_{o}$ .

Values of $λ$ larger than $0.5$ indicate increasing levels of interrater agreement among raters beyond prevalence-based agreement. These values can be interpreted using an agreement scale, where the deviation of $λ$ from the neutral ( $0.5$ ) quantifies the extent of agreement. Values of $λ$ smaller than $0.5$ indicate lower levels of disagreement among raters. These values can be interpreted using a disagreement scale, where the deviation of $λ$ from the neutral ( $0.5$ ) quantifies the extent of disagreement analogously to how $λ$ quantifies agreement.

Moreover, $λ = 0.5$ does not represent chance agreement except in the case of two raters. For more than two raters, chance agreement is indicated by $λ = \frac{1}{R}$ . This follows from the linear transformation linking Fleiss’s Kappa (κ) and coefficient Lambda (λ), where κ ranges between $\frac{- 1}{R - 1}$ and 1, and λ ranges between 0 and 1: $λ = \frac{1 - R}{R} κ + \frac{1}{R}$ . Accordingly, the point of chance agreement, represented by κ = 0, corresponds to λ = $\frac{1}{R}$ . For instance, chance agreement occurs at λ = 0.5, 0.33, 0.25, and 0.20 when R = 2, 3, 4, and 5, respectively. As the number of raters becomes very large (approaching infinity), chance agreement is indicated when λ approaches 0.

The chance agreement defined by Fleiss’s Kappa can be reinterpreted within the framework of coefficient Lambda. Since Fleiss’s κ = 0 corresponds to λ = $\frac{1}{R}$ , chance agreement occurs when the true proportion of interrater agreement equals $\frac{1}{R}$ of the proportion of disagreements among raters (equivalently, when the proportion of disagreements is $R$ times the true proportion of agreement).

Table 1 summarizes the interpretation framework for coefficient Lambda. These categories offer a qualitative framework for interpreting the level of agreement or disagreement between raters based on $λ$ values. Higher values of $λ$ indicate stronger agreement, while lower values reflect greater disagreement. This framework enables a more meaningful interpretation of interrater reliability by incorporating the influence of both agreement intensity and the prevalence-agreement effect.

Table 1.

Framework for Interpreting Coefficient $λ$

Values	Interpretation
$1.00$	100% True Agreement and 0% Disagreement
$0.95 \leq λ < 1.00$	Almost Perfect Agreement
$0.95$	95% True Agreement and 5% Disagreement
$0.90 \leq λ < 0.95$	Extraordinary Agreement
$0.90$	90% True Agreement and 10% Disagreement
$0.80 \leq λ < 0.90$	Substantial Agreement
$0.80$	80% True Agreement and 20% Disagreement
$0.70 \leq λ < 0.80$	Moderate Agreement
$0.70$	70% True Agreement and 30% Disagreement
$0.60 \leq λ < 0.70$	Fair Agreement
$0.60$	60% True Agreement and 40% Disagreement
$0.50 \leq λ < 0.60$	Slight Agreement
$0.50$	50% True Agreement and 50% Disagreement True Agreement is equal to Disagreement
$0.40 \leq λ < 0.50$	Slight Disagreement
$0.40$	40% True Agreement and 60% Disagreement
$0.30 \leq λ < 0.40$	Fair Disagreement
$0.30$	30% True Agreement and 70% Disagreement
$0.20 \leq λ < 0.30$	Moderate Disagreement
$0.20$	20% True Agreement and 80% Disagreement
$0.10 \leq λ < 0.20$	Substantial Disagreement
$0.10$	10% True Agreement and 90% Disagreement
$0.05 \leq λ < 0.10$	Extraordinary Disagreement
$0.05$	5% True Agreement and 95% Disagreement
$0 \leq λ < 0.05$	Almost Perfect Disagreement
$0$	0% True Agreement and 100% Disagreement

Standard Error of $\hat{λ}$

Using the estimators of ${\hat{P}}_{o}$ and ${\hat{P}}_{v}$ , the estimator of coefficient Lambda $λ$ is $\hat{λ} = \frac{{\hat{P}}_{o} - {\hat{P}}_{v}}{1 - {\hat{P}}_{v}}$ . The asymptotic standard error for coefficient Lambda can be obtained using delta method (Radhakrishna, 1948) using the derivatives of $\hat{λ}$ with respect to ${\hat{P}}_{o}$ and ${\hat{P}}_{v}$ as illustrated in the Appendix,

\frac{d \hat{λ}}{d {\hat{P}}_{o}} = \frac{1}{1 - {\hat{P}}_{v}} and \frac{d \hat{λ}}{d {\hat{P}}_{v}} = - \frac{1 - \hat{λ}}{1 - {\hat{P}}_{v}} .

Hence, $\hat{λ}$ follow a normal distribution with a mean $λ$ and a standard error of

SE (\hat{λ}) = \frac{\sqrt{V ({\hat{P}}_{o}) + {(1 - \hat{λ})}^{2} V ({\hat{P}}_{v}) - 2 (1 - \hat{λ}) C ({\hat{P}}_{o}, {\hat{P}}_{v})}}{1 - {\hat{P}}_{v}},

(11)

Here, V stands for variance and Cov is for covariance. The three terms on equation (11) are estimated as

\begin{matrix} V ({\hat{P}}_{o}) = \frac{1}{N R^{2} {(R - 1)}^{2}} V [\sum_{j} y_{ij} (y_{ij} - 1)], \\ where [\sum_{j} y_{ij} (y_{ij} - 1)] = \frac{\sum_{i} {(\sum_{j} y_{ij} (y_{ij} - 1) - \frac{1}{N} \sum_{i} \sum_{j} y_{ij} (y_{ij} - 1))}^{2}}{N - 1}, \end{matrix}

(12)

\begin{matrix} V ({\hat{P}}_{v}) = \frac{1}{N R^{2} {(R - 1)}^{2}} V [\sum_{j} (2 y_{j} - 1) y_{ij}], \\ where V [\sum_{j} (2 y_{j} - 1) y_{ij}] = \frac{\sum_{i} {(\sum_{j} (2 y_{j} - 1) y_{ij} - \frac{1}{N} \sum_{i} \sum_{j} (2 y_{j} - 1) y_{ij})}^{2}}{N - 1}, \end{matrix}

(13)

C ({\hat{P}}_{o}, {\hat{P}}_{v}) = \frac{1}{N R^{2} {(R - 1)}^{2}} Cov (\sum_{j} y_{ij} (y_{ij} - 1), \sum_{j} (2 y_{j} - 1) y_{ij}),

(14)

where

Cov (\sum_{j} y_{ij} (y_{ij} - 1), \sum_{j} (2 y_{j} - 1) y_{ij}) = \frac{\sum_{i} (\sum_{j} y_{ij} (y_{ij} - 1) - \frac{1}{N} \sum_{i} \sum_{j} y_{ij} (y_{ij} - 1)) (\sum_{j} (2 y_{j} - 1) y_{ij} - \frac{1}{N} \sum_{i} \sum_{j} (2 y_{j} - 1) y_{ij})}{N - 1} .

Under the null hypothesis that the population agreement coefficient $λ$ equals a specific value (e.g., $λ = λ_{o}$ ), the sampling distribution of the estimated coefficient $\hat{λ}$ is approximately normal, such that:

z = \frac{\hat{λ} - λ}{SE (\hat{λ})} ~ N (0, 1) .

(15)

This allows for hypothesis testing and the construction of confidence intervals for $λ$ using standard normal theory. The $(1 - α) %$ confidence interval of coefficient Lambda is $\hat{λ} \pm Z_{α / 2} SE (\hat{λ})$ , where $Z$ is obtained from the standard normal distribution.

Numerical Examples

Table 2 presents an agreement matrix for a hypothetical example in which three coders ( $R = 3$ ) independently classify a set of 14 news articles covering the same political event. Each article is categorized according to its tone, using five ordinal categories: very positive (A), positive (B), neutral (C), negative (D), and very negative (E). The objective is to evaluate the consistency with which media coverage conveys sentiment about the event. The left-hand side of Table 2 displays the raw category assignments made by the coders for each article, while the right-hand side summarizes the frequency of assignments per category across the three raters.

Table 2.

Agreement Matrix for Three Raters With Five Categories

Subject	Raters			Category
Subject	$r = 1$	$r = 2$	$r = 3$	$A$	$B$	$C$	$D$	$E$
1	$A$	$A$	$A$	3	0	0	0	0
2	$C$	$A$	$A$	2	0	1	0	0
3	$A$	$A$	$A$	3	0	0	0	0
4	$D$	$D$	$D$	0	0	0	3	0
5	$A$	$A$	$A$	3	0	0	0	0
6	A	$A$	$A$	3	0	0	0	0
7	$E$	$E$	$E$	0	0	0	0	3
8	$A$	$A$	$A$	3	0	0	0	0
9	$C$	$C$	$C$	0	0	3	0	0
10	$C$	$C$	$C$	0	0	3	0	0
11	$A$	$A$	$A$	3	0	0	0	0
12	$D$	$D$	$D$	0	0	0	3	0
13	$A$	$A$	$A$	3	0	0	0	0
14	$D$	$D$	$D$	0	0	0	3	0
Average				1.643	0	.500	.643	.214
$p_{j}$				.548	0	.167	.214	.071

$P_{o} = . 952$ , $P_{v} = . 068$ , $\hat{λ} = . 949, \hat{κ} = . 923, \hat{α} = . 925, AC 1 = . 944$ .

The data exhibit extreme category prevalence, with category A dominating—having a prevalence of 1.643, while the other categories (B through E) show very low or zero prevalence values (0, .5, .643, .214, respectively). This imbalance in category usage produces a prevalence-agreement effect of $P_{v} = . 068$ , indicating that the observed proportion agreement ( $P_{o}$ ) underestimates the true agreement among raters by a small margin. The resulting coefficient $\hat{λ}$ is .949, suggesting almost perfect agreement after accounting for category prevalence. For comparison, Fleiss’s Kappa applied to the same data yields a chance agreement of $P_{e} = . 377$ , producing a Kappa value of .923. Krippendorff’s alpha and Gwet’s AC1 yield values of .925 and .944, respectively.

The estimated standard error of $\hat{λ} = . 949$ is $SE (\hat{λ}) = . 048$ . Testing the null hypothesis of no agreement ( $λ = . 5$ ), the resulting z-value is 9.354, with a p-value of p<.001. This result demonstrates that the interrater agreement is statistically significantly different from .5, suggesting higher agreement than the reference value.

Using the dataset reported in Fleiss (1971) to assess the consistency of psychiatric diagnoses, each of 30 patients was evaluated by six psychiatrists, selected from an original pool of 43. This represents a standardized adjustment from the original study by Sandifer et al. (1968), where patients had been rated by between six and ten psychiatrists. The standardization to six raters per patient ensured uniformity in evaluating interrater agreement. Under this setup, the observed proportion agreement among the psychiatrists was .556, while the expected agreement by chance was calculated at .220, resulting in a Fleiss’s Kappa of .430—commonly interpreted as moderate agreement. However, a prevalence-agreement effect of 0.064 was also identified, indicating that the observed agreement slightly overestimated the true agreement by 0.064. This inflation is attributed to variations in category prevalence across the five diagnostic categories, whose prevalence values were reported as 0.867, 0.867, 1.000, 1.833, and 1.433, respectively. After adjusting for the prevalence-agreement effect, the resulting coefficient $\hat{λ} = . 525$ . This highlights the importance of accounting for category prevalence in agreement analysis, as traditional chance-corrected measures may misrepresent true interrater consistency when prevalence is unevenly distributed.

Table 3 presents a hypothetical example of an agreement matrix in which four clinical psychologists ( $R = 4$ ) independently assess a sample of 25 patients using a standardized depression assessment tool. Each psychologist rates the severity of depressive symptoms using a six-point categorical scale (1 to 6). The objective is to evaluate the consistency of diagnostic judgments across raters. The observed proportion agreement among the four raters was .393, while the chance agreement was estimated at .180, yielding a Fleiss’s Kappa of .260, which reflects fair agreement by conventional interpretation. Similar interrater agreement values were obtained with Krippendorff’s $α = . 268$ and Gwet’s $AC 1 = . 274$ .

Table 3.

Agreement Matrix for Four Raters With Six Categories and Estimates of Interrater Agreement

Subject	Category
Subject	$j = 1$	$j = 2$	$j = 3$	$j = 4$	$j = 5$	$j = 6$
1	0	1	2	1	0	0
2	1	1	1	1	0	0
3	0	0	0	0	3	1
4	2	2	0	0	0	0
5	0	0	2	2	0	0
6	2	2	0	0	0	0
7	0	0	3	1	0	0
8	0	0	0	0	1	3
9	0	0	2	2	0	0
10	1	3	0	0	0	0
11	0	0	0	2	2	0
12	3	1	0	0	0	0
13	0	0	3	1	0	0
14	0	0	0	0	2	2
15	0	0	2	2	0	0
16	2	2	0	0	0	0
17	0	0	1	3	0	0
18	0	2	2	0	0	0
19	0	0	0	0	3	1
20	0	2	2	0	0	0
21	3	1	0	0	0	0
22	0	0	1	3	0	0
23	0	0	0	0	1	3
24	0	0	2	2	0	0
25	1	3	0	0	0	0
$y_{j}$	.60	.80	.92	.80	.48	.40
$p_{j}$	.15	.20	.23	.20	.12	.10

$P_{o} = . 393$ , $P_{v} = - . 094$ , $\hat{λ} = . 445$ , $\hat{κ} = . 260$ , $\hat{α} = . 268$ , $AC 1 = . 274$ .

However, the data exhibit variation in category usage, with prevalence values across the six categories of .60, .80, .92, .80, .48, and .40, respectively. This distribution results in a prevalence-agreement effect of $P_{v} = - . 094$ . This comes from adding the following expected proportion agreement from each category (–.02, –.013, –.006, –.013, –.021, –.02), indicating that the observed agreement underestimates the true agreement by .094. In other words, the skew in category prevalence led to lower-than-actual observed agreement. After adjusting for this effect, the resulting coefficient $\hat{λ} = . 445$ provides an estimate of interrater consistency. $λ$ exceeds the observed proportion agreement and the three interrater agreement coefficients, further illustrating the impact of prevalence imbalance and the value of accounting for it in agreement analysis.

The standard error of $\hat{λ} = . 445$ is $SE (\hat{λ}) = . 040 .$ Testing the same null hypothesis of $λ = . 5$ , the z-value is –1.375, with a p-value of .085. This finding indicates that the interrater agreement is not statistically significantly different from .5.

Number of Raters and True Agreement

When the observed proportion agreement is fixed across conditions, the true proportion agreement and interrater agreement coefficients should be smaller for more raters. This is because achieving the same level of observed agreement with more raters generally requires stronger agreement patterns, leading to a relative decrease in true agreement. Among common indices, the coefficient Lambda uniquely reflects this principle.

This can be understood by considering the different agreement patterns that emerge with varying numbers of raters. With two raters, there is only one possible agreement pattern ( $y_{ij} = 2$ ). With three raters, two distinct patterns exist: unanimous agreement ( $y_{ij} = 3$ ) or partial agreement with one dissenter ( $y_{ij} = 2$ ). With four raters, three distinct patterns occur: unanimous agreement ( $y_{ij} = 4$ ), partial agreement with one dissenter ( $y_{ij} = 3$ ), or partial agreement with two dissenters ( $y_{ij} = 2$ ). As the number of raters increases, the range of possible agreement patterns expands. Consequently, achieving equal true agreement requires higher levels of observed agreement when more raters are involved. Likewise, maintaining the same level of observed agreement with a larger number of raters generally necessitates stronger agreement patterns, which results in a relative decrease in true agreement.

Consider three agreement matrices (Table 4) constructed with an equal observed proportion agreement ( $P_{o} = 0.5$ ), the same number of categories ( $J = 3$ ), and a sample size of $N = 12$ . The first matrix involves two raters, while the second and third involve three raters but differ in the specific assignment patterns. Coefficient Lambda values were 0.611 for the two-rater matrix, and decreased to 0.497 and 0.431 for the two three-rater matrices. In contrast, Fleiss’s Kappa and Krippendorff’s $α$ were 0.222 and 0.254 for two raters, then rose to 0.245 and 0.266 for the first three-rater matrix, but dropped to 0.147 and 0.171 for the second. Gwet’s AC1 was 0.263 for two raters, decreased to 0.253 for the first three-rater matrix, and increased to 0.293 for the second.

Table 4.

Interrater Agreement Coefficients for Two and Three Raters With Equal Observed Proportion Agreement

Subject	Two raters			Three raters			Three raters
Subject	A	b	c	a	b	C	a	b	c
1	0	2	0	3	0	0	3	0	0
2	0	0	2	0	0	3	0	0	3
3	0	0	2	3	0	0	3	0	0
4	2	0	0	0	3	0	0	3	0
5	0	2	0	2	0	1	2	1	0
6	2	0	0	2	0	1	2	1	0
7	1	1	0	1	2	0	2	1	0
8	1	0	1	1	2	0	2	1	0
9	1	0	1	0	1	2	2	0	1
10	1	0	1	0	1	2	2	1	0
11	1	0	1	1	1	1	1	1	1
12	1	0	1	1	1	1	1	1	1
$y_{j}$	0.833	0.417	0.75	1.167	0.917	0.917	1.667	0.833	0.5
	$P_{o} = . 5$			$P_{o} = . 5$			$P_{o} = . 5$
	$\hat{λ} = . 611$			$\hat{λ} = . 497$			$\hat{λ} = . 431$
	$\hat{κ} = . 222$			$\hat{κ} = . 245$			$\hat{κ} = . 147$
	$\hat{α} = . 254$			$\hat{α} = . 266$			$\hat{α} = . 171$
	$AC 1 = . 263$			$AC 1 = . 253$			$AC 1 = . 293$

Although all three matrices yielded the same observed proportion agreement ( $P_{o}$ ), this was obtained from different agreement patterns: one pattern with two raters, and two distinct patterns with three raters (either unanimous agreement or partial agreement with one dissenter). Thus, obtaining equal true agreement requires larger observed agreement for three raters than for two. The numerical example confirms that coefficient Lambda alone consistently captures this relationship, in contrast to the other coefficients.

Monte Carlo Simulation

Scope

A Monte Carlo simulation study was conducted to evaluate coefficient Lambda and compare it with three interrater agreement coefficients, Fleiss’s Kappa, Gwet’s AC1, and Krippendorff’s Alpha (α). The simulation manipulated three types of agreement patterns: balanced agreement, where categories had equal prevalence rates, and three levels of imbalanced agreement, where category prevalence rates differed (moderate prevalent and large prevalent). For each agreement pattern, data were generated under combinations of the following simulation factors: seven levels of true interrater agreement (those corresponds reliability values of 0, 0.1, 0.2, 0.3, 0.4, 0.5, and .9), three numbers of rater ( $R$ = 2, 4, and 6), two rating scale conditions (one with three categories and another with five categories), and three sample sizes ( $N$ = 50, $N$ = 100 and $N$ = 300).

Simulation Procedure

The Underlying Variable Approach (UVA; Muthén, 1984) was employed to simulate ordinal rating data for interrater agreement analysis. This method posits that observed discrete ratings arise from latent continuous variables following a standard multivariate normal distribution. The following steps were followed:

Step 1: Rater assessments were modeled as latent variables drawn from a multivariate standard normal distribution (M = 0, SD = 1) and reliability levels ( $ρ$ ) were fixed at seven levels 0, 0.1, 0.2, 0.3, 0.4, 0.5, and 0.9 to represent different true agreement levels from null to near-perfect level. True scores ( $T$ ) and error scores ( $E$ ) were generated for $R$ raters and a sample size of $N$ . Then, the generated true scores and error scores using reliability levels were used to obtain the observed scores ( $X$ ) with the formula: $X = \sqrt{ρ} T + \sqrt{1 - ρ} E$ .

Step 2: Prevalence patterns were induced by shifting the mean of the latent distributions: M = 0 (symmetric category usage) for balanced patterns, M = 1.0 (skewed toward higher categories) for moderate prevalence, and M = 1.5 (strong skew toward higher categories) for large prevalence.

Step 3: For each agreement pattern, observed scores ( $X$ ) were discretized using thresholds derived from quantiles of the standard normal distribution: thresholds at Φ⁻¹(0.33) and Φ⁻¹(0.67) for three categories ( $J$ = 3): and thresholds at Φ⁻¹(0.20), Φ⁻¹(0.40), Φ⁻¹(0.60), and Φ⁻¹(0.80) for five categories ( $J$ = 5), and all raters used identical thresholds to ensure scale equivalence.

Analyses and Evaluation Criteria

A total of 336 unique data set combinations were generated out of 378 combinations by systematically varying five factors: number of raters (3 levels), true agreement levels (7 levels), prevalent agreement patterns (3 levels), number of categories (2 levels), and sample size (3 levels). It should be noted that no data sets (42 sets) were generated for a sample size of 50 under the third prevalent agreement pattern owing to incompatibility in the data generation process. Each combination was replicated 5,000 times using Monte Carlo simulations. The following analyses and evaluation criteria were then applied to summarize the results:

Estimates and standard errors were computed using the corresponding formulas for coefficient Lambda and for each of the three interrater agreement coefficients across all generated data sets (5,000 × 336). These values were then averaged across the 5,000 replications within each of the 336 conditions. The average of the estimates for each coefficient represents the mean of its sampling distribution and was used to examine how each interrater agreement coefficient responds to the five factors and their interactions. Similarly, the average of the estimated standard errors for each coefficient represents the estimated standard deviation (via the direct formulas) of its sampling distribution, enabling an assessment of how the standard errors are affected by the five factors and their interactions.

The standard deviation of the estimates of coefficient Lambda, as well as those of the other interrater agreement coefficients, was computed across 5,000 replications for each condition using: $SE = \sqrt{\frac{1}{L - 1} \sum_{l} {(a_{l} - \bar{a})}^{2}}$ , where $a_{l}$ is estimate of the interrater agreement coefficient for the l-th replication ( $l = 1, 2, \dots, L$ ), L = total number of replications (5,000) and $\bar{a} = \frac{1}{L} \sum_{l} a_{l}$ is the mean of $a_{l}$ across all combinations. This yielded the empirical standard error for each coefficient. The bias of the average estimated standard error was then obtained as $Bias (SE) = \frac{1}{L} \sum_{l} (S E_{l} - SE)$ , where $S E_{l}$ is estimated standard error for each l-th replication and $SE$ is the empirical standard error. An effective interrater agreement coefficient should provide an unbiased estimate of its standard error, meaning that the average estimated standard error closely approximates the empirical standard error.

The third analysis aimed to evaluate the sensitivity and specificity of coefficient Lambda and to compare these properties with other interrater agreement coefficients across different conditions of the five factors. Specificity was assessed by examining the Type I error rate, which is the probability of rejecting the null hypothesis of interrater chance agreement when it is actually true. A coefficient is regarded as specific if it maintains a Type I error rate below the statistical significance threshold (0.05 in this study). For coefficient Lambda, the null hypothesis of a chance agreement is H0: λ=1/R, whereas for the other coefficients it is H0: $κ = α = AC 1 = 0$ . This evaluation was conducted under the condition ρ = 0, which represents the chance agreement or agreement by random assignment using right-tailed standard normal distribution. Sensitivity refers to the ability of an interrater agreement coefficient to reject the null hypothesis when it is false at the 0.05 significance level (Cousineau & Laurencelle, 2017). To examine this, progressively more realistic false null hypotheses of a chance agreement were tested at six different agreement levels ( $ρ = 0.1, 0.2, 0.3, 0.4, 0.5, 0.9$ ). A coefficient is considered sensitive if it demonstrates power above the significance level and exhibits an increasing trend as agreement levels rise.

Results

Estimates of Interrater Agreement

Figure 1 presents the average estimates of coefficient Lambda alongside three other interrater agreement coefficients—Fleiss’s Kappa, Gwet’s AC1, and Krippendorff’s Alpha—calculated over 5000 replications under all conditions defined by the five factors. Rows correspond to sample sizes (N = 50, 100, 300), and columns to the number of categories (J = 3 on the left, J = 5 on the right). The x-axis orders conditions hierarchically by number of raters (2, 4, 6), prevalence levels (balanced, moderate, strong), and three true agreement levels (0, 0.5, 0.9). Only three agreement levels are shown to save space. The results for the other four levels were found to be similar. It is important to note that the observed proportion agreements were fixed across different numbers of raters, for all conditions for other factors.

Coefficient Lambda and other coefficients showed higher estimates of interrater agreement as true agreement levels increased at all conditions of other factors. Coefficient Lambda and other coefficients were not influenced by sample size as they produced comparable average estimates at the three sample sizes for different conditions for other factors. Also, coefficient Lambda and other coefficients showed similar average estimates when moving from J=3 to J=5 for different conditions for other factors.

Coefficient Lambda exhibited a consistent decrease in average estimates of interrater agreement as the number of raters increased (R = 2, 4, 6) across all conditions. In contrast, the other three interrater agreement coefficients yielded roughly equal average estimates with increasing numbers of raters. This aligns with the earlier discussion on the relationship between interrater agreement and the number of raters when the observed proportion agreement is held constant across varying numbers of raters. Unlike the other coefficients, which fluctuated upward or downward but ultimately averaged out to equality across rater conditions, coefficient Lambda clearly reflects the expected decrease in agreement as the number of raters increases.

With respect to prevalence patterns, the observed proportion agreement increased as category prevalence became stronger, across all conditions of the other factors. When certain categories are more common (i.e., imbalanced prevalence), raters are more likely to agree because they are more frequently assigning the same common category. This naturally boosts the observed proportion agreement, even though the agreement might not reflect true consistency in judgment.

Similarly, the AC1 coefficient showed increasing average estimates of interrater agreement with higher category prevalence. In contrast, coefficient Lambda, Fleiss’s Kappa, and Krippendorff’s Alpha maintained stable average estimates regardless of prevalence levels. Notably, coefficient Lambda consistently yielded larger average estimates of interrater agreement than Fleiss’s Kappa and Krippendorff’s Alpha across all prevalence patterns, particularly when the number of raters was small.

Comparing the values from the interrater agreement measures to the observed proportion agreement reveals distinct patterns. Under balanced agreement conditions, coefficient Lambda produced values that were sometimes higher and sometimes lower than the observed proportion agreement, while under imbalanced conditions, it consistently yielded slightly lower values. In contrast, Fleiss’s Kappa and Krippendorff’s Alpha consistently produced lower values than the observed proportion agreement across both balanced and imbalanced conditions, with the difference being more pronounced under imbalanced conditions. Gwet’s AC1 showed a mixed pattern: it yielded slightly lower values than the observed proportion agreement under balanced conditions, but much higher values under imbalanced conditions, closely approximating the observed proportion agreement.

Under imbalanced agreement conditions, coefficient Lambda, Fleiss’s Kappa, and Krippendorff’s Alpha all yielded lower values compared to their counterparts under balanced agreement conditions across all true levels of interrater agreement. Among these, coefficient Lambda showed the smallest decrease, while Fleiss’s Kappa and Krippendorff’s Alpha exhibited more pronounced reductions. In contrast, under imbalanced agreement conditions, Gwet’s AC1 produced significantly higher values compared to its counterparts under balanced conditions and closely approximated the observed proportion agreement across all levels of true interrater agreement.

Under chance agreement (ρ = 0), coefficient Lambda took a value of 0.5 with $R = 2$ , 0.25 with $R = 4$ , and 0.20 with $R = 6$ . These values were observed under both balanced and prevalent agreement patterns, and remained consistent across different sample sizes and numbers of categories. This confirms that chance agreement corresponds to $ρ = 1 / R$ , regardless of the number of categories.

Standard Error and Its Bias

Figure 2 presents the average standard errors of coefficient Lambda alongside three other interrater agreement coefficients—Fleiss’s Kappa, Gwet’s AC1, and Krippendorff’s Alpha—calculated over 5,000 replications under all conditions defined by the five factors. The organization of Figure 2 follows the same structure as Figure 1. Results indicate that the standard error of coefficient Lambda is consistently smaller than that of any other coefficient, reflecting its relatively narrower scale (0 to 1) across all conditions. The average standard errors of Fleiss’s Kappa and Krippendorff’s Alpha were nearly identical, whereas Gwet’s AC1 generally produced smaller standard errors than the other coefficients, including Lambda, as prevalence patterns increased.

Across all coefficients, results showed that the average standard errors systematically decreased with larger sample sizes, more categories, and greater numbers of raters. Conversely, the average standard errors of Lambda and the other coefficients—except Gwet’s AC1—increased with higher true agreement levels and stronger prevalence patterns.

Figure 3 illustrates the bias of the standard errors for coefficient Lambda alongside three other interrater agreement coefficients—Fleiss’s Kappa, Gwet’s AC1, and Krippendorff’s Alpha—across all conditions defined by the five factors. Bias was calculated as the difference between the average estimated standard errors over 500 replications and the empirical standard error of the interrater agreement coefficient estimates. The layout of Figure 3 mirrors that of Figure 1. Results indicate that the bias of the standard errors was generally small for all interrater agreement coefficients, with coefficient Lambda consistently showing the smallest bias across all conditions. Under stronger prevalence patterns, Gwet’s AC1 also demonstrated relatively small bias. Furthermore, the bias of standard errors systematically decreased as sample size increased. For the remaining factors, no clear pattern in the bias of standard errors was observed for any of the coefficients.

Specificity and Sensitivity

Figure 4 presents the sensitivity and specificity of coefficient Lambda alongside three other interrater agreement coefficients—Fleiss’s Kappa, Gwet’s AC1, and Krippendorff’s Alpha—based on 5,000 replications across all conditions defined by the five factors. The layout of Figure 4 follows the same structure as Figure 1. Coefficient Lambda demonstrated acceptable specificity rates (below 0.05) across all conditions involving variations in the number of categories, raters, sample sizes, and prevalence patterns. In terms of sensitivity, Lambda was responsive to changes in true agreement levels, showing gradually increasing power as true agreement increased across all conditions. At each true agreement level, the sensitivity of Lambda decreased when moving from balanced agreement patterns to stronger prevalence patterns, increased when the number of categories rose from three (J = 3) to five (J = 5), increased as the number of raters rose from two (R = 2) to four (R = 4) and six (R = 6), and increased as sample size enlarged from 50 (N=50), to 100 (N=100) and 300 (N=300). Similar results were found with Cohen Kappa for two raters by Cousineau and Laurencelle (2017).

Fleiss’s Kappa and Krippendorff’s Alpha exhibited trends in sensitivity and specificity that were largely consistent with those of coefficient Lambda across all conditions involving variations in the number of categories, raters, sample sizes, and prevalence patterns. Similar results about Cohen Kappa were also reported by Cousineau and Laurencelle (2017). In contrast, Gwet’s AC1 showed trends similar to Lambda only under the balanced agreement pattern. However, as prevalence patterns became stronger, AC1 lost specificity, yielding inflated Type I error rates above 0.05. This inflation distorted sensitivity estimates, which reached values of 1.0 even at low levels of true agreement.

Conclusion and Discussion

This paper introduces coefficient Lambda (λ) as a novel measure of interrater agreement applicable to multiple raters or methods. It is defined as the ratio of the observed proportion agreement to the overall proportion agreement, with both quantities adjusted for the prevalence-agreement effect. Unlike traditional chance-corrected coefficients, Lambda is grounded in the assumption that a portion of the observed agreement is systematically influenced by variations in category prevalence—not random chance. These prevalence variations may result from differences in category ambiguity, clarity, attractiveness, social desirability, the number of available categories, or other factors linked to the nature of the measured construct and the context in which the ratings occur. By accounting for this prevalence-agreement effect, coefficient Lambda offers a more realistic and interpretable estimate of the true agreement among raters. Coefficient Lambda estimates the extent of interrater true agreement among raters with relative to disagreement. Also, it gives how much interrater agreement among raters represents from the total agreement after correcting both for category prevalence.

Coefficient Lambda addresses the scaling and interpretation of Fleiss’s Kappa and other Kappa-like coefficients including Krippendorff’s Alpha and Gwet’s AC1, which complicates interpretation across different numbers of raters. A well-known concern with Fleiss’s Kappa is that its scale is not fixed between –1 and 1, but rather varies depending on the number of raters $R$ . Specifically, the lower bound of Fleiss’s Kappa is $- \frac{1}{R - 1}$ , while the upper bound remains 1. Negative Kappa values are generally interpreted as indicating agreement below the level expected by chance, without distinguishing between different degrees of such disagreement. For example: When $R = 2$ , the scale is from –1 to 1, when $R = 3$ , the scale narrows to –.5 to 1, and when $R = 4$ , it compresses further to –.33 to 1. As $R$ increases, the lower bound approaches 0, making the scale increasingly asymmetric and compressed.

This non-constant scaling creates interpretative challenges: the same Kappa value can imply different levels of agreement depending on the number of raters. Although some researchers suggest disregarding negative Kappa values, these negative estimates are not rare and should not be ignored. Negative values are generally interpreted as agreement worse than chance, which may signal issues in rater training, scale design, or the clarity of categories being rated. Therefore, researchers must exercise caution when interpreting negative Kappa values and consider the study context, including the construct being measured, the characteristics of the rating scale, and the raters’ expertise. According to guidelines from Altman (1991) and McHugh (2012), Kappa values below .60 indicate inadequate agreement, and limited confidence should be placed in study findings based on such results.

Coefficient Fleiss’s Kappa has been widely criticized in the literature for consistently yielding values that are lower than the observed proportion agreement ( $P_{o}$ ). This discrepancy stems from its reliance on an expected agreement due to chance, which often leads to underestimation of true agreement, especially when category prevalence is skewed. In contrast, coefficient Lambda offers greater flexibility and realism in its behavior. Depending on the prevalence-agreement effect ( $P_{v}$ ), Lambda can be greater than, less than, or equal to the observed proportion agreement if. is negative, $λ > P_{o}$ ; if $P_{v}$ is positive, $λ < P_{o}$ ; and if $P_{v} = 0$ , $λ = P_{o}$ . This sensitivity to category prevalence allows Lambda to reflect how differences in category usage influence the level of agreement among multiple raters or methods. As a result, coefficient Lambda provides a more nuanced and context-sensitive measure of interrater agreement, especially in situations where prevalence effects are non-negligible.

Lambda, Fleiss’s Kappa, and Krippendorff’s Alpha all produced lower values in imbalanced conditions compared to balanced ones. Fleiss’s Kappa and Krippendorff’s Alpha correct for chance agreement, and when category prevalence is skewed, the chance agreement increases. As a result, these coefficients discount more of the agreement as being due to chance, leading to lower agreement scores. Among these, Lambda showed the smallest decrease, suggesting it may be less sensitive to category imbalance than Kappa or Alpha.

It is important to note that a coefficient Lambda value lower than the observed proportion agreement ( $P_{o}$ ) is not inherently undesirable, especially in cases where there are substantial differences in category prevalence. Such variations—often influenced by factors like social desirability, category ambiguity, or the inherent appeal of certain response options—can lead to inflated $P_{o}$ values that overestimate the true level of interrater agreement. In these situations, Lambda provides a corrected and more accurate reflection of agreement by accounting for the prevalence-agreement effect ( $P_{v}$ ). Therefore, a lower Lambda relative to $P_{o}$ should be interpreted not as a shortcoming of the coefficient, but as an indication of potential bias in the observed agreement due to uneven category usage across raters.

As the number of raters increased from two to four to six, it is noted that coefficient Lambda showed decreasing estimates of interrater agreement; whereas Fleiss’s Kappa, Krippendorff’s Alpha, and Gwet’s AC1 did not exhibit this pattern. This trend held true across different numbers of categories and varying true levels of agreement. From a conceptual standpoint, interrater agreement tends to decrease as the number of raters increases. This is because achieving consensus among a larger group of individuals is inherently more challenging than among a smaller group. With fewer raters, the likelihood of aligning on the same category or judgment is higher due to less variability in perspectives, interpretations, and decision-making criteria. However, as more raters are introduced, the diversity of viewpoints and potential for disagreement naturally increase, making it more difficult to reach a high level of uniformity in ratings. This phenomenon is particularly evident in complex or subjective tasks, where different raters may interpret criteria or rating scales differently. As a result, interrater agreement measures often reflect lower values when more raters are involved—not necessarily because the quality of the raters has declined, but because the probability of unanimous or near-unanimous agreement decreases with group size.

The proposed coefficient Lambda differs from traditional agreement coefficients in several important ways. First, its values range from 0 to 1, and it does not produce negative values, avoiding the interpretative issues associated with negative Kappa-type coefficients. Second, Lambda maintains a fixed and consistent scale, regardless of the number of raters or categories involved in the agreement study. Third, it provides a uniform interpretation of agreement levels across different study designs and contexts, making it easier to compare findings across diverse applications. These properties make coefficient Lambda a more practical and intuitive tool for measuring interrater agreement, particularly for researchers and practitioners across various scientific disciplines who seek a stable, interpretable, and meaningful metric of agreement.

The interpretation of coefficient Lambda is based on comparing the true proportion of agreement—defined as $p_{o} - p_{v}$ (the observed proportion of agreement corrected for category prevalence)—to the proportion of disagreement, $1 - p_{o}$ . Lambda reflects the proportion of true agreement relative to disagreement between raters. Values greater than 0.5 indicate that true agreement exceeds disagreement, values less than 0.5 indicate that disagreement exceeds true agreement, and a value of 0.5 represents the point at which true agreement and disagreement are equal.

In addition, the value of chance agreement for Lambda is given by $λ = 1 / R$ , which depends on the number of raters. Only when does $λ = 0.5$ correspond to the chance agreement level as well as true agreement and disagreement are equal. This means that, for more than two raters, chance agreement is not synonymous with equality between true agreement and disagreement. Importantly, a value of $λ = 0.5$ carries a unique interpretation that is independent of the number of raters: it always represents the point at which true agreement and disagreement are balanced.

In our analysis, we found that the new agreement coefficient is more sensitive to variations in category prevalence than Fleiss’s Kappa and other Kappa-like statistics. This behavior reflects the design of the new coefficient, which explicitly adjusts for the prevalence-agreement interaction, accounting for how certain categories may elicit higher or lower agreement due to clarity, frequency, or social salience. In contrast, Fleiss’s Kappa and other Kappa-like statistics, which adjust for expected agreement purely through marginal probabilities, does not fully capture this interaction and remains less responsive to structural imbalance in the data.

Moreover, coefficient Lambda demonstrated acceptable levels of sensitivity and specificity to chance agreement, comparable to those of Fleiss’s Kappa and Krippendorff’s Alpha, across varying conditions of true agreement levels, number of raters, number of categories, sample size, and prevalence patterns. Future research is recommended to further investigate the sensitivity and specificity of Lambda at different population values, as well as to develop hypothesis-testing procedures for comparing Lambda values obtained from independent samples (e.g., males vs. females) and paired samples (e.g., across time points).

Future research on coefficient Lambda should further investigate its statistical properties across diverse rating conditions, including varying numbers of raters, categories, and prevalence distributions. In particular, studies are needed to examine its robustness in unbalanced settings and its behavior relative to other agreement coefficients when disagreement is systematically structured. Simulation and empirical work could also clarify how Lambda performs in large-scale assessments. Such investigations would contribute to establishing Lambda as a more comprehensive and reliable tool for evaluating interrater agreement.

Footnotes

Appendix

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Material preparation, data collection, and analysis were performed by me.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

Ethical Approval and Informed Consent Statements

The manuscript did not require ethics approval since it is a methodology paper that does not have direct contact with human or animal subjects. The manuscript did not require consent to participate since it is a methodology paper that does not involve direct contact with human subjects.

ORCID iD

Rashid Saif Almehrizi

Data Availability Statement

There is no data to share since the study does not require data collection.

References

Almehrizi

R. S.

(2025). Coefficient of agreement between two raters corrected for category prevalence: Alternative to kappa. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000732

Altman

D. G.

(1991). Practical statistics for medical research. Chapman & Hall/CRC.

Brennan

R. L.

Prediger

D. J.

(1981). Coefficient Kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3), 687–699.

Byrt

Bishop

Carlin

J. B.

(1993). Bias, prevalence and Kappa. Journal of Clinical Epidemiology, 46(5), 423–429.

Cohen

(1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46. https://doi.org/1.1177/001316446002000104

Cousineau

Laurencelle

(2017). An unbiased estimate of global interrater agreement. Educational and Psychological Measurement, 77, 721–742.

Dawson

(2017). Assessment rubrics: Towards clearer and more replicable design, research and practice. Assessment & Evaluation in Higher Education, 42(3), 347–336. https://doi.org/1.1080/02602938.2015.1111294

Delgado

Tibau

(2019). Why Cohen’s Kappa should be avoided as performance measure in classification. PLOS ONE, 14(9), Article e0222916. https://doi.org/1.1371/journal.pone.0222916

Feinstein

A. R.

Cicchetti

D. V.

(1990a). High agreement but low Kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. https://doi.org/1.1016/0895-4356(90)90158-L

10.

Feinstein

A. R.

Cicchetti

D. V.

(1990b). High agreement but low Kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6), 551–558.

11.

Fleiss

J. L.

(1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378. https://doi.org/1.1037/h0031619

12.

Fleiss

J. L.

Cohen

Everitt

B. S.

(1969). Large sample standard errors of Kappa and weighted Kappa. Psychological Bulletin, 72(5), 323–327. https://doi.org/1.1037/h0028106

13.

Fleiss

J. L.

Nee

J. C.

Landis

J. R.

(1979). Large sample variance of Kappa in the case of different sets of raters. Psychological Bulletin, 86(5), 974–977. https://doi.org/1.1037/0033-2909.86.5.974

14.

Gwet

(2001). Statistical methods for inter-rater reliability measurement: Advance analytics. LLC.

15.

Kraemer

H. C.

Periyakoil

V. S.

Noda

(2002). Kappa coefficients in medical research. Statistics in Medicine, 21(14), 2109–2129. https://doi.org/1.1002/sim.1180

16.

Krippendorff

(2004). Measuring the reliability of qualitative text analysis data. Quality & Quantity, 38(6), 787–780.

17.

Landis

Koch

(1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.

18.

Mandrekar

(2011). Measures of interrater agreement. Journal of Thoracic Oncology, 6(10), 6–7. https://doi.org/1.1097/JTO.0b013e318200f983

19.

McHugh

(2012). Interrater reliability: The Kappa Statistic. Biochemia Medica, 22(3), 276–282.

20.

Moons

Vandervieren

(2022). Handwritten math exams with multiple assessors: Researching the added value of semi-automated assessment with atomic feedback. Twelfth Congress of the European Society for Research in Mathematics Education (CERME12), TWG21(14). https://hal.science/hal-03753446.

21.

Muthén

(1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49(1), 115–132.

22.

Radhakrishna

R. C.

(1948). Large sample tests of statistical hypotheses concerning several parameters with applications to problems of estimation. Mathematical Proceedings of the Cambridge Philosophical Society, 44(1), 50–57. https://doi.org/10.1017/S0305004100023987

23.

Sandifer

M. G.

Fleiss

J. L.

Green

L. M.

(1968). Sample selection by diagnosis in clinical drug evaluations. Psychopharmacologia, 13, 118–128. https://doi.org/10.1007/BF00404809

24.

Vach

(2004). The dependence of Cohen’s Kappa on the prevalence does not matter. Journal of Clinical Epidemiology, 58, 655–661. https://doi.org/1.1016/j.jclinepi.2004.02.021

25.

Warrens

M. J.

(2010). A formal proof of a paradox associated with Cohen’s Kappa. Journal of Classification, 27(3), 322–332. https://doi.org/1.1007/s00357-010-9060-x

26.

Zec

Soriani

Comoretto

Baldi

(2017). High agreement and high prevalence: The paradox of Cohen’s Kappa. The Open Nursing Journal, 11, 211–218. https://doi.org/1.2174/1874434601711010211

Coefficient Lambda for Interrater Agreement Among Multiple Raters: Correction for Category Prevalence

Abstract

Keywords

Introduction

Coefficient Fleiss’s Kappa

A New Coefficient of Agreement

Interpretation of Coefficient Lambda

Standard Error of λ ^

Numerical Examples

Number of Raters and True Agreement

Monte Carlo Simulation

Scope

Simulation Procedure

Analyses and Evaluation Criteria

Results

Estimates of Interrater Agreement

Standard Error and Its Bias

Specificity and Sensitivity

Conclusion and Discussion

Footnotes

Appendix

Declaration of Conflicting Interests

Funding

Ethical Approval and Informed Consent Statements

ORCID iD

Data Availability Statement

References

Standard Error of $\hat{λ}$