When you compare two raters, two methods, or a test against a reference on a categorical result, the obvious summary is the proportion of cases they agree on. Some of that agreement is free. If a condition is common, two raters who both lean towards “positive” will coincide often without either being any good. Raw agreement counts that luck as skill. The rarer or commoner the condition, the more it misleads.
Kappa rescales observed agreement against chance agreement. Observed agreement, written po, is simply the proportion of cases the two raters classified the same way — the diagonal of the table divided by the total, sometimes called the percent agreement or overall percent agreement. The agreement expected by chance, pe, follows from the raters’ marginal rates: how often each rater calls positive overall, read from the row and column totals of the table. Then κ = (po − pe) / (1 − pe).
The numerator is how much you beat chance. The denominator is how much you could have beaten it. κ = 1 is perfect agreement. κ = 0 is agreement no better than chance. Negative values mean the raters disagree more than chance would predict.
Take 100 samples where two raters agree on 45 positives and 45 negatives, and split on 10. Observed agreement is 0.90. Each rater called 50 positive and 50 negative, so chance agreement is (0.5 × 0.5) + (0.5 × 0.5) = 0.50. Kappa is (0.90 − 0.50) / (1 − 0.50) = 0.80. A kappa of 0.5 is strong: half the possible above-chance agreement has been earned.
Plain kappa treats every disagreement as equally wrong, which is right for nominal categories and wasteful for ordered ones. If a grade can be 1, 2, 3 or 4, a 3-versus-4 split is a near miss and a 1-versus-4 split is a serious conflict. Those two errors should not count the same.
Weighted kappa assigns partial credit by distance from the diagonal. Linear weights penalise in proportion to how many categories apart the two calls are. Quadratic weights penalise the square of that distance, forgiving near misses more sharply and punishing distant ones harder. State which weighting you used, because the two can give materially different numbers on the same table.
Which to choose depends on how the scale behaves. Linear weights suit a scale where each step is about as serious as the next. Quadratic weights suit one where large disagreements matter disproportionately, and they are the convention in much of the clinical and educational literature, so quadratic is often the expected default where a field has one. Quadratic weighting also has a useful property: on an ordered scale it equals the intraclass correlation coefficient computed on the category numbers.
One question comes up often enough to answer directly. With only two categories, weighted kappa and unweighted kappa are the same number. Weighting adjusts the credit given to disagreements by how far apart they are, and in a binary table every disagreement is exactly one category apart, so there is nothing left for the weights to distinguish. Weighting only changes the result once there are three or more ordered categories.
Kappa depends on the marginal totals, and that dependence has a well-known consequence. Keep observed agreement at 0.90 but skew the table: 85 samples both-positive, 5 both-negative, 10 split. Now each rater calls 90 positive and 10 negative. Chance agreement climbs to (0.9 × 0.9) + (0.1 × 0.1) = 0.82. Kappa collapses to (0.90 − 0.82) / (1 − 0.82) = 0.44.
Same 90% agreement, half the kappa. Nothing went wrong with the raters. The base rate did it. When a condition is very common or very rare, chance agreement is already high, so there is little room left for kappa to reward.
Always read kappa next to the raw agreement and the marginal rates, never on its own. The output below reports all of them together for two qualitative methods.
Report kappa with a confidence interval, because a single figure hides how much the estimate would move on a fresh sample. For an ordered scale, report weighted kappa and name the weights. Give the raw agreement and the marginal totals alongside, so a reader can see whether a low kappa reflects poor agreement or merely a skewed base rate.
The familiar interpretive bands come from Landis and Koch (1977):
| κ | Conventional label |
|---|---|
| < 0.00 | Poor — worse than chance |
| 0.00 – 0.20 | Slight |
| 0.21 – 0.40 | Fair |
| 0.41 – 0.60 | Moderate |
| 0.61 – 0.80 | Substantial |
| 0.81 – 1.00 | Almost perfect |
Quote those labels lightly. The bands are a convention rather than a law, they were never derived from any clinical requirement, and they take no account of the prevalence effect above — which is how the same 90% agreement can be labelled “almost perfect” on one table and “moderate” on another. What counts as adequate agreement depends on what the classification will be used for.
Download the CLSI EP12-A2 example workbook (.xlsx) — two qualitative methods compared, with observed and chance agreement, kappa, and its Wald confidence interval, ready to open in the Analyse-it trial.
The example workbook is downloading.
It opens in Excel on its own — the data and the finished results are both in it. Analyse-it is what lets you change the analysis and re-run it, try the same study on your own data, or work through it to see how the software handles it.
Every feature from all five editions for 15 days, with no sign-up and no licence key.
Reporting kappa without prevalence. A kappa of 0.44 can mean poor agreement or excellent agreement on a lopsided sample. Publish the agreement and margins next to it.
Using unweighted kappa on an ordered scale. Grades and stages carry order. Unweighted kappa throws it away and understates real agreement. Weight it.
Using it alone to characterise a diagnostic test. FDA discourages relying on kappa, or on overall percent agreement, on its own to describe diagnostic performance. Both can stay high while the test fails on one category. See performance studies for an IVD 510(k).
Leaning on the interpretive bands. “Substantial agreement” sounds definitive but rests on arbitrary cut-offs. Treat the label as shorthand, not evidence.
Ignoring the interval. Kappa from a small sample is unstable. A confidence interval that spans two of the interpretive bands is telling you not to over-read the point estimate.
Analyse-it reports kappa as part of qualitative test evaluation (EP12-A2), inside Excel:
Every feature from all five editions for 15 days, with no sign-up and no licence key. Qualitative agreement is in the Method Validation and Ultimate editions, from US$ 475 a year. Validated against NIST and CLSI reference datasets. See PPA and NPA vs sensitivity and specificity for agreement without a gold standard, and the contingency reference guide for the technical detail.