Run six groups and compare every pair with a t-test at the 5% level, and you have made fifteen comparisons. If all six means were truly identical, the chance of at least one coming out “significant” is about 54%, more likely than not. That inflation is the problem, and a multiple comparison procedure is the fix. Such a procedure controls the error rate across the family of comparisons rather than across each one separately.
The procedures are not weaker or stronger versions of one another. Each is built for a specific structure of comparison. Choosing the one that matches your question gives you tighter intervals than a general-purpose method would. Narrowing what you ask is the cheapest power you will ever buy.
The comparison-wise error rate is the chance of a false positive on any single comparison, the familiar 5%. The family-wise error rate is the chance of at least one false positive across the whole set. Every procedure below controls the family-wise rate at your chosen level. What differs is how efficiently it does that, given the structure of the family.
The confidence intervals a procedure produces are simultaneous. You can be 95% confident that all of them cover their true differences at once. Simultaneous intervals are necessarily wider than individual ones, and how much wider depends on how many comparisons the procedure is protecting against.
When every group is of equal interest and you genuinely need every pairwise difference, comparing five suppliers with no single one designated as the reference, Tukey-Kramer is the procedure. Tukey-Kramer controls the family-wise rate exactly for the set of all pairwise comparisons, handles unequal group sizes, and is the standard follow-up to a one-way ANOVA.
Tukey-Kramer is also the most commonly over-used procedure. Suppose you only ever intended to compare four treatments against a control. Asking for all ten pairs and then reading four of them means paying for six comparisons you did not want. You pay in the form of wider intervals on the four you did.
For the rank-based equivalent, all pairs without the normality assumption, the Dwass-Steel-Critchlow-Fligner procedure does the same job following a Kruskal-Wallis test.
Dose-response studies, new formulations against the current one, treatments against placebo: one group is the reference and the others are compared only to it. With k treatments that is k comparisons rather than k(k − 1)/2, and Dunnett’s procedure takes advantage of that directly. You get noticeably tighter intervals than Tukey-Kramer on the comparisons you care about, because it is not defending the ones you do not.
Dunnett also comes in a one-sided form, which is the right choice when only one direction is meaningful. A new treatment that is worse than control usually needs no interval, just a conclusion. Steel’s procedure is the rank-based counterpart for comparisons against a control.
Sometimes the question is not which groups differ but which ones could plausibly be the best. Screening candidate formulations, selecting a supplier. Hsu’s multiple comparisons with the best compares each group to the largest of the others and returns a set of groups that cannot be ruled out as best.
The procedure answers a selection question directly. Otherwise you would run all pairs and try to infer which groups could be best from which differences failed to reach significance. If your actual decision is “which of these do we take forward”, this is the procedure that matches it.
Scheffé’s method protects every possible linear contrast among the means, not just pairwise differences but combinations such as “the average of groups 1 and 2 against the average of 3, 4 and 5”. That generality is expensive. For pairwise comparisons alone, Scheffé gives wider intervals than Tukey-Kramer, so it should not be used for them.
Scheffé’s real virtue is that the protection holds for contrasts you decide on after looking at the data. If a pattern in the means suggests a comparison you had not planned, Scheffé is the procedure that lets you test it with valid family-wise control. Every other method on this page assumes the family was fixed in advance.
Where a small number of specific comparisons were specified before the data existed, written into the protocol rather than chosen afterwards, individual Student’s t or Wilcoxon-Mann-Whitney comparisons are justified without a family-wise adjustment. No family of opportunistic tests exists to protect against. The discipline this requires is real, though. The difference between “we planned this comparison” and “we noticed this comparison” is the difference between a valid result and a rediscovery of the 54% problem above.
Whichever procedure you use, read the intervals, not the significance flags. A difference of 0.4 units with a simultaneous interval of 0.1 to 0.7 is a different finding from the same 0.4 with an interval of −0.2 to 1.0. Both are different again from a significant difference of 0.02 that is far too small to act on. Significance and importance are separate judgements, as confidence intervals and p-values sets out.
A mean-mean scatter plot shows every pairwise difference at once, as circles whose overlap encodes significance. That makes the whole family readable at a glance, in a way a table of fifteen rows never does.
Download the one-way ANOVA with multiple comparisons example (.xlsx) — seven brands with Levene’s test, one-way ANOVA, Tukey-Kramer across all 21 contrasts, and a mean-mean scatter plot. The ANCOVA example shows Dunnett used correctly, comparing each dose against placebo.
The example workbook is downloading.
It opens in Excel on its own — the data and the finished results are both in it. Analyse-it is what lets you change the analysis and re-run it, try the same study on your own data, or work through it to see how the software handles it.
Every feature from all five editions for 15 days, with no sign-up and no licence key.
Using Tukey-Kramer when the question was against a control. You pay in interval width for comparisons you never wanted. Dunnett is both correct and tighter.
Deciding which comparisons to make after seeing the means. Unless you use Scheffé, the procedures assume the family was fixed in advance. Choosing afterwards simply reintroduces the inflation.
Using Scheffé for pairwise comparisons. It is protecting against contrasts you are not testing, so its intervals are needlessly wide.
Skipping the procedure because the ANOVA was significant. The F-test licenses the follow-up. It does not perform it, and unadjusted t-tests afterwards are the original problem.
Reporting only which pairs were significant. The differences and their simultaneous intervals are the result. The flags are just a reading aid.
Analyse-it offers each procedure directly, in Compare Groups and Fit Model:
Every feature from all five editions for 15 days, with no sign-up and no licence key. ANOVA and multiple comparisons are part of the Standard edition, so it is in every Analyse-it edition, from US$ 155 a year. Validated against NIST Standard Reference Datasets. See one-way, two-way and multi-factor ANOVA for the model behind them.