Multiple comparison procedures for the means/medians of independent samples

Methods of controlling the Type I error and dependencies between parameters when making multiple comparisons between many means/medians of independent samples.

Procedure Purpose
Student's t (Fisher's LSD) Compare the means of each pair of groups using the Student's t method.

When making all pairwise comparisons this procedure is also known as unprotected Fisher's LSD, or when only performed following significant ANOVA F -test known as protected Fisher's LSD.

Control the type I error rate is for each contrast.

Wilcoxon-Mann-Whitney Compare the median/means each pair of groups using the Wilcoxon nonparametric method.

Control the type I error rate for each contrast.

Tukey-Kramer Compare the means of all pairs of groups using the Tukey-Kramer method.

Controls the error rate simultaneously for all k(k+1)/2 contrasts.

Steel-Dwass-Critchlow-Fligner Compare the median/means of all pairs of groups using the Steel-Dwass-Critchlow-Fligner pairwise ranking nonparametric method.

Controls the error rate simultaneously for all k(k+1)/2 contrasts.

Hsu Compare the means of all groups against the best of the other groups using the Hsu method.

Controls the error rate simultaneously for all k contrasts.

Dunnett Compare the means of all groups against a control using the Dunnett method.

Controls the error rate simultaneously for all k-1 contrasts.

Steel Compare the medians/means of all groups against a control using the Steel pairwise ranking nonparametric method.

Controls the error rate simultaneously for all k-1 comparisons.

Scheffé Compare the means of all groups against all other groups using the Scheffé F method.

Controls the error rate simultaneously for all possible contrasts.

Student-Newman-Keuls (SNK) Not implemented.

As discussed in Hsu (1996) it is not a confident inequalities method and cannot be recommended.

Duncan Not implemented.

As discussed in Hsu (1996) it is not a confident inequalities method and cannot be recommended.

Dunn Not implemented in Analyse-it version 3 onwards. Implemented in version 1.

A joint-ranking nonparametric method but as discussed in Hsu (1996) it is not a confident inequalities method and cannot be recommended. Use the Dwass-Steel-Critchlow-Fligner or Steel test instead.

Conover-Iman Not implemented in Analyse-it version 3 onwards. Implemented in version 2.

As discussed in Hsu (1996) applying parametric methods to the rank-transformed data may not produce a confident inequalities method even if the parametric equivalent it mimics is. Use the Dwass-Steel-Critchlow-Fligner or Steel test instead.

Bonferroni Not a multiple comparisons method. It is an inequality useful in producing easy to compute multiple comparisons. In most scenarios, there are more powerful procedures such as Tukey, Dunnett, Hsu.

A useful application of Bonferroni inequality is when there are a small number of pre-planned comparisons. In this case, use the Student's t (LSD) method with the significance level (alpha) set to the Bonferroni inequality (alpha divided by the number of comparisons). In this scenario, it is usually less conservative than using Scheffé all contrast comparisons.