Standard Edition

Hypothesis testing for comparing groups and pairs t-tests, Wilcoxon, Mann-Whitney, Welch’s ANOVA, Kruskal-Wallis, Friedman, nine multiple comparison procedures, and Cohen’s d effect sizes — parametric and non-parametric.

The right test for the comparison you need to make

Two groups or ten. Independent or paired. Normal or not. Equal variances or not. Each combination calls for a different test, and choosing the wrong one invalidates the conclusion. Use a t-test when you should use Welch’s and you understate the uncertainty. Use a parametric test on skewed data and the p-value is unreliable. The assumption checks aren’t optional — they determine which test gives a valid answer.

And knowing that groups differ is only the start. A significant p-value doesn’t tell you which groups differ, by how much, or whether the difference is large enough to matter. That takes multiple comparison procedures, effect sizes with confidence intervals, and plots that show the data alongside the statistics — all in the same analysis, not pieced together from separate tools.

I’ve tried different statistical packages (e.g. SPSS, Minitab) and none are as easy or intuitive and the fact it is an add in to Excel is also very handy. The statistical guides are also really useful.

What’s included

Compare independent groups

Two groups: Student’s t-test when variances are equal, Welch’s t-test when they’re not, Wilcoxon-Mann-Whitney when normality is in doubt. Three or more groups: one-way ANOVA, Welch’s ANOVA, or Kruskal-Wallis. Side-by-side dot plots with confidence diamonds show the group distributions before you commit to a test.

Compare paired and repeated measures

Before-and-after, matched pairs, repeated observations on the same subjects. Paired t-test, Wilcoxon signed ranks, and Sign test for two time points. Within-subjects ANOVA and Friedman for three or more. The difference plot with histogram shows the distribution of individual changes — not just the average.

Find out which groups differ, not just that something does

A significant ANOVA tells you the groups aren’t all equal — but not which ones differ. Nine multiple comparison procedures answer the specific follow-up question: all pairs with Tukey-Kramer, against a control with Dunnett, with the best using Hsu. Non-parametric alternatives with Steel and Dwass-Steel-Critchlow-Fligner. The Mean-Mean scatter plot shows every pairwise difference at a glance.

Quantify the size of the difference

A p-value tells you whether a difference exists. Cohen’s d and Hedges’ g tell you how large it is, on a standardized scale with non-central t confidence intervals. Mean difference with CI for the raw scale. Hodges-Lehmann location shift for a robust non-parametric estimate. Report both the significance and the magnitude.

Check the assumptions before committing

The choice between Student’s t and Welch’s t depends on whether variances are equal. F-test for two groups, Bartlett, Levene, and Brown-Forsythe for three or more. When the assumption fails, the alternative is right there in the same analysis — switch to Welch’s t-test, Welch’s ANOVA, or a non-parametric test without starting over.
Compare plain Compare plain

Example analyses

See hypothesis test results in detail — t-tests, ANOVA, multiple comparisons, and effect sizes.

Compare Groups 1 Compare groups — t-test
Calcium supplementation and blood pressure. 2 groups (11 calcium, 10 placebo). Side-by-side box plots, Fisher F-test, independent-samples t-test against hypothesised difference of 3.
Compare Groups 2 Compare groups — one-way ANOVA
Y by brand, 7 groups. Dot plots with confidence diamonds, Levene’s test, one-way ANOVA, Tukey-Kramer with 21 contrasts, Mean-Mean scatter plot.
Compare Pairs Compare pairs
Body fat before and after exercise. 28 paired observations. Box plots with paired lines, difference plot with Hodges-Lehmann shift and 95% CI, Wilcoxon signed-ranks.

Part of the Standard edition

Hypothesis testing is one part of a complete statistical analysis toolkit. The Standard edition also includes ANOVA and ANCOVA, simple and multiple regression, logistic regression, PCA and factor analysis, descriptive statistics, correlation, and categorical data analysis. See everything in the Standard edition →

Validated, reliable, trusted for over 30 years

Validated calculations Every statistic tested against NIST Standard Reference Datasets, published datasets and thousands of internal test-cases. No reliance on Excel'sbuilt-in functions. See how we develop and validate Analyse-it →
Data stays on your PC No cloud processing, no uploads, no third-party access. Your data never leaves your computer — essential when working with sensitive, confidential, or patient-identifiable data.
Standard Excel workbooks Analyses are ordinary Excel workbooks that you can share with colleagues, archive for audit, and open on any machine with Excel — no Analyse-it licence required.
No formulas to break Results contain no formulas, so they can't be accidentally edited or corrupted. The results you reported will be exactly what you find when you reopen the workbook.

Technical details

Compare groups (independent)

  • Descriptive statistics by group
  • Side-by-side dot plots, mean plots, box plots
  • Z test for difference in means (known SDs)
  • Student’s independent samples t-test
  • Welch’s t-test for unequal variances
  • Wilcoxon-Mann-Whitney test
  • 1-way between-subjects ANOVA
  • Welch’s ANOVA for unequal variances
  • Kruskal-Wallis test

Compare pairs (dependent)

  • Descriptive statistics for each group
  • Side-by-side dot plots, mean plots, box plots
  • Difference plot with identity line and histogram
  • Z test for difference in means (known SDs)
  • Student’s paired t-test
  • Wilcoxon signed ranks test
  • Sign test for median
  • 1-way within-subjects ANOVA
  • Friedman test

Multiple comparisons

  • All pairs: Tukey-Kramer, Dwass-Steel-Critchlow-Fligner
  • Against control: Dunnett, Steel
  • With best: Hsu
  • All contrasts: Scheffé
  • Individual: Student’s t, Wilcoxon-Mann-Whitney
  • Mean-Mean scatter plot

Variance tests

  • F-test for variance ratio
  • Bartlett test for homogeneity
  • Levene test for homogeneity
  • Brown-Forsythe test for homogeneity

Effect size estimators

  • Mean difference with t-based, Welch-Satterthwaite, or Z-based CI
  • Cohen’s d and Hedges’ g with non-central t CI
  • Hodges-Lehmann location shift with Moses CI (independent)
  • Hodges-Lehmann location shift with Tukey CI (paired)
  • Median difference with Thompson-Savur CI