Every t-test compares a difference against the uncertainty in that difference. What separates the variants is what they assume about where the numbers came from. Those assumptions are not interchangeable. One choice is fixed entirely by your study design. Get it wrong and the analysis is invalid. The other is a judgement about variances. The conventional answer to it is the wrong one.
One confusion is worth clearing up before anything else, because it turns up constantly: Welch’s is not a third option alongside paired and unpaired. Welch’s is a variant of the unpaired test. The design decides paired against unpaired first, and only then, if the design is unpaired, does the variance question decide Student’s against Welch’s. There is no such thing as a paired Welch’s t-test, because the paired test works on one set of differences and so has only one variance to estimate.
| Test | Design | Assumes equal variances |
|---|---|---|
| Paired t-test | Same subject measured twice | Not applicable — one set of differences |
| Student’s t-test (unpaired) | Different subjects in each group | Yes — pools them into one estimate |
| Welch’s t-test (unpaired) | Different subjects in each group | No — uses each group’s own variance |
Neither Student’s nor Welch’s is a non-parametric test. Both assume approximate normality; what Welch’s drops is the equal-variance assumption, not the normality one.
Independent samples, also called unpaired, means different subjects in each group. Think of a treated group and a control group, or samples from two suppliers. Paired means the same subject, specimen or unit measured twice: before and after, two methods on one sample, matched case-control pairs.
A paired test does not compare two groups at all. The test computes the difference within each pair and asks whether those differences average to something other than zero. That single change is why pairing is so effective: differences between subjects, often the largest source of variation in the whole study, cancel out of the comparison completely.
The gain can be dramatic. If subjects vary widely but each responds consistently, a paired design can detect with twenty pairs what an independent design would need hundreds of subjects to see. Analysing paired data as independent throws all of that away and will often miss a real effect. Analysing independent data as paired is not just wasteful but wrong, since there are no genuine pairs to difference.
For independent samples, Student’s t-test assumes both groups have the same variance and pools them into a single estimate of spread. Welch’s t-test does not: it uses each group’s own variance and adjusts the degrees of freedom accordingly. Use Welch’s by default. Where the variances really are equal, Welch’s loses almost nothing. Where they are not, and especially when the group sizes are unequal too, Student’s test can be badly wrong, with a real error rate well away from the 5% it claims.
The preliminary variance test is itself a problem, not just an inconvenience. Choosing which test to run based on the outcome of another test on the same data makes the properties of the final result something other than advertised. The same objection applies to screening with a normality test, covered in testing normality. Variance tests are worth running to understand your data. They are not worth running to select your t-test.
Three assumptions, in descending order of importance.
Independence of observations. Non-negotiable, and unfixable after the fact. Three measurements on each of ten subjects are not thirty observations. No variant of the t-test will notice if you pretend otherwise.
Equal variances, for Student’s only. This is what Welch’s exists to sidestep.
Approximate normality. The least demanding of the three. It applies to the sampling distribution of the mean, not to the raw data. With a few dozen observations per group the central limit theorem does most of the work and moderate skew is tolerable. What does still bite at small n is heavy tails and outliers. The mean and standard deviation are both sensitive to extreme values.
For a paired test, the assumption applies to the differences, not to either set of original measurements. That distinction matters, because differences are often well behaved even when the underlying values are skewed.
When normality is genuinely doubtful and the sample is small, the Wilcoxon-Mann-Whitney test replaces the independent-samples t-test. The Wilcoxon signed-ranks test replaces the paired one. Both work on ranks and assume nothing about shape. The sign test goes further still, using only the direction of each difference, which makes it robust to anything and correspondingly blunt.
They cost less power than people expect, but they answer a subtly different question, about distributions and typical values rather than means. That is worth understanding before switching. Non-parametric tests covers the trade.
A t-test tells you whether a difference is distinguishable from noise. What it does not tell you is how big the difference is, which is usually what you actually needed. The mean difference with a confidence interval should be the headline. It is on the scale of the measurement, so it is directly interpretable, and its width tells you how well the study pinned the answer down.
Standardised effect sizes, Cohen’s d or Hedges’ g with its small-sample correction, express the difference in standard deviations. That is useful when comparing across studies or when the raw units mean nothing to the reader. For rank-based analyses the Hodges-Lehmann location shift is the natural companion estimate, and it comes with a confidence interval too.
The two failure modes here are symmetrical and both common. A significant result that is far too small to matter, because the sample was large. And a non-significant result reported as “no difference”, when the interval runs from −8 to +15 and the study was simply too small to say anything. See confidence intervals and p-values.
If there are three or more groups, do not run t-tests on every pair. Three groups give three comparisons, six give fifteen, and the family-wise error rate climbs fast. One-way ANOVA is the right starting point. Welch’s ANOVA is the unequal-variance version, following the same reasoning above. Follow with a multiple comparison procedure matched to the comparisons you need.
Download the independent-samples example workbook (.xlsx) — calcium supplementation and blood pressure in two groups, with box plots, an F-test for the variance ratio, and the t-test. The paired example covers 28 before-and-after observations with paired lines, a difference plot with Hodges-Lehmann shift, and the Wilcoxon signed-ranks test.
The example workbook is downloading.
It opens in Excel on its own — the data and the finished results are both in it. Analyse-it is what lets you change the analysis and re-run it, try the same study on your own data, or work through it to see how the software handles it.
Every feature from all five editions for 15 days, with no sign-up and no licence key.
Using an independent-samples test on paired data. The most expensive error here. Pairing removes between-subject variation; ignoring it usually costs the result.
Testing variances to decide which t-test to use. Use Welch’s by default and skip the screening step entirely.
Reporting only the p-value. The mean difference and its interval are the finding. The p-value is one thing you can read off them.
Reading non-significant as no difference. A wide interval means the study could not tell, which is not the same as an answer.
Running every pairwise t-test with three or more groups. Use ANOVA and a multiple comparison procedure.
Checking normality of the raw values in a paired design. It is the within-pair differences that need to be well behaved.
Analyse-it runs the test and reports the estimate you should be quoting, inside Excel:
Every feature from all five editions for 15 days, with no sign-up and no licence key. Hypothesis testing is part of the Standard edition, so it is in every Analyse-it edition, from US$ 155 a year. Validated against NIST Standard Reference Datasets.