Analysis of variance has an unhelpful name: it compares means, and it does so by partitioning variance. If several groups really do have the same mean, the variation between the group means should be no larger than the variation within the groups. Both are just noise. When the between-group variation is substantially larger than the within-group variation, the group means are not all the same.
The F statistic is the ratio of those two, and the p-value is how often a ratio that large would arise by chance if every group shared one mean.
One-way ANOVA is the direct extension of the two-sample t-test: one factor (supplier, dose, operator, batch) at three or more levels, and one continuous response. One-way ANOVA assumes the observations are independent, the residuals are approximately normal, and the groups have similar variance.
There are two escape routes if those last two do not hold. If the variances differ markedly, Welch’s ANOVA relaxes the equal-variance assumption the same way Welch’s t-test does, and it is the safer default whenever group sizes are unequal. If normality is in real doubt, not just unproven, the Kruskal-Wallis test compares the groups through ranks instead. Levene’s and Brown-Forsythe tests check the variance assumption directly, and so does Bartlett’s, though it is itself sensitive to non-normality.
What you get out is a single F-test saying “these means are not all equal.” The F-test does not say which ones differ. The temptation to follow it with a string of t-tests is exactly the trap multiple comparison procedures exist to avoid.
Real experiments rarely vary just one thing. You might have three primers and two application methods, four suppliers and two shifts, a treatment and a baseline covariate. Fitting a separate one-way ANOVA for each factor is not the same analysis, and it is usually worse. Each one ignores the variation the other factor explains, inflating the residual and making both effects harder to see.
A multi-factor model puts them in together. Each factor gets a test that accounts for the others, the residual shrinks down to what neither factor explains, and you can ask whether the factors interact.
An interaction means the effect of one factor depends on the level of another. Primer A might outperform primer B when dipped and underperform it when sprayed. Both main effects could look modest or absent while the interaction is large, because averaging over the other factor cancels the two opposite effects out.
Interactions are the main reason to fit a multi-factor model, and they come with a reporting rule that is easy to miss. When an interaction is present, do not interpret the main effects on their own. “Primer A is better” is not true if it is only better when sprayed. Report the effect at each level of the other factor, and let the interaction plot carry the finding. Two lines that cross say more than any table could.
The same structure shows up in method validation, where the factors are run, day and operator, and the goal is not to test them but to estimate how much variation each contributes. That is the precision components: a variance-component analysis rather than an F-test, but the same nested design sits underneath it. The same machinery underpins robustness testing with factorial designs, where the point of the design is to vary several factors at once.
Sometimes something continuous and measurable influences the response without being what you are testing: baseline severity, ambient temperature, sample age. Adding it to the model as a covariate turns the analysis into ANCOVA. The covariate absorbs variation that would otherwise sit in the residual. The factor you care about is then tested against a smaller residual, and the group means are reported adjusted to a common covariate value.
Adjusted means are what you should quote from an ANCOVA. Raw group means still carry whatever imbalance in the covariate the groups happened to have.
In a multi-factor model with unequal cell counts, the factors overlap: some variation could be credited to either one. How you split it is what the sums-of-squares types settle. Type I (sequential) gives each term only the variation left after the terms entered before it, so the answer depends on the order you list them in. Type III (partial) gives each term the variation it uniquely explains after every other term, so it does not depend on order.
For a balanced design, with equal counts in every cell, the two agree and the distinction never comes up. For an unbalanced design they can differ, sometimes substantially. Type III is the sensible default for asking “does this factor matter, given everything else in the model?”, which is nearly always the question you actually want answered. Type I is the right tool when the terms have a genuine hierarchy and you want to test them in a deliberate sequence. Adding a quadratic term after the linear one is the standard case.
An ANOVA table summarises a fitted model, and it is only as trustworthy as that model is. The same diagnostics that apply to regression apply here too, because underneath it is the same machinery. Residual plots expose non-constant variance and curvature. A residual distribution plot and a normal Q-Q plot address the normality assumption. An outlier and influence plot, Cook’s D against studentised residuals, shows whether a handful of observations are driving the whole result. Reading residual diagnostics covers what each pattern means.
Download the two-way ANOVA example workbook (.xlsx): aircraft primer paint adhesion across three primers and two application methods, with Type III F-tests, adjusted effect means, main effect plots, and Tukey-Kramer comparisons on both factors. The 2³ full factorial example adds all two-way and three-way interactions, and the ANCOVA example shows a three-level factor adjusted for a continuous covariate.
The example workbook is downloading.
It opens in Excel on its own — the data and the finished results are both in it. Analyse-it is what lets you change the analysis and re-run it, try the same study on your own data, or work through it to see how the software handles it.
Every feature from all five editions for 15 days, with no sign-up and no licence key.
Reporting main effects in the presence of an interaction. If the effect of one factor depends on another, its average effect describes no single condition. Report it at each level instead.
Running separate one-way ANOVAs instead of one multi-factor model. You lose the interaction entirely and inflate the residual in both analyses.
Following a significant F with unadjusted t-tests. Six groups give fifteen pairwise comparisons; at 5% each, a false positive somewhere is more likely than not.
Quoting raw means from an ANCOVA. The point of the covariate is adjustment. Report the adjusted effect means.
Assuming Type I and Type III agree. In an unbalanced design they need not, and reporting whichever the software defaulted to is not really a decision.
Treating repeated measurements as independent. Several measurements on the same subject belong in a within-subjects design, not stacked up as separate observations.
Analyse-it fits the model your design calls for, inside Excel:
Every feature from all five editions for 15 days, with no sign-up and no licence key. ANOVA and ANCOVA are part of the Standard edition, so it is in every Analyse-it edition, from US$ 155 a year. Validated against NIST Standard Reference Datasets. Read on for choosing a multiple comparison procedure.