When you measure many things on the same samples, the measurements are usually related. Several move together because they reflect a common underlying property. Others carry information of their own. Principal component analysis takes that tangle of correlated variables and rebuilds it as a smaller set of uncorrelated ones: the principal components. The components are ordered so the first captures as much of the variation as possible, the second as much of what remains, and so on.
Nothing is discarded in the process itself. With p variables you get p components. Together they contain exactly the information you started with. The reduction happens later, when you decide the last few components are just noise and stop looking at them.
Three quantities do the work here. Keeping them straight is most of understanding the output.
Loadings (the coefficient matrix) say how each original variable contributes to each component. They are what you read to interpret a component. If six variables all load heavily and positively on the first component, that component is a general “size” or “overall level” axis. If some load positively and others negatively, it contrasts the two sets instead.
Scores are each observation’s position on each component: the new coordinates. Plotting the first two scores against each other is often the single most informative picture you can get of a multivariate dataset. Clusters, gradients and outliers that no pair of original variables would reveal.
Eigenvalues tell you how much variance each component accounts for. Expressed as a percentage of the total, they tell you how much of the structure the first few components have captured. By implication, they also tell you how much you lose by ignoring the rest.
A biplot puts observations and variables in the same picture. The biplot is where PCA becomes useful for interpretation rather than just a tidy summary. In the classic Gabriel biplot, observations appear as points and variables as vectors radiating from the origin.
That last rule is what makes a biplot worth the trouble. You can look at a cluster of samples and read off directly which measurements characterise it. A Gower-Hand biplot goes further, drawing variables as calibrated axes with scale markings rather than as vectors. You can then read approximate values off the plot instead of just directions. Where only the variable structure is of interest, a correlation monoplot shows the variables alone.
PCA maximises variance. A variable with a large numeric spread dominates the components simply because of its units. Measure one thing in metres and another in millimetres and the second will look a thousand times more important than it is.
Running PCA on the correlation matrix is equivalent to standardising every variable to unit variance first, which removes the problem. Standardising is the right default whenever variables are measured on different scales. Use the covariance matrix only when all variables share the same units and their relative variances are genuinely part of what you want the analysis to reflect.
There’s no test for this. Treating any of the conventions as a firm rule is a mistake. Three approaches are used. They are best used together.
The scree plot shows eigenvalues in order. You look for the elbow where the curve flattens, and keep the components above it. The cumulative variance criterion keeps enough components to reach a chosen percentage. 80% is common and entirely arbitrary. The eigenvalue-greater-than-one rule, applied to a correlation matrix, keeps components explaining more than a single original variable would. It’s the most mechanical of the three and tends to keep too many.
Keep a component if you can interpret it. A component that no combination of your subject knowledge can make sense of is usually noise, whatever its eigenvalue says.
The two are routinely conflated, and they answer different questions. PCA is descriptive: it re-expresses the total variance in the observed variables, with no model behind it. The components are constructions rather than claims about anything real.
Common factor analysis assumes a model: a small number of unobserved latent factors generate the shared variance among your variables, with the rest unique to each. Factor analysis separates common from unique variance, which PCA does not, and it supports rotation to make the factors interpretable. Orthogonal rotations such as Varimax keep factors uncorrelated. Oblique ones such as Oblimin allow them to correlate, which is often more realistic.
The practical rule: if you want to compress correlated measurements into a few summary axes, use PCA. If you believe in underlying constructs and want to estimate them (the usual situation with questionnaire or rating data), use factor analysis. Check internal consistency with Cronbach’s alpha while you are there.
PCA finds linear combinations, so a curved structure gets spread awkwardly across several components rather than captured by one. PCA is also unsupervised: components are chosen to explain variance among the predictors, with no reference to any outcome. So the component that explains the most variance need not be the one that predicts anything. And PCA is sensitive to outliers, because a single extreme observation can inflate a variance and tilt a component towards itself. Check a score plot for exactly that before you interpret anything.
Download the multivariate example workbook (.xlsx) — a principal component analysis with eigenvalues, a scree plot, the coefficient matrix, and biplots, ready to open in the Analyse-it trial. The correlation example covers the matrix and scatter plot matrix that usually precede it.
The example workbook is downloading.
It opens in Excel on its own — the data and the finished results are both in it. Analyse-it is what lets you change the analysis and re-run it, try the same study on your own data, or work through it to see how the software handles it.
Every feature from all five editions for 15 days, with no sign-up and no licence key.
Not standardising variables on different scales. The components will follow the units rather than the structure.
Treating the eigenvalue-greater-than-one rule as a decision. It’s a convention, and a generous one. Read the scree plot and ask whether the component means anything.
Naming a component and then reasoning as if it were real. A label is a convenience for discussion, not evidence that an underlying quantity exists.
Confusing PCA with factor analysis. One re-expresses total variance with no model; the other estimates latent factors from shared variance. They answer different questions. Factor analysis explained takes the other side of the distinction.
Reading vector length as importance without checking the plane. A short vector means the variable is poorly represented by these two components, not that it’s unimportant. It may load heavily on the third.
Interpreting components before checking for outliers. One extreme observation can define an entire component.
Analyse-it computes the components and draws the biplots, inside Excel:
Every feature from all five editions for 15 days, with no sign-up and no licence key. PCA and factor analysis are part of the Standard edition, so it is in every Analyse-it edition, from US$ 155 a year. Validated against NIST Standard Reference Datasets. See Pearson, Spearman or Kendall for the correlations underneath, or the reference guide for the technical detail.