Assessing linearity of a measurement procedure Linearity fixes the range over which you can trust a result. How to establish it under CLSI EP06-Ed2 — and why a correlation coefficient is the wrong tool for the job.

A quantitative method is linear over the range where its measured result follows a straight line against the true concentration. Equal steps in concentration should produce equal steps in the reading, whatever the starting point. That range is the interval over which you are entitled to report numbers. Outside it, results curve away from the truth and mislead.

Establishing the linear range is the subject of CLSI EP06-Ed2. A high correlation coefficient is a common shortcut, but it does not do the job, for much the same reason it fails in method comparison.

The dilution series

Linearity is assessed with a series of samples of known relative concentration spanning the claimed measuring interval. Typically you mix a high and a low pool in known proportions, or use dilution. Measure each level in replicate. The levels should reach the ends of the interval and space out across it. Nonlinearity most often appears at the extremes: saturation at the top, or a disproportionate response near the bottom. Too few levels, or levels clustered in the middle, leave the regions where curvature hides unexamined.

Fitting and testing for curvature

The quantity that matters is the deviation from linearity at each level: how far the measured response departs from a straight line fitted through the data. EP06-Ed2 takes the deviation as the difference between the mean result at each level and the straight line. The earlier EP06-A fitted second- and third-order polynomials as well, and took the difference between the best-fitting curve and the line. That polynomial approach is still widely used. A straight-line fit with no comparison of this kind cannot expose curvature, because it draws a line through curved data and leaves the curvature in the residuals.

Judge against an allowable nonlinearity

Judge the deviation from linearity at each level against an allowable nonlinearity, not against zero, because statistical significance is not the same as clinical importance. A large enough study will flag a trivial, clinically irrelevant curvature as significant.

The allowable nonlinearity is a criterion set from clinical or analytical requirements, often expressed as a percentage. Use an equivalence test rather than a test against zero. A difference plot of each level’s deviation against the allowable band makes the judgement immediate. Where the deviations sit inside the band, the method is linear enough; where they fall outside it, the method is not.

A linearity plot showing measured values against equally spaced dilution levels: a dashed straight-line fit, a shaded allowable-nonlinearity band around it, and a red curve of the true response that bends away and leaves the band at the top two levels.
Linearity is judged as deviation from a straight line, not proportionality. The measured response (red) curves away from the fitted straight line at the top. Where a level’s deviation falls outside the allowable-nonlinearity band, the nonlinearity is too large to accept, whether or not it is statistically significant.

Report the linear interval

The output is not a single pass or fail for the whole range but the interval over which the method is linear within the allowable nonlinearity. If the extremes deviate but the middle holds, the measuring interval is narrower than claimed. Results outside it should be diluted into range or flagged. The calcium linearity workbook shows exactly this pattern. Over the full interval, five of the six dilutions exceed the ±0.2 mg/dL allowable nonlinearity. A refit over a reduced interval passes at every level, and that reduced interval is the one to report.

Linearity difference plot showing each dilution's deviation from the best-fit line against an allowable nonlinearity band.
A linearity evaluation with deviations judged against an allowable nonlinearity band (IgM, dataset from CLSI EP06-A Appendix C). Only dilution 4 of the five meets the ±5% allowable nonlinearity, so the method fails linearity over the full interval at that limit.

Downloads

Two worked examples, ready to open in the Analyse-it trial:

  • IgM linearity — five dilutions with linear and polynomial fits and a ±5% allowable nonlinearity band.
  • Calcium linearity — the full and a reduced measuring range compared against allowable nonlinearity.

Common mistakes

Using a correlation coefficient or R² as the test. Both can look excellent while the data curve. Fit polynomials and judge the deviation from linearity against an allowable limit.

Too few levels, or none at the extremes. Curvature hides where you did not sample. Span the interval and reach its ends.

Reading statistical significance as clinical significance. A large study flags trivial curvature. Judge the deviation against an allowable nonlinearity, not against zero.

Treating linearity as a one-off. A linearity study establishes the interval when the method is introduced. Under the CAP checklists, recurring AMR verification then checks every six months that the interval still holds.

Confusing linearity with calibration. A linearity study asks whether the response stays on a straight line across the interval. A calibration curve is the fitted relationship you invert to report results. See fitting a calibration curve.

Reporting outside the linear interval. If the extremes fail, the measuring interval is narrower than claimed. Dilute into range or flag. Do not report numbers the method cannot support.

Assess linearity with Analyse-it

Analyse-it runs the linearity assessment on your own dilution series, inside Excel:

  • EP06-Ed2: a straight-line fit, optionally weighted, with the deviation from linearity taken as each level’s mean minus the line
  • The earlier EP06-A procedure as well: the best second- or third-order polynomial, a forward stepwise fit by lack of fit, or a polynomial up to fifth order
  • Hsieh–Liu confidence intervals and equality or equivalence tests of the polynomial’s deviation from the line
  • A difference plot against an allowable nonlinearity, absolute or relative, constant across the interval or set level by level

Every feature from all five editions for 15 days. Linearity is in the Method Validation and Ultimate editions, from US$ 475 a year. Validated against NIST and CLSI reference datasets. The linear interval established here is the basis of the analytical measuring interval and reportable range. Full detail in the trueness and linearity reference guide.