Statistical software is only useful if its results are correct. Tested against the NIST Statistical Reference Datasets (StRD) — the industry-recognised benchmark for the numerical accuracy of statistical software — Analyse-it performs consistently among the best, matching or outperforming several well-known commercial statistical packages.
This page describes the benchmark, our methodology, and the results. You can download the Excel workbooks used to run the benchmark and re-run them on your own installation as part of your validation activities.
Download Analyse-it NIST StRD validation workbooks (.zip)
In response to industry concerns about the numerical accuracy of statistical software, the Statistical Engineering Division and the Mathematical and Computational Sciences Division of NIST’s Information Technology Laboratory developed a collection of reference datasets with certified computational results. These datasets let the results produced by any statistical software package be compared directly against values of known accuracy.
The certified values are accurate to 15 significant digits and were computed using ultra-high precision floating-point arithmetic. The datasets cover univariate summary statistics, analysis of variance, linear regression, and non-linear regression, graded by difficulty from “lower” to “higher” based on conditions known to stress floating-point algorithms.
For more information about the datasets see https://www.itl.nist.gov/div898/strd/.
Most statistical packages, including Analyse-it, use IEEE 754 double-precision (64-bit) floating-point arithmetic. Because of finite precision and the rounding and truncation errors that occur in numerical operations, no double-precision software can reproduce a NIST-certified value exactly. A good measure of how closely a computed result x matches a certified value c is the log relative error (LRE):
LRE = −log10 (|x − c| / |c|), if c ≠ 0; otherwise LRE = −log10 |x|.
LRE is the number of significant digits the computed result has in common with the certified value. Higher is better, and the maximum value obtainable with double-precision arithmetic is 15. Results at or near 15 effectively mean the software has matched the certified value to the limits of what double-precision arithmetic allows.
The NIST StRD is the public, independent benchmark for the numerical accuracy of the statistical procedures it covers. Because the datasets and certified values are maintained by NIST, they provide an objective check on software that any third party can verify. For the procedures within its scope, we treat it as a reference benchmark and track performance release to release.
The NIST StRD does not cover every statistical procedure Analyse-it implements. For procedures outside its scope — such as the CLSI method-validation analyses, diagnostic performance analyses, control charting, and multivariate procedures — numerical correctness is verified against independently reproduced reference results, against other statistical packages, and against a proprietary test suite of more than 1,000 multi-faceted test cases maintained and re-run at every release. See our development and validation process for the full picture.
We test Analyse-it against the NIST StRD as part of our release process. The results below summarise performance across the univariate, analysis of variance, and non-linear regression dataset groups. No statistical package achieves perfect accuracy on every test, and no single package performs best on every test — but in our testing:
Some developers of statistical software have published their own NIST StRD benchmarks, and several independent authors have compared results across packages:
You can download the Analyse-it validation workbooks, run them on your own installation, and compare the results against the published NIST certified values and against other packages’ published benchmarks.
The univariate tests consist of nine datasets graded by difficulty. The mean and standard deviation are computed using the Distribution analysis in Analyse-it and compared against the NIST-certified values. The lag-1 autocorrelation is not computed by Analyse-it and is not reported here.
| Test | Difficulty | LRE | |
|---|---|---|---|
| Mean | SD | ||
| PiDigits | Lower | 15.0 | 15.0 |
| Lottery | Lower | 15.0 | 15.0 |
| Lew | Lower | 15.0 | 15.0 |
| Mavro | Lower | 15.0 | 13.1 |
| Michelson | Lower | 15.0 | 13.8 |
| NumAcc-1 | Lower | 15.0 | 15.0 |
| NumAcc-2 | Average | 15.0 | 15.0 |
| NumAcc-3 | Average | 15.0 | 15.0 |
| NumAcc-4 | Average | 15.0 | 15.0 |
| NumAcc-5 | Average | 15.0 | 15.0 |
| NumAcc-6 | Average | 15.0 | 15.0 |
| NumAcc-7 | Average | 15.0 | 15.0 |
| NumAcc-8 | Average | 15.0 | 15.0 |
| NumAcc-9 | Average | 15.0 | 15.0 |
The analysis of variance tests include three dataset families representing increasing model complexity. All tests are run using the Fit Model analysis in Analyse-it and compared against the NIST-certified F statistic, sum of squares for treatments, and sum of squares for error.
| Test | Dataset | LRE | ||
|---|---|---|---|---|
| F statistic | SS treatment | SS error | ||
| SmLs01 | SmLs01 | 15.0 | 15.0 | 10.7 |
| SmLs02 | 15.0 | 15.0 | 14.2 | |
| SmLs03 | 12.5 | 10.6 | 11.8 | |
| AtmWtAg | SmLs04 | 15.0 | 15.0 | 15.0 |
| SmLs05 | 14.7 | 12.7 | 12.6 | |
| SmLs06 | 10.5 | 10.0 | 11.5 | |
| SiRstv | SmLs07 | 14.8 | 14.7 | 14.9 |
| SmLs08 | 14.7 | 13.2 | 11.1 | |
| SmLs09 | 13.5 | 10.4 | 9.7 | |
The non-linear regression tests include datasets representing increasing model complexity. All tests are run using the Fit Model analysis in Analyse-it and compared against the NIST-certified parameter estimates and predictions.
| Test | Dataset | LRE | |
|---|---|---|---|
| Parameters | Predictions | ||
| Gauss1 | Gauss1 | 15.0 | 15.0 |
| Gauss2 | 15.0 | 15.0 | |
| Gauss3 | 14.9 | 15.0 | |
| Mavro | Mavro | 15.0 | 15.0 |
| Chwirut | Chwirut | 14.6 | 13.8 |
The validation workbooks contain the NIST datasets with the corresponding Analyse-it analyses already configured. You can open them, recalculate the analyses on your installation, and compare the results against the NIST-certified values printed in each workbook. This provides documented evidence that Analyse-it is computing correct results in your environment — suitable either as operational qualification evidence under a traditional Computer System Validation approach, or as the vendor-benchmark element of a risk-based approach under the FDA’s Computer Software Assurance guidance (September 2025).