1. Tutorials
  2. Verifying the precision of a measurement procedure against a performance claim and estimating the bias (CLSI EP15-A3)

Verifying the precision of a measurement procedure against a performance claim and estimating the bias (CLSI EP15-A3)

Learn how to verify that a measurement procedure meets performance requirements before introducing a new measurement procedure into the laboratory, after maintenance to the measuring system, or upon failing a proficiency test or inspection.

In this tutorial you will use the CLSI EP15-A3 procedure to verify the precision against the manufacturer’s claim and estimate the bias using proficiency testing materials.

  1. Estimating precision
    Estimate the precision of the measurement procedure.
  2. Testing precision against a performance claim
    Test if the precision meets the manufacturer's performance claim.
  3. Identifying and excluding outliers
    Dealing with outliers and assessing their impact.
  4. Estimating bias
    Estimate the bias using reference materials.

Estimating precision

Estimate the precision of the measurement procedure.

  1. Open the file tutorials\EP15-A3.xlsx.

    The worksheet opens showing 3 columns. The Sample column identifies the levels of the analyte in the sample (3 levels). Run identifies the day number of each run (5 runs). Ferritin (ug/L) identifies the measured value for 3 replicates of sample in each run.

  2. Click a cell in the dataset.
  3. On the Analyse-it ribbon tab, in the Statistical Analyses group, click Precision drop-down, and then click 1 Factor.
    The analysis task pane opens.
  4. In the Y drop-down list, select Ferritin.
  5. In the Factor A drop-down list, select Run.
  6. In the By drop-down list, select Sample.
  7. In the Estimator drop-down list, select SD.
  8. Clear the Detailed components check box.
  9. On the Analyse-it ribbon tab, in the MSA group, click Variability plot.
  10. On the Analyse-it ribbon tab, in the MSA group, click Identify Outliers.
  11. Click Calculate.

    The analysis report opens.

The variability plots show a simple visual assessment of the closeness of agreement between the measured quantity values. The purple lines show the mean of each run and the blue line the overall grand mean, allowing you to see the variation between and within each run.
EP15 variability plot

You should observe the scatter of the points to ensure there are no obvious problems. If a result appears spurious then you should investigate and correct any mistakes. If the data appear to be unusable, stop the evaluation and begin an expanded evaluation of the sources of measurement error or contact the manufacturer. We note that one observation in Sample 1, Run 1 is highlighted as a potential outlier but do not know the reason for the aberrant result. We will continue with this observation included in the analysis and return to it later.

The abbreviated variance components table shows precision expressed numerically as the standard deviation (SD) and coefficient of variation (CV).
EP15 abbreviated components when failing goal

Testing precision against a performance claim

Test if the precision meets the manufacturer's performance claim.

You should use this procedure when you already have a performance claim from a manufacturer's package insert and you want to test whether your precision is significantly greater than the claim. It is possible for the imprecision from a study to be greater than the manufacturer's claim due to the chance alone. This procedure ensures that the manufacturer's claims are only falsely rejected 5% of the time when they are in fact true.

  1. On the Analyse-it ribbon tab, in the Precision group, click Test Claim.
    The hypothesis test section on the Precision panel opens.
  2. In the Hypotheses grid, under the Repeatability SD/CV column, type the performance claims as an absolute value (for example, type 5 for a SD of 5 mg/dL) or relative value (for example, type 5% for a CV of 5%).
    Level Repeatability CV
    1 3.3%
    2 2.0%
    3 1.6%
  3. In the Hypotheses grid, under the Laboratory SD/CV column type the performance claims as an absolute(for example, type 5 for a SD of 5 mg/dL) or relative value (for example, type 5% for a CV of 5%).
    Level Laboratory CV
    1 5.3%
    2 3.4%
    3 2.8%
  4. In the Significance level edit box, type 5%, and then select the Familywise error rate check box.

    CLSI EP15 uses a familywise significance level so the overall significance level is a maximum of 5% regardless of the number of levels (for example, for 1 level the significance level is 5%, for 2 levels the significance level is 5%/2 = 2.5% for each level, for 3 levels the significance level is 5%/3 = 1.6% for each level).

  5. Click Recalculate.

    The analysis report updates.

The detailed variance components table shows the observed and expected SD/CV along with the hypothesis test p-value for each level.
variance hypothesis testfail

All the hypothesis tests are not significant and highlighted green in this example. If any are significant they are highlighted in red and you should contact the manufacturer for further assistance in diagnosing the problem. If the hypothesis test is not statistically significant but the imprecision estimate is much larger than the claim, you may want to repeat the study with more data to be able to detect smaller departures from the claim.

Identifying and excluding outliers

Dealing with outliers and assessing their impact.

Even after correcting or excluding all result known to be spurious, there may sometimes be results marked as statistical outliers. This may be due to a nonperformance-related cause, which, if known, would have justified excluding the result. Alternatively, the apparently extreme results may genuinely represent the performance. There is always a trade-off between retaining the result which will inflate the imprecision estimates, or exclude it which may overly optimistic estimates. It is often good practice to calculate the results before and after excluding the outlier.

  1. On the Analyse-it ribbon tab, in the Report group, click Clone.

    The dataset worksheet activates and the analysis task pane remains open.

  2. On the Descriptives panel, select the Exclude identified outliers check box.
  3. On the analysis task pane, click Calculate.

    A new analysis report open.

The variability plot show the excluded observation as a red cross.


EP15 variability plot with outlier

The detailed variance components table shows the observed and expected SD/CV along with the hypothesis test p-value for each level.
variance hypothesis test

All the hypothesis tests are not significant and highlighted green. Because the exclusion of the outlier changes the outcome of the study it is essential to assess the clinical effect of the outlier and investigate further to try and determine its cause, or contact the manufacturer for further assistance in diagnosing the problem.

Estimating bias

Estimate the bias using reference materials.

You should use this procedure when you already have a sample with a known assigned value and you want to estimate the bias and test whether it is significantly different to zero. It is possible for the bias from a study to be greater than zero due to the chance alone. This procedure ensures that the assumption that the bias is 0 is only falsely rejected 5% of the time when it is in fact true.

  1. On the Analyse-it ribbon tab, in the MSA group, click Test Equality.
    The Trueness panel opens.
  2. In the Assigned values grid, in the Values column type the known values of the materials.
    ID Value
    1 25.0
    2 142.5
    3 650
    Note: CLSI EP15-A3 only gives the assigned value for Sample 2. We have fabricated values for the other samples for demonstration purposes.
  3. Select the Uncertainty in assigned values check box, and then in the Assigned values grid, alongside sample 2, in the SE column type 0.69 and in the DF column type 42.

    The standard error (SE) of the assigned value depends on the source of the assigned value. Reference materials usually are accompanied by a standard uncertainty, whereas proficiency testing materials usually specify the SD and number of laboratories. The standard error (SE) should be computed using the formula in the CLSI document. The degrees of freedom (DF) is usually only used when the sample is PT material and is equal to the number of laboratories minus 1.

  4. In the Significance level edit box, type 5% and select the Familywise error rate check box.

    CLSI EP15-A3 uses a familywise significance level so the overall significance level is a maximum of 5% regardless of the number of comparisons (for example, for 1 level the significance level is 5%, for 2 levels the significance level is 5%/2 = 2.5% for each level, for 3 levels the significance level is 5%/3 = 1.6% for each level).

  5. Click Recalculate.

    The analysis report updates.

The table shows the observed and expected value along with the bias and the hypothesis test p-value for each level.
biashypothesis test

The hypothesis tests for level 1 and 2 are not significant in this example, but for level 3 the bias is significantly different from zero and is highlighted in red. You should therefore determine if the bias is acceptable for your laboratories needs by comparing it to user-specified allowable bias, or contact the manufacturer for further assistant in diagnosing the problem. If the hypothesis test is not statistically significant but the bias estimate is much larger than zero, you may want to repeat the study with more data to be able to detect smaller departures from the zero.

Tutorials v6.15

Administrator's Guide

User's Guide

Statistical Reference Guide