You are viewing documentation for the old version 2.30 of Analyse-it. If you are using version 3.00 or later we recommend you go to the ROC plot.

ROC Curve Plot

ROC, or Receiver Operator Characteristic, is used to examine the performance of a diagnostic test over a range of decision levels (medical decision points). Performance is the test's ability to correctly identify positive and negative cases. Individual decision levels can be evaluated using Qualitative (Sensitivity / Specificity).

The requirements of the test are:

  • A dichotomous scale variable containing the true state of the subject: positive or negative.
  • A diagnostic test measured on a continuous scale.


Arranging the dataset

Data in existing Excel worksheets can be used and should be arranged in the List dataset layout. The dataset must contain a nominal scale variable indicating the true state of the case, positive or negative, and a continuous scale variable containing the observations of the diagnostic test.

When entering new data we recommend using New Dataset to create a new Test performance dataset.

Using the test

To start the test:

  1. Excel 2007:
    Select any cell in the range containing the dataset to analyse, then click Test performance on the Analyse-it tab, then click ROC curve
  2. Excel 97, 2000, 2002 & 2003:
    Select any cell in the range containing the dataset to analyse, then click Analyse on the Analyse-it toolbar, click Test performance then click ROC curve.

  3. Click True classification then select the dichotomous variable containing the true state (positive or negative) of the case.
  4. Click Positive indicated by and select the group indicating a positive case. The list shows the two groups of the classification variable.
  5. Click Test and select the variable containing the observations of the diagnostic test.
  6. Click Positive if and select whether positive cases produce diagnostic test results that are less than, less than or equal, greater than or equal, or greater than a decision level.
  7. Enter Confidence level to compute for the area under the curve (AUC, see below) of the diagnostic test. The level should be entered as a percentage between 50 and 100, without the % sign.
  8. Click OK to run the test.

The report shows the number of observations analysed, how many missing values were listwise excluded, and the number of positive and negative cases.

The area under the curve (AUC) is a measure of the ability of the diagnostic test to correctly identify cases. Diagnostic tests with higher AUCs are generally better and should always be higher than 0.5, indicating the test is better at diagnosing than chance. A hypothesis test is used to statistically test if the diagnostic test is better than chance at correctly diagnosing state. A significant p- value indicates the diagnostic test is better at diagnosing than chance.

The performance of the test at each decision level is evaluated. The tables show the TP rate (true positive rate / sensitivity) and TN rate (true negative rate / specificity) with confidence intervals, and the positive and negative likelihood ratios.

Understanding the ROC plot

The ROC plot (see below) shows False Positive rate (1-specificity) (X axis), the probability of incorrectly diagnosing a case as positive when it's true state is negative, against True positive rate (sensitivity) (Y axis), the probability of correctly diagnosing a positive case, across all decision levels for the diagnostic test. Ideally the curve will climb quickly toward the top-left meaning the test correctly identifies cases. The diagonal grey line is a guideline for a test that is unable to correctly identifying cases.

(click to enlarge)

Additional secondary axes can be shown on the ROC plot to show True negative rate (specificity) (secondary X axis), the probability of correctly identifying a negative case, and False Negative rate (1-sensitivity) (secondary Y axis), the probability of incorrectly diagnosing a case as negative when it's true state is positive.

To change the axes shown on the ROC plot:

  1. If the ROC curve dialog box is not visible click Edit on the Analyse-it tab/toolbar.
  2. Click ROC plot and select 1-Specificity vs Sensitivity or All axes to show specificity and 1-sensitivity as well.
  3. Click OK.

Predictive values 

When evaluating the results of a diagnostic test we are usually interested in the probability of the classification been correct given the test result.

Predictive values are affected by the prevalence of the condition in the population. When evaluating test performance the cases studied are usually hand selected to ensure the sample contains enough normal and abnormal cases to fully evaluate the test's performance. Therefore the prevalence is not representative of the population and the predictive values would be too optimistic. To calculate the correct predictive values you should use the pre-test probability (prevalence) of case been truly positive before the test is performed.

To show positive & negative predictive values:

  1. If the ROC Curve dialog box is not visible click Edit on the Analyse-it tab/toolbar.
  2. Tick Predictive value.
  3. Enter Pre-test probability (prevalence) of the condition in the population as a proportion between 0 and 1. For example if the prevalence is 5% of the population enter 0.05. If the prevalence is left blank Analyse-it will use the prevalence in the sample to calculate predictive values.
  4. Click OK.

The positive and negative predictive values are now shown in the table showing performance at each decision level. A Positive Predictive value (PV+) is probability of the case been truly positive given a positive test result, the Negative Predictive value (PV-) is the probability of the case been truly negative given a negative test result.

Costs

The cost of a diagnosis, whether in terms of financial cost or in terms of the cost to the health of the subject, can be taken into account when evaluating decision levels. For some diagnosis the cost of treating a subject may be high financially, or the impact of a treatment might be such that treating false-positive subjects should be minimised even at the risk of missing some positive cases. In other situations the treatment cost financially or to health might be minimal meaning a higher false-positive rate can be tolerated to catch more positive cases.

To specify costs of diagnosis: 

  1. If the ROC Curve dialog box is not visible click Edit on the Analyse-it tab/toolbar.
  2. Tick Cost.
  3. Enter Pre-test prevalence of the condition in the population as a proportion between 0 and 1. For example if the prevalence is 5% of the population enter 0.05. If the prevalence is left blank Analyse-it will use the prevalence in the sample to calculate predictive values.
  4. Enter Cost FP (false positive cost) and Cost FN (false negative cost) in financial terms (e.g. as dollar values) or as relative costs of a misdiagnosis (with higher values indicating severity).
  5. Optionally enter Cost TP (true positive cost) and Cost TN (true negative cost).
  6. Click OK.

The average cost of a diagnosis at each decision level is now shown in the table. The lowest cost indicates the best decision level minimizing the costs. If only the FP / FN costs were specified then the costs given are the cost of mistakes.

Interpreting the Sensitivity / Specificity decision plot

A decision plot can be shown to help in choosing optimum decision levels in terms of sensitivity (true positive rate) and specificity (true negative rate).

To show the sensitivity / specificity decision plot: 

  1. If the ROC Curve dialog box is not visible click Edit on the Analyse-it tab/toolbar.
  2. Click Decision plot and select Sensitivity / Specificity.
  3. Click OK.

The decision plot (see below) shows the sensitivity (true positive rate) and specificity (true negative rate) (Y axis) over all decision levels (X axis). The point where the lines cross is the optimum decision level, where the maximum number of cases are correctly diagnosed as positive/negative. From the plot you may choose a decision level that correctly identifies a given proportion of the true positive or true negative cases.

(click to enlarge)

Interpreting the Likelihood ratio decision plot

A decision plot can be shown to help in choosing optimum decision levels in terms of likelihood ratios. 

To show the likelihood ratio decision plot: 

  1. If the ROC Curve dialog box is not visible click Edit on the Analyse-it tab/toolbar.
  2. Click Decision plot and select Likelihood ratio.
  3. Click OK.

The decision plot (see below) shows the positive (Y axis 1) and negative likelihood ratios (Y axis 2) over all decision levels (X axis). The negative likelihood ratio TODO

(click to enlarge)

Interpreting the Predictive value decision plot

A decision plot can be shown to help in choosing optimum decision levels in terms of predictive values.  

To show the predictive value decision plot: 

  1. If the ROC Curve dialog box is not visible click Edit on the Analyse-it tab/toolbar.
  2. Click Decision plot and select Predictive value.
  3. Enter Pre-test prevalence of the condition in the population as a proportion between 0 and 1. For example if the prevalence is 5% of the population enter 0.05. If the prevalence is left blank Analyse-it will use the prevalence in the sample to calculate predictive values.
  4. Click OK.

The decision plot (see below) shows the positive and negative predictive values (Y axes) over all decision levels (X axis). The point where the lines cross is the optimum decision level, where the maximum number of cases are correctly diagnosed as positive/negative.

(click to enlarge)

Interpreting the Cost decision plot

A decision plot can be shown to help in choosing optimum decision levels in terms of costs. Costs can be expressed in financial terms or in terms of the cost to the health of the subject (see Costs above).

To show the costs decision plot: 

  1. If the ROC Curve dialog box is not visible click Edit on the Analyse-it tab/toolbar.
  2. Click Decision plot and select Cost.
  3. Enter Pre-test prevalence of the condition in the population as a proportion between 0 and 1. For example if the prevalence is 5% of the population enter 0.05. If the prevalence is left blank Analyse-it will use the prevalence in the sample to calculate predictive values.
  4. Enter Cost FP (false positive cost), Cost FN (false negative cost), Cost TP (true positive cost) and Cost TN (true negative cost) in financial terms (e.g. as dollar values) or as relative costs of a misdiagnosis (with higher values indicating severity).
  5. Click OK.

The decision plot (see below) shows the cost (Y axis) over all decision levels (X axis). If costs were expressed in financial terms the cost shows the average financial cost of each diagnosis, including the cost to treat positive cases (if the Cost TP & Cost FP include treatment cost). If relative costs are specified, for example indicating the relative damage to the health of the subject, the lowest cost indicates the best decision level.

(click to enlarge)

 References to further reading

  1. Receiver-Operating Characteristic (ROC) Plots: A Fundamental Evaluation Tool in Clinical Medicine
    Zweig M.H., Campbell G. Clin Chem 1993; 39:561-77.
  2. The use of Relative Operating Characteristic (ROC) Curves in Test Performance Evaluation
    Beck J.R., Schultz E.K. Arch Pathol Lab Med 1986; 110:13-20.
  3. Assessing test accuracy and its clinical consequences: a primer for receiver operating characteristics curve analysis.
    A Ralph Henderson Ann. Clin. Biochem. 1993; 30; 521-539
  4. Meaning and Use of the Area under an ROC Curve.
    Hanley J.A., McNeil B.J. Diagnostic Radiology 1982; 143:28-36.
  5. Assessment of the Clinical Accuracy of Laboratory Tests Using Receiver Operating Characteristic (ROC) Plots.
    CLSI GP10-A 2001; ISBN 1-56238-285-3.
  6. ROC Curves in Clinical Chemistry: Uses, Misuses, and Possible Solutions.
    Nancy A. Obuchowski, Michael L. Lieber, Frank H. Wians Jr; Clin Chem. 50:7 2004; 1118-1125
  7. Statistical Methods in Diagnostic Medicine.
    Xiao-Hua Zhou, Nancy A. Obuchowski, Donna K. McClish 2002; ISBN 0-471-34772-8

(click to enlarge)