The area under the ROC curve summarises a test’s discrimination without committing to a threshold. The moment the test enters routine use, someone has to draw a line and call each result positive or negative.
For most markers a higher value is the more suspicious one, so results above the line are positive and those below it negative. For some the association runs the other way, and it is a low value that signals disease, which simply mirrors the rule.
Where that line goes is not a statistical detail. The cut-off fixes the balance of missed cases against false alarms. Different clinical situations want that balance struck in very different places.
The most common rule is Youden’s index: choose the cut-off that maximises sensitivity plus specificity minus one. Geometrically it is the point on the ROC curve furthest above the diagonal, the point of greatest total correctness. Youden’s index is simple and needs nothing but the ROC data. That makes it a reasonable default when you have no strong reason to prefer catching cases over avoiding false alarms.
Even-handedness is Youden’s assumption. The index weights a false negative and a false positive equally. In medicine they rarely are equal. Missing an aggressive, treatable cancer is not interchangeable with an unnecessary follow-up scan. A rule that treats them as such will put the cut-off in the wrong place for the clinical reality.
When the two errors carry different costs, the optimal cut-off shifts. The direction is intuitive. If a missed case is far worse than a false alarm, you move the cut-off to catch more cases, accepting more false positives. Prevalence works the same way. When disease is rare, the same relative cost produces many more false positives per true case, which again moves the optimal point.
The formal version chooses the cut-off where the ROC curve’s slope equals a value set by the ratio of costs and the odds of disease. The practical version is simply to state your relative cost of the two errors and let it move the line, rather than defaulting to equal weighting without having chosen it.
Often the cleaner approach is to fix the operating characteristic the clinical role demands, and read off the other. A screening test that must not miss cases is set to a required sensitivity (say 95% or 99%) and you report the specificity that buys. A confirmatory test that must not falsely label healthy people is set to a required specificity instead. Fixing the requirement in advance makes the clinical priority explicit and the trade-off visible, rather than hiding both inside a single “optimal” number.
One mistake often inflates a large share of the cut-offs reported in the literature. Choose the threshold that maximises performance on your data, then report the sensitivity and specificity at that threshold from the same data, and both figures are biased upward. The cut-off optimises on noise in that sample. The reported performance overestimates how the test will perform. On a fresh sample the same cut-off will look worse.
The correction is to separate choosing the cut-off from estimating its performance. Select on one dataset and evaluate on another, or use cross-validation or bootstrapping to estimate the optimism and discount for it. At the very least, report the sensitivity and specificity at your chosen cut-off with their confidence intervals. With the small diseased groups typical of these studies, the operating point is often not very firmly pinned down.
The workbook below determines the optimal cut-off for an oxidised-LDL marker and reports the sensitivity and specificity at that point. Open it in the trial and move the threshold. Watch the two rates trade against each other, and see how a cut-off chosen to maximise one criterion performs against the others.
Download the CLSI EP24-A2 diagnostic performance example workbook (.xlsx) — a ROC analysis with optimal cut-off determination and the sensitivity and specificity at that point, ready to open in the Analyse-it trial.
The example workbook is downloading.
It opens in Excel on its own — the data and the finished results are both in it. Analyse-it is what lets you change the analysis and re-run it, try the same study on your own data, or work through it to see how the software handles it.
Every feature from all five editions for 15 days.
Defaulting to Youden without thinking about costs. Youden is right only when a false negative and a false positive are equally bad. Decide whether they are before you accept its cut-off.
Reporting performance at a cut-off chosen on the same data. The optimism is built in. Validate the cut-off on independent data, or estimate and discount the optimism.
Quoting a cut-off without its confidence intervals. The sensitivity and specificity at the operating point are estimates. With a small diseased group they can be very uncertain. Report the intervals.
Using one cut-off for two purposes. A screening threshold and a confirmatory threshold answer different questions. One number rarely serves both.
Analyse-it selects and reports the cut-off from your own data, inside Excel:
Every feature from all five editions for 15 days. Diagnostic performance is in the Medical, Method Validation and Ultimate editions, from US$ 340 a year. Validated against NIST and CLSI reference datasets. Start with the ROC and AUC guide, or the diagnostic performance reference guide for the technical detail.