AUC Robustness Lab

Load your scores and labels (or an example) and see how much each observation drives the AUC and its confidence interval. Move, remove or relabel cases and watch the result change. This is a sensitivity analysis, not data cleaning: the original result stays on screen and every edit is logged.

Original AUC
—
—
Current AUC
—
—
Conclusion
—
—
Fragility
—
—
Your edits
—
—

Observations: dot size = effect on the AUC

Y case N case removal moves AUC ≥ ½ SE or changes the conclusion excluded / original position

Click a dot to inspect it · drag it sideways to change its score · Del exclude/include · F flip label · R restore · ←/→ previous/next case · Ctrl+Z undo

ROC staircase

Y case: step up 1/nY N case: step right 1/nN tied Y and N original

Click a step to select its case.

Sensitivity curve (original data)

most damaging edits most flattering edits solid = AUC, dashed = lower CI limit your edits

All observations

Edit log

Methods & formulas

Placement values

For a Y case, V = share of N cases scored below it; for an N case, V = share of Y cases scored above it (ties count ½). V is the share of the other class that the case is correctly ordered against. The AUC is the mean placement value within either class, and the DeLong variance is s²(VY)/nY + s²(VN)/nN.

Removing one case (exact)

Removing a Y case changes the AUC by (AUC − V)/(nY − 1), and an N case by (AUC − V)/(nN − 1). The new DeLong SE is also exact: removing a Y case leaves the other Y placements unchanged and takes one pair out of each N placement. The required sums come from prefix sums over the sorted scores, so every case costs O(log N). "Conclusion" means the lower limit of the logit-scale DeLong CI exceeds the chosen value.

Why the small class dominates

Moving a Y case from the bottom to the top of the ranking changes the AUC by exactly 1/nY, and an N case by 1/nN. In the ROC staircase each Y case is a vertical step of height 1/nY and each N case a horizontal step of width 1/nN.

Flipping a label (exact)

The case loses its pairs with its old class's opponents and gains pairs against its former classmates: e.g. for a Y case, AUC' = (U − nN·V + a) / ((nY − 1)(nN + 1)), where U is the number of correctly ordered pairs and a counts the other Y cases scored above it (ties ½).

Sensitivity curve and fragility

Starting from the original data, a greedy search repeatedly applies the single edit (removal, or optionally label flip) that lowers (red) or raises (green) the lower CI limit the most. Fragility is the number of steps after which the conclusion changes. Greedy search finds an achievable set of edits, so the true minimum can only be smaller: read "k edits suffice" as an upper bound.

A caution

Removing cases because they lower the AUC biases it upward, and the CI after such a selection is no longer valid. Use edits to understand how much a result depends on particular observations, and report the original analysis together with a justified, pre-specified sensitivity analysis.