Observations: dot size = effect on the AUC
Click a dot to inspect it · drag it sideways to change its score · Del exclude/include · F flip label · R restore · ←/→ previous/next case · Ctrl+Z undo
ROC staircase
Click a step to select its case.
Sensitivity curve (original data)
All observations
Edit log
Methods & formulas
Placement values
For a Y case, V = share of N cases scored below it; for an N case, V = share of Y cases scored above it (ties count ½). V is the share of the other class that the case is correctly ordered against. The AUC is the mean placement value within either class, and the DeLong variance is s²(VY)/nY + s²(VN)/nN.
Removing one case (exact)
Removing a Y case changes the AUC by (AUC − V)/(nY − 1), and an N case by (AUC − V)/(nN − 1).
The new DeLong SE is also exact: removing a Y case leaves the other Y placements unchanged and takes one pair out of each N placement.
The required sums come from prefix sums over the sorted scores, so every case costs O(log N).
"Conclusion" means the lower limit of the logit-scale DeLong CI exceeds the chosen value.
Why the small class dominates
Moving a Y case from the bottom to the top of the ranking changes the AUC by exactly 1/nY, and an N case by 1/nN. In the ROC staircase each Y case is a vertical step of height 1/nY and each N case a horizontal step of width 1/nN.
Flipping a label (exact)
The case loses its pairs with its old class's opponents and gains pairs against its former classmates: e.g. for a Y case, AUC' = (U − nN·V + a) / ((nY − 1)(nN + 1)), where U is the number of correctly ordered pairs and a counts the other Y cases scored above it (ties ½).
Sensitivity curve and fragility
Starting from the original data, a greedy search repeatedly applies the single edit (removal, or optionally label flip) that lowers (red) or raises (green) the lower CI limit the most. Fragility is the number of steps after which the conclusion changes. Greedy search finds an achievable set of edits, so the true minimum can only be smaller: read "k edits suffice" as an upper bound.
A caution
Removing cases because they lower the AUC biases it upward, and the CI after such a selection is no longer valid. Use edits to understand how much a result depends on particular observations, and report the original analysis together with a justified, pre-specified sensitivity analysis.