Score distributions by class
ROC curve
How uncertain is this AUC? — bootstrap distribution (this sample)
Repeat the experiment — the AUC is a random variable
In practice you get one sample. Here we rerun the same study (same distributions, same nY, nN) many times to see how much the sample AUC moves around, and whether the confidence intervals really capture the true AUC as often as they claim.
Confusion matrix at threshold t (this sample)
AUC as a probability — the pair game
Draw one random Y case and one random N case from the sample. How often does the Y case get the higher score? (Ties count ½.) In the long run that fraction is the AUC — nothing more, nothing less.
Methods & formulas
Simulation
Scores are drawn independently: nY from P(score | Y) and nN from P(score | N), using a seeded generator (mulberry32; Box–Muller normals; Marsaglia–Tsang gammas; Beta = Gα/(Gα+Gβ)). The same seed gives the same sample. While you change parameters, the seed stays fixed, so the random numbers underneath stay the same and differences come from the parameters, not from noise. An optional monotone score transform is applied to both classes after sampling.
True (population) ROC and AUC
ROC(t) = (1 − FN(t), 1 − FY(t)) using exact CDFs (erfc, regularised incomplete gamma and beta functions). Thresholds are refined adaptively until no interval holds more than 0.05% of either class's probability, and AUC = ∫ FN dFY = P(SY > SN) is the trapezoidal area (accurate to ~1e-6). For two normals this matches the closed form Φ((μY − μN) / √(σY² + σN²)).
Sample ROC and AUC
One ROC vertex per distinct score (tied Y/N scores produce a diagonal segment). The trapezoidal area equals the Mann–Whitney statistic
P(sY > sN) + ½ P(sY = sN) over all nY·nN pairs.
DeLong confidence interval
Structural components V10(i) = fraction of N scores below Y case i, V01(j) = fraction of Y scores above N case j (ties ½);
Var(AUC) = s²(V10)/nY + s²(V01)/nN. Computed in O(N log N) with one sort per class (equivalent to the mid-rank algorithm of Sun & Xu, 2014).
The interval is formed on the logit scale, logit(AUC) ± z·SE / (AUC(1−AUC)), and mapped back, so it stays inside (0, 1).
If the sample AUC is exactly 0 or 1 the SE is 0 and the interval collapses to a point (shown as "degenerate").
Bootstrap confidence interval
Stratified: Y and N cases are resampled separately (so prevalence is fixed), B replicates, percentile interval. Each replicate costs O(N): each class is sorted once, and each replicate only redraws multiplicity weights.
Repeated experiments
Each replicate draws a fresh sample and computes the AUC and its DeLong (and optionally bootstrap) CI. Coverage = share of CIs containing the true AUC (± Monte-Carlo SE √(p(1−p)/R)). Power vs chance = share of CIs whose lower limit exceeds 0.5. "Empirical SE" is the SD of the replicate AUCs, i.e. the quantity the DeLong SE tries to estimate from a single sample.
Threshold metrics
Ŷ = Y iff s ≥ t. Sensitivity = TP/(TP+FN), Specificity = TN/(TN+FP), PPV = TP/(TP+FP), NPV = TN/(TN+FN), Youden J = Sens + Spec − 1. PPV and NPV depend on the prevalence nY/(nY+nN); the ROC and AUC don't.