ROC AUC Calculator

Measure How Well Your Scores Separate Two Classes

ROC AUC Formula

Formula

\[ \text{AUC} = \frac{\#\{\text{pairs with } s_+ > s_-\} + \tfrac{1}{2}\,\#\{\text{pairs with } s_+ = s_-\}}{P \cdot N} \] \[ G = 2\,\text{AUC} - 1, \qquad J = \text{sensitivity} + \text{specificity} - 1 = \text{TPR} - \text{FPR} \]

Where:

\(P\) — number of positive cases (label 1)
\(N\) — number of negative cases (label 0)
\(s_+\) — score of a positive case; \(s_-\) — score of a negative case
\(G\) — Gini coefficient (Somers' D), a rescaling of the AUC
\(J\) — Youden's J statistic, used to find the best cutoff

The AUC is the probability that a randomly drawn positive case receives a higher score than a randomly drawn negative case (ties count as half a win). It equals the trapezoidal area under the ROC curve and the Mann–Whitney U statistic divided by \(P \cdot N\).

ROC AUC – Calculation Example

Example: 10 cases, 5 positive and 5 negative, with the default scores. The positive scores are 0.9, 0.85, 0.8, 0.75, 0.6 and the negative scores are 0.7, 0.4, 0.3, 0.2, 0.1.

There are \(P \cdot N = 25\) positive–negative pairs. In 24 of them the positive case scores higher; the only mis-ranked pair is (0.6, 0.7). No pairs are tied, so:

\[ \text{AUC} = \frac{24 + \tfrac{1}{2}\cdot 0}{5 \times 5} = 0.96 \]

Result: AUC = 0.96, Gini coefficient \(2 \times 0.96 - 1 = 0.92\). The best cutoff by Youden's J is score ≥ 0.6 (sensitivity 100%, specificity 80%, \(J = 0.8\)). At the default threshold 0.5 the confusion matrix is TP = 5, FP = 1, FN = 0, TN = 4 (accuracy 90%).

How to use the calculator

Paste your Actual class labels (0 or 1) into the first field — one label per case, separated by spaces, new lines or semicolons. Use 1 for the positive class and 0 for the negative class.
Paste your Prediction scores into the second field, in the same order as the labels. Any numeric scale works (probabilities, model outputs, ratings); decimal commas are accepted.
Optionally change the Threshold for the confusion matrix (default 0.5). A case is predicted positive when its score is ≥ this cutoff. The threshold does not affect the AUC.
Read the results: the Area under the ROC curve, the Gini coefficient, the best cutoff (Youden's J), the confusion matrix at your threshold, and a table of ROC curve points you can plot.

The two lists must have the same length, and the data must contain at least one positive and one negative case — otherwise no ranking can be evaluated.

What is the ROC AUC?

The ROC curve (Receiver Operating Characteristic) shows how a binary classifier performs at every possible cutoff. Each point on the curve is a (false-positive rate, true-positive rate) pair obtained by classifying all cases with score ≥ some cutoff as positive. Sweeping the cutoff from high to low traces the curve from the origin (nothing predicted positive) to the top-right corner (everything predicted positive).

The AUC — the area under that curve — summarises the whole curve in one number. It has a direct probabilistic meaning: the chance that a randomly chosen positive case scores above a randomly chosen negative case. A perfect ranking gives AUC = 1; a useless (random) ranking gives AUC = 0.5; a consistently inverted ranking gives AUC < 0.5, which means you should reverse the direction of your scores.

A useful rough guide: 0.9–1 excellent, 0.8–0.9 good, 0.7–0.8 fair, 0.6–0.7 poor, 0.5 no better than chance. The Gini coefficient \(2\cdot\text{AUC}-1\) rescales this so that 0 means chance and 1 means perfect; it is common in credit scoring, where it is also called Somers' D.

Unlike accuracy or precision, the AUC is threshold-free: it measures how well the scores rank cases, independent of any cutoff you might later choose. That makes it the natural metric for comparing scoring models, but it also means it says nothing about calibration — a score of 0.8 need not correspond to an 80% probability. If you need confusion-matrix metrics at a specific cutoff, pair this tool with the Confusion Matrix calculator.

Choosing a cutoff

The calculator reports the cutoff that maximises Youden's J statistic, \(J = \text{sensitivity} + \text{specificity} - 1\) — the point on the ROC curve farthest from the diagonal. It is a sensible default, but not always the operationally best choice. In medical screening you may want high sensitivity and accept more false positives; in a confirmatory test or fraud flagging with costly investigations, specificity may matter more. Use the ROC points table to see the sensitivity/specificity trade-off at every distinct score and pick the cutoff that fits your costs.

Because the AUC is a ranking measure, it is closely related to other rank statistics. If you are analysing rank agreement more generally, the Kendall's Tau calculator covers concordant versus discordant pairs between two variables.

Assumptions, limitations and tips

Binary outcomes only. Labels must be exactly 0 or 1. For multi-class problems compute one AUC per class (one-versus-rest).
Independence. Each case contributes one label and one score, and cases are assumed independent. Repeated measurements of the same subject violate this.
Ties are handled by counting each tied positive–negative pair as half a success, which matches the trapezoidal AUC and the Mann–Whitney U convention.
Small samples. With few cases the AUC has a wide confidence interval; a Hanley–McNeil or DeLong interval is worth reporting alongside it. The default 10-case example is for illustration only.
Class imbalance. The AUC is insensitive to the positive/negative mix, which can be an advantage or a blind spot — a model can have a high AUC while missing many actual positives. Check the confusion matrix and metrics such as the odds ratio to see the practical impact.
AUC < 0.5 means your scores are inverted: either use "score ≤ cutoff" logic or swap the labels.

Frequently Asked Questions

How do I calculate the AUC by hand without any software?

List every positive score and every negative score, then compare all positive–negative pairs. Count how many pairs have the positive score strictly higher; tied pairs count as 0.5. Divide by the total number of pairs. With 4 positives and 6 negatives there are 24 pairs, so if 21 rank correctly and 2 tie, \(\text{AUC} = (21 + 1)/24 \approx 0.92\). This is exactly the Mann–Whitney U statistic divided by \(P \cdot N\).

Can the AUC be greater than 1 or negative?

No. The AUC is a probability, so it always lies between 0 and 1. If your manual calculation gives something outside that range, you have miscounted pairs or mislabeled classes. An AUC below 0.5 is still valid mathematically — it just means your scores rank in the wrong direction, and reversing the comparison (treating lower scores as more positive) turns it into 1 − AUC.

How many cases do I need for a reliable AUC?

There is no fixed minimum, but precision grows slowly. With 10 positives and 10 negatives the AUC moves in steps of 1/100, and the 95% confidence interval can easily span 0.2 or more. As a rule of thumb, aim for at least 20–30 cases per class for a rough estimate, and report a DeLong or Hanley–McNeil confidence interval so readers can judge the uncertainty rather than trusting a single number.

Is a higher AUC always a better model?

Not necessarily. The AUC only measures ranking quality, so a model can score highly while performing poorly in practice — for example, with rare positives it may still miss many actual cases while keeping a strong AUC. It also ignores calibration and the real costs of errors. Always pair the AUC with a confusion matrix at a realistic cutoff, and consider precision, recall, or lift for your specific decision problem.

Why does my AUC not change when I change the threshold?

That is expected. The AUC summarises the model across all possible cutoffs at once, so no single threshold can affect it. Changing the threshold only moves you along the ROC curve, changing which point you operate at — and therefore the confusion matrix, accuracy, precision and recall. Use the threshold field to explore that trade-off, and the AUC to compare models before choosing one.

How is the Gini coefficient related to the AUC?

It is a linear rescaling: \(G = 2\times\text{AUC} - 1\). Chance performance (AUC 0.5) maps to Gini 0, and a perfect ranking maps to Gini 1. Credit-scoring and economics publications often quote the Gini or the equivalent Somers' D, while machine learning papers use the AUC — they carry identical information, so converting between them is just this one formula.

Can I use this calculator for more than two classes?

Not directly — the ROC curve is defined for binary outcomes, so labels must be 0 or 1. For a multi-class problem, compute one AUC per class using a one-versus-rest scheme: treat that class as 1, merge all other classes into 0, and run the calculation separately. To summarise the result, average the per-class AUCs, typically with equal weights (macro average) or weighted by class frequency.