Calibration

Starting with the everyday version. Let’s say you have a weather forecaster that says “30% chance of rain” for a particular day. To check calibration of this forecaster in this situation, you collect each historical 30% forecast and check: did it rain on about 30% of them? Do the same for the 80%-days, the 5%-days, the x%-days (usually with a buffer around x). If every group checks out, the forecaster’s probabilities mean what they say, and it is considered calibrated.

Two features of the definition that are important to understand.

First, it is a per-bucket measurement, not an average over the whole population. A forecaster that says 10% half the time and 90% the other half, while the event historically always occurred at a 50% rate, averages out perfectly while being badly wrong in both buckets, when looking at calibration.

Second, calibration is a property of the forecaster over repetition. A single “30%” is never right or wrong on its own; only the population of 30%-forecasts can be judged. This is why calibration can only ever be measured against history, and why this whole subject reduces to a question about how much history you have.

When history is abundant, checking calibration is exactly the bucket exercise, formalised:

When every bin holds plenty of outcomes, this works comfortably, and most of the work on this site uses it wherever the data allows. But 2 cautions apply before trusting the picture that it paints.

Now break the abundant case. You want to check a “1%” claim with 50 observations. On average, a bucket constructed to hold events from 0%-1% expects half an event, so there will be no data for you to measure against. This is a significant issue, especially when the tail of the distribution is what matters to you most, as it is sometimes the rare events that carry the most consequence.

The right move for small p is not to pretend the point estimate settles anything. It is to report the confidence interval on the coverage estimate itself, so whoever consumes the claim sees the width. “Empirical coverage 9% [3%, 21%]” and “9%” contain the same point estimate but completely different amounts of information: the first says what the data can and cannot support; the second launders a wide interval into false precision. All following sections in this article are a response to that width.

One more thing counting cannot see is the temporal dimension of the data. 10 events scattered over 10 years and 10 events in a single week produce the same count while describing completely different risks. Christoffersen’s test asks the follow-up question: do the events cluster? 2 2Christoffersen (1998) [5]: a likelihood-ratio test on the transition frequencies of the hit sequence; does a violation today make a violation tomorrow more likely? The standard companion to Kupiec in VaR backtesting. Clustering is the signature of regime-driven markets, so a coverage claim worth trusting states both facts: the count is right, and the events don’t bunch.

Section 03 left us with intervals too wide to act on. The following are 4 ways to narrow them, and they form a ladder, each rung gaining precision at the cost of a stronger assumption. Knowing which rung you are standing on, and saying so, is most of the relevant information.

Binning throws information away: it reduces a rich forecast to “which bucket was it in”. If your forecasts are full distributions (a predicted range with probabilities over it, not just one number), there is a check that uses every observation: 3 3Three modes of “calibrated” are in play across this tutorial: event-probability calibration (§ 01), quantile/coverage calibration (the counting of § 03), and full-distribution calibration, which is what PIT tests. The definitions nest; the data demands differ.

Assumption. Forecasts are full distributions (CDFs), not point probabilities, and observations independent enough to test uniformity.

If you forecast many related series (several assets, venues, time windows), test calibration jointly across all of them. This is the standard trick in weather verification [8], and it multiplies your effective sample. But a model that is fine on 9 series and broken on the 10th can hide inside the average, and because related series move together, treating their observations as independent claims more certainty than you have; the fixes (clustering the errors, block methods) exist, but must actually be used.

Assumption. Miscalibration is shared across the pooled series, and between-series dependence is handled.

For tail probabilities specifically, there is a way to change what is being validated. Instead of testing the 1% estimate against 1% events you don’t have, fit a tail model to the 5–10% of observations beyond a moderate threshold, data you do have, and let a theorem carry the shape out to the far tail. 5 5The Pickands–Balkema–de Haan theorem: for a very broad class of distributions, excesses over a high threshold converge to a generalised Pareto. Stated, explained, and priced in background/extreme-value-theory. What the extrapolation costs, and how the tail model’s own uncertainty compounds with distance, is the subject of its own tutorial, background/extreme-value-theory; § 08 covers what to do when even this rung is unavailable.

Assumption. Threshold exceedances are regular enough to follow a generalised Pareto (much weaker than trusting a full model of everything, but genuine), and the tail model’s uncertainty compounds with extrapolation distance.

Scores like CRPS and the log score compare whole forecast distributions against outcomes [9], and they converge much faster than bucket-counting. The catch: they don’t test calibration in isolation; they mix it with how decisive the forecasts are (§ 09 makes that mixing exact). So use them the way you use a smoke detector: fast, sensitive, unspecific. A model that is badly wrong in the tails usually loses on CRPS long before you have enough tail events for a counting test to say anything. The scoring rule raises the alarm; the coverage test is the inspection that follows.

Assumption. That a good score means good calibration. It does not: the score blends calibration with sharpness, so read it as a fast, unspecific alarm, not an isolated calibration test.

Every rung of the ladder still estimates calibration from history: collect data, measure, and hope the interval is narrow. Conformal prediction takes a different route entirely: build the interval so that its coverage is guaranteed by arithmetic, not estimated from outcomes.

2 honest caveats apply, and neither belongs in a footnote:

1. The guarantee is marginal: it holds on average over everything, not within every subgroup. That gap is § 06.
2. It degrades under distribution shift: if the world changes between calibration and use, the arithmetic no longer applies.

The shift problem has its own answer: adaptive variants (adaptive conformal inference [12], conformal PID control [13]) watch the realised miss rate and adjust the interval width to steer it back to target. That reverses the § 03 problem entirely: calibration stops being a fixed property you estimate from history and becomes a running property you enforce, day by day.

Calibration becomes a running property you enforce rather than a fixed property you estimate.

Everything so far audits the margin: the track record pooled over all forecasts. Here is the failure that pooling hides: suppose a system’s “10%” forecasts come true 18% of the time on one venue and 3% of the time on another, in proportions that average to 10%. The marginal audit passes perfectly. Every single user is misled, each in a different direction. Nobody consumes a forecast at the margin: a holder is on a particular venue, in a particular regime, holding a particular coin, and the calibration statement they need is the one conditional on where they actually stand.

Papers on this site that promise “group-conditional (Mondrian) coverage” are invoking exactly this section, and both halves of it must be read together. A group-conditional guarantee is only informative when it states both its groups and their sample sizes.

Suppose the audit rejects: the stated probabilities are systematically off. The good news is that systematic distortion can be undone: relearn the map from the model’s raw outputs to honest probabilities, using held-out data. The standard options, from most to least assumed:

3 caveats keep this honest:

1. Recalibration spends data: the held-out set it learns from is data the audit can no longer use.
2. It repairs only stable distortion: a map learned in one regime transfers to the next only if the distortion does.
3. It inherits both earlier problems: in the tails it faces § 03’s widths, and across subgroups it faces § 06’s question, recalibrated for whom?

A recalibrated forecaster is a new forecaster, and it queues for the same audit as the old one.

Some regimes defeat the whole ladder: the event has essentially never happened, and no pooling or tail-fitting manufactures the missing observations. What remains is a protocol, not a calculation:

Bayesian shrinkage toward a conservative prior is the same move in a different vocabulary: with 2 tail events, your posterior is mostly prior, and saying so out loud is a strength. The failure mode both vocabularies guard against is the same one: presenting a model’s opinion as if it were a measured fact.

What it lacks is sharpness: how concentrated, how decisive, the forecasts are, a property of the forecasts alone, before any outcome arrives. A sharp forecaster says 2% on most days and 85% when the front is coming in; it takes stands. The right goal, in the standard formulation due to Gneiting and colleagues [19], is to maximise sharpness subject to calibration: calibration is the constraint that keeps the probabilities honest; sharpness is the objective that makes them worth having.

The division of labour matters for how this tutorial gets used. Calibration is the licence: the property that makes a stated probability a claim about the world rather than an opinion. It is not the achievement. Papers on this site that pin here are, almost always, invoking the licence half; whether the forecasts in question are also worth having is a separate question, with a separate literature, and a separate background article when the rule of two demands it.