Tail Risk Backtesting: When the Test Improves and the Model Does Not

A VaR model promises that losses will exceed a stated level on 1% of days. Counting exceedances tests that promise directly, and 2,172 trading days of counting is a reasonable amount of evidence. Expected shortfall is harder. It asks how bad the bad days were, and there are only about 24 of them at α = 0.01, so any statistic is an average over a couple of dozen numbers.

The Acerbi-Székely conditional statistic, which they call Z₁, takes the realised loss on each exceedance day, divides by the expected shortfall the model predicted that morning, averages over the exceedance days, and adds 1. A correctly specified model gives 0 on average. Negative means realised losses ran deeper than predicted.

The same paper defines an unconditional Z₂, which divides the same sum by the number of exceedances the model expected (Tα) instead of the number that happened. The difference matters later in this article. Z₁ averages over the exceedances themselves, so it cannot see whether there were too many of them, and Acerbi and Székely pair it with an exceedance-count test for that reason. Z₂ responds to both the number and the depth of the exceedances. § 02 to § 05 are about Z₁; § 04 also re-runs the sweep with Z₂.

Here is the subtle part: the question is what counts as too negative, and there is no table to look it up in. The standard construction simulates the answer: draw many alternative loss paths from the model’s own predicted distribution, compute Z₁ for each, and take the 5th percentile as the green threshold.

The zones in this article are defined as follows: green when the statistic is above the 5th percentile of those simulated values, red when it is below the 0.1th percentile, and yellow between the two. These are my own zones for an ES statistic. They are not the Basel traffic light, which counts VaR exceptions over 250 days.

In my implementation the null draws from a Student-t whose degrees of freedom come from the median excess kurtosis the model predicted over the reporting window, via ν = 4 + 6/κ. Read back in plain words: the fatter the tails a model claims, the fewer degrees of freedom its null gets, and the wider the reference distribution it is graded against. That is a defensible engineering choice. It is also the whole ballgame, and I did not treat it as a modelling decision when I changed it.

The May revision did 3 things. It swapped a Gaussian copula for a t-copula in the base model, so joint extreme moves carried the tail thickness of the marginals; it swapped a Gaussian null for the moment-matched Student-t, on the grounds that a Gaussian reference over-rejects when the truth is fat-tailed. And it added a fifth method.

The expected-shortfall verdict improved. My changelog recorded that the copula and the null “did most of the work”, with the copula named first, and that the plain conformal method had reached green on its own merits.

2 of those 3 changes push Z₁ toward passing, and the design does not separate them. A t-copula does not lower the bar; it widens the modelled joint tail, which moves the statistic. A Student-t null does not move the statistic; it widens the reference distribution, which lowers the bar. Both make green likelier, for entirely different reasons, and I had run them together.

Separating them needs a factorial: both copulas crossed with both nulls, everything else fixed. That takes 2 backtest runs, because the null is applied after the fact and costs nothing to vary. It was not hard. It simply was not done, because the result already looked right.

Holding the null fixed and switching only the copula moves the statistic by +0.0096, from −0.0004 to +0.0092. Holding the copula fixed and switching only the null moves the green threshold by −0.104, from +0.019 to −0.085.

The change to the test did roughly 11 times what the change to the model did.

Here is the stronger version, in the bottom-left cell. Scored against the original Gaussian null, the t-copula run is still yellow at α = 0.01 and still red at α = 0.025. The model change on its own never reached green at either level.

So I re-scored every model against 6 nulls, holding the model, the data, the VaR and the ES fixed. The statistic cannot move under this treatment. Only the threshold can.

For Z₁, across the 10 model-and-level cells (5 models at 2 confidence levels), the green counts are as follows:

• Gaussian null: 0 green, 3 yellow, 7 red.
• Student-t at df 15: 5 green.
• The shipped moment-matched t, df ≈ 8.1: 6 green.
• Student-t at df 5: 10 green.
• Vol-adjusted empirical bootstrap: 10 green.

Here is the whole sweep read through one model. For Conformal-PID at α = 0.01 the statistic is +0.0092 throughout. The threshold runs from +0.019 under a Gaussian null, where the verdict is yellow at p = 0.025, to −0.085 under the shipped null, where it is green at p = 0.656, to −0.190 at df 5, where p = 0.942. Same number, 3 different published answers.

With the conditional statistic, the zone grades the reference distribution someone chose for the model, not the model itself.

The obvious question is whether this is a property of ES backtesting or of the statistic I used. Acerbi and Székely report that the critical value of Z₂ stays roughly fixed across Student-t tails from 3 to 100 degrees of freedom, which would make it much less exposed to the choice of null. So I re-ran the sweep with Z₂, scored on the same simulated paths, and added 2 models I know are broken. Switch the statistic in Fig. 02 to see it.

The re-run gives 3 results:

1. The threshold of Z₂ moves as much as the threshold of Z₁. For Conformal-PID at α = 0.01 it runs from −0.353 under the Gaussian null to −1.052 at df 5. That does not contradict Acerbi and Székely, whose result assumes the null is the model’s exact predictive distribution. Replicated in this code with exact nulls over 2,172 days, the 5th percentile of Z₂ stays between −0.44 and −0.36 from df 3 to Gaussian at α = 0.01, while that of Z₁ moves from −0.23 to −0.04. The nulls in this sweep are moment-matched approximations whose breach rates do not match the model’s own VaR, and Z₂, which responds to the number of breaches, moves with that mismatch.
2. The verdicts of Z₂ barely move. Under each of the 5 parametric nulls, the same 6 of the 10 cells are green. The other 4 are red, or yellow at df 5. Those 4 are the Historical and Parametric baselines, which the exceedance-count test also rejects (p ≤ 0.015). They miss the threshold by more than the threshold travels.
3. The empirical null passes everything under both statistics. That includes a model breaching at 2.5 times its stated rate. Its residual pool is built from the realised series, so a frequency error in the model is copied into the reference.

The 2 broken models separate what each statistic can see:

• VaR × 0.8. Conformal-PID with its VaR, ES and predicted scale all multiplied by 0.8. It breaches 54 times against about 22 expected, and the exceedance-count test rejects it at p < 0.001. Z₂ puts it in red under all 5 parametric nulls. Z₁ is −0.035 against Conformal-PID's +0.009 and passes under the shipped null at p = 0.276, because it averages over the breaches and cannot see how many there were.
• Gaussian tail. A Gaussian whose VaR matches Conformal-PID's, so it breaches on the same 24 days, but whose ES is the thinner Gaussian one. Z₂ passes it under all 6 nulls. Z₁ flags it only under its own Gaussian null, yellow at α = 0.01 and red at α = 0.025, and the Gaussian null puts the working Conformal-PID model in exactly the same zones. The flag does not discriminate.

The headline therefore holds in a narrower form than version 1.0 stated. Under both statistics the threshold depends on the null. The verdict depended on it heavily for Z₁ and hardly at all for Z₂ in this data, because the models Z₂ rejects breach too often by a wide margin. Neither statistic, scored against a simulated null, caught the thin-tailed model with the right breach rate.

The degrees of freedom in the null come from the model’s own predicted excess kurtosis. A model claiming fat tails produces a wide null and a threshold far below zero. A model claiming Gaussian tails produces a tight one and a threshold close to zero.

The Parametric EWMA baseline makes this stark. It predicts a Gaussian portfolio distribution, so its median predicted excess kurtosis is exactly 0, and the guard that requires κ > 0.1 sends it back to a Gaussian null. Switch panels in Fig. 02 to see it.

• Conformal-PID, claimed κ = 1.47, null df 8.08, green threshold −0.0847.
• Parametric EWMA, claimed κ = 0.00, Gaussian null, green threshold −0.0451.

Here is what that pair of thresholds measures: not how the 2 models performed, but the kurtosis each of them claimed. The model claiming fatter tails is judged against a threshold about twice as permissive.

Under the vol-adjusted empirical null, all 10 cells are green for Z₁. That includes the Parametric baseline, which fails the exceedance-count test at p < 0.001 and breaches at 2.07% against a 1% target. Its Z₁ of −0.167 sits comfortably inside a realistically shaped null: threshold −0.233, p = 0.464.

Version 1.0 of this paper read that as a limit of sample size, and that reading was wrong. Z₁ averages over the exceedance days, so it cannot detect an excess number of them at any sample size. Acerbi and Székely say so, and pair it with an exceedance-count test for exactly this reason. Here the pair worked as designed: the count test rejected the Parametric baseline, which Z₁ was never meant to do. The VaR × 0.8 model in § 04 shows the same thing under controlled conditions.

What is left for Z₁ to detect is tail depth, and there the sample size does bind. With 17 to 79 exceedances, Z₁ is an average over a few dozen numbers. In § 04 it could not separate the thin-tailed Gaussian model from the working Conformal-PID model: it flagged both under the Gaussian null and passed both under the df 5, df 8 and empirical nulls. The red and yellow Z₁ zones in § 04 are generated mainly by the narrowness of the parametric null, not by detected misspecification.

Passing the ES test is not evidence that the model is adequate.

The obvious temptation was to drop the metric, or to quietly fix the attribution before anything shipped. I did neither. The zone stays in the code, demoted to a weak consistency check with this sensitivity attached to it, and the verdict was tested against its removal: the same models pass with or without that gate. The gate was never carrying anything.

The methodology manual ships carrying the corrected attribution under a heading saying what the earlier conclusion got wrong, and a tuning register recording, per choice, what was selected on which data. Here is its worst row: the May revision was made after seeing an unfavourable result on the same dataset, which cannot be distinguished from specification search using that dataset alone. Shipping the register is the point. A methodology that only records the decisions that flatter it is not a record.