Tail-Risk Backtesting, in One Page

The consequence

If your expected-shortfall backtest compares a statistic to a simulated threshold, the pass or fail you report can depend on a modelling choice inside the test that nobody recorded as one. In the case audited here, a model moved from fail to pass and the change log credited the model. The model had barely moved. The test had.

A validation pack that reports the zone without the reference distribution beside it is reporting the reference.

3 findings, in the order a committee would ask

1. The same models pass or fail depending on the reference. 5 models at 2 confidence levels give 10 verdicts. With the models, the data, the VaR and the ES held fixed, the conditional ES statistic passes 0 of 10 under a Gaussian reference, 6 under the reference as shipped, and 10 under a Student-t with 5 degrees of freedom. Sample: 2,172 trading days, 17 to 79 breach days per verdict, 2,000 simulated paths per reference.

2. The improvement was credited to the wrong change. One revision changed the model and the reference together. Separated, the model change moved the statistic by 0.0096 and the reference change moved the pass threshold by 0.104, about 11 times as far. Against the original reference the improved model is still yellow. These are point values from seeded runs; no interval is attached to the ratio.

3. A pass is not evidence that the tail is right. A model built with the correct number of breaches and a deliberately thin tail passes the unconditional statistic at both levels (p = 0.18 and 0.24). The conditional statistic cannot see an excess number of breaches at all, by construction, and has to be read with an exceedance count.

In money, as an illustration

The breach count is where a backtest meets capital, for a firm with internal-model permission. On the same data, the Parametric model's 99% VaR was breached on 2.07% of days [1.43%, 2.77%] and sat at 5 or more overshootings in 54% of 1,923 rolling 250-day windows [28%, 86%]. On an assumed EUR 100 million book, the resulting addend raised the VaR component of the own funds requirement by EUR 1.6m on average [EUR 0.6m, EUR 2.6m].

The book value is an assumption, only the VaR component is computed, and the rule applied is the one in force until 1 January 2027. The less comfortable half of the result: with the penalty included, that model still carried EUR 3.5m less than the model that held its level. The working, the rule text and its version are in the specimen report.

3 questions to put to your own backtest

None of them needs new data.

Where does the threshold come from? If it was simulated or estimated, it is a model, and it has assumptions and a change log.
Did the reference and the model ever change in the same revision? If so, the attribution is unidentified until a 2-by-2 separates them.
What does the test do to a model you know is broken, one kind of breakage at a time? If it passes, it is not carrying the weight placed on it.

What this does not support

One implementation on one three-asset book. It does not show that the Acerbi-Székely tests are bad tests: with a model's exact predictive distribution as the reference, the threshold moves much less. It does not show that your backtest has this problem, only how to find out. The euro figures are an illustration and apply only where the multiplication factor does. Nothing here is a certification or a regulatory opinion.

If you want it run on your model

The Model Health Check is this review on your own series, with the specimen as the format of what arrives. Reference price: fixed, set by the number of models. About EUR 6,500 for one model with its backtest. Timing: 3 weeks from data delivery. One paragraph describing the model and its backtest is enough to start: matthew@mtanti.com.