Tail Risk Backtesting: When the Test Improves and the Model Does Not
I changed 3 things about my own backtest at once, the expected-shortfall diagnostic improved, and my changelog credited the model. Running the experiment that separates them shows the reference distribution did about 11 times the work. Re-scored against 6 nulls with every statistic held fixed, the same 5 models go from 0 passes to 10.
Version history
- v0.1 · 2026-08-26 · first draft, from the de-confounding factorial and null sensitivity runs of 2026-08-25
- v1.0 · 2026-09-02 · first public release. Content unchanged from 0.1; the version marks the paper as issued rather than circulating for review.
A validation metric can improve because the model improved or because the reference it is graded against moved, and nothing in the improvement itself distinguishes the two. This audits a case where it was the second. Three results. First, a factorial separating a copula change from a reference change shows the copula moved the statistic by +0.0096 while the reference moved the acceptance threshold by -0.104, so the test did roughly 11 times the work of the model; against the original reference the improved model still fails. Second, re-scoring 5 methods at 2 alpha levels against 6 references, with every statistic held fixed, moves the count from 0 passes to 10. Third, under a distribution-free reference nothing is rejected at all, including a baseline that breaches at twice its stated rate, which bounds what the test can detect at 24 to 79 exceedances.
- Calibration background/calibration · v1.1 · tutorial
I have been building a backtest of 5 tail-risk models. In May I changed 3 things about it at once and the expected-shortfall diagnostic improved sharply. My development log recorded that the change to the model had done the work.
That attribution was wrong. The factorial that shows it took 2 runs and had been sitting there the whole time.
Separating the changes shows the improvement came almost entirely from the reference distribution I was grading against, not from the model I had credited. The statistic barely moved. The bar moved.
None of this is exotic. The test is standard and the null is a reasonable choice. The trap is available to anyone who has ever improved a metric and assumed the thing being measured improved with it. I am writing it up because a moved threshold leaves no trace in the statistic, and because the check that catches it costs 2 runs.
The argument runs in the order the evidence forced: what the factorial separates (§ 02), how far the verdict moves when only the reference changes (§ 03), why the construction rewards claiming fat tails (§ 04), and what the test can detect at all (§ 05). § 06 is what I changed as a result, including the part where I had already tried to correct one piece of this and got the direction backwards.
A Test That Grades Its Own Reference
Expected shortfall is hard to test, and the standard answer simulates from the model being tested. That is correct; it is also where the trouble starts.
A VaR model promises that losses will exceed a stated level on 1% of days. Counting exceedances tests that promise directly, and 2,172 trading days of counting is a reasonable amount of evidence. Expected shortfall is harder. It asks how bad the bad days were, and there are only about 24 of them at α = 0.01, so any statistic is an average over a couple of dozen numbers.
The Acerbi-Székely Z₂ statistic takes the realised loss on each exceedance day, divides by the expected shortfall the model predicted that morning, averages, and adds 1. A correctly specified model gives 0 on average. Negative means realised losses ran deeper than predicted.
Here is the subtle part: the question is what counts as too negative, and there is no table to look it up in. The standard construction simulates the answer: draw many alternative loss paths from the model’s own predicted distribution, compute Z₂ for each, and take the 5th percentile as the green threshold.
The reference distribution is generated from the model under test. That is not a flaw. The hypothesis being tested is “the distribution this model claimed is the right one”, so the null has to encode the model’s claim. A specification test necessarily conditions on the specification.
The flaw is elsewhere: the simulated null is not the model’s actual predictive distribution. It is a moment-matched approximation to it, and the approximation has a free parameter.
In my implementation the null draws from a Student-t whose degrees of freedom come from the median excess kurtosis the model predicted over the reporting window, via ν = 4 + 6/κ. Read back in plain words: the fatter the tails a model claims, the fewer degrees of freedom its null gets, and the wider the reference distribution it is graded against. That is a defensible engineering choice. It is also the whole ballgame, and I did not treat it as a modelling decision when I changed it.
3 Changes at Once, and the One I Credited
A t-copula, a Student-t null, and a new method shipped in the same revision. 2 of them push the verdict the same way.
The May revision did 3 things. It swapped a Gaussian copula for a t-copula in the base model, so joint extreme moves carried the tail thickness of the marginals; it swapped a Gaussian null for the moment-matched Student-t, on the grounds that a Gaussian reference over-rejects when the truth is fat-tailed. And it added a fifth method.
The expected-shortfall verdict improved. My changelog recorded that the copula and the null “did most of the work”, with the copula named first, and that the plain conformal method had reached green on its own merits.
2 of those 3 changes push Z₂ toward passing, and the design does not separate them. A t-copula does not lower the bar; it widens the modelled joint tail, which moves the statistic. A Student-t null does not move the statistic; it widens the reference distribution, which lowers the bar. Both make green likelier, for entirely different reasons, and I had run them together.
The reasoning that felt obvious at the time: the number got better, the changes were principled, therefore the changes worked. Each step is fine. The conclusion does not follow, because “the number got better” is consistent with the model improving and with the test getting easier, and nothing in the observation distinguishes them.
Here is the tell, and it was sitting in the changelog the whole time: the null was one of the things I changed, and a changed null changes the verdict without touching the model.
Separating them needs a factorial: both copulas crossed with both nulls, everything else fixed. That takes 2 backtest runs, because the null is applied after the fact and costs nothing to vary. It was not hard. It simply was not done, because the result already looked right.
The Null Did 11 Times the Work
Reading down a column isolates the model change. Reading across a row isolates the test change. The two effects are not the same size, and not by a little.
copula, down a column +0.0096 moves the statistic
null, across a row -0.1039 moves the threshold
Both changes shipped together, and the improvement was credited to the copula. The column comparison is the copula's actual contribution. Against the original Gaussian null the t-copula run is still yellow, so the model change on its own never reached green.
Holding the null fixed and switching only the copula moves the statistic by +0.0096, from −0.0004 to +0.0092. Holding the copula fixed and switching only the null moves the green threshold by −0.104, from +0.019 to −0.085.
The change to the test did roughly 11 times what the change to the model did.
Here is the stronger version, in the bottom-left cell. Scored against the original Gaussian null, the t-copula run is still yellow at α = 0.01 and still red at α = 0.025. The model change on its own never reached green at either level.
1The copula change is not worthless. It is a better description of joint tail behaviour and I would make it again. The claim here is narrower and only about attribution: it did not cause the result I attributed to it.6 Nulls, 10 Verdicts
If the threshold is doing the work, it is worth seeing how far it can be pushed by choices that all look reasonable.
So I re-scored every model against 6 nulls, holding the model, the data, the VaR and the ES fixed. The statistic cannot move under this treatment. Only the threshold can.
The dashed rule is the observed statistic. It is identical in all 6 rows, because the model, the data, the VaR and the ES are all held fixed. Only the shaded reject region moves, and it is what decides the verdict on the right. Conformal-PID predicts a median excess kurtosis of 1.47.
Across the 10 cells, 5 models at 2 confidence levels:
- Gaussian null: 0 green, 3 yellow, 7 red.
- Student-t at df 15: 5 green.
- The shipped moment-matched t, df ≈ 8.1: 6 green.
- Student-t at df 5: 10 green.
- Vol-adjusted empirical bootstrap: 10 green.
Here is the whole sweep read through one model. For Conformal-PID at α = 0.01 the statistic is +0.0092 throughout. The threshold runs from +0.019 under a Gaussian null, where the verdict is yellow at p = 0.025, to −0.085 under the shipped null, where it is green at p = 0.656, to −0.190 at df 5, where p = 0.942. Same number, 3 different published answers.
The zone is not a verdict on the model; it is a verdict on the reference distribution someone chose for it.
The empirical bootstrap passes everything, and it is tempting to call it the neutral choice because it makes no parametric assumption about shape. It is not neutral. Its residual pool is drawn from the same realised series the test then evaluates, so it is distribution-free about shape but not independent of the data. That is a different circularity, not an absence of one.
It is reported here because it brackets the parametric nulls from the permissive side, which is useful. It is not the correct null. There is no correct null on the list.
The Test Rewarded the Claim, Not the Performance
Because the null is calibrated from the model's own predicted kurtosis, a model that claims fatter tails is graded more leniently.
The degrees of freedom in the null come from the model’s own predicted excess kurtosis. A model claiming fat tails produces a wide null and a threshold far below zero. A model claiming Gaussian tails produces a tight one and a threshold close to zero.
The Parametric EWMA baseline makes this stark. It predicts a Gaussian portfolio distribution, so its median predicted excess kurtosis is exactly 0, and the guard that requires κ > 0.1 sends it back to a Gaussian null. Switch panels in Fig. 02 to see it.
- Conformal-PID, claimed κ = 1.47, null df 8.08, green threshold −0.0847.
- Parametric EWMA, claimed κ = 0.00, Gaussian null, green threshold −0.0451.
Here is what that pair of thresholds measures: not how the 2 models performed, but the kurtosis each of them claimed. The model claiming fatter tails is judged against a threshold about twice as permissive.
An early version of my methodology carried a caveat saying exactly this: a model understating tail thickness gets an artificially easy test. A later revision withdrew it. My argument was that a thinner claimed tail tightens the threshold, which makes the test harder, not easier, so the caveat had the sign backwards.
That argument identifies a real channel. There is a second channel running the other way: understating tails shrinks the ES denominator, which pushes the observed Z₂ down and makes failure likelier. I asserted the second dominated. The measurement says the first does, and by a wide margin.
The original caveat was right, my withdrawal of it was wrong, and it is reinstated. I am recording this because the failure mode is the same one as the article’s: I reasoned about which effect was larger instead of measuring it, twice, in opposite directions.
What the Test Can Actually Detect, and What I Changed
The most useful result is not about bias. It is that at these exceedance counts the test barely discriminates at all.
Under the vol-adjusted empirical null, all 10 cells are green. That includes the Parametric baseline, which fails the exceedance-count test at p < 0.001 and breaches at 2.07% against a 1% target. Its Z₂ of −0.167 sits comfortably inside a realistically shaped null: threshold −0.233, p = 0.464.
A model breaching at twice its stated rate is not a subtle failure. The coverage test sees it immediately. The shortfall test, once its reference has a realistic shape, cannot see it at all.
With 24 to 79 exceedances, Z₂ is an average over a few dozen numbers, and its sampling distribution is wide enough to swallow that breach whole. The red and yellow zones on the scoreboard are generated mainly by the narrowness of the parametric null, not by detected misspecification. That is not a test grading a model; it is a test grading its own reference.
3 questions, none of which need new data.
- What is the reference? If a verdict comes from comparing a statistic to a threshold, write down where the threshold came from. If it was estimated, it is a model, and it has assumptions.
- Did I change the reference and the thing being measured in the same revision? If so the attribution is unidentified. The fix is a factorial, and if the reference is applied after the fact it is nearly free.
- What does the test do to a model I know is broken? Score a deliberately bad model. If the test passes it, the test is not carrying the weight you are putting on it. This is the single cheapest check here and it is the one that produced the § 06 finding.
The obvious temptation was to drop the metric, or to quietly fix the attribution before anything shipped. I did neither. The zone ships, demoted to a weak consistency check with this sensitivity linked from it, and the verdict was tested against its removal: the same models pass with or without that gate. The gate was never carrying anything.
The methodology document ships carrying the corrected attribution under a heading saying what the earlier conclusion got wrong, and a tuning register recording, per choice, what was selected on which data. Here is its worst row: the May revision was made after seeing an unfavourable result on the same dataset, which cannot be distinguished from specification search using that dataset alone. Shipping the register is the point. A methodology that only records the decisions that flatter it is not a record.
This is one implementation scored on one dataset. 2 things it does not license.
It does not show the Acerbi-Székely test is bad; the construction here follows the standard recipe, and the weakness is in a moment-matched approximation I chose. It does not show ES backtesting is hopeless either. It does show what 2,172 days buys at α = 0.01: not enough to defend an adequacy claim, and the way out is far more data or a pooled cross-sectional design.
What generalises is narrower and duller. Every adaptive component in a stack is a place where the score can move without the model moving, and a reference distribution estimated from the data is an adaptive component whether or not anyone calls it one.
Every figure here regenerates from public price data with 2 commands, seeded. The live scoreboard, the full methodology including the parts that do not flatter it, and the reproduction instructions are at var-backtest.mtanti.com.
If you run a model whose uncertainty estimates are load-bearing, Recipe 6.1 is where I would start. I take this work independently. Get in touch.
@article{tanti2026,
author = {Matthew Tanti},
title = {Tail Risk Backtesting: When the Test Improves and the Model Does Not},
journal = {Kwantil},
year = {2026},
number = {KW-2026-04},
version = {1.0},
url = {https://kwantil.com/papers/did-the-model-get-better/}
}Matthew Tanti (2026). Tail Risk Backtesting: When the Test Improves and the Model Does Not. Kwantil, KW-2026-04, v1.0.