VaR and expected-shortfall backtest: a three-asset book

2 of the 4 VaR models were breached more often than the level they state allows, and the conditional expected-shortfall test passes 0 of 10 model-and-level cells under one reference distribution and 10 of 10 under another. GARCH with a t-copula also misses its stated level, in the cautious direction: at the 97.5% level it was breached on 1.84% of days against 2.5% stated, which is rated Significant. The unconditional test passes the same 6 of 10 cells under every parametric reference, and it passes a model built with too thin a tail. Of 7 findings, 5 are Material, 1 is Significant and 1 is an Observation. On an illustrative EUR 100,000,000 book, the breach record of Parametric EWMA raised the VaR component of the own funds requirement by EUR 1.621m on average [EUR 0.629m, EUR 2.643m]. With that penalty included it still held EUR 3.522m less than the model that kept its level.

ID	Finding	Severity	Own funds effect, illustration
F-01	Parametric EWMA VaR at the stated 99.0% level was breached on 2.07% of days, against 1.0% stated	Material	EUR 1.621m
F-02	Historical simulation VaR at the stated 99.0% level was breached on 1.75% of days, against 1.0% stated	Material	EUR 1.623m
F-03	The conditional ES test passes 0 of 10 model-and-level cells under one reference and 10 under another	Material	—
F-04	The unconditional ES test passes the same 6 of 10 cells under every parametric reference and passes a thin-tailed model with the right breach count	Material	—
F-05	The reference distribution was changed in the same revision as the model	Material	—
F-06	GARCH with a t-copula VaR at the stated 97.5% level was breached on 1.84% of days, against 2.5% stated	Significant	none
F-07	Conformal-PID VaR at the stated 99.0% level was breached on 1.10% of days, against 1.0% stated	Observation	EUR 0.382m

The last column is the average rise in the VaR component of the own funds requirement caused by the backtesting addend, on an assumed EUR 100,000,000 book. It is an illustration, not evidence: the public book has no euro value. The rule, its version and its scope are under Sources.

What a review did not look at is the half a reader cannot infer, so it is stated before the findings.

Series integrity, checked before anything is scored

A backtest of a series that is not a forecast and its own outcome grades nothing. Every report opens with the same checks: the breach flag matches the two series, the P&L and the VaR are on the same scale, and the stated ES is at least as deep as the stated VaR. All 5 models pass.

Model	Flag matches	Mean |P&L| over mean |VaR|	Days ES shallower than VaR	Verdict
Parametric EWMA	yes	0.34	0	passes
Historical simulation	yes	0.30	0	passes
GARCH with a t-copula	yes	0.27	0	passes
Conformal-PID	yes	0.25	0	passes
Conformal-PID, joint VaR and ES	yes	0.25	0	passes
Control: Conformal-PID, joint VaR and ES, as released in 0.1.x	yes	0.33	1,880 (87%)	fails

The control row is a real failure, not an illustration. It is this check run on the previous release of the joint model, which stated an expected shortfall shallower than its own VaR on 1,880 of 2,172 days at its worst level. An expected shortfall is the average loss beyond the VaR, so that pair is impossible. The first run of this report found it, the model was corrected, and the old output is kept because a check nobody has seen fail is a check nobody should trust. Under the rule in the method note it would have been recorded Material.

The scale ratio must lie between 0.10 and 0.70; a P&L in percent against a VaR in fractions would sit near 30. The shallower-day count is at the model's worst audited level.

Breach frequency against the level each VaR states

Intervals are 95% month-block bootstrap, resampling whole calendar months 2,000 times. Independence is the Christoffersen test with a Monte Carlo p-value. A VaR fails when the stated rate falls outside the interval or independence is rejected at 0.05.

Model	States	Breaches	Stated	Rate	95% interval	Independence p	Severity
Parametric EWMA	99.0%	45	21.7	2.07%	[1.43%, 2.77%]	0.040	Material
Parametric EWMA	97.5%	79	54.3	3.64%	[2.80%, 4.57%]	0.527	Significant
Historical simulation	99.0%	38	21.7	1.75%	[0.98%, 2.57%]	0.001	Material
Historical simulation	97.5%	73	54.3	3.36%	[2.35%, 4.56%]	0.012	Significant
GARCH with a t-copula	99.0%	17	21.7	0.78%	[0.42%, 1.20%]	0.051	Observation
GARCH with a t-copula	97.5%	40	54.3	1.84%	[1.33%, 2.39%]	0.829	Significant
Conformal-PID	99.0%	24	21.7	1.10%	[0.69%, 1.58%]	0.573	Observation
Conformal-PID	97.5%	53	54.3	2.44%	[1.84%, 3.10%]	0.149	Observation

Breaches by year at the 99% level

The table above averages over 9 calendar years, and a risk committee lives in one of them at a time. A 99% VaR states about 2.5 breaches in a full year.

Model	2018	2019	2020	2021	2022	2023	2024	2025	2026
Parametric EWMA	5	0	9	7	7	2	5	4	6
Historical simulation	4	6	7	0	7	1	5	4	4
GARCH with a t-copula	3	0	2	2	1	1	3	3	2
Conformal-PID	2	2	3	2	1	3	6	3	2

2026 is a part year of 161 trading days.

Time spent outside the green zone

The supervisory rule counts overshootings of the one-day 99% VaR over the most recent 250 business days. Fewer than 5 carries no addend. From 5 upward an addend is added to the multiplication factor of 3. A correctly stated model, with independent breaches, would still show 5 or more in 10.8% of windows. The period holds 1,923 complete windows, of which only 8 do not overlap, so the intervals are wide.

Model	No addend	5 to 9	10 or more	Outside green	95% interval	Highest count
Parametric EWMA	46%	51%	3%	54%	[28%, 86%]	11
Historical simulation	55%	39%	7%	45%	[13%, 72%]	13
GARCH with a t-copula	100%	0%	0%	0%	[0%, 19%]	4
Conformal-PID	84%	16%	0%	16%	[0%, 38%]	7

What the addend costs, as an illustration

The euro figures below are an illustration, not evidence. The breach counts and the VaR series are measured. The book value is assumed at EUR 100,000,000, the one-day VaR is scaled to 10 days by the square root of time, and only the VaR component of the requirement is computed: the higher of the previous day's VaR and the 60-day average multiplied by the factor.

Model	Without addend	With addend	Added by the addend	95% interval	Peak
Parametric EWMA	EUR 13.928m	EUR 15.550m	EUR 1.621m	[EUR 0.629m, EUR 2.643m]	EUR 5.936m
Historical simulation	EUR 15.705m	EUR 17.328m	EUR 1.623m	[EUR 0.296m, EUR 2.576m]	EUR 8.418m
GARCH with a t-copula	EUR 17.314m	EUR 17.314m	EUR 0.000m	[EUR 0.000m, EUR 0.525m]	EUR 0.000m
Conformal-PID	EUR 18.690m	EUR 19.072m	EUR 0.382m	[EUR 0.000m, EUR 1.138m]	EUR 3.742m

The model that breached most still carried the smallest requirement after its penalty. Parametric EWMA averaged EUR 15.550m with the addend, against EUR 19.072m for Conformal-PID, because its VaR is smaller every day and the addend applied in only part of the period. On this evidence the addend prices a poor breach record. It does not make an understated VaR as expensive as an honest one.

The expected-shortfall test against 6 reference distributions

Each model's ES verdict comes from comparing a statistic to a threshold simulated from a reference distribution. Here the models, the data, the VaR and the ES are held fixed and only the reference changes. The table counts how many of the 10 model-and-level cells pass.

Reference distribution	Conditional statistic, Z1	Unconditional statistic, Z2
Gaussian	0 of 10	6 of 10
Student-t, df 15	5 of 10	6 of 10
Moment-matched t, as used	6 of 10	6 of 10
Student-t, df 8	7 of 10	6 of 10
Student-t, df 5	10 of 10	6 of 10
Vol-adjusted empirical	10 of 10	10 of 10
Verdicts that change zone across the 5 parametric references	10 of 10	4 of 10

A cell passes only in the green zone, so a verdict can change zone without changing between pass and fail. Under Z2, 4 of 10 cells change zone and none of them changes between pass and fail; they move between yellow and red and fail under every parametric reference. That is why the Z2 column shows the same count in each parametric row.

What the test does to a model known to be broken

2 models were built to be wrong in one way each, from Conformal-PID. One has the right number of breaches and a tail that is too thin. The other has its VaR and ES scaled by 0.8, so it breaches far too often. A test that passes a model broken in the way it is meant to detect is not carrying the weight placed on it.

Statistic	Known-broken model	Level	Breaches	Zone, reference as used	p	Caught
Z1	Right breach count, thin tail	99.0%	24	yellow	0.008	yes
Z1	Right breach count, thin tail	97.5%	53	red	0.000	yes
Z2	Right breach count, thin tail	99.0%	24	green	0.182	no
Z2	Right breach count, thin tail	97.5%	53	green	0.240	no
Z2	VaR and ES scaled by 0.8	99.0%	54	red	0.000	yes
Z2	VaR and ES scaled by 0.8	97.5%	100	red	0.000	yes

Z1 averages over the breach days and cannot see how many there were, so the scaled model is not held against it. Z1 does flag the thin-tailed model here, but under that model's own Gaussian reference, which puts the working Conformal-PID model in the same zones. The flag does not separate the two.

What moved the result: the model or the test

One revision changed the model's copula and the test's reference distribution together, and the ES verdict improved. The 2-by-2 below separates them for Conformal-PID at the 99% level.

Change	From	To	Moved
Model changed, reference held: the statistic Z1	−0.0004	+0.0092	+0.0096
Reference changed, model held: the pass threshold	+0.019	−0.085	−0.104

The reference moved the threshold about 11 times as far as the model moved the statistic. Against the original reference the changed model is yellow; with both changes it is green.

Every number above rests on these. The second column says which way the headline moves if the assumption is wrong.

Assumption	If it is wrong
The book is worth EUR 100,000,000	Illustrative. Every euro figure scales in proportion, and none of them is evidence about any real book.
The firm holds internal-model permission, so the multiplication factor applies	If it does not, the euro column does not apply at all. The breach, clustering and test findings stand without it.
Only the VaR component is computed; the stressed-VaR component is left out	Understates. The stressed component carries the same addend, so the effect on a real requirement is larger.
One-day VaR scaled to 10 days by the square root of time	Either direction. The Regulation permits scaling by an appropriate methodology; a directly computed 10-day VaR would differ.
One P&L series stands for both hypothetical and actual changes in value	The rule counts the higher of the two. A real desk with fees and intraday trading has 2 series, and the count can only be equal or higher.
The multiplication factor is exactly 3 plus the table's addend	Understates where a supervisor sets a factor above the minimum. The rule says "at least 3".
Calendar months are the bootstrap block	Carries clustering within a month, not across months. Longer dependence would widen every interval further.
The addend is re-read every day	The rule requires the count "at least quarterly". A firm that updates quarterly would see the same counts later and in steps.
The rule in force on the date of this run applies throughout the period	It changes on 1 January 2027. The count is unchanged; the add-on table is different. Under it the same counts raise the factor by 11.24% on average for Parametric EWMA, against 11.17% under the rule applied here.

How severity is assigned

Severity is derived by rule, not asserted. There are 4 kinds of finding and each has its own rule, fixed in the script and tested there.

Series integrity

Material when a model states an ES shallower than its own VaR on more than 1% of days at any audited level. Observation at or below that.

Model

Observation when the stated breach rate lies inside the month-block interval and independence is not rejected. Material when either fails and the model sat outside the green zone in at least 25% of windows and the lower bound of that share exceeds the 10.8% a correct model would spend there. Significant otherwise. The zone limb exists only at the 99% level, because the rule counts there and nowhere else.

The test

Material when the test passes a model built to be broken in the way the statistic is meant to detect, or when more than 50% of its verdicts change zone across the parametric references. Significant when some change and every control is caught. Observation when none change.

Process

Material when the reference and the model changed in one revision, the reference moved the threshold further than the model moved the statistic, and the model change alone does not reach a pass.

A model takes the most severe rating it reaches at any audited level. The 25% floor is a commercial judgement and not a statistical one. It is stated here so it can be argued with.

Why the intervals are bootstrapped by month and not binomial

Breaches cluster: a bad week contributes several. Treating 2,172 days as independent trials gives an interval tighter than the evidence supports. Every interval here resamples whole calendar months with replacement, 2,000 times, seed 20261111. Each estimate has its own random stream, keyed by what is being estimated, so adding a statistic does not move the others.

Rule applied	Regulation (EU) No 575/2013, Article 364(1)(a), Article 365(1) and Article 366(2) and (3) with Table 1, consolidated version of 9 January 2024 (CELEX 02013R0575-20240109). That is the text in force on 8 July 2024.
Why that version	Regulation (EU) 2024/1623 deleted Part Three, Title IV, Chapter 5. Article 520a, as replaced by Commission Delegated Regulation (EU) 2025/1496 of 12 June 2025, reads: 'Until 1 January 2027, institutions shall continue to apply Part Three, Title IV, and the market risk requirements of Articles 430, 430b, 445 and 455 of this Regulation in the version in force on 8 July 2024.'
From 1 January 2027	From 1 January 2027: Article 325bf(6) with Table 3, consolidated version of 26 June 2026 (CELEX 02013R0575-20260626). Same count (overshootings of a one-day 99% VaR over the most recent 250 business days), a base of 1,5 and an add-on of 0,20 to 0,50.
Zone names	Zone names (green, yellow, red) and the probability a correct model lands in each: Basle Committee on Banking Supervision, Supervisory framework for the use of 'backtesting' in conjunction with the internal models approach to market risk capital requirements, January 1996, Table 2. The Regulation's table carries addends only.
Scope	The multiplication factor applies to institutions granted permission to use internal models under Article 363. It does not apply to a firm on the standardised approach.
Retrieved	2026-11-11, from the Publications Office of the European Union and bis.org

Consolidated texts of EU law are documentation tools and have no legal effect. The authentic versions are those published in the Official Journal.

4 conclusions do not follow from it.

1. That any real firm would have paid these euro amounts. The book value is assumed, only one component of the requirement is computed, and the factor applies only to a firm with internal-model permission.
2. That these models rank this way on another book. One three-asset book, 2,172 days, 8 non-overlapping windows.
3. That the expected-shortfall tests are bad tests. The weakness found is in a moment-matched reference this implementation chose. With a model's exact predictive distribution as the reference, the threshold moves much less.
4. That a model scored Observation is adequate. It says the stated level sits inside the interval on this series and period, at a sample size the report states.

A client engagement replaces the public series and my models with your own and changes nothing else. The analysis behind the test findings is the paper Tail Risk Backtesting: When the Test Improves and the Model Does Not; scope, timing and price are on the services page.