Methodology Manual

1. Orientation

Banks running an Internal Models Approach (IMA) for market-risk capital under Basel III / FRTB must produce a daily Value-at-Risk (VaR) figure for each trading book. Supervisors then judge whether that VaR is statistically honest, predicted often enough, deep enough, and robust under stress. A model that systematically under-predicts risk is a material model-risk signal, and may become a supervisory finding depending on institution, materiality, and remediation history; one that over-predicts wastes capital.

This dashboard runs five candidate VaR/Expected-Shortfall (ES) forecasting methods on the same USD-denominated 3-asset equal-weight portfolio (SPY equities, TLT long-duration Treasuries, GLD gold), and grades each on the coverage and tail-adequacy tests a market-risk validation pack would carry. The dashboard is a snapshot of that scoreboard, updated with the paper; this manual is the methodology handbook.

Data period versus graded period

These are two different numbers and an earlier version of this manual conflated them, describing the study as running over "~16 years of daily returns" in a way that implied 16 years of graded results. It does not.

Every test statistic, zone, and capital figure on the dashboard is computed on the 2,172-day window, which is about 8.6 years. The earlier history is real and is used, but only to fit and warm up the forecasters. See §11 for why the consumed block is 2,013 days rather than the 1,000 that the fit window and warmup alone would imply.

What this is, and which framework each layer belongs to

The tests below are drawn from three different regulatory generations plus a governance framework. They are related but they are not one continuous stack, and conflating them is the most common way a document like this loses a technical reader. They separate as follows.

Scope in one sentence. This is a framework-neutral market-risk validation benchmark that borrows the legacy supervisory VaR machinery as a comparable capital yardstick. It is not an FRTB IMA implementation, and no number on the dashboard is a regulatory capital figure.

Reading the evidence tags

Components below carry one of three tags, because they do not have equal standing and the difference matters more than any individual result.

• established A published result with proof, applied within its stated conditions. Example: strict consistency of the pinball loss for the quantile functional.
• adapted A published method used outside the exact setting its guarantees were proved in. Example: the conformal PID controller, proved for a symmetric-coverage target, applied here to a one-sided tail with volatility normalisation.
• heuristic An engineering choice made for the implementation to work, with no theoretical backing claimed. Example: the shared copula degrees-of-freedom rule, the ES scale clipping bounds, the gain values.

2. Core concepts

Value-at-Risk (VaR)

For a confidence level α (e.g. α=0.01), VaR_α(t) is the loss level the portfolio expects to breach on day t with probability α. In our sign convention, P&L is negative for losses, and VaR_α < 0. A "99% VaR" corresponds to α = 0.01, i.e. the 1% lower-tail quantile.

Breach

A breach on day t at level α is the event realised P&L_t < VaR_α(t). Under correct calibration, breaches arrive as iid Bernoulli with rate α.

Expected Shortfall (ES)

ES_α(t) = E[X_t | X_t ≤ VaR_α(t)], the conditional expectation of P&L given a breach occurred. By construction, ES_α < VaR_α (more negative). FRTB IMA capital is keyed off ES at α=0.025, not VaR.

Sign convention, stated once

Mixing loss-magnitude and P&L conventions is a reliable source of confusion in this literature, so the rule here is fixed and absolute:

Everything on the dashboard and in this manual uses the P&L convention. Losses are negative. A return of −3% is a loss of 3%. VaR_α < 0 and ES_α < VaR_α, so ES is the more negative of the pair. A breach is realised < VaR, not >.

The one place this collides with published work is the Acerbi-Szekely statistics, which are derived in the loss-magnitude convention where ES is positive. Rather than flipping the whole document, the AS formulas in §8 divide by |ES|. That is the only place an absolute value appears for this reason, and it changes nothing about the statistic or the zones. Where the dashboard displays a capital or VaR magnitude it shows |VaR| as a positive percentage, because a negative capital number would be absurd; the underlying series is still signed.

Calibration

A VaR predictor is calibrated if its empirical breach rate over time matches the target α. Statistical tests like Kupiec POF and Christoffersen check this. "Calibrated" ≠ "accurate": a model can be calibrated (right breach rate on average) and still produce wide intervals that cost capital, or tight intervals that don't reflect the true tail.

3. The five candidate methods

Parametric Gaussian / EWMA (baseline)

Per-asset RiskMetrics EWMA volatility with λ=0.94, sample correlation matrix on raw returns, joint Gaussian, zero-mean drift. This is the intentionally simple parametric baseline, included as a lower bound rather than as a claim about what any institution runs. It is expected to break under regime shifts because its tails are too thin.

Historical simulation 250-day

Empirical joint distribution from the trailing 250-day window; portfolio P&L is the empirical projection of asset returns onto the weights vector. Historical simulation is widely used in bank market-risk practice, though we make no claim about its prevalence in any particular jurisdiction or institution class. It freezes whatever tail the lookback last saw, and so predictably miscalibrates after stress events.

Distributional (t-GARCH + t-copula)

Per-asset GARCH(1,1) with Student-t innovations via the arch package; sample correlation matrix on standardised residuals. Day-ahead conditional covariance Σ_t = D_t · R · D_t. Portfolio P&L computed by 5,000-sample Monte Carlo through a multivariate Student-t copula since sums of t-distributed marginals have no closed form. The copula df defaults to the average of the per-asset GARCH-fitted ν values (floored at 4 for MC stability), so joint extreme moves carry the same tail thickness as the per-asset marginals, which a Gaussian copula would silently lose. GARCH parameters refit every 20 trading days; conditional variance updates daily by the GARCH recursion using cached parameters between refits.

heuristic Dependence-model assumptions, stated explicitly. Two choices here are engineering simplifications rather than estimators, and a reviewer should treat them as such.

• Shared copula df by averaging. The joint df is the arithmetic mean of the per-asset fitted ν. This is a moment-style plug-in, not a joint estimate: it is not the MLE for a common copula df, not a pseudo-likelihood estimate, and not a hierarchical treatment. It was chosen because it is stable and cheap at every refit. A joint pseudo-likelihood df is the documented next step.
  
  The averaging is not neutral about tail risk. For a Student-t, smaller ν means heavier tails, so averaging lets a thin-tailed asset (large ν) pull the joint df upward and thin the modelled joint tail. Taking the minimum per-asset ν would let the heaviest-tailed asset govern joint extremes, which is the conservative choice and arguably the right one for a capital application. This project uses the mean, so the reported joint tail dependence should be read as the less conservative of the two defensible rules. The df floor of 4 partially offsets this by capping how thin the joint tail can get; it does not remove the directional bias.
• Correlation is constant between refits, and its uncertainty is ignored. R is the sample correlation of standardised residuals over the fit window, recomputed at each 20-day refit and held fixed in between (a CCC-GARCH structure). It is not exponentially weighted, not shrunk toward a target, and no positive-definite repair is applied because the full-rank sample estimate does not need one at 3 assets. Sampling error in R is not propagated into the reported VaR or ES, so interval widths are conditional on R being known. At 3 assets this is minor; at desk scale it would not be.

Conformal-PID (VaR)

Wraps the Distributional method with an adaptive online-conformal controller per α target. The controller adjusts the predicted VaR using PID feedback on the empirical breach indicator e_t = b_t − α.

heuristic What the pinball loss does and does not buy here. The pinball / quantile loss is strictly consistent for the quantile functional (Gneiting 2011), which makes it a principled scoring objective for a VaR forecast. It does not follow that this controller minimises expected pinball loss, or that its update is derived from that objective. The PID update is a feedback-control heuristic on the breach indicator: it is motivated by the quantile target but is not an optimisation procedure over it, and it carries no optimality claim. Separately, the quantile objective gives the controller no signal at all about the conditional tail mean, which is why the ES variant below exists. ES is read off the base distribution conditional on the calibrated VaR. See section 9 for the full update law.

Conformal-PID + ES (joint VaR / ES)

Same VaR controller as Conformal-PID, plus a per-α ES corrector: a feedback loop on the realised tail-mean residual e_t^{ES} = (realised_t / |ES_t|) + 1, computed against the ES the model stated, that updates a multiplicative scale (in log space, for positivity) on every breach day. The scale applies to the excess of the base ES beyond the calibrated VaR, not to the whole ES: final reported ES is VaR_t + s_t · (ES_base_t − VaR_t) with the scale clipped to [0.5, 3.0], so the stated ES is always at least as deep as the stated VaR.

heuristic This is not joint elicitability in practice. The architecture is motivated by the fact that the pair (VaR, ES) is jointly elicitable (Fissler & Ziegel 2016), but it does not implement the Fissler-Ziegel scoring gradient. It is better described as a VaR conformal controller plus a separate multiplicative ES feedback correction, with the two updated independently rather than from a joint objective. The clipping bounds and the log-space parameterisation are stability choices with no theoretical support. The benefit claimed is narrow and empirical: ES gets an explicit signal about realised tail-mean error instead of inheriting the base distribution's tail shape, so the AS Z₁ column responds to the controller rather than only to the base. A strict FZ-gradient updater would occupy the same architectural slot and is the documented next step.

4. Kupiec POF test (proportion-of-failures)

Hypothesis

H₀: P(breach) = α. Under H₀, breach count x ~ Binomial(N, α).

Test statistic

Likelihood ratio LR = -2 [x log(α/p̂) + (N-x) log((1-α)/(1-p̂))] where p̂ = x/N. Asymptotically LR ~ χ²(1) under H₀.

Why we use Monte-Carlo p-values

The χ² approximation breaks when expected cell counts in the LR's contingency table fall below ~5. At the regulatory α=0.01 over 2,172 days, expected breaches are ~22, fine, but expected "breach-after-no-breach" transition counts can be ~0.2, which destroys the χ² approximation in the related Christoffersen test. We therefore use a Monte-Carlo p-value uniformly: simulate 5,000 sequences of N iid Bernoulli(α) draws under H₀, compute LR for each, and report the fraction at-or-above the observed LR.

Worked example, with the arithmetic

Conformal-PID at α=0.01 over the full graded window. Every intermediate value is shown so the result can be checked by hand.

The MC p-value is the fraction of the 5,000 simulated LR values that equal or exceed 0.2338, which is 3,352 of 5,000. Reading it plainly: if the model were perfectly calibrated, a discrepancy at least this large would arise about two times in three. There is no evidence of miscalibration. For contrast, the Parametric baseline over the same window gives x=45, p̂=2.07%, and an MC p-value below 0.001, meaning a discrepancy that large would arise less than once in a thousand calibrated runs.

Interpretation

A method passes calibration at the 5% level iff Kupiec p ≥ 0.05. A failure is strong evidence of miscalibration and, in a supervised institution, the kind of result that triggers investigation and remediation; whether it becomes a formal finding is a matter for the institution and its supervisor, not for this test. Passes are a necessary but not sufficient calibration condition (Christoffersen catches methods that have the right marginal rate but cluster their breaches in time).

5. Christoffersen independence + conditional coverage

Independence

Tests H₀: P(breach_t | breach_{t-1}=0) = P(breach_t | breach_{t-1}=1): breaches don't cluster in time. Define the transition counts n_{ij} for the breach Markov chain; the LR statistic is LR_ind = -2 [ ll_null - ll_alt ] where the null log-likelihood uses one shared transition probability and the alternative uses π_{01} and π_{11} separately. χ²(1) asymptotically.

Worked example, and why the χ² approximation cannot be trusted here

Same method and window as §4. The 2,172-day breach series gives 2,171 consecutive pairs:

From these, π₀₁ = 24/2147 = 0.011178, π₁₁ = 0/24 = 0, and the pooled π = 24/2171 = 0.011055, giving LR_ind = 0.5366 and LR_cc = LR_pof + LR_ind = 0.2338 + 0.5366 = 0.7704. The MC p-values are 0.579 and 0.655 respectively: no evidence of clustering, and none against joint calibration.

Look at n₁₁ = 0. Not once in 2,172 days did a breach follow a breach. This is the concrete case the Monte-Carlo machinery exists for. The expected count in that cell under H₀ is 2171 × 0.01² ≈ 0.22, far below the rule-of-thumb 5 needed for the χ² approximation, and the cell is empty so π₁₁ = 0 sits on the boundary of the parameter space where the asymptotic χ² result does not hold at all. Quoting a χ²(1) p-value here would be indefensible; the MC p-value is unaffected, because it compares like with like by construction.

Conditional coverage

LR_cc = LR_pof + LR_ind, combined test of both calibration AND independence. χ²(2) asymptotically, with the same caveat as above.

MC p-values for both, stated precisely

"Same treatment as Kupiec" is too loose to reproduce, and the null being simulated differs by statistic. The exact construction is below; all three use 5,000 replications.

The independence null resamples at p̂ rather than α because the independence hypothesis is about serial structure, not level: a model with the wrong rate but no clustering should pass it. Under that null the transition probabilities are equal, π₀₁ = π₁₁ = p̂, since iid sampling is exactly the "no dependence on yesterday" restriction the LR tests.

Four implementation details a validator will ask about, answered directly:

• Is p̂ treated as fixed? Yes. It is a plug-in: estimated once from the observed series and held constant across all 5,000 replications. Estimation uncertainty in p̂ is therefore not propagated, which makes the independence p-value mildly anti-conservative. A parametric bootstrap that re-estimated the rate within each replication would be stricter.
• Is the simulation conditional on the observed breach count? No. Each replication draws N independent Bernoulli variates, so the simulated breach count varies. The test is therefore unconditional, not a conditional-on-x permutation test.
• How are zero cells handled? By the 0 · log 0 = 0 convention, applied through a guarded log that returns 0 for non-positive arguments. Empty transition classes contribute nothing to either log-likelihood rather than producing a non-finite statistic. All LR values are floored at 0 to absorb floating-point noise where the alternative fits marginally worse than the null.
• How is the first observation treated? Dropped from the transition counts. Transitions are formed from consecutive pairs, so the independence statistic uses N−1 transitions and the likelihood is conditional on the first observation rather than modelling its marginal.

Because LR_cc = LR_pof + LR_ind is recomputed inside each replication rather than combining the two marginal p-values, the conditional-coverage p-value correctly reflects the joint null.

6. Legacy supervisory traffic light (CRR Article 366)

Layer B. This is the legacy exception-counting mechanism, retained under CRR Article 366 and inherited from the Basel II market-risk amendment. It is a VaR backtesting layer. It is not the FRTB IMA capital engine, and the multiplier it produces does not feed an FRTB calculation anywhere in this project.

Definition

Count breaches in the trailing 250 trading days at α=0.01 (99% VaR):

Why this isn't enough on its own

With only 250 observations at α=0.01, expected breach count is 2.5 with std ≈ 1.6, meaning 0–4 breaches happens by chance even when the method is moderately under-calibrated over the long run. A Basel-green finding does NOT imply Kupiec passes. The dashboard's "Coverage timeseries" chart shows methods that pass Basel TL today but fail Kupiec when integrated over 2,000+ days.

7. Illustrative VaR-based capital proxy

Layer B, and read this before quoting any number from it. This section was previously titled "IMA capital path". That title was wrong and has been corrected. What is computed here is a VaR-based capital proxy in the shape of the legacy Basel/CRR construction. It is not an FRTB IMA capital calculation, and it is not a regulatory capital figure. See §7A for what an actual IMA number would additionally require.

What is actually computed

For a VaR series {VaR_t}, the dashboard computes

proxy_t = m_c(t) · max( |VaR_t|, mean_60(|VaR|) )

where m_c(t) is the trailing-250-day traffic-light multiplier as of date t, and mean_60(|VaR|) is the 60-trading-day rolling average. The max means short stretches of small VaR don't reduce the proxy, so a model cannot look cheap merely by following a calm week.

How this differs from the legacy CRR formula

It differs in one specific and material way, disclosed here because the resemblance is close enough to mislead. CRR Article 364(1) sets the VaR-based own funds requirement as the higher of the previous day's VaR and the multiplier applied to the 60-day average:

CRR: max( |VaR_t|, m_c(t) · mean_60(|VaR|) )

The proxy above instead applies m_c(t) to the whole maximum. On days when the current VaR exceeds the multiplied average, the proxy is roughly m_c times larger than the CRR quantity would be. The levels reported by this dashboard are therefore inflated relative to any real capital requirement, and should not be read as percentages of a book that anyone would actually hold.

The second departure: holding period

Every VaR figure in this project is a one-day forecast. The legacy CRR own-funds requirement is built on a ten-day VaR, obtained either by direct ten-day simulation or by scaling a one-day figure, commonly by √10 ≈ 3.16 under an iid assumption that daily returns do not satisfy. No such scaling is applied here, and none should be inferred: the numbers are one-day throughout.

This is a second reason the levels are not capital figures, and it runs opposite to the multiplier discrepancy above, which inflates them. The two do not cancel in any principled way and no attempt is made to net them. For method comparison it is irrelevant, since a horizon scaling is a single constant applied identically to all five methods and cannot change their ordering.

Why it is still useful. Every method is put through the identical transformation, and the transformation is monotone in both inputs that distinguish the methods, the multiplier ladder and the quoted VaR magnitude. The ranking and the relative comparisons the dashboard actually reports are consequently robust to this discrepancy even though the absolute levels are not. Aligning the formula with CRR 364 is a queued correction; it will move every level and is expected to preserve the ordering.

What "mean capital proxy" means

The dashboard's capital column averages this path over the reporting window. It captures both traffic-light zone changes and the magnitude of predicted VaR, which is the point: a method with excellent calibration but very wide intervals is still expensive. This is the trade-off the dashboard exists to make visible, and it is a relative statement, not a dollar figure.

Interpreting the headline

The Hero card "Capital impact vs replaced baseline" reports (mean_proxy_adopt − mean_proxy_replace) / mean_proxy_replace as a percentage. A positive number means adopting the new method costs more than the baseline it replaces. A negative number means it saves. On the current run Conformal-PID is +10.9% against Distributional and +25.2% against the cheapest method (Parametric). That is honest calibration carrying a real cost: the cheaper baselines are cheaper precisely because they under-cover, and the proxy does not charge them enough for it within this window.

7A. Not implemented: what FRTB IMA would additionally require

This project implements a coverage-and-tail-adequacy benchmark, not a regulatory capital calculation. Centralising what is absent is more useful to a reviewer than discovering it piecemeal, so the full list is here. Nothing below is computed anywhere in this codebase.

Consequence. Anyone reading this as evidence about FRTB readiness should stop at this table. The defensible claim is narrower and, for the purpose of comparing forecasters, more useful: these five methods are graded on identical data with identical tests, and the calibration differences between them are real.

8. Model-conditional Acerbi-Szekely ES diagnostic

Layer A, and deliberately not called a backtest zone. The zones below are computed against a null generated from each method's own predicted distribution. That is the construction Acerbi and Szekely intend, but it means the zone is a model-conditional diagnostic rather than an externally calibrated threshold, and it should not be read as carrying the same inferential standing as a fixed regulatory cut-off. See the null-specification note below.

Test 1: Z₁, conditional on breach

Z₁ = (1 / N_breach) · Σ_breach [ X_t / |ES_t| ] + 1

Average shortfall ratio on breach days only. Expected value 0 under H₀ (correctly specified ES). Negative Z₁ indicates realised tail-mean magnitude exceeds prediction (under-estimated ES). Because it divides by the number of breaches that happened, Z₁ cannot see whether there were too many of them; Acerbi and Szekely pair it with an exceedance-count test for that reason, and so should any reading of it. This is the statistic in the dashboard's ES column.

Test 2: Z₂, unconditional

Z₂ = (1 / (T · α)) · Σ_t [ X_t · 1[X_t < VaR_t] / |ES_t| ] + 1

The same sum divided by the number of breaches the model expected, not the number that happened, so it responds to both breach frequency and breach depth. Expected value 0 under H₀. The two are tied exactly: Z₂ = 1 + N_breach / (T · α) · (Z₁ − 1).

Naming correction, 2026-09-30. Earlier revisions of this manual and of the code had the two names the wrong way round: what they called Z₂ is the conditional statistic, Acerbi and Szekely's Z₁. Every number is unchanged, and the names now follow the paper throughout, including in the changelog entries below. See §15.

Sign convention adjustment

Acerbi & Szekely's original derivation uses positive ES (loss magnitude). Our P&L convention has ES < 0; we divide by |ES| to match the published formulas. This is cosmetic, the test statistic and zone mapping are identical.

Bootstrap critical values, and what conditions on what

Instead of arbitrary thresholds (e.g. green if Z₁ > −0.10), zones come from the null distribution of Z₁ under each method's own predicted marginal:

1. Per (method, α), each step's (μ_t, σ_t) describes a Gaussian H₀ marginal.
2. Simulate 2,000 alternative realised-P&L paths from these per-step marginals.
3. Compute Z₁ for each simulated path; this builds the empirical null distribution.
4. Zones: green = observed Z₁ above the 5th percentile of null; yellow = between 5th and 0.1th; red = below 0.1th.

This is method-specific calibration rather than a one-size-fits-all threshold, and it is the right shape for a specification test: the question being asked is "is the realised data consistent with the distribution this model claimed", which necessarily conditions on the model's claim.

heuristic Where the construction is genuinely vulnerable. The vulnerability is not that the null is model-conditional, which is by design. It is that the simulated null does not use the model's actual predictive distribution. VaR and ES are computed from the full copula Monte-Carlo predictive, but the null P&L is drawn from a moment-matched univariate Student-t whose df is a single shared value derived from the median predicted excess kurtosis across the whole reporting window. Three consequences follow, and they do not all point the same way:

• If the moment-matched t is fatter-tailed than the true predictive, the null Z₁ distribution is wider, its 5th percentile is further from 0, and the test is too easy.
• If it is thinner-tailed, the thresholds tighten and the test is too strict. Note that a model understating its tails is pushed in this direction, not the lenient one, because both its null becomes tighter and its observed Z₁ becomes more negative through a smaller |ES_t| denominator.
• Using a single median-derived df discards the time variation in predicted kurtosis, so the null is least accurate exactly in the high-kurtosis periods where breaches concentrate.

Where df = 4 + 6/κ comes from. It is not a citation, it is the inversion of a moment identity, and it is stated here because reproducibility requires it. A Student-t with ν degrees of freedom has excess kurtosis 6/(ν−4) for ν > 4. Setting that equal to a target excess kurtosis κ and solving gives ν = 4 + 6/κ. Precisely, in this implementation: κ is excess kurtosis, not total (total = excess + 3); it is computed per step from the predicted portfolio distribution and the median across all 2,172 steps is taken, so a single scalar df applies to the whole window; the daily frequency is used throughout with no annualisation; the result is floored at 4.5 so that both the kurtosis and the variance rescaling stay finite; and if the median κ falls at or below 0.1 the sampler falls back to Gaussian rather than producing an implausibly large df.

Measured: the zone is mostly a property of the null

This has now been run rather than argued about. Each method's Z₁ was re-scored against six nulls, holding the model, the data, the VaR and the ES fixed so that only the reference distribution changes. The statistic never moves; only the threshold does. Reproduce with scripts/as_null_sensitivity.py.

Ten cells, five methods at two α levels. Under a Gaussian null nothing passes; under a df-5 null everything passes. The zone the dashboard reports is therefore closer to a statement about the assumed reference distribution than about the model being tested.

Concretely, for Conformal-PID at α=0.01 the statistic is Z₁ = +0.0092 under every null. The green threshold moves from +0.019 (Gaussian, giving yellow, p=0.025) to −0.085 (shipped t, giving green, p=0.656) to −0.190 (df 5, p=0.942). Same model, same data, three different verdicts.

The same sweep with the unconditional Z₂

Re-scored on the same simulated paths with Z₂ in place of Z₁, same ten cells:

The threshold moves at least as far as Z₁'s (for Conformal-PID at α=0.01, from −0.353 under the Gaussian null to −1.052 at df 5), but the verdicts barely move: the same six cells pass under every parametric null. The four that do not are Historical simulation and the Gaussian baseline, which Kupiec also rejects. With exact nulls, where the VaR and ES are the true ones for the sampling distribution, the Z₂ threshold is close to fixed across tail shapes, as Acerbi and Szekely report; the movement here comes from the moment-matched nulls misstating the breach rate, which Z₂ counts.

Two deliberately broken models built from Conformal-PID show what each statistic can see. With VaR, ES and predicted scale all shrunk by 0.8 (54 breaches against about 22 expected; Kupiec p < 0.001), Z₂ is red under every parametric null while Z₁ passes under the shipped null (p = 0.276). A Gaussian with the same VaR, so the same breach days, but the thinner Gaussian ES, passes Z₂ under all six nulls. Z₁ is blind to frequency, and in this data Z₂ was blind to depth. Reproduce with scripts/as_null_sensitivity.py in the published analysis.

Correction: the original caveat was right and my withdrawal was wrong

A previous revision of this manual carried a caveat saying a method that understates tail thickness "will get an artificially easy AS Z₁ test". A later revision withdrew it, on the argument that a thinner claimed tail tightens the threshold and so makes the test harder. That argument identified a real channel but weighed it wrongly, and the measurement settles it against me. The original caveat was directionally correct and is reinstated.

The decisive case is the Parametric baseline. It predicts a Gaussian portfolio distribution, so its median predicted excess kurtosis is exactly 0, and the sampler's κ > 0.1 guard makes its "moment-matched t" null fall back to Gaussian. The consequence is stark:

The model claiming fatter tails is judged against a threshold roughly twice as permissive. Both channels described above are real, but the threshold channel dominates the |ES|-denominator channel, so on net claiming fatter tails buys a more lenient test. That is the circularity in its original form, now with a number attached.

What the conditional statistic can detect

Under the vol-adjusted empirical null, which is distribution-free about shape, all ten cells are green for Z₁, including the Parametric baseline that fails Kupiec at p < 0.001 and breaches at 2.07% against a 1% target. Its Z₁ of −0.167 sits comfortably inside a realistically shaped null (threshold −0.233, p = 0.464).

An earlier revision read this as a lack of power at 24 to 79 breaches, and that reading was wrong. Z₁ averages over the breach days, so it cannot detect an excess number of them at any sample size. The count test it is designed to be paired with rejected the baseline, which is the pair working as designed. What is left for Z₁ to detect is tail depth, and there the sample size does bind: in the Z₂ sweep above it could not separate a thin-tailed model with the right breach rate from Conformal-PID. The red and yellow Z₁ zones on the dashboard are generated mainly by the narrowness of the parametric null, not by detected tail misspecification.

How to read the ES zone column, given all this. Treat it as a rough consistency check between a method's claimed tail shape and its realised breaches, never on its own: it cannot see breach frequency, so read it with Kupiec and Christoffersen, and where they disagree believe those. Passing it is not evidence that the ES is adequate. For a check on frequency and depth together, Z₂ read with the count test gave verdicts that did not depend on the null in this data (above). A defensible claim about tail depth at α=0.01 needs many more years of data or a pooled cross-sectional design, and this project has neither.

One honest limitation of the empirical null itself: its residual pool is built from the same realised series the test evaluates. It is distribution-free about shape but not independent of the data, so it is a different circularity rather than an absence of one. It is reported because it brackets the parametric nulls from the permissive side, not because it is the correct null.

Why VaR can pass but ES can fail

Conformal-PID calibrates the VaR quantile; ES is the conditional tail mean above the calibrated quantile. The PID gets the breach rate right but doesn't directly correct the conditional tail shape. Methods inheriting a thin-tailed base distribution (Gaussian innovations, even t-GARCH at α=0.025) can land in red on AS Z₁ even when their VaR Kupiec is pristine. A future revision could add a paired VaR-ES calibrator using the Fissler-Ziegel scoring function.

9. Conformal-PID controller

Reference

Angelopoulos, Candès & Tibshirani (2024), "Conformal PID Control for Time Series Prediction." This implementation extends the original symmetric-coverage PID to a one-sided lower-tail target with vol-EMA normalisation and width-scale gain modulation (the ConformalPIDController class in backtest-uq/src/var_backtest_uq/conformal/controllers.py. see the public source).

State

• residuals: deque of length 500 (per α target) of vol-EMA-normalised residuals (realised - forecast_mean) / vol_ema.
• q_t: the adjusted percentile in residual-distribution space. Starts at the target α; PID nudges it.
• integral, prev_error: PID memory.
• vol_ema: exponential moving average of |residual| with span 168 observations (~7 months on daily data).

Predict step

Given a base forecast central value μ_t and a width signal (q_lo, q_hi) (e.g. base's predicted 5th and 95th percentiles):

1. Read off the residual deque the percentile at the adjusted level q_t.
2. Multiply by vol_ema to undo normalisation.
3. Multiply by an MDN-style scale (q_hi - q_lo) / median_historical_width clamped to [0.3, 3.0], so the offset stretches when the base predicts wider tails.
4. Apply optional AR(1) smoothing on the offset (disabled by default for daily data).
5. Final VaR_α(t) = μ_t + offset_α(t).

Update step (after observing realised P&L)

q_{t+1} = q_t − ( K_p · e_t + K_i · I_t + K_d · (e_t − e_{t-1}) )

where e_t = b_t − α is the breach error, with default gains K_p=0.03, K_i=5e-4, K_d=3e-3. Integral clipped at ±50 to prevent windup.

Why PID and not a simpler proportional rule?

The proportional term tracks fast changes; the integral is intended to absorb persistent bias (a model that consistently under-covers by 0.5pp); the derivative damps oscillation. A pure-P controller (the original ACI of Gibbs & Candès 2021) leaves a steady-state bias proportional to the model's intrinsic miscalibration, and the integral term is intended to reduce it.

adapted Why "reduce" and not "eliminate". Zero steady-state error under integral action is a result about linear time-invariant systems with a constant setpoint. Essentially none of those conditions hold here: the percentile state enters non-linearly, the integral is clipped at ±50, the residual deque is finite and rolling, the base forecaster is refit every 20 days, and the disturbance is a volatility process that does not settle. Persistent coverage bias is empirically small in the reported run, and that is an observation about this dataset, not a property the controller guarantees.

10. Per-asset calibration

Concern: a portfolio-only calibrator can hide individual-asset miscalibration via diversification cancellation. If asset A under-covers and asset B over-covers, their joint distribution can have the right portfolio breach rate while neither marginal does. This breaks under correlation regime change.

The Conformal-PID calibrator therefore runs n_assets × n_alpha_targets additional one-sided PID controllers, one per (asset, α). Each calibrates that single asset's marginal at the target level, using the asset-level central forecast, width signal, and realised return.

The Per-asset Diagnostics panel shows the resulting per-asset breach rates. For Conformal-PID these should land near the target across all three assets, distinct from the under-/over-covering rates of the uncalibrated baseline.

11. Out-of-sample warmup

Split-conformal coverage guarantees are finite-sample and distribution-free, but only under exchangeability of the calibration residuals, which daily financial returns violate through volatility clustering. The adaptive conformal literature (Gibbs & Candès 2021; Angelopoulos et al. 2024) replaces that with long-run average coverage under adversarial sequences, which is an asymptotic statement. This implementation inherits neither guarantee cleanly, because it adds volatility normalisation, width-scale modulation, clipping, and a finite rolling deque on top. Treat the warmup below as removing a known and avoidable bias, not as restoring a theoretical guarantee. If the controller's residual deque is seeded with in-sample (window-fit) residuals, the early-period coverage is biased optimistically.

The runner therefore performs an explicit warmup_walk(returns_full, weights, fit_window, n_steps=500) before the main reporting loop begins. Each warmup step refits the base forecaster, predicts, observes the realised return, and updates each PID controller, without recording outputs. By the end of warmup, the controller's residual deque contains exclusively out-of-sample residuals.

Where the reporting window actually starts

An earlier version of this manual stated the reporting period was "exactly [fit_window + warmup_steps, end_of_returns)". That was wrong, and the arithmetic did not close: 4,185 available days minus a 500-day fit window minus 500 warmup steps would give roughly 3,185 graded days, but the dashboard reports 2,172.

The run takes an explicit --start date, defaulting to 2018-01-01, and the true reporting period is

[ max( start_date, fit_window + warmup_steps ), end_of_returns )

On the current configuration the start date binds and the fit window does not. Fit plus warmup need 1,000 days and would reach 2013-12; the 2018-01-01 start discards a further 1,013 days beyond that. The forecasters therefore begin the graded period with more history behind them than the stated 500-day fit window suggests, which is conservative in the right direction but was undocumented.

Reconciliation. 4,185 loaded − 2,013 consumed = 2,172 graded, matching the dashboard's n_steps and the as_of of 2026-08-24 exactly.

12. Stress windows

Five named windows shaded in the coverage timeseries chart, hardcoded in data/stress_windows.py:

• COVID crash, 2020-02-19 to 2020-04-30
• UK gilt / Fed terminal, 2022-09-01 to 2022-11-15
• SVB / Credit Suisse, 2023-03-08 to 2023-03-31
• Yen carry unwind, 2024-08-01 to 2024-08-15
• Tariff vol, 2025-04-02 to 2025-04-30

These are the editor's choice, not regulatory definitions. They anchor visual interpretation: a method that drifts above target during a stress band and stays elevated afterwards is not regime-robust.

13. Caveats and known limitations

• The AS bootstrap H₀ swapped one parametric assumption for another. It is now a moment-matched Student-t rather than Gaussian (per-step location/scale from the predicted marginal; df from the median empirical excess kurtosis via df = 4 + 6/κ, floored at 4.5). The Gaussian H₀ was over-rejecting under fat-tailed data, and the t H₀ rejects less. Fewer rejections under a wider null is the arithmetically expected outcome of widening the null; it is not evidence that the models are correctly specified. This has now been measured rather than asserted: across ten (method, α) cells the zone count goes from 0 green under a Gaussian null to 10 green under a df-5 null, with the shipped null at 6. A method that claims fatter tails is judged against a materially more permissive threshold, and the Parametric baseline is scored against a Gaussian null because it predicts zero excess kurtosis. See §8 for the full table, the de-confounding factorial, and the power result.
• PID gains use defaults from a different domain (energy hourly data). Performance might improve with finance-tuned gains; we deliberately did not fit them to avoid overfitting on the same backtest we report.
• ES corrector is feedback-based, not strict FZ-gradient. The Conformal-PID + ES variant updates a multiplicative ES scale on breach days from the realised tail-mean residual rather than from the gradient of a Fissler-Ziegel scoring function. The two are directionally consistent but not identical; a strict FZ-gradient controller is the documented next research step.
• Single-currency, three single-name assets. No FX hedging, no dispersion within asset class, no counterparty valuation adjustments.
• Stress windows are subjective. A regulator may impose different windows (e.g. CCAR scenarios), and the regime-robustness verdict can shift.
• Capital savings are sensitive to the as-of date. Basel TL is a rolling 250-day window; running the dashboard on a different end-date can produce a different multiplier and therefore a different headline capital impact.
• VaR is elicitable; ES is not. Pure VaR calibration via quantile loss is strictly consistent for the quantile functional (Gneiting 2011) but gives the controller no signal about ES, the conditional tail mean. The Conformal-PID + ES variant added a feedback loop on the realised tail-mean residual to address this; joint VaR-ES via strict Fissler-Ziegel (2016) gradient updates is the more rigorous next step (their pair is jointly elicitable).

14. OCC 2026-13 documentation cross-reference

Layer D. This is a cross-reference, not an alignment claim. The table records where each topic is discussed in these artefacts. It does not demonstrate that any governance control is satisfied, because most of them are organisational rather than analytical and this project has no organisation around it. This section was previously headed "lifecycle mapping" and described as alignment; both overstated what a pointer table can show.

OCC Bulletin 2026-13, issued 17 April 2026 as the revised interagency model risk management guidance, describes expectations across model development and use, validation, ongoing monitoring, and governance and controls. The guidance is explicitly risk-based and tailored, and states that it is guidance rather than an enforceable standard, so "compliance" is not a property a document like this could have in the first place. With that established, the topic pointers are:

Disclosure. The claim above is a documentation cross-reference and nothing more: every OCC 2026-13 topic has a pointer to where it is discussed. It is not a certification, a regulator review, or evidence of a functioning model risk management framework. Of the eight rows, seven point at analytical artefacts and one is explicitly out of scope, which is a fair summary of the whole table's standing: the analysis is here, the governance is not. A bank's internal MRM function would need to perform its own assessment against its model inventory standard and risk-tier definitions, and would find no part of that work done here.

15. Methodology changelog

Static reports and SaaS feature lists don't carry a methodology changelog. For a methodology-fluent reviewer, a date-stamped record of every methodology change is a non-trivial trust signal, because it makes the development history auditable rather than asking anyone to take the current state on trust. Entries below predating this document's release are development history, not prior publications: this is the first public version, and the corrections it carries were made before release rather than retracted after one.

2026-11-12, ES corrector fixed: stated ES could be shallower than stated VaR

An integrity check run for the specimen Model Health Check found that the Conformal-PID + ES variant stated an expected shortfall that was a smaller loss than its own VaR on 463 of 2,172 graded days at the 99% level and 1,880 of 2,172 at 97.5%. An expected shortfall is the mean loss beyond the VaR, so that pair is impossible. Two defects caused it:

• The correction multiplied the whole ES, so a scale below 1 pulled it inside the VaR. It now multiplies only the excess beyond the VaR.
• The corrector computed its error against the uncorrected ES, so a correction never changed the next error and the scale drifted to a bound. It now uses the ES the model stated.

Every ES figure for that one method changed, and the scoreboard was regenerated from the same 2026-08-24 price data. Its VaR, its breach counts and the other four methods are unchanged to the last digit, and its ES zone is the same under all six reference distributions at both levels. The earlier output is kept as the control in the specimen report.

2026-09-30, Acerbi-Szekely names corrected, Z₂ re-run, moved to kwantil.com

• The two Acerbi-Szekely statistics were named the wrong way round in the code and in this manual. The conditional statistic in the ES column is Z₁ (their Test 1), not Z₂. Renamed throughout, including in the entries below; no reported figure changed, because each statistic kept its bootstrap seed.
• §8 no longer attributes Z₁'s failure to reject the Parametric baseline to sample size: Z₁ cannot see breach frequency by design. The null sweep was re-run with the unconditional Z₂ and two known-broken models (§8).
• The dashboard moved from var-backtest.mtanti.com to kwantil.com/var-backtest/. It is a snapshot of the 2026-08-24 run, not a live feed.

2026-05-07, t-copula base, Student-t H₀ bootstrap, joint VaR-ES variant

Three concurrent changes addressing the Acerbi-Szekely Z₁ tail-shape finding the prior baseline left visible:

• t-copula on the standardised residuals (replaces Gaussian copula). Gaussian-copula portfolios under-price joint extreme moves; the t-copula preserves upper-tail dependence with a single shared df fit from the average of per-asset GARCH-fitted ν, floored at 4 for MC stability. See §3.
• Student-t H₀ in the AS Z₁ bootstrap (replaces Gaussian H₀). Per-step (μ, σ) from the predicted distribution; shared df from the median empirical excess kurtosis across the reporting window via df = 4 + 6/κ, floored at 4.5. Eliminates the over-rejection bias the Gaussian H₀ caused under fat-tailed truth. See §13.
• Joint VaR-ES calibrator added (Conformal-PID + ES), new fifth method in the comparison. Same VaR controller as Conformal-PID, plus a per-α multiplicative ES scale updated on breach via PID feedback on the realised tail-mean residual. Pragmatic decoupled stand-in for a strict Fissler-Ziegel gradient updater. See §3.

heuristic Three changes were made at once, and they are confounded. The base distribution, the ES null, and the method set all changed in the same revision, so the improvement in AS Z₁ cannot be attributed to any one of them from this run. Worse, two of the three act on the same quantity in the same direction: a t-copula widens the modelled joint tail, and a Student-t null widens the reference distribution the result is judged against. Both make AS Z₁ easier to pass, and the design does not separate them. Resolved 2026-08-25. The factorial rerun has now been done and the attribution is reversed: the null change accounts for essentially all of the movement and the copula change for almost none. See the correction immediately below and §8.

heuristic Corrected 2026-08-25, before release: the attribution below was wrong. The factorial has since been run, crossing the Gaussian and t copula with the Gaussian and t null, and it separates the two cleanly. Switching the copula moved Conformal-PID's Z₁ at α=0.01 by +0.0096, from −0.0004 to +0.0092. Switching the null moved the green threshold by −0.104, from +0.019 to −0.085. The null change did roughly eleven times as much work as the copula change. Held against the Gaussian null, the t-copula run is still yellow at α=0.01 (p=0.025) and still red at α=0.025 (p=0.000): the copula change alone would not have moved anything to green. The claim that "the t-copula and t-H₀ fixes did most of the work" is retained below only as the record of what was believed at the time, and is corrected here rather than quietly edited out. Reproduce with --copula gaussian and scripts/as_null_sensitivity.py.

Empirical finding from the re-baseline (as recorded on 2026-05-07, since superseded): the t-copula and t-H₀ fixes did most of the work. The plain Conformal-PID (VaR) moved to GREEN on AS Z₁ at both α targets with a Z₁ statistic essentially zero at α=0.01, before any joint VaR-ES correction. The Conformal-PID + ES variant also lands GREEN at both α but slightly over-corrects on this base / dataset (the corrector responds to historical breach residuals, but with the t-copula in place those residuals were not under-estimating ES). On a less well-specified base, or under regime shifts the t-copula doesn't fully capture, the corrector would close more of the gap. Both variants are kept on the dashboard so the delta is visible, not hidden.

2026-04, First complete build

Four candidate methods (Parametric (EWMA), Historical 250d, GARCH + Gaussian copula, Conformal-PID); 2,096-day backtest on SPY / TLT / GLD equal-weight portfolio; full Kupiec / Christoffersen / Basel Traffic Light / Acerbi-Szekely ES backtests with Monte-Carlo p-values. AS Z₁ disclosure included from the first build, see the methods-comparison panel for the original framing of the tail-shape question that drove the 2026-05-07 re-baseline.

Future entries will be added here as the methodology evolves. adding a joint Fissler-Ziegel gradient updater, replacing the shared-df t-copula with a per-asset hierarchical-t copula, benchmarking against deep-learning-based ES estimators, etc.

15A. Tuning and leakage controls

This project contains several adaptive mechanisms stacked on top of one another: the base forecaster, the conformal controller, its gains, the width-scale modulation, the ES corrector, and a bootstrap whose null is itself estimated. Every one of them can improve a displayed metric while making that metric harder to interpret. The question a reviewer should ask is not whether the numbers improved but whether the model improved or the score did, and that question is only answerable if the tuning history is disclosed. It is disclosed below, including where the answer is unflattering.

What would actually resolve the last two rows

Only out-of-sample data collected after the methodology was frozen. The re-baseline was frozen on 2026-05-07; the reporting window now extends to 2026-08-24, which supplies roughly 76 trading days the revised methodology has never been fitted or tuned against. That is currently too short for a 1% tail test to say much on its own, with under one expected breach, but it accumulates automatically and requires no further decisions. Reporting coverage on the post-freeze segment separately, once it is long enough to carry a Kupiec test, is the single change that would most improve this document's standing.

Interim reading. Treat the coverage results (§4, §5) as substantially trustworthy, since the controller's gains were never fitted here and coverage is what they would have been fitted to. Treat the AS Z₁ zones (§8) as the weakest numbers on the dashboard, because both the null and the base distribution were revised in response to earlier AS Z₁ results.

15B. Glossary of technical terms

Terms used elsewhere in this manual without stopping to define them, collected here rather than expanded inline so the argument stays readable.

Elicitable

A statistical functional is elicitable if some scoring function is minimised, in expectation, by reporting the true value of that functional. VaR is elicitable, using the pinball loss. ES is not elicitable on its own, which is the technical reason ES cannot simply be "optimised for" the way a quantile can.

Strictly consistent scoring function

A scoring function whose expected value is minimised uniquely by the true functional value. Strictness matters: without it, wrong forecasts could score just as well as right ones and the score would not identify the truth.

Jointly elicitable, and Fissler-Ziegel

Although ES is not elicitable alone, the pair (VaR, ES) is jointly elicitable: Fissler and Ziegel (2016) construct scoring functions that are strictly consistent for the pair taken together. This is what a principled joint VaR-ES calibrator would optimise, and what the ES corrector in §3 deliberately does not do.

Exchangeability

A sequence is exchangeable if its joint distribution is unchanged by reordering. It is weaker than iid but strong enough to support split-conformal's finite-sample coverage guarantee. Daily financial returns violate it, because volatility clusters and therefore the order carries information. This is precisely why the adaptive conformal literature exists.

Excess kurtosis (κ)

Kurtosis minus 3, so a Gaussian has excess kurtosis 0 and positive values mean fatter tails. Every κ in this manual is excess, never total.

Backtest, in the market-risk sense

Comparing realised outcomes against the forecasts that were made before them, to test whether the forecast distribution was right. Note this is a narrower thing than "backtest" in a trading-strategy sense, where it usually means simulating a strategy's profitability.

Plus factor / multiplier ladder

The supervisory add-on applied to the capital multiplier as the trailing 250-day exception count rises, taking it from 3.0 in the green zone to 4.0 in the red. See §6.

Windup, and integral clipping

In a PID controller, windup is the accumulation of a large integral term during a period when the controller cannot act on it, causing a violent correction later. Clipping the integral, at ±50 here, is the standard defence.

16. References

• Acerbi, C., & Szekely, B. (2014). Backtesting expected shortfall. Risk, December 2014, pp. 76–81.
• Angelopoulos, A. N., Candès, E. J., & Tibshirani, R. J. (2024). Conformal PID control for time series prediction. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). arXiv:2307.16895.
• Christoffersen, P. F. (1998). Evaluating interval forecasts. International Economic Review, 39(4), 841–862.
• Fissler, T., & Ziegel, J. F. (2016). Higher order elicitability and Osband's principle. Annals of Statistics, 44(4), 1680–1707.
• Gibbs, I., & Candès, E. J. (2021). Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems 34, 1660–1672.
• Gneiting, T. (2011). Making and evaluating point forecasts. Journal of the American Statistical Association, 106(494), 746–762.
• Kupiec, P. H. (1995). Techniques for verifying the accuracy of risk measurement models. Journal of Derivatives, 3(2), 73–84.
• McNeil, A. J., Frey, R., & Embrechts, P. (2015). Quantitative Risk Management: Concepts, Techniques and Tools, revised ed. Princeton University Press. Ch. 9 (risk measures) and Ch. 7 (copulas).
• Basel Committee on Banking Supervision (2019). Minimum capital requirements for market risk (FRTB).
• Capital Requirements Regulation (CRR), Article 364, VaR-based own funds requirement; Article 366, backtesting plus-factor.
• Basel Committee on Banking Supervision (1996). Amendment to the Capital Accord to incorporate market risks (origin of the multiplier and traffic-light construction).
• OCC Bulletin 2026-13, Model Risk Management, issued 17 April 2026 (revised interagency guidance). Cross-referenced by topic in §14; the bulletin is guidance, not an enforceable standard.