I test whether a model's stated uncertainty deserves to be trusted.

Kwantil is my independent practice, and the papers here are its working evidence. Each one takes a model that somebody relies on, checks its stated uncertainty against what actually happened, and publishes the code that did the checking.

A confident wrong answer is the one that gets acted on. So these papers do not ask whether the average error is small. They ask how often a stated range actually contained the outcome, and whether it still did in the regime where a miss is expensive. Any dataset is mostly calm days, and an average taken across them is an average of the hours that did not matter.

3 rules the papers follow

Evidence or illustration, stated. Every figure is labelled as one or the other. A chart drawn to explain a mechanism and a chart drawn from measured outcomes are different objects, and a reader should never have to guess which one is in front of them.

Numbers arrive with their uncertainty. A coverage figure computed on 16 cutting inserts is not the same claim as one computed on 2,172 trading days, and the interval says so. Where the sample is small the interval is wide, and the width is the finding rather than an embarrassment to be narrowed.

Every paper ends with what the result does not support. This is the rule that costs something. It is easy to write an audit that implies more than it establishes, and the section at the end of each paper exists to take that back.

Corrections are published, not quietly applied

The paper on tail-risk backtesting exists because I got something wrong. An expected-shortfall diagnostic improved, my development log credited the model, and the experiment that separates the two shows the reference distribution did about 11 times the work. It was caught before release, and it would have been easy to fix the attribution quietly and publish the improved number.

A methodology that only records the decisions that flatter it is not a record. So the register documenting the confound ships with the result, and the article is about the mistake rather than around it.

How to read them

Each paper is written to be attacked. The analysis code, the data-fetch scripts and a notebook that walks the experiment step by step live in github.com/m-tanti/kwantil-research, because a validation finding you cannot reproduce is an assertion. Each paper links to its own directory there. If a number here does not match a number you compute, that is a bug worth telling me about.

The background notes are the other half: plain-language tutorials on the concepts the papers lean on, so that a paper can reference a concept once and move on rather than re-teaching it every time.

Who

Matthew Tanti. I work as a data scientist specialising in probabilistic modelling and uncertainty quantification, and I run Kwantil from Malta.

Write in

Corrections, disagreements, and questions about anything published here: matthew@mtanti.com.

Corrections are the most useful thing anyone sends. If you find an error it gets fixed in public, with a note saying what changed.