Skip to content

ENH: Add VaR backtests (Kupiec, Christoffersen, Basel traffic light) - #965

Open
bashtage wants to merge 1 commit into
mainfrom
a91-f904-var-backtests
Open

bashtage wants to merge 1 commit into
mainfrom
a91-f904-var-backtests

Conversation

@bashtage

Copy link
Copy Markdown
Owner

Target 9.1: merge after v9.0.0 is tagged. xref #904. This implements the five standard
backtests from the proposal. max_undetectable_understatement is left out, pending a
decision on the issue.

Adds the new module arch/univariate/backtest.py, exported from arch.univariate:

  • var_exceptions(returns, var): a bool array of -returns > var. VaR is a positive loss,
    and a loss equal to the VaR is not an exception.
  • kupiec_pof(returns, var, p): the Kupiec (1995) proportion-of-failures likelihood-ratio
    test, chi2(1).
  • christoffersen_independence(returns, var): Christoffersen's (1998) Markov independence
    test, chi2(1).
  • christoffersen_conditional_coverage(returns, var, p): LR_uc + LR_ind, chi2(2).
  • traffic_light(returns, var, p=0.01, window=250): the Basel Committee (1996) zone,
    returned as a frozen TrafficLightResult with the zone, the count, the cumulative
    probability, nobs and p.

The three tests return WaldTestStatistic.

Inputs:

  • The data must be finite and 1-d.
  • Two Series are aligned on the index. Otherwise the lengths must match.
  • 0·log 0 = 0 through xlogy.

Validation:

  • R 4.6.1, rugarch 1.5.6 VaRTest, on 11 cases: largest difference in the statistics
    1.2e-14.
  • Degenerate series (zero or all exceptions, first-only or last-only, 2 observations)
    cannot go through rugarch, which indexes a 2x2 table. They are checked against an
    independent R likelihood implementation, cross-checked with rugarch.
  • The traffic-light probabilities match scipy and R pbinom to 1e-12, and the published
    Table 2 of the BIS 1996 backtesting framework, including the zone boundaries at 5 and
    10.
  • The Kupiec 99% non-rejection regions (255/510/1000 observations) match the usually quoted
    values.
  • The references are stored as constants in the tests.

Tests:

  • test_backtest.py (198) and test_validation_backtest.py (299 table rows).
  • Each of 8 deliberately introduced bugs is caught.
  • Fast suite 27,472 passed. The new tests pass in random orders. numpydoc lint 0, and the
    docs build has no new warnings (a new "Backtesting Value-at-Risk" section with an
    example).

For review:

  1. traffic_light raises for a series shorter than window, rather than using what is
    there.
  2. Zero exceptions is always green.
  3. p is required except in traffic_light.
  4. TrafficLightResult is public.
  5. var_exceptions returns an ndarray, not a Series.
  6. WaldTestStatistic.pval is 1 - cdf, which is imprecise below about 1e-15. A survival
    function would fix it, as a separate change.

Thanks to @sarthakguptaquant for the proposal.

🤖 Generated with Claude Code

Add arch.univariate.backtest with standalone Value-at-Risk backtests, exported from
arch.univariate and added to the utility page of the documentation (xref #904).
VaR is a positive loss and an exception is -returns > var.

- var_exceptions returns the bool array of exceptions.
- kupiec_pof is the proportion-of-failures test of Kupiec (1995), chi2(1).
- christoffersen_independence is the Markov independence test of Christoffersen
  (1998), chi2(1), and christoffersen_conditional_coverage is the sum of the two
  statistics, chi2(2). The tests return a WaldTestStatistic.
- traffic_light is the Basel Committee (1996) zone over the last window observations
  (250 by default) with the exception count and the cumulative binomial probability,
  returned as a TrafficLightResult. For 250 observations and p=0.01 the zones are
  those of Table 2: green 0-4, yellow 5-9, red 10 or more.

The inputs must be finite. Series are aligned on the index when both are Series, and
must otherwise have the same length. A series with no exceptions, only exceptions or no
1 -> 1 transition is handled with the limits of the likelihood (0 log 0 = 0). A series
that is shorter than the window raises.

The tests compare the statistics and p-values with rugarch::VaRTest (1.5.6) and with an
independent likelihood implementation in R, the zones with the binomial distribution
and the published Tables 1 and 2 of the Basel framework, and the non-rejection regions
with Kupiec (1995).

def test_p_must_be_given():
with pytest.raises(TypeError, match="p"):
kupiec_pof(RETURNS, VAR) # type: ignore[call-arg]
with pytest.raises(TypeError, match="p"):
kupiec_pof(RETURNS, VAR) # type: ignore[call-arg]
with pytest.raises(TypeError, match="p"):
christoffersen_conditional_coverage(RETURNS, VAR) # type: ignore[call-arg]
@codecov

codecov Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.97%. Comparing base (d2c90b4) to head (aac6fce).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #965      +/-   ##
==========================================
+ Coverage   98.95%   98.97%   +0.01%     
==========================================
  Files         136      139       +3     
  Lines       30043    30596     +553     
  Branches     2331     2360      +29     
==========================================
+ Hits        29729    30282     +553     
  Misses        256      256              
  Partials       58       58              
Flag Coverage Δ
adder 98.91% <100.00%> (+0.01%) ⬆️
subtractor 98.91% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants