Skip to content

TST: Do not compare separate calls bit for bit on macOS x86-64 - #959

Merged
bashtage merged 2 commits into
mainfrom
a90-fix-macos-intel-tests
Oct 17, 2026
Merged

bashtage merged 2 commits into
mainfrom
a90-fix-macos-intel-tests

Conversation

@bashtage

@bashtage bashtage commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Branch a90-fix-macos-intel-tests, two commits on main e677722.

The macos-15-intel wheel job fails on main, with a different test in each run:

  • Run at e677722:
    test_valid_results_are_unchanged_by_numpy_scalars fails. It asserts that
    ZivotAndrews(Y, trim=np.float32(0.25)).stat ==
    ZivotAndrews(Y, trim=0.25).stat, and the two differ in the last place
    (-4.3285219553671475 vs -4.328521955367147).
  • Run at a93ff73: the same test fails, and so does
    test_float32_is_computed_in_float64[arch_model]. That second test passed in
    the later run.

Cause. Not float32 itself: float32 to float64 is exact, and both trims give
the same trim index. On macOS on x86-64, two calls with the same inputs are not
always bit-identical. This is most likely because the BLAS kernels' summation
order depends on memory alignment, which changes from call to call.

  • The Zivot-Andrews comparison used ==.
  • The float32 test of arch_model fits a GARCH(1,1) to i.i.d. data. The
    estimate is on the boundary (alpha = 0, beta = 1), where the likelihood is
    flat, so the optimizer turned a difference in the last bits into a relative
    difference of 3e-8 in mu and omega, against a tolerance of 1e-12.

Fix (tests only):

  • Separate calls are compared with assert_allclose(..., rtol=1e-12): the
    ADF, Zivot-Andrews, mackinnonp and mackinnoncrit comparisons, and the
    float32 against float64 long-run covariance in test_covariance_dtypes.py.
  • The two GARCH targets of the data type tables (arch_model and
    arch_model(mean="ARX")) evaluate the model at fixed parameters (the
    log-likelihood and the conditional volatility) instead of fitting it. They
    still exercise the float64 conversion and still detect a calculation in
    float32 (an error of about 1e-7), without the optimizer.

Fast suite 26,955 passed. No library code changes.

Also fixed: an order-dependent test on the Windows legs. The first CI run
of this PR failed on the two Windows legs:

Cause:

  • The tests shared one model fixture for the whole module, and fit and fix
    store the sample on the model.
  • When the result fixture had been created before the test that fits with
    first_obs=-20, the model kept that sample. Then forecast(start=-25)
    failed the check that start is not before the first fitted observation.

The tests now use a new model each, and result fits its own. Reproduced
locally with that order and fixed. The file passes in 8 random orders, and the
full fast suite passes in 2 random orders (26,955 passed). The other modules
with shared models passed in 5 random orders.

🤖 Generated with Claude Code

Separate calls are not bit-identical on macOS on x86-64: ZivotAndrews with
trim=np.float32(0.25) and trim=0.25 differed in the last place. The float32 data type
test of arch_model fitted a GARCH(1,1) to i.i.d. data, whose estimate is on the boundary
(alpha=0, beta=1), so the optimizer turned a difference in the last bits into a relative
difference of 3e-8 in the mean and omega, against a tolerance of 1e-12.

Compare the results of separate calls with a tolerance of 1e-12, and evaluate the GARCH
targets of the data type tests at fixed parameters instead of fitting them, so that the
test still detects a calculation in float32.
@codecov

codecov Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.97%. Comparing base (e677722) to head (c71ed59).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #959   +/-   ##
=======================================
  Coverage   98.97%   98.97%           
=======================================
  Files         136      136           
  Lines       30021    30028    +7     
  Branches     2327     2327           
=======================================
+ Hits        29712    29719    +7     
  Misses        255      255           
  Partials       54       54           
Flag Coverage Δ
adder 98.91% <100.00%> (+<0.01%) ⬆️
subtractor 98.91% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

fit and fix store the sample on the model, and forecast checks start against it. The
tests shared one model, so when the result fixture was created before a test that fit
with first_obs=-20, the forecasts with start=-25 or start=250 raised a ValueError. This
failed on the Windows legs, where the tests ran in that order.
@bashtage
bashtage merged commit 80d3688 into main Oct 17, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant