Repository navigation
ENH: One data type rule, with complex data in the covariance estimators - #957
Merged
Merged
Conversation
…of the distributions The standardized residuals of a degenerate model, such as one fit to a constant series, can be 0/0, which is NaN on macOS on x86-64. The check of std_resid in starting_values rejected them, so fit raised a ValueError there instead of the ConvergenceWarning that test_no_variance expects. NaN and inf are allowed again, and the distributions return their default starting values for them, as before.
…uals The kurtosis of residuals that contain inf is NaN, which gives the default starting value, and SciPy warned about the invalid value.
A merge of main brought back the copy of the examples before the references, so the section appeared twice and the docstring test failed.
as_float64, ensure1d, ensure2d, check_array and to_array_1d now share one rule for
the data type of numeric data, defined once in as_float64 and _rejected_dtype:
* bool, integer, unsigned integer, float32, float64 and longdouble data are converted
to float64, and the calculation is done in float64. float32 is promoted and not
computed in float32 (see below).
* float16 data are never accepted. They raise a TypeError that names the argument, the
data type received and the data types accepted, for example
"y must be a 1-d array of finite real numbers (integer, Boolean, float32 or float64
data); got float16 data with shape (120,) (ndarray)". A DataFrame with one float16
column is rejected, which np.asarray(frame) would hide by merging the columns.
* complex data raise a TypeError, unless the new keyword allow_complex is True
(check_array, ensure1d, ensure2d, as_float64). Then complex64 and complex128 data,
and object data with a complex element, are converted to complex128 and real data
are still converted to float64. The finite check covers both parts.
* to_array_1d no longer discards the imaginary part of complex data (it used astype)
and raises the same TypeError.
np.longdouble is equal to np.float64 where it has 64 bits (Windows), so that
`dtype == np.float64` passed such an array through as it was, and LAPACK rejects it
("array type float64 is unsupported in linalg"). The fast path now also checks the
type character, and pandas objects with a longdouble column are rebuilt, since astype
does not convert equal types.
Why float32 is promoted. float32 data computed in float32, emulated with every array
in float32 (OLS, lagmat, add_trend and the data), against the same values in float64:
the kernel long-run covariance has a relative error of 1e-7 to 1e-6 for well scaled
data, 4e-4 at 25,000 observations and 8e-3 at 250,000 for data with a mean of 1,000 and
a standard deviation of 1 (the centering loses digits), and the bandwidth changes by
5%; the ADF, DF-GLS, KPSS and Zivot-Andrews statistics differ by 1e-7 to 1e-6, the
Phillips-Perron statistic by up to 0.04 and the variance ratio by up to 1e-4 for a
series with a large level; the ADF lag search raises a singular matrix error in every
replication for a level of 1,000, and with a trend for a level of 5 with steps of 0.01
and 2,500 observations and for a random walk with 5,000 observations; and the
cointegrating regressions differ by up to 6e-5. Selected lags were the same in the
runs that completed. NumPy promotes in numpy.cov for the same reason. The work is not
faster either: the float32 and float64 times of the kernel estimators were the same.
The tests are tables of data types for each helper, container (ndarray, Series,
DataFrame) and allow_complex.
The kernel covariance estimators, cov_nw and add_trend accept complex data. The estimators are Hermitian: every product uses the conjugate transpose, so that the lag-j autocovariance is the sum of x[t] times the conjugate of x[t - j], the long-run covariance is Gamma_0 + sum of w_j (Gamma_j + Gamma_j^H) and the estimates are complex128 matrices that are exactly Hermitian, with a real diagonal. * The automatic bandwidth combines the columns with the (real) weights and uses the real part of the autocovariance of the complex series, since the sums over j and -j add a value and its conjugate. The product v[t] * v[t - j] of the earlier code is the pseudo-covariance, which is not what the bandwidth formula needs. * cov_nw returns a real float for a 1-d complex series, and a complex Hermitian matrix otherwise. * add_trend adds the real trend columns to complex data and returns complex128. Constant columns of complex data are detected. * PreWhitenedRecolored raises a TypeError for complex data. A VAR for complex data is not one model: a VAR with complex coefficients cannot represent real and imaginary parts that depend on each other in a way that is not a complex multiplication, the VAR of the stacked real and imaginary parts is a different model with more parameters, and the order that the information criteria select depends on which is used. The Notes say to use the stacked data or a kernel estimator. * The weights of the bandwidth are real. The estimates for float32 data are float64, and for complex64 data they are complex128. The tests of antonsoo's last commit (#942) expected float32 to be preserved, which the estimators did not do consistently (short_run was float32 and long_run float64) and the float32 estimates are not accurate enough, and expected the complex estimates to be the products without the conjugate. test_preserve_floating_kernel_dtype and test_preserve_inexact_newey_west are replaced by tests of the same values with float64 and complex128 dtypes and the conjugate transpose. The reference for the tests is independent of the complex arithmetic. For z = a + ib the Hermitian long-run covariance of the estimators equals S_aa + S_bb + i (S_ba - S_ab), where S is the long-run covariance of the real data [a, b] with the same kernel, bandwidth, centering and degrees of freedom, and the automatic bandwidth equals the one computed from the sums of the autocovariances of the real and imaginary parts. The tests cover all ten kernels, every estimate (long run, short run, one sided, one sided strict), centering and degrees of freedom, a rotation of the data (Hermitian estimates are invariant to it, the pseudo-covariance is not), real data stored as complex, labels of a DataFrame, and an exact rational reference for float32 data with a large mean. They pass with and without numba.
The replications of a statistic were stored in a float64 array, so that a function that returns complex values lost its imaginary part with a ComplexWarning (apply, cov, var) or gave complex intervals that mean nothing (conf_int). The data of the bootstraps are not converted and are resampled as given, so complex data worked until the statistic. * apply, cov and var store the replications as complex128 for a complex statistic. * cov is the Hermitian covariance (errors.T @ errors.conj()), and var is the real variance, the mean of the squared absolute errors. * conf_int raises a TypeError for a complex statistic, before the replications are run, since a region and not an interval is needed. A reused result is checked too. For z = a + ib the covariance of the bootstrap mean equals S_aa + S_bb + i (S_ba - S_ab) where S is the covariance of the mean of the real data [a, b] computed with the same seed, for the iid, stationary, circular and moving block bootstraps. Real statistics are unchanged.
test_validation_dtypes.py has a row for each entry point, argument and data type, built on validation_table.py: the unit root tests, the cointegration tests and estimators, the mean models, the volatility processes, the distributions, the covariance estimators, cov_nw, the multiple comparison procedures, optimal_block_length and add_trend. Each argument is replaced by float16 (rejected, with the data type in the message), complex64 and complex128 (rejected, except where the estimator is Hermitian) and by bool, integer, unsigned integer, float32, float64 and longdouble data (accepted). pandas Series and DataFrames with one bad column are rejected too. test_dtype_inputs.py runs the cases of test_integer_inputs.py, 41 calls of the public API, with float32 data (the results equal those of float64), float16 data and complex data (TypeError), and runs each entry point with random float32 data, rounded to float32 first, and requires the result to equal the float64 result for the same values to 1e-12. A calculation in float32 differs by 1e-7 or more.
contributing.rst: rule 5 of the input validation section points to a new "Data types" section, which has the table of data types by the default and by allow_complex, why float32 is promoted, why float16 is rejected, where complex data are accepted and why the others are not, and how to add complex support to a function and to test it against the real and imaginary parts. 9.0.rst: the entry about the conversion of integer, unsigned integer and Boolean data states the rules for all data types and credits antonsoo for the first float32 and complex support of the covariance estimators (#942, #943), and a new entry in the new features describes the Hermitian covariance estimators, add_trend and the bootstraps.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+41
to
+46
| from arch.utility.array import ( | ||
| _numbers_rule, | ||
| _rejected_dtype, | ||
| as_float64, | ||
| cutoff_to_index, | ||
| ) |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #957 +/- ##
========================================
Coverage 99.00% 99.01%
========================================
Files 129 134 +5
Lines 28773 29563 +790
Branches 2231 2290 +59
========================================
+ Hits 28488 29271 +783
- Misses 233 239 +6
- Partials 52 53 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
bashtage
added a commit
that referenced
this pull request
Oct 17, 2026
bashtage
added a commit
that referenced
this pull request
Oct 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Branch
a90-dtype1-dtypes, commits ona90-fix-main-tests. Merge #953 first.It merges cleanly with #955 and #956. This settles release decision 65 and the
maintainer's request: "someone might want to use complex data in a covariance
estimator ... Arch models must be float64. Unit root tests are also float64 or
float32 (I think float32 is OK, but you check). Same for cointegration ... Float16
is never OK. Find other places though where complex might make sense." It also
fixes the 7 tests that fail on main since the last commits of #942.
One rule, in one place (
_rejected_dtypeinarch/utility/_validation.py,used by
as_float64,ensure1d,ensure2d,check_arrayandto_array_1d):converted to float64, and every calculation is done in float64.
TypeErrorthat names the argumentand the accepted types, also for a float16 DataFrame column.
TypeError, except in the functions that estimate aHermitian covariance (below), where it becomes complex128.
float32 is accepted as input but promoted. Computing in float32 is not
accurate enough. With every array in float32:
but 8e-3 for long series with a large level. The automatic bandwidth was off by
4.8%.
of 1000, and the ADF-AIC and Zivot-Andrews lag searches raised "singular
matrix" in every replication for large levels or trends.
float32 is also not faster. So float32 data give exactly the float64 result for
the same values, which is tested. The scripts and outputs are in the release
folder (
work/dtype1/prec_*).Complex data, where they make sense:
cov_nwestimate the Hermitianlong-run covariance. They use the conjugate transpose (the lag-j autocovariance
is the sum of
x[t] conj(x[t-j])), the automatic bandwidth uses the real partof the autocovariances, and the estimates are Hermitian complex128 with a real
diagonal.
Omega_aa + Omega_bb + i(Omega_ba - Omega_ab), computed from the stacked real
data [a, b] with the same kernel and bandwidth (10 kernels x center x
df_adjust, max error 4e-13). Real float64 results are bit-identical to before.
apply,covandvaraccept statistics with complexvalues.
covis Hermitian andvaris real; the imaginary part used to bedropped with a
ComplexWarning.conf_intraises aTypeErrorfor a complexstatistic, since it has no interval.
add_trendaccepts complex data and adds real trend columns.PreWhitenedRecoloredrejects complex data. A complex VAR is eitherstrictly linear or the stacked real VAR, and the selected order depends on that
choice. Its notes say to stack the real and imaginary parts, or to use a kernel
estimator.
TypeError: unit root and cointegration tests, ARCHmodels, volatility processes, distributions, MCS/SPA/StepM,
optimal_block_length,auto_bandwidthandto_array_1d.to_array_1dusedto drop the imaginary part silently.
ensure1d/ensure2dgain anallow_complexkeyword (default False).Changes to #942's tests (antonsoo; credit kept in the release notes):
test_preserve_floating_kernel_dtypebecomestest_floating_kernel_dtype_is_float64. float32 input gives float64 for allfour estimates; on main, BUG: Promote discrete covariance inputs before products #942 left
short_runfloat32 andlong_runfloat64.test_preserve_inexact_newey_westbecomestest_inexact_newey_west. Itsexpected value now uses the conjugate, since
data.T @ datais thepseudo-covariance, not the covariance.
Also fixed: a crash with longdouble data on Windows, and a leftover
z = asarray(y)incov_nwfrom a merge.Tests:
test_validation_dtypes.py(571) andtest_dtype_inputs.py(200),plus the Hermitian identity, float32 equality and float16 rejection tests.
Checks: the golden battery is identical, numpydoc lint finds nothing, and
style is clean. The contributing guide has a new "Data types" section, and the
release notes have the rules, the complex support and the float16 change.
🤖 Generated with Claude Code