Repository navigation
Reduce additive NTT work in retained and lifted transforms - #550
Merged
tcoratger merged 1 commit intoOct 8, 2026
Merged
Conversation
Skip zero-twiddle products, prune discarded WHIR transpose outputs, and apply the original GF8 butterfly network directly to the reduced E-valued Flock vector. Preserve all point orders and proof bytes. Native PCS/Flock tests and AArch64 NEON emulation pass. Hash and leanXMSS leaf proofs plus first and recursive aggregation proofs are byte-identical to upstream at 1, 4 and 32 workers. The isolated size-64 lifted helper measured 51.889 us before and 0.998 us after on Ryzen 7950X3D with rustc 1.97.0; this is not an end-to-end prover speed claim.
Benchmarks: base against this PR, on one runnere5cc3b0 merged into its base, against the base edafd39: each benchmark run in turns on one runner, base and PR 5 times each. A size is shown when it changed; a time or a peak memory when it moved the same way in every round and its median by at least 1% (the change is the median ratio, with the rounds' range). Proving time
Prover stages
Verifying time
Time per operation
Every resultProving time
Prover stages
Verifying time
Time per operation
Proof size
Peak memory
allocations
heap-peak
|
Collaborator
Author
|
Adding to backlog since change is not that large |
tcoratger
approved these changes
Oct 8, 2026
tcoratger
left a comment
tcoratger
left a comment
Collaborator
There was a problem hiding this comment.
We should merge no? Since this brings an improvement without many lines of code changed
This was referenced Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reduce arithmetic in three existing additive-NTT paths without changing domains, point order, transcript values, proof format, or verifier logic. Based on upstream
edafd396120f453de6f75013f231bfda2049f217.Update the existing protocol annexes with the implementation identities and add independent Lagrange-interpolation and retained-output regression coverage.
Arithmetic and storage
For a dense transpose of N entries retaining n=N/2^r outputs, the discarded suffix previously performed rN/2 E-by-K products and rN E additions. Its replacement has zero products and N-n E additions. Each E-by-K product is three K products, and each E addition is three K XORs. Sparse-prefix cost still depends on query occupancy; these are dense-suffix counts, not a claim that every sparse opening pays the dense bound.
At the production Flock domain size 64, the old reduced-vector helper performed 12,288 full E products and E additions, plus Boolean packing and table work. The lifted network needs 384 E-by-K products and 768 E additions before eliminating zero twiddles; the standard input domain eliminates 63 products and additions, leaving 321 E-by-K products (963 K products) and 705 E additions. It uses one output allocation rather than three temporary/output allocations. The cached lifted twiddles add one allocation and 1,008 bytes per table at size 64; table construction is included separately in the experiment below. There is no new allocation in the PCS transpose or row-butterfly shortcut.
Why this is not a sub-n-log-n NTT claim
The investigation read the supplied
int-mul.pdfin full. Its integer-multiplication theorem is stated in a fixed finite-tape Turing-machine model; cheaper tape address interchange does not itself prove a cheaper finite-field arithmetic circuit or a RAM speedup. Its Gaussian-dyadic synthetic butterflies use division by two and fourth-root phases, which do not transfer to leanVM's characteristic-two fields. The paper separately cites a companion exact-complex-DFT arithmetic result; that companion was not available for independent inspection.Exact executable matrix models rejected the small generic radix-4/radix-8 candidates tested. A same-twiddle tensor can share products, but sibling twiddles in the actual additive NTT differ. Accounting for that difference raised the radix-4 count from 4 products / 8 XORs to 5 / 9 or 6 / 14, and the radix-8 count from 12 / 24 to 14 / 26 or 16 / 36. Those networks were not shipped. The implementation here instead removes provably unused outputs, redundant zero-twiddle work, and an unnecessarily expanded representation of the same small-field linear map. It remains an O(N log N) additive NTT.
This does not duplicate the open #464 register-tile/few-worker scheduling work. It changes arithmetic and retained outputs rather than selecting a wider fused tile.
Local experimental evidence
Host: Ryzen 9 7950X3D, rustc 1.97.0, native ISA. Runs use the shared benchmark lock, a 16 GiB systemd memory scope with swap disabled, start below load 12, and reject load above 20. One worker is pinned to CPU 2, four workers to CPUs 8-11; 32 workers use the full host.
A standalone executable comparing the old and new reduced-vector algorithms on exactly equal E inputs measured the size-64 extension at 51,889 ns versus 998 ns. Its table construction measured 20,346 ns versus 19,921 ns, within setup noise. Sizes 8, 16, 32 and 128 also produced identical outputs and lower extension time. These are isolated helper measurements, not a 52x prover-speed claim. The measured helper used two twiddle allocations; the committed layout consolidates them into one with the same arithmetic.
Final-head local ABBA results, five measured passes per process and two processes per side, are below. Values are arithmetic means in milliseconds, base -> head; percentages include every measured pass. Load stayed below the rejection threshold. Rate here denotes
log_inv_rate, and PCS log-size is 22.The cleanest local evidence is the isolated lifted helper and single-worker NTT/commit reduction, not a uniform prover win. The 4-worker measurements remain unstable, including the rate-2 NTT regression. The 32-worker recursive aggregation regression is retained in the table: its large excursions occurred in
stage.reduce-claims(up to 135 ms on head, with first-level base also reaching 151 ms), whilestage.prove-recursionremained around 63-65 ms. An earlier alternating experiment had excursions in the opposite comparisons as well. These shared-host results do not justify attributing every percentage to the NTT change; the dedicated native comparison below is stronger evidence.Native CI performance
Bench run 37705721691 completed for exact head
e5cc3b0cfd8a57318110cd7f3d8a7d43b1f58ea1, testing merged685cc47f943f648598f102017da5b66335661d1against baseedafd396120f453de6f75013f231bfda2049f217. Each case alternates base/head for five rounds on one dedicated runner. Both use rustc 1.99.0 andtarget-cpu=native, 4 KiB pages, THPmadvise, and the workflow's 16 GiB/no-swap process scopes. x86 is a 32-vCPU Xeon Gold 6548N with AVX-512, VPCLMULQDQ and GFNI on Linux 6.8.0-142; ARM is a 16-vCPU Neoverse-V2 with PMULL/SHA3 on Linux 7.0.0-1014-aws. These are native ARM measurements, not QEMU timings or an Apple Silicon claim.Means in milliseconds, base -> head:
The default-worker leanXMSS base/head ranges do not overlap on either runner: Xeon 116.836-117.901 versus 114.039-116.197 ms, V2 213.376-215.902 versus 210.753-212.241 ms. NTT median changes are -1.30% on Xeon and -3.48% on V2. Other gains are small and not universal: in particular, the table retains the Xeon 2-to-1 aggregation regressions, and ARM PCS-open is effectively unchanged (median -0.26%). The full artifacts also cover all 2-to-1 and 4-to-1 aggregation cases at 1/4/8/default workers and every other standard benchmark. Do not interpret the helper's isolated speedup as the speedup of the prover.
One-worker leanXMSS uses 73 fewer allocations on both platforms. Its measured heap peak is unchanged on Xeon and increases by precisely the 1,008-byte twiddle table on V2. The paired native traces show Xeon Round 1 at 92.2 -> 88.3 ms, but V2 at 308 -> 308 ms at the trace's displayed precision: this does not resolve the bulk ARM Round 1 bottleneck. Proof sizes are identical across both sides of all 52 native workload/architecture proof metrics.
Correctness evidence
cargo test -p pcs -p flock: 43 PCS tests and 49 Flock tests pass.neoverse-n2,+aes,+sha3passes. QEMU AArch64/NEON runs all 92 PCS/Flock tests successfully, then reruns both lifted-extension tests after consolidating twiddle storage. QEMU is correctness evidence only. Its pure-Rust test executables do not link the bench allocator; an empty archive override bypasses the unused jemalloc cross build, not any tested arithmetic implementation. The initial all-target GNU check could not build jemalloc because this host lacksaarch64-linux-gnu-gcc; native ARM CI covers that build.cargo clippyallandcargo docallpass.cargo fmt --allapplied. The revised LaTeX document builds with Tectonic, and both changed pages were visually inspected; existing warnings are in unchanged sections.latexmkis not installed locally.log_inv_rate=2for leaves and 1 for tree proofs. Final branch proofs are byte-identical to upstream and across 1/4/32 workers, not merely equal in size. Full-size workload hashes:e17f3f04c9025369ad75dbbd9b900fd7d8b073fb0eb13792c6db3db40b7f09b807c0d84852ad15193811f90c968df9e8ff637d2d30ff09c8d178e75e89ed47a5c6c9503072535f592c331a5c276d3b6f1c8abe8f03fb51f68032e172b3348a40fe6a64543e5ad5596fd324ef779834aea250b343ac83b15ec2656bee55b4e46fExact-head CI reports 63 successful checks and one intentionally skipped PDF-publication job. This includes the full release workspace test job, MSRV, both native Clippy targets, AVX2/GFNI/VPCLMUL and NEON tests, Miri, ACT4, Rustdoc, formatting, dependency checks, Python verifier checks, counts, PDF validation and every native benchmark job. Rust run, benchmark run and raw artifacts.