Repository navigation
Fuse the ARM PMULL dense-grid extension and products - #561
Merged
tcoratger merged 5 commits intoOct 8, 2026
Merged
Conversation
Benchmarks: base against this PR, on one runner299bcef merged into its base, against the base f52bd99: each benchmark run in turns on one runner, base and PR 5 times each. A size is shown when it changed; a time or a peak memory when it moved the same way in every round and its median by at least 1% (the change is the median ratio, with the rounds' range). Proving time
Prover stages
Verifying time
Every resultProving time
Prover stages
Verifying time
Time per operation
Proof size
Peak memory
allocations
heap-peak
|
Collaborator
Author
|
Normal PR benchmark37775824371, PR#561 final head299bcef05c0ca716bdfecd086ebdecb6484f76b2, merge78e2ea0cfc29cd99d9adb995fffebb0ea32518ca, basef52bd991c8623894e7b03a2e50bede6c248ea1ad. All46native jobs passed;90result cases,5alternating pairs percase. Negative percent is faster. Standard workflow results are separate from the instrumented controlled captures.
|
kevaundray
marked this pull request as ready for review
October 8, 2026 12:35
tcoratger
approved these changes
Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fuse the final ternary-grid extension digit with the mixed products on AArch64 PMULL, retaining the three unreduced F192 coefficient accumulators for each of the three generated points in registers across the eight offsets. Store the accumulated coefficients in the existing ProductRow representation. Earlier extension digits, packing, reduction, chunk scheduling, weight filling and scratch layouts stay unchanged. The R0 and non-PMULL paths retain their original implementation and captured loop bound.
This is a prover kernel change only. It does not change the protocol, transcript, guest, proof format or public API. The arithmetic/layout regression test checks all ternary-grid values against an independent bit-serial field definition across rounds, block sizes, partial groups and padded tails. Existing module documentation explains the fused path.
Hardware evidence
Baseline:
f52bd991c8623894e7b03a2e50bede6c248ea1ad, including #549 and #550, excluding #552 and guest rewrites. Final source:299bcef05c0ca716bdfecd086ebdecb6484f76b2.Actual Neoverse V2 and Xeon measurements used rustc 1.99.0 (
b940084d7),-C target-cpu=native, thin LTO and one codegen unit. Each comparison used five alternating paired runs on the same machine, at 1, 4, 8 and default workers. Default means 16 workers on V2 and 32 on Xeon. Runs were serialized under the benchmark lock and load-checked. No hardware performance counters were available. Component timings are not added together.The screening run compared baseline, final-digit fusion alone and fusion with register-resident accumulators. Register-resident accumulation was selected. Fusion alone improved the V2 dense pass by about 9%, versus about 20% for the selected variant. The missing-shapes confirmation measured the selected clean implementation at
2872ebe0b0ae403213082c4e9f8fe72cff002266, adding leaf rate 2, first and higher aggregation nodes at both rates, and the captured seven-lane R3 tail.V2 paired median changes in reported whole-proof latency, with negative values meaning faster:
Every V2 scenario median improved. All 20 leaf-rate-1 pairs improved; 97 of 100 pairs in the missing-shapes matrix improved. Three individual default-worker observations were slower, so this is not a claim that every observation improves.
The actual captured V2 dense pass changed from 90.893/22.946/12.530/7.096 ms to 72.764/18.466/10.007/5.625 ms at 1/4/8/default workers. Paired median changes were -19.97%/-19.63%/-20.03%/-20.73%. The captured seven-lane R3 tail improved by 21.45% to 22.40%. The leaf-rate-1 Basis span improved by 5.13% to 5.80%. These are separate measurements, not additive contributions to whole-proof latency.
Xeon guard and final source relationship
The first clean candidate unnecessarily routed the non-PMULL implementation through a generic wrapper. Although its arithmetic was unchanged, that altered native loop code generation and three higher-node rows were consistently slower: rate 1 at one worker +0.488%, rate 2 at one worker +0.326%, and rate 1 at eight workers +0.299%, each with all five pairs slower. The final commit restores the original inline non-PMULL loop and captured point bound, rather than shipping this avoidable x86 code change.
The targeted final-head Xeon control repeated exactly those three rows:
The eight-worker control still has a positive median and four of five pairs slower. This is disclosed rather than claiming universal runtime neutrality or assigning a cause. All 5,473,448 disassembled executable bytes and instruction addresses matched baseline exactly in the final native Xeon artifacts, including all R0 through R4 grid closures. This is not a claim that entire ELF files, data or runtime environments are identical.
The final non-PMULL isolation does not change the selected ARM arithmetic. The complete R3 and R4 instantiated ARM closures have identical normalized instruction sequences before and after isolation, after ignoring relocation-dependent target addresses: 1,312 and 1,438 instructions respectively. This is emitted-code provenance supporting reuse of the V2 measurements, not a new hardware timing result.
Normal PR benchmark confirmation on the final head
The completed normal PR Bench matrix tested final head
299bcef05c0ca716bdfecd086ebdecb6484f76b2through merge78e2ea0cfc29cd99d9adb995fffebb0ea32518caagainst the same baseline. All 46 native jobs passed, producing 90 result cases with five alternating pairs each. These are the standard workflow measurements, separate from the custom load-checked capture experiments above. The complete 90-case table retains baseline/head medians and sample ranges, paired-percent medians and ranges, and slower-pair counts for every full-proof and microbenchmark case.Final-head V2 whole-proof paired medians in the normal workflow:
V2 improved 25 of 26 full-proof scenario medians and 108 of 130 individual pairs. The exception is
falcon-7: +0.214% paired median, range -1.388% to +0.582%, with four of five pairs slower. Other default-worker proof medians were fibonacci -1.246%, hash -1.118%, leansphincs -0.924%, shielded -0.768% and stateproof -1.045%. The normalpcs-open-22microbenchmark improved by -3.907%, range -6.003% to -1.484%, with all five pairs faster. Primitive arithmetic microbenchmarks include both signs; their full results are retained, without attributing unrelated primitive timings to this kernel change.Xeon improved 18 of 26 full-proof scenario medians; its median changes range from -2.252% to +0.569%. Retained positive medians include the eight-worker leaf +0.569% [-1.333%, +1.114%], default higher 4-to-1 aggregation +0.499% [-2.783%, +1.155%], and shielded +0.406% [-0.221%, +3.348%]. No universal x86 runtime neutrality is claimed. The earlier executable-byte equality receipt belongs to the targeted control, not these normal PR timing artifacts.
In this additional matrix, V2 median tracked heap differences ranged from -73,488 to +68,096 bytes and allocation differences from -2 to +2; Xeon ranges were -42,720 to +231,424 bytes and -3 to +1 allocations. RSS median differences ranged from approximately -15.58 to +7.76 MiB on V2 and -10.96 to +2.46 MiB on Xeon. These are observations, not claimed allocation savings.
Memory and instruction traffic
Scratch remains 119,040 bytes and the separate accumulator remains 15,552 bytes per state. No new allocation or copy is introduced. Across the V2 whole-proof rows, median tracked heap differences were between -106,440 and +78,264 bytes, allocation-count differences between zero and six, and RSS differences between about -1.95 and +8.41 MiB. One-worker tracked heap values were unchanged. These small multiworker tracking/lifetime variations are retained rather than presented as a memory saving.
The selected R4 product loop has no stack accesses. Its carryless-product count is unchanged. Static logical extension/product traffic per eight-offset R4 row falls from 101,760 to 50,784 bytes; this excludes packing, setup, reduction, cache effects and earlier-extension spills, and is not a counter measurement. Earlier R4 extension spills remain 272 bytes stored and reloaded. R3 introduces 384 bytes stored and reloaded in the earlier extension, despite its measured tail improvement. No SVE or generic SIMD throughput claim is made.
Correctness and validation
The retained capture harness compared literal proof bytes, not just digests: 120 leaf-rate-1 proof files in screening, 560 leaf/first-node/higher-node proof files in the missing-shapes matrix, and 90 additional final-head Xeon control files. For each matching input/rate/node, bytes agree across baseline/candidate, worker counts and architectures; the final control also matches the earlier captures. Every measured production proof path completed verification. The source tree contains no profiling switches, proof-dump hooks or diagnostic workflow changes.
Validation includes native focused first-pass tests, an actual final-head first/higher aggregation proof-and-verify smoke, PMULL QEMU correctness on Rust 1.97 and 1.99 for the selected implementation, and final-head portable ARM fallback tests on both Rust versions. QEMU results are correctness and emitted-code checks, never performance measurements. The exact final-head CI run passed all 13 jobs, including release workspace tests, executed NEON and VPCLMUL kernel tests, formatting, clippy, rustdoc, MSRV, Miri and dependency checks. The GFNI job compiled and linted successfully but skipped execution because its runner lacked GFNI.
The optimization is independent of outer weight-fill/chunk scheduling. Combining it with another optimization requires a new measurement; no combined speedup is claimed.
The normal PR checks also passed: Rust, RISC-V ACT4, Counts and Python verifier lint/types. Both exact-head Rust runs executed NEON and VPCLMUL tests but skipped the dedicated AVX2+GFNI runtime tests because those runners lacked GFNI. Separately, native Xeon production proof-and-verify runs executed successfully with GFNI enabled in the captured native target configuration; that is not a claim that the skipped dedicated AVX2+GFNI suite ran.