Repository navigation
feat(bb): per-proof batch verification responses under bb msgpack run --reactor - #25636
Draft
charlielye wants to merge 19 commits into
Draft
charlielye wants to merge 19 commits into
charlielye wants to merge 19 commits into
Conversation
…e legacy bb::ipc Combined redo of #23612 + #23614 + #23613. Those predate envelope ids (the 8-byte request-id frame prefix), which broke wire compatibility between the old and new stacks, so bb, bb.js and bb-rs have to move together. bb now serves msgpack through ipc-runtime on every transport: `bb msgpack run` maps ""/"-" to a stdio pipe server, .sock to UDS and .shm to shared memory, all dispatching through one generated handler with envelope framing (a plain file keeps the offline bare-frame replay). The command surface comes from a checked-in bb_schema.json in the friendly ipc-codegen dialect, with handlers implemented as thin adapters over the existing domain commands. The typed cbind entrypoint is replaced by the ipc-codegen FFI contract ipc_ffi_entry, so wasm and static-linking consumers speak the same payload as the transports. That makes the legacy machinery consumer-free, so it goes: the in-tree ipc library and its benchmark, the nodejs_module msgpack client (bb.js now uses ipc-runtime's NAPI addon, and nodejs_module only exports LMDBStore), the Command/CommandResponse named unions with their execute dispatcher, and the func_traits/schema reflection behind the CBIND macros. Both client libraries now generate from ipc-codegen rather than forks of it. bb.js/src/cbind (a stale copy of the generator: schema visitor, TS and rust backends, naming, friendly-schema lowering) is deleted in favour of calling ipc-codegen, and barretenberg-rs drops its hand-written Backend trait, error type, Fr/Point types and both backends — codegen emits those, and ipc_runtime::IpcClient plugs in as the Backend, so the crate is ~89% generated with a deprecated BarretenbergApi shim keeping the published surface. The hand-rolled PipeBackend goes with them; its tests now run over ipc-runtime's UDS transport. ipc-codegen gains --strip-type-prefix (bb.js and barretenberg-rs publish unprefixed names while wire tags keep the service prefix) and three rust serde fixes: fq[2] pairs, [u8; N] and above-cutoff byte arrays were encoded as sequences of integers rather than msgpack bin, which the C++ side rejects. ipc-runtime gains the pipe transport (fd pair or stdio, peer EOF requests shutdown), a sync spawned-process backend for callers that cannot await, pre-spawn removal of stale shm segments (they are created O_EXCL, so a killed server's leftovers blocked the next one), and an unref option so a spawned backend cannot hold the Node event loop open while idle. Its rust and zig build files now discover the C++ sources instead of keeping hand-written copies of the CMake list, which had silently drifted. Also fixes two latent breakages from the ts/ -> ts/bb.js/ restructure that blocked wasm and pinned-flow tests: the barretenberg_wasm symlinks and the chonk pinned-inputs repo-root resolution were both one directory short. Raises MAX_FRAME_SIZE from 256 MiB to 1 GiB. bb.js reaches bb over ipc-runtime's socket transport now rather than the deleted nodejs_module msgpack client, so proving requests are newly subject to the frame cap: the rollup circuits exceed it, and bb_prover_full_rollup failed with the server refusing a 314525552-byte frame and the client seeing the closed connection as "write EPIPE". The TS client also checks the outgoing size, so a request over the cap names itself instead of surfacing as an unexplained EPIPE on the next write. Fixes two gaps in unref handling that left the Node event loop open. The shared-memory backends dropped the caller's option -- their new() did not take it and createBackend never passed it -- so unrefStdio was never set and the child's stdout/stderr pipes (which exist whenever a logger is set) kept the loop alive; they thread it through as the socket backend already does. And BarretenbergSync.initSingleton did not force unref: true, unlike its async counterpart, so the sync backend it creates held the loop for the life of the process. Nothing destroys a singleton, so it opts in now too. Together these had a labs pxe test pass all 7 cases and then hang until the job timed out. Rebased onto next, which added two things to the surface this replaces. #25487 hardened the msgpack entrypoints: the generated dispatch already answers a malformed envelope, an unknown command and a throwing handler with an error frame, but an unpack or convert failure still escapes it, so api_msgpack.cpp and ipc_ffi_entry catch that rather than lose the response. Poseidon2AbsorbChain was a new command registered in the NamedUnion this deletes, so it moves to bb_schema.json with a handler alongside the other crypto ones, and its malformed -length test is reexpressed against the FFI entrypoint. barretenberg/acir_tests is its own yarn project and consumes bb.js through a portal, so it has to resolve bb.js's dependencies itself. bb.js declares @aztec-foundation/ipc-runtime with the placeholder specifier that only a resolution gives meaning to, which barretenberg/ts has and acir_tests did not: `yarn install` there failed with "isn't supported by any available resolver". It gets the same resolution. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016einpgfthfjLYGwCB3iQqD
…ull bb build It takes ~35s idle but exceeded its 360s jest timeout (408s) when the lane ran concurrently with a full bb rebuild. It uses the published bb.js, so the slowdown is load, not the tree: raise the per-test timeout to 20 minutes and the lane's to 25. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
…al run() loop The shared-memory transport woke the other side with a futex syscall on every publish and release, about six per bb request, which roughly doubled the pipelined poseidon benchmark against next. Those wakes were made unconditional to fix an intermittent hang, whose cause turned out to be a startup race rather than the conditional wake itself: segments are mappable from ftruncate, before create() initialises them, so a fast client could attach, set its blocked flag on a response ring, and have the server's initialisation reset it; the next publish then skipped the wake and the client slept on published data forever. connect() now refuses a segment until it is fully initialised. create() writes the ring capacity, and the doorbell's num_slots, last with release ordering, and a zero value means not ready: SpscShm::connect and MpscProducer::connect throw and the existing connect-retry loops try again. With that in place the conditional wake is restored on the rings, the doorbell and the reactor's notify(). bb also served every request through run_reactor, paying a request copy, a heap-allocated respond closure, a locked completion queue and a notify() (a self-pipe write on sockets) per request, although every bb handler responds before returning. It now uses the serial run() loop, asserting that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
…and stop the MPSC spin reading the clock every iteration Pipelined bb.js calls over shared memory were still ~50% slower than next on the poseidon benchmark after the futex fix. Layer by layer, the transport itself was at parity; the extra cost was main-thread JavaScript per call, which left the main thread saturated so the API's own encode/decode no longer overlapped with bb's work. Most of it was SpawnedProcessBackend.call: an async function awaiting ensureUp() before every call. A live process now takes a direct path with no async frame; only a missing or dead process goes through ensureUp(). On the benchmark's pattern (the first pipelined burst after 10k sequential calls) this takes the fixed build from 17.3 to 13.4 us/call (min of 8, next 14.1). MpscConsumer::wait_for_data's spin also called clock_gettime on every iteration, about a tenth of bb's instructions while serving; it now checks the clock every 256 iterations, as SpscShm's spin does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
… with small-int ids NapiShmAsyncClient kept pending calls in a Map<bigint>. Each call allocated two heap BigInts (the id and the one napi hands back) and hashed them three times. Under jest, where the main thread saturates during a pipelined burst, that overhead was directly visible in bb.js's poseidon bench. Ids are now numbers below 2^30, so V8 keeps them as small integers. Pending calls sit in an issue-order queue: in-order servers (bb's serial run() loop) always match the head, and out-of-order responses fall back to a search. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
…a per-call .catch SpawnedProcessBackend.call chained .catch(attributeCallError) onto every call, so each successful call paid for an extra promise and microtask. In a pipelined burst under jest that was ~0.8us/call of main-thread CPU in steady state and ~2us in the first burst. The SHM and UDS clients now take an optional mapError hook, applied only on their rejection paths. SpawnedProcessBackend passes attributeCallError for the incarnation when it connects, and the fast path returns the client's promise directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
Every mode was timed on its first run, so the numbers included JIT warm-up for that call pattern. A pipelined burst exercises different code paths from the sequential warm-up that preceded it, so it was the most affected. Its first-burst result swung by several µs between CI runs. Each mode now runs once untimed first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
UdsIpcClient still kept pending calls in a Map<bigint>, with the same per-call BigInt allocation and hashing the SHM client had. Both clients now use one PendingQueue: small-int ids, head-of-queue fast path, search fallback. The UDS wire id stays u64; live ids sit in the low word, and a nonzero high word never matches a call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
The 7.0 line is a breaking release, and older barretenberg-rs versions stay on crates.io for anyone who needs the old API (semver keeps `"6"` manifests off 7.x). Removes the legacy BarretenbergApi/PipeBackend/byte-valued types, the old module paths and the inert `native`/`async` features, so the crate exposes only the generated BbApi. The tests move to BbApi with typed scalars; the pipe tests spawn bb and drive it through ipc_runtime::IpcClient::from_fds. The README gains a 6.x migration note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
barretenberg-rs now depends on the ipc-runtime crate, which had no version and was not on crates.io, so `cargo publish` refused the package. The release step now publishes ipc-runtime at the same version first and pins it exactly. ipc-runtime's build compiles the C++ sources from ../cpp, which a published crate does not have; the release copies them into the crate's cpp/ directory and build.rs falls back to it. Publishing happens from the barretenberg-rs release rather than ipc-runtime's, because the latter also runs for private releases, which must not reach crates.io. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
…release ipc-runtime owns its crate, so its release publishes it. The release also runs for private releases, which publish only to the internal npm registry, so the crates.io step is skipped when assert_public_release refuses. ipc-runtime is released before barretenberg/rust, whose release now only pins the version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
The wake is skipped unless the waiter has flagged that it is about to sleep. Each side stores (head/tail/seq, or the flag) and then loads the other's value, and release/acquire lets each load run ahead of its own store. Both sides could then miss each other, leaving the waiter asleep on published data, with no timeout on the default call path. A seq_cst fence between the store and the load on both sides guarantees one of them sees the other. The futex lock already fenced the waiter in practice; the SPSC producer had no such cover. No measurable cost in a pipelined poseidon A/B on a loaded host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
UDS and pipe frames were capped at 1 GiB (256 MiB until bb proving requests outgrew it). Nothing in a stream transport needs a limit below the u32 length field; the cap only guarded against a corrupt prefix driving a huge up-front allocation. Receivers now grow their buffer as payload bytes arrive (C++ read_payload, TS FrameReader), so that guard is no longer needed and the limit is the wire format's own. SHM stays bounded by half its ring capacity. FrameReader also fixes the TS client and server re-concatenating their whole buffer on every chunk, which made receiving a large frame quadratic. A request too large for the length field now rejects with a non-transport IpcError: it is the caller's mistake on a healthy connection, and as a transport error SpawnedProcessBackend killed the server for it. Senders also check against the payload limit (frame limit minus the 8-byte id) rather than the frame limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
Serving "-" made the process's stdout the frame stream, so any other write to it (std::cout, printf, a library's diagnostics) landed inside a frame and desynced the peer. next's bb guarded this by redirecting std::cout; that guard did not carry over to the ipc-runtime pipe path. make_server now serves on a private duplicate of stdout and points fd 1 at stderr, which also covers C stdio and raw writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
It looked for nodejs_module.node, which no longer ships now that bb.js uses ipc-runtime's addon, so it could only ever return null. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NwZD6M5xfpDdAFkdbkWT67
…t, under bb msgpack run --reactor The batch verifier's results went over a side-channel FIFO because bb's server replied to each request before reading the next: a call that waited for its batch would have stopped more proofs arriving. With --reactor, bb serves with ipc-runtime's run_reactor, so a handler may reply after it returns. ChonkBatchVerifierVerify queues a proof with a batch verifier started without a FIFO and replies with its verdict once checked; a proof bb can't queue (service not running, queue full) gets an error reply, not a verdict. bb.js gains a reactor option on the socket backend. The FIFO commands are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NwZD6M5xfpDdAFkdbkWT67
… let a stop answer queued proofs - An unregistered vk_index is an error, not a verdict. - verify() waits for room in the batch verifier's queue instead of failing it, so callers need no in-flight cap. - Stopping the service, or the server, answers proofs still queued as unchecked errors rather than verifying them all. - run_reactor: an on_stop hook; a handler that throws or a send that throws stops serving without unwinding past responders still in flight; notify() before releasing the request. - serve_msgpack_reactor is factored out, and tested end to end over a socket with pipelined real proofs. - bb.js refuses the reactor option on backends that can't honour it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NwZD6M5xfpDdAFkdbkWT67
charlielye
force-pushed
the
cl/ipc-bb-bbjs-migrate
branch
from
October 13, 2026 20:02
9f50140 to
b8a89ca
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #25362. It needs that PR's ipc-runtime server in bb and ipc-runtime client in bb.js.
Why
aztec-node's p2p batch verifier gets each proof's verdict over a side channel: a named pipe (FIFO) that bb writes results to. It does this because bb's msgpack server reads a request, runs it, and replies before reading the next one. A call that waited for its batch would stop more proofs arriving, so batching would never happen.
The side channel brings a lot with it on the TS side:
aztec-labs-eng/aztec-node#359 has the details.
What
bb msgpack run --reactor. bb serves with ipc-runtime'srun_reactorinstead ofrun(). Handlers still run one at a time, in arrival order, on the reactor thread, so session state needs no locking. A handler may now respond after it returns, from another thread, and responses go out as they complete, matched by request id. Without the flag nothing changes. The flag is opt-in because the reactor copies each request and wakes across threads.ChonkBatchVerifierVerify {vk_index, proof_fields}→{verified, time_in_verify_ms}.fifo_path).verified: falsemeans bb judged the proof: too short to split, wrong size, or failed the check (off-curve commitments included).vk_index, the service not running, or the service stopping before the proof is checked.--reactorthe command is refused, rather than stalling the server.Verify, and one started without refusesQueue.ChonkBatchVerifierStop, or the server shutting down viarun_reactor's newon_stophook, answers proofs still queued as unchecked errors and only finishes the batch in progress. Otherwise it would verify the whole queue for callers that may be gone.run_reactor.on_stophook.send()that throws now stops serving cleanly instead of unwinding past responders still in flight (use-after-free). A request whose handler threw without responding is released, so shutdown doesn't wait for it forever.notify()now runs before the in-flight decrement, so a cross-thread responder can't touch a server that has just been destroyed.reactoroption on the NativeUnixSocket backend passes--reactor. Other backends refuse it.After a bb release with this, aztec-node's
BatchChonkVerifierbecomesawait bb.chonkBatchVerifierVerify(...)per proof. The FIFO, the probe, the request-id map and the in-flight bookkeeping all go away.Not done
batch_checkand then bisected) still read asverified: false. That's the same as the FIFO path today. Separating them means telling proof-derived exceptions from bb's own inside the verifier.run_reactorkeeps until it responds.reactorisn't wired into the shared-memory backends.Testing
api_msgpack_reactor.test.cpp: end to end over a real Unix socket throughserve_msgpack_reactor, with real (small) Chonk proofs.bbapi_chonk_batch_verify.test.cpp: through the generated dispatch: verdicts for good, tampered and malformed proofs; refusal without--reactor; the two modes refuse each other's command.socket.test.cpp: a reactor handler that throws without responding stops the server, runson_stop, still sends the deferred response, and returns. Before the change it crashed the process.batch_verifier_verify.test.ts, against the built bb: several calls in flight on one connection, the verdict-vs-error split, and refusal without--reactor.ipc_runtime_tests32/32,ChonkBatchVerifierTests12/12,bbapi_tests43/43 (excludingChonkPinnedIvcInputsTest, which needs downloaded pinned flows), bb.js tsc (node and browser) and eslint clean.🤖 Generated with Claude Code