Skip to content

feat(bb): per-proof batch verification responses under bb msgpack run --reactor - #25636

Draft
charlielye wants to merge 19 commits into
nextfrom
cl/bb-chonk-batch-verify-deferred
Draft

charlielye wants to merge 19 commits into
nextfrom
cl/bb-chonk-batch-verify-deferred

Conversation

@charlielye

Copy link
Copy Markdown
Contributor

Stacked on #25362. It needs that PR's ipc-runtime server in bb and ipc-runtime client in bb.js.

Why

aztec-node's p2p batch verifier gets each proof's verdict over a side channel: a named pipe (FIFO) that bb writes results to. It does this because bb's msgpack server reads a request, runs it, and replies before reading the next one. A call that waited for its batch would stop more proofs arriving, so batching would never happen.

The side channel brings a lot with it on the TS side:

  • a request-id map;
  • a startup probe to make bb open the FIFO (bb opens it lazily on its first write, and until then its death goes unseen);
  • an in-flight cap matched to bb's queue size;
  • inferring from FAILED results whether bb actually judged a proof.

aztec-labs-eng/aztec-node#359 has the details.

What

  • bb msgpack run --reactor. bb serves with ipc-runtime's run_reactor instead of run(). Handlers still run one at a time, in arrival order, on the reactor thread, so session state needs no locking. A handler may now respond after it returns, from another thread, and responses go out as they complete, matched by request id. Without the flag nothing changes. The flag is opt-in because the reactor copies each request and wakes across threads.
  • ChonkBatchVerifierVerify {vk_index, proof_fields} → {verified, time_in_verify_ms}.
    • Needs a batch verifier started without a FIFO (empty fifo_path).
    • Queues the proof and responds with its verdict once checked. verified: false means bb judged the proof: too short to split, wrong size, or failed the check (off-curve commitments included).
    • Everything else is an error response, not a verdict: an unregistered vk_index, the service not running, or the service stopping before the proof is checked.
    • When the queue is full, bb waits for room instead of refusing, so a caller needs no cap of its own and the wait reaches it as backpressure.
    • Without --reactor the command is refused, rather than stalling the server.
    • The FIFO commands are unchanged. A service started with a FIFO refuses Verify, and one started without refuses Queue.
  • Stopping. ChonkBatchVerifierStop, or the server shutting down via run_reactor's new on_stop hook, answers proofs still queued as unchecked errors and only finishes the batch in progress. Otherwise it would verify the whole queue for callers that may be gone.
  • ipc-runtime run_reactor.
    • New on_stop hook.
    • A handler or send() that throws now stops serving cleanly instead of unwinding past responders still in flight (use-after-free). A request whose handler threw without responding is released, so shutdown doesn't wait for it forever.
    • notify() now runs before the in-flight decrement, so a cross-thread responder can't touch a server that has just been destroyed.
  • bb.js. A reactor option on the NativeUnixSocket backend passes --reactor. Other backends refuse it.

After a bb release with this, aztec-node's BatchChonkVerifier becomes await bb.chonkBatchVerifierVerify(...) per proof. The FIFO, the probe, the request-id map and the in-flight bookkeeping all go away.

Not done

  • bb-internal failures during a batch check (e.g. an allocation failure in reduce, or an exception swallowed by batch_check and then bisected) still read as verified: false. That's the same as the FIFO path today. Separating them means telling proof-derived exceptions from bb's own inside the verifier.
  • Each pending proof is held twice: the decoded proof in the queue, and the request bytes run_reactor keeps until it responds.
  • reactor isn't wired into the shared-memory backends.

Testing

  • api_msgpack_reactor.test.cpp: end to end over a real Unix socket through serve_msgpack_reactor, with real (small) Chonk proofs.
    • Five proofs are pipelined, one of them tampered, followed by a Poseidon hash. The hash comes back first, and each proof gets its own verdict.
    • An unregistered key gets an error.
    • With one proof per batch, a stop answers the queued proofs as errors and the rest as verified.
    • The last two failed with their fixes disabled and pass with them.
  • bbapi_chonk_batch_verify.test.cpp: through the generated dispatch: verdicts for good, tampered and malformed proofs; refusal without --reactor; the two modes refuse each other's command.
  • socket.test.cpp: a reactor handler that throws without responding stops the server, runs on_stop, still sends the deferred response, and returns. Before the change it crashed the process.
  • bb.js batch_verifier_verify.test.ts, against the built bb: several calls in flight on one connection, the verdict-vs-error split, and refusal without --reactor.
  • Regression: ipc_runtime_tests 32/32, ChonkBatchVerifierTests 12/12, bbapi_tests 43/43 (excluding ChonkPinnedIvcInputsTest, which needs downloaded pinned flows), bb.js tsc (node and browser) and eslint clean.

🤖 Generated with Claude Code

charlielye and others added 19 commits October 1, 2026 18:06
…e legacy bb::ipc

Combined redo of #23612 + #23614 + #23613. Those predate envelope ids (the
8-byte request-id frame prefix), which broke wire compatibility between the
old and new stacks, so bb, bb.js and bb-rs have to move together.

bb now serves msgpack through ipc-runtime on every transport: `bb msgpack run`
maps ""/"-" to a stdio pipe server, .sock to UDS and .shm to shared memory,
all dispatching through one generated handler with envelope framing (a plain
file keeps the offline bare-frame replay). The command surface comes from a
checked-in bb_schema.json in the friendly ipc-codegen dialect, with handlers
implemented as thin adapters over the existing domain commands. The typed
cbind entrypoint is replaced by the ipc-codegen FFI contract ipc_ffi_entry,
so wasm and static-linking consumers speak the same payload as the transports.

That makes the legacy machinery consumer-free, so it goes: the in-tree ipc
library and its benchmark, the nodejs_module msgpack client (bb.js now uses
ipc-runtime's NAPI addon, and nodejs_module only exports LMDBStore), the
Command/CommandResponse named unions with their execute dispatcher, and the
func_traits/schema reflection behind the CBIND macros.

Both client libraries now generate from ipc-codegen rather than forks of it.
bb.js/src/cbind (a stale copy of the generator: schema visitor, TS and rust
backends, naming, friendly-schema lowering) is deleted in favour of calling
ipc-codegen, and barretenberg-rs drops its hand-written Backend trait, error
type, Fr/Point types and both backends — codegen emits those, and
ipc_runtime::IpcClient plugs in as the Backend, so the crate is ~89% generated
with a deprecated BarretenbergApi shim keeping the published surface. The
hand-rolled PipeBackend goes with them; its tests now run over ipc-runtime's
UDS transport.

ipc-codegen gains --strip-type-prefix (bb.js and barretenberg-rs publish
unprefixed names while wire tags keep the service prefix) and three rust
serde fixes: fq[2] pairs, [u8; N] and above-cutoff byte arrays were encoded as
sequences of integers rather than msgpack bin, which the C++ side rejects.

ipc-runtime gains the pipe transport (fd pair or stdio, peer EOF requests
shutdown), a sync spawned-process backend for callers that cannot await,
pre-spawn removal of stale shm segments (they are created O_EXCL, so a
killed server's leftovers blocked the next one), and an unref option so a
spawned backend cannot hold the Node event loop open while idle. Its rust and
zig build files now discover the C++ sources instead of keeping hand-written
copies of the CMake list, which had silently drifted.

Also fixes two latent breakages from the ts/ -> ts/bb.js/ restructure that
blocked wasm and pinned-flow tests: the barretenberg_wasm symlinks and the
chonk pinned-inputs repo-root resolution were both one directory short.

Raises MAX_FRAME_SIZE from 256 MiB to 1 GiB. bb.js reaches bb over ipc-runtime's
socket transport now rather than the deleted nodejs_module msgpack client, so
proving requests are newly subject to the frame cap: the rollup circuits exceed
it, and bb_prover_full_rollup failed with the server refusing a 314525552-byte
frame and the client seeing the closed connection as "write EPIPE". The TS client
also checks the outgoing size, so a request over the cap names itself instead of
surfacing as an unexplained EPIPE on the next write.

Fixes two gaps in unref handling that left the Node event loop open. The
shared-memory backends dropped the caller's option -- their new() did not take it
and createBackend never passed it -- so unrefStdio was never set and the child's
stdout/stderr pipes (which exist whenever a logger is set) kept the loop alive;
they thread it through as the socket backend already does. And
BarretenbergSync.initSingleton did not force unref: true, unlike its async
counterpart, so the sync backend it creates held the loop for the life of the
process. Nothing destroys a singleton, so it opts in now too. Together these had
a labs pxe test pass all 7 cases and then hang until the job timed out.

Rebased onto next, which added two things to the surface this replaces. #25487
hardened the msgpack entrypoints: the generated dispatch already answers a
malformed envelope, an unknown command and a throwing handler with an error
frame, but an unpack or convert failure still escapes it, so api_msgpack.cpp and
ipc_ffi_entry catch that rather than lose the response. Poseidon2AbsorbChain was
a new command registered in the NamedUnion this deletes, so it moves to
bb_schema.json with a handler alongside the other crypto ones, and its malformed
-length test is reexpressed against the FFI entrypoint.

barretenberg/acir_tests is its own yarn project and consumes bb.js through a
portal, so it has to resolve bb.js's dependencies itself. bb.js declares
@aztec-foundation/ipc-runtime with the placeholder specifier that only a
resolution gives meaning to, which barretenberg/ts has and acir_tests did not:
`yarn install` there failed with "isn't supported by any available resolver".
It gets the same resolution.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016einpgfthfjLYGwCB3iQqD
…ull bb build

It takes ~35s idle but exceeded its 360s jest timeout (408s) when the lane
ran concurrently with a full bb rebuild. It uses the published bb.js, so the
slowdown is load, not the tree: raise the per-test timeout to 20 minutes and
the lane's to 25.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
…al run() loop

The shared-memory transport woke the other side with a futex syscall on every
publish and release, about six per bb request, which roughly doubled the
pipelined poseidon benchmark against next. Those wakes were made unconditional
to fix an intermittent hang, whose cause turned out to be a startup race rather
than the conditional wake itself: segments are mappable from ftruncate, before
create() initialises them, so a fast client could attach, set its blocked flag
on a response ring, and have the server's initialisation reset it; the next
publish then skipped the wake and the client slept on published data forever.

connect() now refuses a segment until it is fully initialised. create() writes
the ring capacity, and the doorbell's num_slots, last with release ordering,
and a zero value means not ready: SpscShm::connect and MpscProducer::connect
throw and the existing connect-retry loops try again. With that in place the
conditional wake is restored on the rings, the doorbell and the reactor's
notify().

bb also served every request through run_reactor, paying a request copy, a
heap-allocated respond closure, a locked completion queue and a notify() (a
self-pipe write on sockets) per request, although every bb handler responds
before returning. It now uses the serial run() loop, asserting that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
…and stop the MPSC spin reading the clock every iteration

Pipelined bb.js calls over shared memory were still ~50% slower than next on
the poseidon benchmark after the futex fix. Layer by layer, the transport
itself was at parity; the extra cost was main-thread JavaScript per call, which
left the main thread saturated so the API's own encode/decode no longer
overlapped with bb's work. Most of it was SpawnedProcessBackend.call: an async
function awaiting ensureUp() before every call. A live process now takes a
direct path with no async frame; only a missing or dead process goes through
ensureUp(). On the benchmark's pattern (the first pipelined burst after 10k
sequential calls) this takes the fixed build from 17.3 to 13.4 us/call (min of
8, next 14.1).

MpscConsumer::wait_for_data's spin also called clock_gettime on every
iteration, about a tenth of bb's instructions while serving; it now checks the
clock every 256 iterations, as SpscShm's spin does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
… with small-int ids

NapiShmAsyncClient kept pending calls in a Map<bigint>. Each call allocated
two heap BigInts (the id and the one napi hands back) and hashed them three
times. Under jest, where the main thread saturates during a pipelined burst,
that overhead was directly visible in bb.js's poseidon bench.

Ids are now numbers below 2^30, so V8 keeps them as small integers. Pending
calls sit in an issue-order queue: in-order servers (bb's serial run() loop)
always match the head, and out-of-order responses fall back to a search.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
…a per-call .catch

SpawnedProcessBackend.call chained .catch(attributeCallError) onto every
call, so each successful call paid for an extra promise and microtask. In a
pipelined burst under jest that was ~0.8us/call of main-thread CPU in steady
state and ~2us in the first burst.

The SHM and UDS clients now take an optional mapError hook, applied only on
their rejection paths. SpawnedProcessBackend passes attributeCallError for
the incarnation when it connects, and the fast path returns the client's
promise directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
Every mode was timed on its first run, so the numbers included JIT warm-up
for that call pattern. A pipelined burst exercises different code paths
from the sequential warm-up that preceded it, so it was the most affected.
Its first-burst result swung by several µs between CI runs. Each mode now
runs once untimed first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
UdsIpcClient still kept pending calls in a Map<bigint>, with the same
per-call BigInt allocation and hashing the SHM client had. Both clients now
use one PendingQueue: small-int ids, head-of-queue fast path, search
fallback. The UDS wire id stays u64; live ids sit in the low word, and a
nonzero high word never matches a call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
The 7.0 line is a breaking release, and older barretenberg-rs versions stay on
crates.io for anyone who needs the old API (semver keeps `"6"` manifests off
7.x). Removes the legacy BarretenbergApi/PipeBackend/byte-valued types, the
old module paths and the inert `native`/`async` features, so the crate exposes
only the generated BbApi.

The tests move to BbApi with typed scalars; the pipe tests spawn bb and drive
it through ipc_runtime::IpcClient::from_fds. The README gains a 6.x migration
note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
barretenberg-rs now depends on the ipc-runtime crate, which had no version and
was not on crates.io, so `cargo publish` refused the package. The release step
now publishes ipc-runtime at the same version first and pins it exactly.

ipc-runtime's build compiles the C++ sources from ../cpp, which a published
crate does not have; the release copies them into the crate's cpp/ directory
and build.rs falls back to it. Publishing happens from the barretenberg-rs
release rather than ipc-runtime's, because the latter also runs for private
releases, which must not reach crates.io.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
…release

ipc-runtime owns its crate, so its release publishes it. The release also runs
for private releases, which publish only to the internal npm registry, so the
crates.io step is skipped when assert_public_release refuses. ipc-runtime is
released before barretenberg/rust, whose release now only pins the version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
The wake is skipped unless the waiter has flagged that it is about to sleep.
Each side stores (head/tail/seq, or the flag) and then loads the other's
value, and release/acquire lets each load run ahead of its own store. Both
sides could then miss each other, leaving the waiter asleep on published data,
with no timeout on the default call path. A seq_cst fence between the store and
the load on both sides guarantees one of them sees the other.

The futex lock already fenced the waiter in practice; the SPSC producer had
no such cover. No measurable cost in a pipelined poseidon A/B on a loaded host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
UDS and pipe frames were capped at 1 GiB (256 MiB until bb proving requests
outgrew it). Nothing in a stream transport needs a limit below the u32 length
field; the cap only guarded against a corrupt prefix driving a huge up-front
allocation. Receivers now grow their buffer as payload bytes arrive (C++
read_payload, TS FrameReader), so that guard is no longer needed and the
limit is the wire format's own. SHM stays bounded by half its ring capacity.

FrameReader also fixes the TS client and server re-concatenating their whole
buffer on every chunk, which made receiving a large frame quadratic.

A request too large for the length field now rejects with a non-transport
IpcError: it is the caller's mistake on a healthy connection, and as a
transport error SpawnedProcessBackend killed the server for it. Senders also
check against the payload limit (frame limit minus the 8-byte id) rather than
the frame limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
Serving "-" made the process's stdout the frame stream, so any other write to
it (std::cout, printf, a library's diagnostics) landed inside a frame and
desynced the peer. next's bb guarded this by redirecting std::cout; that guard
did not carry over to the ipc-runtime pipe path. make_server now serves on a
private duplicate of stdout and points fd 1 at stderr, which also covers C
stdio and raw writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
It looked for nodejs_module.node, which no longer ships now that bb.js uses
ipc-runtime's addon, so it could only ever return null.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
…t, under bb msgpack run --reactor

The batch verifier's results went over a side-channel FIFO because bb's server
replied to each request before reading the next: a call that waited for its
batch would have stopped more proofs arriving. With --reactor, bb serves with
ipc-runtime's run_reactor, so a handler may reply after it returns.
ChonkBatchVerifierVerify queues a proof with a batch verifier started without a
FIFO and replies with its verdict once checked; a proof bb can't queue (service
not running, queue full) gets an error reply, not a verdict. bb.js gains a
reactor option on the socket backend. The FIFO commands are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NwZD6M5xfpDdAFkdbkWT67
… let a stop answer queued proofs

- An unregistered vk_index is an error, not a verdict.
- verify() waits for room in the batch verifier's queue instead of failing it, so callers need no in-flight cap.
- Stopping the service, or the server, answers proofs still queued as unchecked errors rather than verifying them all.
- run_reactor: an on_stop hook; a handler that throws or a send that throws stops serving without unwinding past
  responders still in flight; notify() before releasing the request.
- serve_msgpack_reactor is factored out, and tested end to end over a socket with pipelined real proofs.
- bb.js refuses the reactor option on backends that can't honour it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NwZD6M5xfpDdAFkdbkWT67
@charlielye
charlielye force-pushed the cl/ipc-bb-bbjs-migrate branch from 9f50140 to b8a89ca Compare October 13, 2026 20:02
Base automatically changed from cl/ipc-bb-bbjs-migrate to next October 15, 2026 17:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant