refactor(bbapi): migrate bb + bb.js to ipc-codegen/ipc-runtime; delete legacy bb::ipc

[charlielye]

Combined redo of #23612 + #23614 off next. Those PRs predate envelope ids (the 8-byte request-id frame prefix), which broke wire compatibility between old and new stacks — bb, bb.js, and bb-rs must move together, so this is one PR. The old PRs were used as the recipe; they'll be closed pointing here.

bb (C++)

• Schema-first bbapi: bbapi/bb_schema.json (64 commands) and bb_curve_constants.json checked in, in the friendly ipc-codegen dialect (same as avm/cdb/wsdb). CMake runs ipc-codegen at build time (client+server, bb::bbapi namespace); the schema is also embedded so bb msgpack schema prints it.
• Handlers: bbapi_handlers.* (+ bbapi_chonk_handlers.cpp, bbapi_ultra_honk_handlers.cpp, bbapi_wire_convert.hpp) implement the generated dispatch (Responder-style, exactly-once ok/error) as thin adapters over the existing domain Cmd{...}.execute(ctx) commands.
• Serve: bb msgpack run serves every transport through ipc-runtime — -/empty → stdio pipe server, .sock → UDS, .shm → shared memory — one generated handler, envelope framing everywhere. It uses ipc-runtime's serial zero-copy run() loop rather than run_reactor(): every bb handler responds inline, so the reactor's per-request copy, closure, lock and wake bought nothing. A plain-file input keeps the offline bare-frame replay for wasm/debug flows.
• FFI/wasm: typed-cbind bbapi() replaced by the ipc-codegen FFI contract ipc_ffi_entry(input, len, out, out_len) — same msgpack payload as the transports, no envelope, output is free()-compatible.
• Deletions (consumer-free after the above): legacy barretenberg/ipc/ + ipc_bench; the whole nodejs_module/ (it only hosted the msgpack_client NAPI wrappers — bb.js now uses ipc-runtime's addon) along with its CMake preset targets, the bb-cpp-yarn Makefile step, and the copies of nodejs_module.node into bb.js; bbapi_execute.* (Command/CommandResponse named unions + dispatcher); func_traits/schema reflection + CBIND macros + msgpack_schema test.

ipc-runtime

• New pipe transport (C++): PipeServer/PipeClient over an fd pair or stdio, identical envelope framing to UDS, peer EOF → shutdown; serve_helper maps - to stdio, serving on a private duplicate of stdout and pointing fd 1 at stderr so stray stdout writes can't corrupt frames (next's bb redirected std::cout for the same reason). Exposed through the C ABI (ipc_client_create_pipe, dup'ing the caller's fds) and the Rust crate (IpcClient::from_fds). Used by bb msgpack run on stdio and by bb-rs. The Rust and Zig builds now discover sources instead of listing them.
• No stream frame cap: UDS and pipe frames are limited only by the u32 length field (the old 256 MiB cap rejected bb proving requests). Receivers grow their buffer as bytes arrive (C++ read_payload, TS FrameReader), so a corrupt prefix can't force a large allocation, and the TS client/server no longer re-concatenate per chunk (receiving a large frame was quadratic). SHM stays bounded by half its ring capacity.
• SHM conditional wakes made safe: the always-wake that SHM ships with was masking a startup race, not a memory-ordering bug — a client could attach to a segment after ftruncate but before create() initialised it, and that init then wiped the client's consumer_blocked flag, so a later publish skipped the wake and the client slept forever. create() now publishes capacity (rings) / num_slots (doorbell) last with release ordering, connect() refuses a segment until it is set, and wakes are conditional again (~6 futex syscalls per request saved). Each side of the wake handshake stores then loads the other's value, so both carry a seq_cst fence between the two; release/acquire alone would let both miss each other. Regression test ShmTest.ConnectRefusesSizedButUninitialisedSegments. The MPSC consumer spin now reads the clock every 256 iterations rather than every iteration.
• TS:
  • SpawnedProcessBackend: unref/unrefStdio options (child + idle-socket unref, re-ref while calls are in flight) so bb.js callers that never destroy() can't hang node; stale shm segments/sockets from a SIGKILLed previous occupant are removed before spawn; a transport error against a still-live server kills it so respawn can replace it; the connect is aborted when the child dies first.
  • New SpawnedProcessBackendSync for the SHM sync client.
  • Call fast path: a live process is called directly, with no async frame or await ensureUp() per call. Errors are attributed to the dead process (exit code, log path) by an IpcErrorMapper hook the clients apply on their rejection paths only, instead of a per-call .catch.
  • SHM and UDS clients pair responses through a shared PendingQueue: small-integer ids (V8 Smis) and an issue-order queue with a head-of-queue fast path and a search fallback for out-of-order servers. This replaces Map<bigint>, which allocated and hashed heap BigInts on every call. The NAPI async client passes ids as numbers; the wire id stays u64.
  • A request too large for the length field rejects up front with a non-transport IpcError, so it doesn't make SpawnedProcessBackend retire a healthy server.
• Sync SHM calls have no deadline. The sync NAPI client gave up on every call after a hard-coded 60s, failing a slow but healthy server and reporting a dead one only as a timeout. A sync call holds Node's main thread, so the child's exit event can't run; instead the client gets the server's pid and checks between 1s waits whether it has exited (waitid with WNOWAIT, so Node still reaps the child). A death fails the call with the exit code or signal, which SpawnedProcessBackendSync reports as IpcProcessExitedError; a live server is waited on however long it takes. A request too large for the ring fails at once, rather than waiting for space that never frees. The wait loop lives in ipc_runtime/blocking_call.hpp. The async SHM client already had no deadline (it is destroyed when the child exits).

ipc-codegen

• --strip-type-prefix: strips the service prefix from generated type and converter names too (BbCircuitProve → CircuitProve); wire tags always keep the full schema name.
• TS: fixed-size arrays of length 2 (Fq2 coordinates) are emitted as tuples.
• Rust: [u8; N] fields get an explicit bytes codec (msgpack bin, not an integer sequence — the C++ side rejects the latter); generated Bin32 keeps the byte-oriented surface callers had (deref to slice, comparisons with Vec/arrays); fq[2] (Fq2) fields get serde_array2_bytes, so G2 coordinates serialize as bin32.

bb.js

• The in-tree src/cbind generator is deleted. scripts/generate.sh runs ipc-codegen on the checked-in bb_schema.json (plain node --experimental-strip-types, ts-node dropped) with --strip-method-prefix --strip-type-prefix, so the public API surface is unchanged while wire tags carry the Bb prefix.
• Backends: UDS, SHM async and SHM sync backends are thin wrappers over ipc-runtime's SpawnedProcessBackend / SpawnedProcessBackendSync and NAPI clients (no separate bb.js NAPI addon); wasm backends call the ipc_ffi_entry export. Unused stdio pipe backend deleted, and findNapiBinary removed (it looked for the no-longer-shipped nodejs_module.node).
• poseidon.bench.test.ts warms each mode with one untimed pass before timing it; previously every mode was timed cold, and the pipelined result in particular swung by several µs between CI runs.
• Public API changes for callers:
  • Field2 is kept as a deprecated alias ([Uint8Array, Uint8Array]) so existing consumers compile; Fq2 coordinates are now typed per curve.
  • chonkStart takes kinds as a Uint8Array, since CircuitKind travels as a u8. AztecClientBackend pipelines its accumulation calls and awaits only the final chonkProve, where any failure surfaces.
  • Command failures are raised as BBApiException, through the generated API's error factory.
  • BarretenbergSync.initSingleton now forces unref: true, like the async singleton. Nothing destroys a singleton, and without it a process using one hung at exit.
  • SHM backends: a call to a dead bb now fails with IpcProcessExitedError (exit code or signal, log path) instead of never returning; a slow but live bb is waited on indefinitely. napiPath is an optional override; by default ipc-runtime's addon is used.
• bb.js depends on @aztec-foundation/ipc-runtime, and so does acir_tests (portal into ipc-runtime/ts).
• Codegen output (src/generated/) is excluded from prettier and eslint: it is gitignored, but CI runners reuse working directories, so it can be present when yarn formatting runs.
• barretenberg/ts and bb.js bootstraps include ipc-runtime/ts/package.json in the node-modules cache key, since the workspaces portal into it.

bb-rs

• The crate ships no transport code of its own: the generated Backend trait has a bridge impl for ipc_runtime::IpcClient (UDS / SHM / pipe), behind the ipc-runtime feature; the FFI backend (binding ipc_ffi_entry) is generated. The in-crate pipe/FFI backends, error and types modules are replaced by the generated backend, bb_client, bb_types, error, ffi_backend.
• Breaking for external users (7.0): the pre-codegen API is removed rather than shimmed: BarretenbergApi → BbApi, PipeBackend → IpcClient::from_fds, typed scalars (Fr/Fq/… as Bin32) in arguments and responses, no shutdown() (bb exits on EOF), native feature → ipc-runtime, async removed. Older versions stay on crates.io and semver keeps "6" manifests off 7.x; the README has a migration note. The only public dependent, noir-zk-backend, pins an exact 7.0 nightly.
• Publishing: barretenberg-rs now depends on the ipc-runtime crate, which wasn't on crates.io (cargo package refused the path-only dependency). ipc-runtime's release now also publishes its crate (it runs before barretenberg/rust's), and the bb-rs release pins it exactly (=$version). The published crate carries a copy of ipc-runtime/cpp/ipc_runtime in cpp/, and build.rs falls back to it when ../cpp is absent. ipc-runtime's release also runs for private releases (internal npm only), so the crates.io step is skipped when assert_public_release refuses.
• Shutdown is gone (pipe EOF replaces it — it was the only sender).
• The crate generates its own bindings (./bootstrap.sh generate, ipc-codegen on bb_schema.json) instead of via bb.js's yarn generate, so its cache hash covers the schema and ipc-codegen rather than the whole ts build. fr_ext.rs keeps the Fr constructors (from_u64, from_buffer_reduce, …) as a separate impl, so the generated types stay a pure regen target.

Schema dialect notes (vs the raw dump the old PRs used)

• Shutdown removed; CircuitKind is a plain u8 on the wire (the alias collided with the domain enum at bb::bbapi scope); Fq2/G2 coords are fq[2]; the shared vk-data struct is VkData (prefix-stripping made the old name collide with CircuitComputeVkResponse).

Drive-by fixes

• Stale docs and comments referring to the deleted nodejs_module, findNapiBinary and bbapi() (bb and vm2 CLAUDE.md, the benchmark-avm skill, -fPIC comments in CMake).
• Two latent breakages from the ts/ → ts/bb.js/ restructure, both one directory short: the src/barretenberg_wasm/*.wasm.gz symlinks resolved to barretenberg/ts/cpp/..., and chonk_pinned_inputs.test.ts's findRepoRoot() produced barretenberg/barretenberg/cpp/chonk-pinned-flows.
• barretenberg/docs recursive example: 1500s lane / 1200s jest budget. It uses the published bb.js, and runs alongside the bb build, which slows it by an order of magnitude when bb rebuilds everything.
• ASAN dsl_tests root rollup Tampering: 900s timeout, up from 600s. It takes ~250s alone, but timed out when it ran alongside the bb cross builds, which miss the cache whenever bb changes. With 150–350 threads ready to run on 64 cores, both ASAN tests ran 3–5x slower.

Performance

The poseidon bench's "Native Shared Pipelined" (2 fields) was ~19–20 µs/call on this branch vs ~12 µs on next. Causes, each measured in isolation: the always-wake futex traffic, the reactor serve loop, the async hop in SpawnedProcessBackend.call, the per-call .catch, and the Map<bigint> pending map (only visible under jest, whose VM realm runs this JS ~3.5x slower and saturates the main thread in a pipelined burst). With the fixes above, CI reports 12.71 µs/call — level with next — and the other modes match next too (shared sync 21.5, shared 34.7, socket pipelined 29.8, socket 48.3).

Validation

• ipc-runtime: C++ tests incl. 6 new pipe tests, the SHM init-race regression test and 4 blocking-call tests (a server that exits or is killed mid-call ends the call in one poll, with its status, and stays reapable; a slow live server is waited out; an oversized request fails at once — the exit/kill/oversize tests took 60s and timed out before the change), + ASAN; TS tests 19/19 (incl. SpawnedProcessBackend exit attribution, kill-on-broken-connection and respawn).
• The SHM hang was reproduced before fixing: with a 20 ms test-only delay before ring init, 100/100 runs hung with conditional wakes and no ready marker, 0/100 with always-wake; the regression test pins the ready-marker fix.
• bb: bbapi_tests green; c_bind exception tests green through ipc_ffi_entry.
• bb.js: new native_shm.test.ts kills bb and calls it in the same tick; the call fails in under a second with IpcProcessExitedError (signal SIGKILL), where before it took 60s. Full suite (66 pass), exercising every transport — wasm, native socket, socket pipelined, SHM async, SHM pipelined, SHM sync.
• Pinned Chonk IVC flows prove and verify through bb.js on both the native and wasm backends.
• Live UDS smoke against real bb: pipelined poseidon2, correct hashes, clean SIGTERM. The wasm ipc_ffi_entry path returns the identical hash. (In wasm a failing command aborts via the throw_or_abort host import rather than returning an error response — unchanged behavior, since the wasm preset defines BB_NO_EXCEPTIONS and the old cbind guarded its catch the same way.)
• bb-rs: pipe tests 14/14 against the new bb binary; FFI tests 72 pass (3 ignored) against libbb-external.a. Release dry run in a scratch copy: the packaged ipc-runtime crate builds with no ../cpp present, barretenberg-rs packages against it, and a consumer built only from the two packages hashes through bb over UDS.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	npm/​@​aztec-foundation/​ipc-runtime@​0.0.0-use.local	N/A	N/A	N/A	N/A	N/A

View full report