Repository navigation
feat: @aztec-foundation/wsdb-ref, a generated in-memory reference world state (wasm + napi) - #25567
Open
charlielye wants to merge 7 commits into
Open
charlielye wants to merge 7 commits into
charlielye wants to merge 7 commits into
Conversation
…e legacy bb::ipc Combined redo of #23612 + #23614 + #23613. Those predate envelope ids (the 8-byte request-id frame prefix), which broke wire compatibility between the old and new stacks, so bb, bb.js and bb-rs have to move together. bb now serves msgpack through ipc-runtime on every transport: `bb msgpack run` maps ""/"-" to a stdio pipe server, .sock to UDS and .shm to shared memory, all dispatching through one generated handler with envelope framing (a plain file keeps the offline bare-frame replay). The command surface comes from a checked-in bb_schema.json in the friendly ipc-codegen dialect, with handlers implemented as thin adapters over the existing domain commands. The typed cbind entrypoint is replaced by the ipc-codegen FFI contract ipc_ffi_entry, so wasm and static-linking consumers speak the same payload as the transports. That makes the legacy machinery consumer-free, so it goes: the in-tree ipc library and its benchmark, the nodejs_module msgpack client (bb.js now uses ipc-runtime's NAPI addon, and nodejs_module only exports LMDBStore), the Command/CommandResponse named unions with their execute dispatcher, and the func_traits/schema reflection behind the CBIND macros. Both client libraries now generate from ipc-codegen rather than forks of it. bb.js/src/cbind (a stale copy of the generator: schema visitor, TS and rust backends, naming, friendly-schema lowering) is deleted in favour of calling ipc-codegen, and barretenberg-rs drops its hand-written Backend trait, error type, Fr/Point types and both backends — codegen emits those, and ipc_runtime::IpcClient plugs in as the Backend, so the crate is ~89% generated with a deprecated BarretenbergApi shim keeping the published surface. The hand-rolled PipeBackend goes with them; its tests now run over ipc-runtime's UDS transport. ipc-codegen gains --strip-type-prefix (bb.js and barretenberg-rs publish unprefixed names while wire tags keep the service prefix) and three rust serde fixes: fq[2] pairs, [u8; N] and above-cutoff byte arrays were encoded as sequences of integers rather than msgpack bin, which the C++ side rejects. ipc-runtime gains the pipe transport (fd pair or stdio, peer EOF requests shutdown), a sync spawned-process backend for callers that cannot await, pre-spawn removal of stale shm segments (they are created O_EXCL, so a killed server's leftovers blocked the next one), and an unref option so a spawned backend cannot hold the Node event loop open while idle. Its rust and zig build files now discover the C++ sources instead of keeping hand-written copies of the CMake list, which had silently drifted. Also fixes two latent breakages from the ts/ -> ts/bb.js/ restructure that blocked wasm and pinned-flow tests: the barretenberg_wasm symlinks and the chonk pinned-inputs repo-root resolution were both one directory short. Raises MAX_FRAME_SIZE from 256 MiB to 1 GiB. bb.js reaches bb over ipc-runtime's socket transport now rather than the deleted nodejs_module msgpack client, so proving requests are newly subject to the frame cap: the rollup circuits exceed it, and bb_prover_full_rollup failed with the server refusing a 314525552-byte frame and the client seeing the closed connection as "write EPIPE". The TS client also checks the outgoing size, so a request over the cap names itself instead of surfacing as an unexplained EPIPE on the next write. Fixes two gaps in unref handling that left the Node event loop open. The shared-memory backends dropped the caller's option -- their new() did not take it and createBackend never passed it -- so unrefStdio was never set and the child's stdout/stderr pipes (which exist whenever a logger is set) kept the loop alive; they thread it through as the socket backend already does. And BarretenbergSync.initSingleton did not force unref: true, unlike its async counterpart, so the sync backend it creates held the loop for the life of the process. Nothing destroys a singleton, so it opts in now too. Together these had a labs pxe test pass all 7 cases and then hang until the job timed out. Rebased onto next, which added two things to the surface this replaces. #25487 hardened the msgpack entrypoints: the generated dispatch already answers a malformed envelope, an unknown command and a throwing handler with an error frame, but an unpack or convert failure still escapes it, so api_msgpack.cpp and ipc_ffi_entry catch that rather than lose the response. Poseidon2AbsorbChain was a new command registered in the NamedUnion this deletes, so it moves to bb_schema.json with a handler alongside the other crypto ones, and its malformed -length test is reexpressed against the FFI entrypoint. barretenberg/acir_tests is its own yarn project and consumes bb.js through a portal, so it has to resolve bb.js's dependencies itself. bb.js declares @aztec-foundation/ipc-runtime with the placeholder specifier that only a resolution gives meaning to, which barretenberg/ts has and acir_tests did not: `yarn install` there failed with "isn't supported by any available resolver". It gets the same resolution. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016einpgfthfjLYGwCB3iQqD
…ull bb build It takes ~35s idle but exceeded its 360s jest timeout (408s) when the lane ran concurrently with a full bb rebuild. It uses the published bb.js, so the slowdown is load, not the tree: raise the per-test timeout to 20 minutes and the lane's to 25. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
…sport bb.js reached bb's wasm through roughly 1,400 lines of hand-written plumbing (comlink workers, WASI stubs, a heap allocator, pako inflation, base64-embedded modules) sitting next to a generated client. This moves "how a client reaches a service module" into ipc-codegen/ipc-runtime, generates the bb API as its own package, and leaves bb.js a facade over it. ipc-runtime gains a generic wasm FFI backend for any WASI reactor implementing the contract (<svc>_ipc_ffi_entry plus its allocator pair): an async backend that runs the instance in a worker with wasi threads on further workers, a synchronous one that runs on the calling thread, node and browser entries chosen by export condition, streaming compilation with gzip support, and a WASI shim. Workers are spawned through factories holding the literal new URL(..., import.meta.url) that bundlers must see. ipc-codegen emits the FFI entry itself for C++ and Rust, with symbols carrying the service name so several services can link into one binary, and its TS --package gains a wasm transport. The generated package owns backend selection: the spawned process when the binary resolves, else the wasm module, with an explicit choice forcing one and no silent downgrade. A failed command raises a generated error class rather than an injected factory. bb's wasm is now a plain WASI reactor. It imported a logger, an abort hook and a thread count because barretenberg/wasi stubbed out the WASI calls the old host did not implement; ipc-runtime implements them, so the stubs go and the three functions are defined against stderr, the environment and proc_exit instead. bb.js keeps its public API. Its own uds/shm/wasm backend files, the base64 wasm embedding and the bundled bb binary are gone; bb resolves through bb.js-api's per-platform packages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`imports` merges another schema's aliases and types, sharing vocabulary between services. `extends` makes a schema a strict superset of another service's contract: it inherits the service name, error, types and commands, and may only add to them. Aliases and types may repeat an inherited definition verbatim; commands may never be redefined. The TS output of an extending schema gains Async<Name>Api/Sync<Name>Api interfaces per parent, which the full API base interfaces extend. They are structural, so code typed against a parent accepts any extension's client.
`--lang cpp --server --ffi --napi` emits <service>_napi.cpp, a Node-API addon over <service>_ipc_ffi_entry: call() on the calling thread and callAsync() on the libuv pool, serialized by one mutex as the FFI entry requires. It depends only on the FFI header. A TS package with `--package-transports napi --package-napi-addon <file>` gains a napi backend (async and sync) that loads the addon from an explicit path, an env var or its arch package. create() tries the process, then napi, then wasm. A package that spawns nothing no longer declares a bin, and arch packages carry the addon via ipcRuntime.nativeFiles. ipc-runtime gains NapiFfiBackend/NapiFfiBackendSync, a resolver for any per-platform native file, and a pickServiceBackend that orders process, napi and wasm, with tests.
world_state_reference/wsdb_base_schema.jsonc is the world-state contract the protocol needs: tree reads, membership and low-leaf witnesses, appends, sequential inserts, archive updates and checkpoints. Its definitions are copied verbatim from the node world state's schema, so that schema can `extends` this one. ReferenceWorldState adds the archive tree, prefilled genesis leaves and the genesis block header hash to the in-memory trees. It starts at the canonical genesis: the protocol contracts' registration nullifiers, which regenerate_genesis_constants.sh now also writes into genesis_protocol_nullifiers.hpp here. Its tests check the genesis against GENESIS_NULLIFIER_TREE_ROOT, GENESIS_BLOCK_HEADER_HASH and GENESIS_ARCHIVE_ROOT. The wsdb_ref module serves the contract over the generated dispatch and FFI entry: a Node-API addon (wsdb_ref_napi.node) in native builds and a WASI reactor (wsdb-ref.wasm) in wasm builds. Requests the reference does not model (other forks, block history) are answered with errors rather than exceptions, which a wasm build cannot throw. The reference now compiles for wasm32.
The in-memory reference world state as a fully generated client package, from wsdb_base_schema.jsonc with --package-transports napi,wasm: the wasm module in node and browsers, or the Node-API addon in node, which create() prefers when its arch package resolves. Built by `make bb-wsdb-ref`, cross-copied and published with the other barretenberg packages, and tested on both backends by `make bb-wsdb-ref-tests`.
charlielye
requested review from
iakovenkos,
ledwards2225 and
ludamad
as code owners
October 1, 2026 18:07
charlielye
force-pushed
the
cl/bbjs-api-codegen
branch
3 times, most recently
from
October 8, 2026 13:15
49f4fab to
81a7021
Compare
charlielye
force-pushed
the
cl/bbjs-api-codegen
branch
from
October 15, 2026 17:47
81a7021 to
a1a0938
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #25433, review that first. This PR is the four commits on top.
Why
wsdb now lives in
native-packages/, headed for the labs repo. The foundation still needs a world state of its own, for example to build Chonk flows from the protocol circuits without depending on labs. Barretenberg already has an in-memory reference world state. This PR exposes it as a service and generates a TS package for it,@aztec-foundation/wsdb-ref, with a wasm backend and an opt-in Node-API backend.What changes
ipc-codegen:
importsandextends(SCHEMA_SPEC.md)importsreuses another schema's aliases and types, so a type is defined once. Each generated program still keeps its own copy.extendsmakes a schema a strict superset of another service's contract. It inherits the service name, error, types and commands, and can only add to them. Identical repeats of aliases and types are allowed; commands can never be redefined.Async<Name>Api/Sync<Name>Apiinterfaces for each parent. They are structural, so code typed against a parent accepts any extension's client.ipc-codegen + ipc-runtime: a
napitransport over the generated FFI entry--lang cpp --server --ffi --napiemits<service>_napi.cpp. It is a Node-API addon over<service>_ipc_ffi_entry, withcallon the calling thread andcallAsyncon the libuv pool, serialized by one mutex.--package-transports napi --package-napi-addon <file>adds sync and async napi backends to a generated package. The addon ships in the arch packages, listed underipcRuntime.nativeFiles.create()now tries the process, then napi, then wasm. A package that spawns nothing no longer declares abin.NapiFfiBackend/NapiFfiBackendSync, a resolver for any per-platform native file, and backend-selection tests.bb: the wsdb base contract and the
wsdb_refmoduleworld_state_reference/wsdb_base_schema.jsoncholds the 22 protocol-level wsdb commands: reads, witnesses, appends, sequential inserts, archive update and checkpoints. They are copied verbatim from wsdb's schema, so wsdb can switch its schema toextendsthis one.ReferenceWorldStateadds the archive tree, prefilled genesis leaves and the genesis block header hash. It starts at the canonical genesis, so callers never configure one.regenerate_genesis_constants.shnow also writes the protocol-contract seed nullifiers into bb, which lets bb's tests checkGENESIS_NULLIFIER_TREE_ROOT,GENESIS_BLOCK_HEADER_HASHandGENESIS_ARCHIVE_ROOTagain.wsdb_refserves the contract over the generated dispatch and FFI entry, aswsdb_ref_napi.node(native builds) andwsdb-ref.wasm(wasm builds, WASI only). Requests the reference doesn't model, such as other forks or historical blocks, get error responses rather than exceptions, so a wasm instance survives them. The reference now compiles for wasm32.bb.js:
@aztec-foundation/wsdb-refbarretenberg/ts/wsdb-refbymake bb-wsdb-ref, cross-copied and released with the other barretenberg packages.make bb-wsdb-ref-tests.Testing
extends/importscases: 30 pass.world_state_reference_tests: 9 pass.wsdb_ref_tests(through the real FFI entry): 5 pass.barretenberg/ts/scripts/wsdb_ref_test.sh napiand... wasm: 5 pass each.wsdb-refinterface is expected.Follow-ups
extendsthe base, once a foundation release containing it is pinned, and drop its copy of the base definitions.