feat(ffi): measure the bindings on the binding benchmark, and commit the numbers

.github/workflows/nightly.yml

@@ -224,7 +224,7 @@ jobs:
           script: nix develop .#android --command just ffi android
 
   ffi-bench:
-    name: Bench the C export against moq-c
+    name: Bench the bindings
     runs-on: ubuntu-24.04-arm
     timeout-minutes: 90
 
@@ -236,8 +236,7 @@ jobs:
 
       - uses: ./.github/actions/nix
 
-      # Frame writes through each C API. The numbers go in the run's summary: no threshold
-      # fails the job.
+      # The binding benchmark's tables go in the run's summary: no threshold fails the job.
       - name: Bench
         # Named, so a failing bench fails the pipe.
         shell: bash

bench/ffi/README.md

@@ -1,11 +1,186 @@
 # Binding benchmark
 
-What moq's bindings cost over Rust, measured before [rust2-ffi](../../ffi/README.md) replaced UniFFI.
-The harness that built moq-ffi against the UniFFI fork was deleted with the fork; [its quest](/quest/m1/ffi-export/bench.md) rebuilds it on rust2-ffi, against the targets below.
+What [rust2-ffi](../../ffi/README.md)'s bindings cost over Rust, per language, on the moq
+stand-in `just ffi bench` exports, against BoltFFI's numbers below as targets.
+
+```bash
+nix develop .#bench --command just ffi bench  # build, run 3 rounds, and write ffi/target/bench/report.md
+nix develop .#bench --command just ffi bench-run 5  # rerun what bench-build built, 5 rounds
+```
+
+`ffi/moq/bench/export.*` time the same cases in each language, `export.rs` the floor
+on the stand-in's API from Rust, and `moq-c.c` moq's hand-written rs/moq-c, which
+has only the stamped write. Each program runs its cases twice and prints the second,
+so a JIT has warmed up, in wall time and in the process's CPU time on every thread.
+`just ffi bench-run` runs each program once a round, records the 1-minute load average
+over each run, and [`report.py`](report.py) prints each case's median over the rounds
+beside the median load, then the export's sizes. Nightly runs it on arm64 Linux into
+the run's summary; nothing fails on a number.
+
+| case | what it times | BoltFFI's nearest |
+|---|---|---|
+| handle call | `track.is_closed()`: one borrowed handle, a bool back, and moq's own read lock | `noop()` |
+| u64 getter | `demand.sequence()`: one borrowed handle and a field read, so all but 0.4 ns is the binding's | `group.sequence()` |
+| record return | `track.largest()`: an `Option` of two `u64`s, copied across | `track.info()`, a record |
+| object return | `track.info()`: a new handle, freed | none |
+| write | `group.write_frame(at, frame)` with one `Timestamp` handle and 100 B, copied into Rust | `write_frame` |
+| stamped write | the same with a new `Timestamp` per frame, as a caller holding a `u64` must | none |
+| ready read | a 100 B frame written, then read through a new future ready at its first poll | `await noop_async()`, plus a write |
+| woken read | a read started before its frame is written, as the language starts work concurrently, so it suspends and the write's wake resumes it | `await yield_once()`, plus a write |
+| payload | `frame.payload` of 100 B, handed to the language | none |
+| throughput | groups of 100 frames written and read, each payload's first byte touched, about 64 MiB | throughput |
+
+A woken read starts its read the language's own way: by polling in C, C++, and Rust; a
+`Task` and a yield in Swift and Python; a goroutine and a yield in Go; an undispatched
+coroutine in Kotlin; and an eager call in Dart. So it measures a task's start and its
+wake, and in Swift, Python, and Go, a read the scheduler had not yet started is ready
+instead of woken.
+
+## rust2-ffi
+
+Measured 2026-11-10 at rust2 `7383790d` plus this harness, moq `ffcb7cd9`, on an Intel
+Core Ultra 7 270K Plus (24 threads), Linux, **on a loaded shared host**: the 1-minute load
+average, in parentheses beside each number, ran 22 to 29 throughout, so each cell is the
+median of 5 rounds. CPU time leaves out time the program waited preempted, so it moves less
+under load than wall time, but caches and SMT siblings stay shared; take the CPU tables
+first. Rust 1.101.0-nightly 2026-10-14 (the export's default `release` profile, no LTO), GCC 16.2.0,
+CPython 3.14.8, OpenJDK 21.0.12.1 with Kotlin 2.4.21 and kotlinx-coroutines 1.10.2,
+Swift 6.3.3, Go 1.26.10, Dart 3.13.0.
+
+CPU time counts every thread of the process: the JVM's JIT and GC threads, Go's GC, and the
+Swift global executor's workers. So Kotlin's CPU column runs above its wall column, and
+Swift's woken read spends 16.6 us of CPU for 5.7 us of wall. Throughput from 16 KiB up is
+bound by memory bandwidth, which the host's load shares: its cells moved 2x between runs,
+the Rust floor's included, so read only the 100 B and 1 KiB rows as the bindings' cost.
+
+### Calls (ns per call, CPU time; load average)
+
+| call | Rust | C | C++ | moq-c | Kotlin | Python | Swift | Go | Dart |
+|---|---|---|---|---|---|---|---|---|---|
+| `track.is_closed()`, a handle call | 9.1 (26) | 18 (25) | 18 (26) |  | 60 (26) | 53 (25) | 26 (27) | 49 (26) | 47 (26) |
+| `demand.sequence()`, a u64 getter | 0.4 (26) | 10 (25) | 10 (26) |  | 50 (26) | 47 (25) | 18 (27) | 41 (26) | 29 (26) |
+| `track.largest()`, a record return | 18 (26) | 31 (25) | 31 (26) |  | 230 (26) | 97 (25) | 36 (27) | 76 (26) | 56 (26) |
+| `track.info()`, an object return | 1.1 (26) | 25 (25) | 25 (26) |  | 210 (26) | 97 (25) | 50 (27) | 263 (26) | 242 (26) |
+| `group.write_frame(at, frame)`, 100 B | 132 (26) | 148 (25) | 161 (26) |  | 230 (26) | 196 (25) | 194 (27) | 270 (26) | 209 (26) |
+| the same, with a new `Timestamp` each | 133 (26) | 169 (25) | 177 (26) | 267 (28) | 480 (26) | 336 (25) | 221 (27) | 526 (26) | 454 (26) |
+| a 100 B frame written and read, ready | 205 (26) | 303 (25) | 337 (26) |  | 690 (26) | 560 (25) | 509 (27) | 841 (26) | 735 (26) |
+| the same, read after a wake | 329 (26) | 427 (25) | 461 (26) |  | 1,630 (26) | 6,912 (25) | 16,580 (27) | 4,711 (26) | 1,776 (26) |
+| `frame.payload`, 100 B | 11 (26) | 23 (25) | 24 (26) |  | 70 (26) | 150 (25) | 96 (27) | 165 (26) | 67 (26) |
+
+### Throughput (MB per CPU second, write plus ready read; load average)
+
+| frame | Rust | C | C++ | Kotlin | Python | Swift | Go | Dart |
+|---|---|---|---|---|---|---|---|---|
+| 100 B | 472 (26) | 303 (25) | 300 (26) | 105 (26) | 138 (25) | 162 (27) | 106 (26) | 120 (26) |
+| 1 KiB | 3,444 (26) | 2,394 (25) | 2,267 (26) | 1,341 (26) | 1,240 (25) | 1,135 (27) | 713 (26) | 991 (26) |
+| 16 KiB | 7,324 (26) | 14,822 (25) | 6,923 (26) | 6,554 (26) | 9,460 (25) | 2,156 (27) | 1,820 (26) | 5,536 (26) |
+| 64 KiB | 2,552 (26) | 10,612 (25) | 6,441 (26) | 3,277 (26) | 9,468 (25) | 2,206 (27) | 2,382 (26) | 6,574 (26) |
+| 1 MiB | 3,647 (26) | 3,596 (25) | 5,546 (26) | 3,146 (26) | 3,464 (25) | 3,354 (27) | 2,767 (26) | 3,812 (26) |
+
+### Calls (ns per call, wall time; load average)
+
+| call | Rust | C | C++ | moq-c | Kotlin | Python | Swift | Go | Dart |
+|---|---|---|---|---|---|---|---|---|---|
+| `track.is_closed()`, a handle call | 9.1 (26) | 18 (25) | 18 (26) |  | 32 (26) | 53 (25) | 26 (27) | 48 (26) | 42 (26) |
+| `demand.sequence()`, a u64 getter | 0.4 (26) | 10 (25) | 10 (26) |  | 24 (26) | 47 (25) | 18 (27) | 41 (26) | 28 (26) |
+| `track.largest()`, a record return | 18 (26) | 31 (25) | 31 (26) |  | 113 (26) | 97 (25) | 37 (27) | 73 (26) | 56 (26) |
+| `track.info()`, an object return | 1.1 (26) | 25 (25) | 25 (26) |  | 137 (26) | 97 (25) | 50 (27) | 255 (26) | 243 (26) |
+| `group.write_frame(at, frame)`, 100 B | 132 (26) | 148 (25) | 161 (26) |  | 239 (26) | 196 (25) | 194 (27) | 274 (26) | 208 (26) |
+| the same, with a new `Timestamp` each | 133 (26) | 172 (25) | 178 (26) | 233 (27) | 331 (26) | 336 (25) | 224 (27) | 539 (26) | 447 (26) |
+| a 100 B frame written and read, ready | 205 (26) | 303 (25) | 341 (26) |  | 544 (26) | 560 (25) | 517 (27) | 903 (26) | 731 (26) |
+| the same, read after a wake | 330 (26) | 431 (25) | 474 (26) |  | 1,600 (26) | 6,912 (25) | 5,697 (27) | 5,061 (26) | 1,766 (26) |
+| `frame.payload`, 100 B | 11 (26) | 23 (25) | 24 (26) |  | 70 (26) | 150 (25) | 96 (27) | 150 (26) | 67 (26) |
+
+### Throughput (MB/s, write plus ready read; load average)
+
+| frame | Rust | C | C++ | Kotlin | Python | Swift | Go | Dart |
+|---|---|---|---|---|---|---|---|---|
+| 100 B | 472 (26) | 299 (25) | 290 (26) | 170 (26) | 138 (25) | 162 (27) | 105 (26) | 121 (26) |
+| 1 KiB | 3,444 (26) | 2,394 (25) | 2,267 (26) | 1,384 (26) | 1,239 (25) | 1,135 (27) | 769 (26) | 1,019 (26) |
+| 16 KiB | 7,324 (26) | 14,824 (25) | 6,923 (26) | 8,822 (26) | 9,464 (25) | 2,156 (27) | 2,072 (26) | 5,618 (26) |
+| 64 KiB | 2,552 (26) | 10,612 (25) | 5,466 (26) | 6,048 (26) | 9,469 (25) | 2,206 (27) | 2,622 (26) | 6,679 (26) |
+| 1 MiB | 3,647 (26) | 3,591 (25) | 5,253 (26) | 3,369 (26) | 3,464 (25) | 3,353 (27) | 2,693 (26) | 3,861 (26) |
+
+### Size
+
+| binding | files | raw | gzip | Rust glue, raw / gzip |
+|---|---|---|---|---|
+| C | 1 | 1,604 KiB | 235 KiB | 2,346 KiB / 125 KiB |
+| C++ | 1 | 2,664 KiB | 299 KiB | none |
+| Kotlin | 2 | 2,373 KiB | 318 KiB | 1,649 KiB / 72 KiB |
+| Python | 1 | 1,424 KiB | 217 KiB | 3,678 KiB / 282 KiB |
+| Swift | 3 | 2,231 KiB | 295 KiB | none |
+| Go | 19 | 3,429 KiB | 404 KiB | none |
+| Dart | 1 | 3,921 KiB | 360 KiB | 1,606 KiB / 59 KiB |
+
+| native library | as built | stripped |
+|---|---|---|
+| `libmoq.a`, the C ABI that C, C++, Swift, and Go link | 191,403 KiB | 102,985 KiB |
+| `libmoq.so`, with the Kotlin and Dart glue | 60,222 KiB | 45,126 KiB |
+| `python/libmoq.so`, the CPython extension | 54,694 KiB | 41,023 KiB |
+
+The stand-in exports all of moq-net, kio, and moq-pattern, and hang, moq-mux, and
+moq-tokio's session side, far more than moq-ffi's UniFFI interface, so its sizes do not
+compare with the fork's 653 / 914 / 684 KiB below. "Rust glue" is the generated crate's
+source each binding builds with: the C ABI's `src/lib.rs` and runtime, which every binding
+but Python's links, and `src/jni`, `src/python`, and `src/dart`. The native libraries are
+the export's default `release` build, which links all of moq's crates; "stripped" is
+`strip --strip-all` on a copy.
+
+### What a call spends
+
+From the CPU tables, read against the code each call runs. In order of what it costs moq's
+frame path:
+
+1. **A handle per frame timestamp.** `Timestamp` is `Copy` in Rust, but its private fields
+   make it an object, so each stamped frame makes and frees a handle: 21 ns in C, 27 in
+   Swift, 140 in Python, 250 in Kotlin, 260 in Go, and 245 in Dart (stamped write minus
+   write). In Go, Dart, and Kotlin that doubles the write.
+2. **Ownership of a returned object.** An object return costs C 25 ns, but Go 263 and
+   Dart 242, ten times C: each new Go object registers `runtime.AddCleanup` and each Dart
+   one a `NativeFinalizer`, beside a handle allocation. Kotlin's 137 ns wall (210 CPU) is a phantom
+   reference linked and unlinked under one global lock. Every returned frame, group, and future pays it.
+3. **A woken read.** C's 427 ns is Rust's 329 plus a future's crossings. Kotlin's 1.6 us and
+   Dart's 1.8 us add their event loops; Go's 5.1 us and Swift's 5.7 us wall (16.6 us CPU)
+   are a goroutine or `Task` and a cross-thread wake per read, and Python's 6.9 us is
+   asyncio's, as BoltFFI's Python measured 10 us. This case starts a task per read; a
+   reader looping on one task would skip the start, and is what to measure before changing
+   a generator.
+4. **A future per read.** A ready read costs C 155 ns over the write where Rust costs 73:
+   the future's three allocations and three crossings. Each language adds 160 (Swift) to 420 (Go) ns
+   more in its own wrapper, so the C ABI's share is not yet where a per-handle poll would pay.
+5. **Kotlin's record codec.** A record return costs Kotlin 113 ns wall against C's 31,
+   through the byte-array codec; Swift, which reads the C struct, costs 37.
+6. **The handle borrow at the C ABI.** The u64 getter costs Rust 0.4 ns and C 10: the
+   C ABI's borrow of each handle, a compare-and-swap in and a `fetch_sub` out of its
+   state, is the whole call. Every handle argument pays it again.
+7. **The binding's own call.** Over C's getter: Swift adds 8 ns (its hold count's two
+   atomics), Kotlin 14 wall (JNI and its hold count), Dart 19 (an `_Arena` per call and
+   its hold), Go 31 (cgo, `defer`, and its hold), and Python 37 (argument parsing and the
+   GIL released and taken back, so Rust may call Python from another thread). BoltFFI's
+   Python getter, which holds the GIL, measured 63 ns.
+
+### Per-handle baseline
+
+For [checks-raise](/quest/m1/ffi-export/checks-raise.md)'s one accounting and
+[swift-noncopyable](/quest/m1/ffi-export/swift-noncopyable.md)'s hold count: the u64
+getter is one borrowed handle and nothing else, so its row less Rust's 0.4 ns is what a
+handle costs a call. C, C++: 10 ns, the runtime's two atomics. Swift: 18, its hold count
+adding two more. Kotlin: 24 wall, Go 41, Dart 29, Python 47, each with its call's own
+transition besides. A change to the accounting moves this row and the handle call's by the
+same amount; measure both, CPU first, at the load beside them.
+
+### Against the targets
+
+Where the cases match, on wall time: Kotlin's getter costs 24 ns against BoltFFI's 5.8,
+its record return 113 against 61, its write 107 ns over Rust against BoltFFI's 56, and 100 B
+throughput 170 MB/s against 105. Python's getter costs 47 ns against 63, its record return
+97 against 1,120, and 100 B throughput 138 MB/s against 12.1. So Python meets BoltFFI's
+and Kotlin's calls trail it by 2 to 4 times, behind JNI's per-call cost and the hold count.
 
 ## BoltFFI comparison
 
-Before forking UniFFI, moq compared it with [BoltFFI](https://github.com/boltffi/boltffi) 0.31.1 on a slice of moq-ffi bound by each generator, measured from Python and Kotlin with a Rust floor.
+The targets. Before forking UniFFI, moq compared it with [BoltFFI](https://github.com/boltffi/boltffi) 0.31.1 on a slice of moq-ffi bound by each generator, measured from Python and Kotlin with a Rust floor.
 Its harness is deleted; [moq-dev/moq@0f986d9](https://github.com/moq-dev/moq/tree/0f986d9fdb7c074b0bd4b34739f7199c3bf2d9e5/bench/ffi-gen) holds it if a decision needs fresh numbers.
 BoltFFI's numbers are the targets for rust2-ffi's generators: its speed comes from calling conventions (a CPython extension instead of `ctypes`, JNI instead of JNA, and bulk byte copies), not from its core.
 

bench/ffi/report.py

@@ -0,0 +1,161 @@
+"""The binding benchmark's tables, in Markdown: the median of each case's rounds per language,
+in CPU and wall time, each beside the median 1-minute load average over its rounds, read
+from `just ffi bench-run`'s raw.tsv, and the sizes of the export's generated bindings and
+native libraries.
+
+Usage: report.py <raw.tsv> <export dir>
+"""
+
+import collections
+import gzip
+import os
+import shutil
+import statistics
+import subprocess
+import sys
+import tempfile
+
+LANGS = [
+	("rust", "Rust"),
+	("c", "C"),
+	("cpp", "C++"),
+	("moq-c", "moq-c"),
+	("kotlin", "Kotlin"),
+	("python", "Python"),
+	("swift", "Swift"),
+	("go", "Go"),
+	("dart", "Dart"),
+]
+
+CALLS = [
+	("call", "`track.is_closed()`, a handle call"),
+	("getter", "`demand.sequence()`, a u64 getter"),
+	("record", "`track.largest()`, a record return"),
+	("object", "`track.info()`, an object return"),
+	("write", "`group.write_frame(at, frame)`, 100 B"),
+	("stamped", "the same, with a new `Timestamp` each"),
+	("ready", "a 100 B frame written and read, ready"),
+	("woken", "the same, read after a wake"),
+	("payload", "`frame.payload`, 100 B"),
+]
+
+SIZES = [100, 1024, 16384, 65536, 1048576]
+
+# Each binding's generated files, and its Rust glue in the export's crate, under the export.
+BINDINGS = [
+	("C", ["include/moq.h"], ["src/lib.rs", "src/rt.rs"]),
+	("C++", ["include/moq.hpp"], []),
+	("Kotlin", ["kotlin/moq.kt", "kotlin/moq.pro"], ["src/jni"]),
+	("Python", ["python/moq.py"], ["src/python"]),
+	("Swift", ["swift"], []),
+	("Go", ["go"], []),
+	("Dart", ["dart/moq.dart"], ["src/dart"]),
+]
+
+# The shared library's suffix, as `just ffi moq-export` names it on each platform it builds on.
+SHARED = "dylib" if sys.platform == "darwin" else "so"
+
+LIBRARIES = [
+	("`libmoq.a`, the C ABI that C, C++, Swift, and Go link", "target/release/libmoq.a"),
+	(f"`libmoq.{SHARED}`, with the Kotlin and Dart glue", f"target/release/libmoq.{SHARED}"),
+	(f"`python/libmoq.{SHARED}`, the CPython extension", f"python/libmoq.{SHARED}"),
+]
+
+# What `strip --strip-all` drops, or on macOS, whose strip lacks it, every local symbol.
+STRIP = ["strip", "-x"] if sys.platform == "darwin" else ["strip", "--strip-all"]
+
+
+def files(root, paths):
+	"""Every file under each of `paths`, from `root`, sorted."""
+	out = []
+	for path in paths:
+		full = os.path.join(root, path)
+		if os.path.isdir(full):
+			for base, _, names in os.walk(full):
+				out += [os.path.join(base, name) for name in names if not name.endswith(".pyc")]
+		else:
+			out.append(full)
+	return sorted(out)
+
+
+def size(paths):
+	"""The raw and gzipped bytes of `paths`, each gzipped alone as a package ships it."""
+	raw = zipped = 0
+	for path in paths:
+		with open(path, "rb") as f:
+			data = f.read()
+		raw += len(data)
+		zipped += len(gzip.compress(data, compresslevel=6, mtime=0))
+	return raw, zipped
+
+
+def kib(n):
+	return f"{n / 1024:,.0f} KiB"
+
+
+def number(value):
+	return f"{value:,.1f}" if value < 10 else f"{value:,.0f}"
+
+
+def table(header, rows):
+	lines = ["| " + " | ".join(header) + " |", "|" + "---|" * len(header)]
+	lines += ["| " + " | ".join(row) + " |" for row in rows]
+	return "\n".join(lines)
+
+
+def main(raw_path, export):
+	rounds = collections.defaultdict(list)
+	with open(raw_path) as f:
+		for line in f:
+			lang, _, case, wall, cpu, load = line.rstrip("\n").split("\t")
+			rounds[(lang, case)].append((float(wall), float(cpu), float(load)))
+	median = {
+		key: tuple(statistics.median(values[i] for values in samples) for i in range(3))
+		for key, samples in rounds.items()
+	}
+	langs = [(key, name) for key, name in LANGS if any(lang == key for lang, _ in median)]
+
+	def cell(lang, case, i):
+		if (lang, case) not in median:
+			return ""
+		values = median[(lang, case)]
+		return f"{number(values[i])} ({values[2]:.0f})"
+
+	thru = [(key, name) for key, name in langs if (key, f"mbps:{SIZES[0]}") in median]
+	labels = {100: "100 B", 1024: "1 KiB", 16384: "16 KiB", 65536: "64 KiB", 1048576: "1 MiB"}
+	for i, clock, rate in [(1, "CPU", "MB per CPU second"), (0, "wall", "MB/s")]:
+		print(f"### Calls (ns per call, {clock} time; load average)\n")
+		header = ["call"] + [name for _, name in langs]
+		print(table(header, [[label] + [cell(lang, case, i) for lang, _ in langs] for case, label in CALLS]))
+		print(f"\n### Throughput ({rate}, write plus ready read; load average)\n")
+		header = ["frame"] + [name for _, name in thru]
+		rows = [[labels[n]] + [cell(lang, f"mbps:{n}", i) for lang, _ in thru] for n in SIZES]
+		print(table(header, rows))
+		print()
+
+	print("### Size\n")
+	rows = []
+	for name, generated, glue in BINDINGS:
+		raw, zipped = size(files(export, generated))
+		glue_raw, glue_zipped = size(files(export, glue))
+		count = len(files(export, generated))
+		glue_cell = f"{kib(glue_raw)} / {kib(glue_zipped)}" if glue else "none"
+		rows.append([name, str(count), kib(raw), kib(zipped), glue_cell])
+	print(table(["binding", "files", "raw", "gzip", "Rust glue, raw / gzip"], rows))
+
+	print()
+	rows = []
+	for label, path in LIBRARIES:
+		full = os.path.join(export, path)
+		with tempfile.TemporaryDirectory() as tmp:
+			copy = os.path.join(tmp, os.path.basename(path))
+			shutil.copyfile(full, copy)
+			subprocess.run([*STRIP, copy], check=True)
+			rows.append([label, kib(os.path.getsize(full)), kib(os.path.getsize(copy))])
+	print(table(["native library", "as built", "stripped"], rows))
+
+
+if __name__ == "__main__":
+	if len(sys.argv) != 3:
+		sys.exit(__doc__)
+	main(sys.argv[1], sys.argv[2])