Skip to content

CLI: bench-coordinator times the coordinator's own share of release-to-paste - #77

Open
dinooo13 wants to merge 1 commit into
mainfrom
cli/bench-coordinator
Open

dinooo13 wants to merge 1 commit into
mainfrom
cli/bench-coordinator

Conversation

@dinooo13

Copy link
Copy Markdown
Owner

Part of #73, point 5.

The CLI benchmarks time the engine and the processors; until now the coordinator's own share of release-to-paste (the feed hand-over, endRecording, the hops between stages on the main actor) showed up only in the app's log line. This adds a harness that gives it a number of its own, and that PRs E and F (the coordinator splits) use for their before/after.

What

swift run -c release pladder-cli bench-coordinator [--runs N] [--hold S] [--engine-latency MS] [--paste-latency MS] [--polish] [--live]

  • Sources/PladderCLI/BenchCoordinator.swift, self-contained like BenchProcess.swift: it needs only its line in main.swift's dispatch, so it can be copied onto an older commit for a before/after.
  • It builds the real DictationCoordinator with StandardProcessors.pipeline(for:) on a fresh install's DictationSettings, and private stand-ins for the rest:
    • FixedLatencyEngine: a StreamingTranscriptionEngine on UtteranceSlot<Void>. endUtterance, warmPass, livePass and transcribe each take the latency and ignore cancellation, as a CoreML pass does.
    • SilentCapture: silence at the real rate. A drain or a stop returns what a microphone would have captured since the last one.
    • TimedOutput, FixedLatencyRefiner (300 ms, with --polish), and a hotkey monitor that does nothing.
  • It touches no microphone, pasteboard or hotkey, so it can run beside a copy of Pladder in use.
  • Defaults (as decided): 21 runs with the first discarded, 1.5 s hold, 150 ms engine, 0 ms paste.

Each run presses, holds, releases and waits for inFlight, stamping .recordingStopped and .inserted in onEvent:

  • total = inserted − stopped
  • share = total − (stop + engine + process + polish + paste), with the stages taken from the inserted event
  • engine excess = timing.engine − engine latency

The output is one line per run, then a table with printBenchHeader's machine line and the load average at start and end.

The stand-ins sleep on a strict DispatchSource timer. Task.sleep overshot a 150 ms sleep by about 8 ms on the M1 even with zero tolerance, which would have swamped the engine-excess column. The strict timer overshoots by about 0.1 to 0.2 ms; the header prints the measured overshoot.

Docs:

TimedOutput also has a no-op copy(_:). #76 makes TextOutput.copy a requirement, and this keeps whichever of the two lands second building.

First numbers

Apple M1, 16 GB, macOS 27.0, 21 runs each, load average about 2 (the other PRs' builds had just finished):

Variant Total (median) Share (median / p90 / max) Engine excess (median)
default 153 ms 0.02 / 0.02 / 0.03 ms 0.15 ms
--polish 453 ms 0.04 / 0.04 / 0.04 ms 0.15 ms
--live 153 ms 0.02 / 0.07 / 0.09 ms 0.14 ms

Spread check, --runs 51 twice: medians of 156 / 153 ms total, 0.05 / 0.02 ms share, 0.17 / 0.15 ms engine excess. The few milliseconds of total above 150 ms are the processors (2 to 3 ms, the language recogniser).

No app code changes, so the critical-path rule does not apply here.

For the developer

  • Replace the draft ANECCompile() FAILED (11) paragraph in docs/BENCHMARKS.md "Baseline" with real numbers once the paced baseline is re-run on a quiet day.

🤖 Generated with Claude Code

…o-paste

The real coordinator and processors, with stand-ins of fixed latency for the
engine, microphone, paste and polish; what is left of the total is the
coordinator's hand-over and its hops between stages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant