Repository navigation
No gate runs the benchmark suite: two breakages reached master unseen #326
Description
What
No gate runs the benchmark suite, so it can break without anyone noticing. Two breakages reached master on 2026-10-14/15, and both were found only by the #185 work:
- No benchmark could be built under BenchmarkDotNet since
5730c893(2026-10-14, "the generator marks the upstream-experimental surface [Experimental]").UnionDeserializationBenchmarks.DeserializeConcreteLogSyncedreturnedEventLogSynced, which became[Experimental("OPENCODE003")].- BenchmarkDotNet's generated runner (
BenchmarkDotNet.Autogenerated.csproj) names each benchmark's return type. It does not inherit the performance project'sOPENCODE00xNoWarn arbitration. - The result:
error OPENCODE003in the generated code, and// BenchmarkDotNet has failed to build the auto-generated boilerplate code, for every benchmark of the run.
EventStreamBenchmarkscould not complete since6f5334f4(feat: the event bus reconnects as upstream's client does, and gives up after a minute #315, 2026-10-15, the event bus reconnects as upstream's client does).- The fixture had no
server.connectedfirst frame, soStreamAttempt.Establishrefuses the connection. - The bus now reconnects at the end of a body instead of ending.
- Its
GlobalSetupwould throw or retry until it gave up. - Found by the Sol 1M cross-family review of the 1.0 benchmark coverage: correctness review, missing scenarios, comparable runs, net11.0 leg #185 slice (internal ledger
platform-review-185a).
- The fixture had no
Both are fixed in the #185 slice. Nothing would have caught a third.
Why
quality-gates.md makes the suite the evidence for performance claims and the 1.0 performance gate. Two silent breakages in two days mean that a baseline or release-candidate run can be the first to discover the suite is broken. The run then costs hours before it fails, or measures the wrong thing.
Cheapest guard that would have caught both
Both breakages fail before any measurement: the first in BDN's build of the generated runner, the second in a GlobalSetup. The cheapest exact guard therefore builds the generated runner and runs every GlobalSetup once. That is BenchmarkDotNet's Dry job over the whole suite.
| Option | Catches | Cost |
|---|---|---|
A. A CI step on one Linux leg: dotnet run --project tests/OpenCode.Sdk.Performance.Tests -c Release -f net10.0 -- --filter '*' --job Dry --runtimes net10.0 |
both classes, and any other generated-runner build failure or setup refusal | Measured on the Windows workstation, 2026-10-15: the whole suite, 320 cases on net10.0 and net472, took 258 s. net10.0 alone is about 160 cases, so roughly 2 to 2.5 min plus the performance project's build, on a leg that has already built the solution. |
B. A reflection test in OpenCode.Sdk.Tools.Tests: no [Benchmark] signature names an [Experimental] type |
class 1 only | negligible (one test) |
C. Run GlobalSetup of each class in-process from a TUnit test |
class 2 only (no BDN codegen) | seconds, but it re-implements BDN's lifecycle |
Recommendation: A, on the Linux leg only (net472 adds nothing to these failure classes), with the step failing on BDN's own "has failed" lines and on any // * Exceptions * from a setup. B can be added later if A's cost is unacceptable.
Done when
- The chosen guard runs in CI on every pull request that touches
tests/OpenCode.Sdk.Performance.Tests,src/, or the generator. quality-gates.mdsays which check keeps the suite runnable.- Reverting either fix makes the guard fail.
Activity