Skip to content

M1/PR16: performance test infrastructure — BenchmarkDotNet + MemoryDiagnoser baselines #18

Closed

Description

@Blind-Striker

Source: maintainer directive + multi-agent review of PR #16 (perf cluster P1-P5 CONFIRMED/PLAUSIBLE), 2026-08-13.

Standing posture (recorded in AGENTS.md): performance is a standing concern in both the SDK and the tooling — SDK targets speed and minimal avoidable allocation, tooling targets speed. No speculative optimization, but obvious waste is always evaluated, starting with low-hanging fruit. Perf claims are settled by benchmarks, not argument.

Deliverable

tests/OpenCode.Sdk.Performance.Tests — BenchmarkDotNet console project (mirrors the BadgeSmith shape: https://github.com/localstack-dotnet/badge-smith/blob/master/tests/BadgeSmith.Api.Performance.Tests/BadgeSmith.Api.Performance.Tests.csproj), riding the repo's full convention set. MemoryDiagnoser on every benchmark — speed and allocation move together.

First benchmark set = the baseline for the confirmed P-findings:

  • End-to-end GetMessageAsync via a canned HttpMessageHandler (no network, deterministic bytes) — P1+P2+P3 composite
  • Union converter walk over a deep assistant-message fixture — P1 (DOM materialization + second traversal per union level; nested content multiplies)
  • Envelope ReadDataPayload parse (ResponseAdapter.cs:52-59) — P2 (full envelope DOM + payload re-traversal)
  • Pure Pipeline overhead via GetHealthAsync — P3 (UTF-16 body buffering on success paths)

Rules

  • Not a CI gate (shared-runner variance); run manually, wire into the M6 nightly canary lane when it exists. Perf-touching PRs carry before/after numbers.
  • ADR-0004 immutable-model allocations and ADR-0007 RawBody retention are contracts, not waste — benchmarks measure around them, never against them.
  • Baseline numbers land in this issue when first measured; subsequent optimizations (issue on M3 union redesign) cite them.

Activity

  1. Blind-Striker commented on Aug 13, 2026

    @Blind-Striker
    CollaboratorAuthor

    Baseline measured at 959c4f0 (pre-fix M1 surface), Linux x64, .NET 10.0.10, BenchmarkDotNet 0.15.8, MemoryDiagnoser:

    Benchmark Mean Allocated
    ClientOperationBenchmarks.GetMessageAsync (end-to-end, deep 4-level assistant fixture) 67.4 μs 27.42 KB
    ClientOperationBenchmarks.GetHealthAsync (end-to-end, bare payload) 1.56 μs 2.08 KB
    UnionDeserializationBenchmarks.DeserializeDeepAssistantMessage (union walk only) 52.8 μs 17.37 KB

    Reading: the union converter walk alone accounts for ~78% of the whole end-to-end message operation (52.8 of 67.4 μs) and 17.4 of 27.4 KB — P1 is confirmed as the dominant cost, exactly as the review claimed. The pipeline itself (health path: request build + auth + buffer + bare parse) costs ~1.6 μs / 2 KB. These are the numbers the M3 single-pass redesign (#23) must beat.

  2. Blind-Striker commented on Aug 13, 2026

    @Blind-Striker
    CollaboratorAuthor

    Follow-up noted with the maintainer: add parameterized payload sizes to the deserialization benchmarks — [Params] over deterministic, shape-representative fixtures (current ~2.6 KB small; add a mid-KB ~30-50 KB transcript with many tool parts). Lands together with the M3 single-pass redesign (#23) so the scaling behavior (P1 nesting multiplier, P3 linear UTF-16 detour) is visible in the before/after.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceSpeed / allocation workready-for-agentSpecified enough that the work is mechanical

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions