Skip to content

ci: add .NET SDK performance reports - #358

Merged
dustinbyrne merged 8 commits into
mainfrom
feat/dotnet-performance-benchmarks
Nov 4, 2026
Merged

dustinbyrne merged 8 commits into
mainfrom
feat/dotnet-performance-benchmarks

Conversation

@dustinbyrne

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

Extend SDK PR performance reporting to .NET, alongside Node and Python, using the shared reporting action and trusted publisher from PostHog/.github#84.

  • Run on every opened, reopened or updated PR. Compare its merge base and exact head using the same head-side benchmark sources, isolated processes and six alternating paired repetitions.
  • Add 17 public-API workloads: single/all local flags, concurrent evaluation, reloads, enqueueing and queue draining. Measure elapsed time, process CPU, managed allocation volume, retained heap and RSS separately. Exact allocation counts are explicitly unsupported.
  • Keep performance deltas informational. Fail on build/execution, fixture or delivery errors; stop after two worker timeouts while preserving diagnostics.
  • Keep PR jobs read-only. A separate pinned workflow_run publisher produces sticky comments once the caller workflow lands on the default branch.

The C# workers use .NET 9.0.8 with workstation GC and non-tiered JIT; the Python coordinator uses only the standard library. SDK dependencies restore in locked mode. Offline HTTP injection retains the SDK's public lifecycle without requiring credentials. Documentation lives in the existing contributing guide.

No SDK runtime/public API changes or NuGet release are included; no changeset is needed.

💚 How did you test it?

  • Final same-revision run: 372 successful workers, 17 workloads, six pairs, 160.94 seconds on macOS ARM64. Largest timing/CPU median delta was 8.85% with unchanged SDK code: this is variability, not a performance improvement claim.
  • Native controls passed for fixture correctness, concurrent/reload workloads, delivery, managed allocation churn, retained heap, RSS and strictly increasing checkpoints.
  • 9 coordinator tests passed, including timeout termination/budget, alternating pairs, wrong-build rejection, invalid measurements and dependency metadata classification.
  • Real shared reporter accepted the resulting schema-v1 documents and produced 57 numeric comparisons. Integration controls verified changed SDK/dependency identities remain comparable, different runtime configurations do not, and failed sources suppress numeric comparisons. Reproduced and fixed the dependency-identity issue found in independent review.
  • dotnet restore --locked-mode, bin/fmt --check, and the existing net8.0 unit suite: 1,403 passed, 2 skipped.
  • Python Ruff/Black checks, Actionlint and git diff --check passed. All action/workflow SHA references were verified through GitHub.
  • Linux execution is covered by this PR's benchmark workflow. Live comment publication still requires default-branch rollout.

📝 Checklist

  • All commits are signed.
  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

🤖 Agent context

Autonomy: Human-driven (agent-assisted).

Implemented with Pi and checked by a fresh read-only reviewer. Validation used .NET, Python, Actionlint and the shared benchmark reporter. The design keeps instrumentation native to .NET and reporting shared across SDKs; this PR requires human review.

@dustinbyrne dustinbyrne self-assigned this Oct 28, 2026
@dustinbyrne
dustinbyrne requested a review from a team as a code owner October 28, 2026 18:09

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 166d3e7: 2 Should fix.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

Comment thread benchmarks/Program.cs Outdated
workloads = Fixtures.Workloads(args.Contains("--smoke")).Select(spec => new
{
spec,
fixture_digest = Fixtures.Digest(Fixtures.Definitions(spec.FlagCount) + JsonSerializer.Serialize(Fixtures.Options, Json) + JsonSerializer.Serialize(spec, Json)),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Hash explicit fixture inputs instead of the SDK options schema

When a PR adds a serializable property to FeatureFlagOptions or its base class, serializing Fixtures.Options includes that property only in the head build, even with its default value. The resulting fixture_digest mismatch makes the coordinator reject both inventories before running benchmarks. Hash a benchmark-owned representation of the explicit inputs instead.

Comment thread benchmarks/run.py
Comment on lines +88 to +92
subprocess.check_output(
["dotnet", str(worker), "describe", *(["--smoke"] if smoke else [])],
env=environment(),
text=True,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Apply a deadline to the describe subprocess

When SDK construction or disposal hangs, describe() blocks indefinitely because this check_output has no timeout. The describe command constructs and disposes PostHogClient, so lifecycle regressions bypass the worker deadline and timeout budget, consuming the workflow's 30-minute limit before either results document exists. Bound this subprocess with TIMEOUT.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

posthog-dotnet Compliance Report

Date: 2026-11-03T21:31:06.709600+00:00
Duration: 124605ms

✅ All Tests Passed!

47/47 tests passed


Capture Tests

✅ 30/30 tests passed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 203ms
Format Validation.Event Has Uuid ✅ 121ms
Format Validation.Event Has Lib Properties ✅ 108ms
Format Validation.Distinct Id Is String ✅ 110ms
Format Validation.Token Is Present ✅ 109ms
Format Validation.Custom Properties Preserved ✅ 113ms
Format Validation.Event Has Timestamp ✅ 108ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc ✅ 118ms
Retry Behavior.Retries On 503 ✅ 8119ms
Retry Behavior.Does Not Retry On 400 ✅ 2114ms
Retry Behavior.Does Not Retry On 401 ✅ 2112ms
Retry Behavior.Respects Retry After Header ✅ 8116ms
Retry Behavior.Implements Backoff ✅ 22130ms
Retry Behavior.Retries On 500 ✅ 6115ms
Retry Behavior.Retries On 502 ✅ 6118ms
Retry Behavior.Retries On 504 ✅ 6116ms
Retry Behavior.Max Retries Respected ✅ 22122ms
Deduplication.Generates Unique Uuids ✅ 113ms
Deduplication.Preserves Uuid On Retry ✅ 6113ms
Deduplication.Preserves Uuid And Timestamp On Retry ✅ 13127ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 6117ms
Deduplication.No Duplicate Events In Batch ✅ 111ms
Deduplication.Different Events Have Different Uuids ✅ 109ms
Compression.Sends Gzip When Enabled ✅ 107ms
Batch Format.Uses Proper Batch Structure ✅ 106ms
Batch Format.Flush With No Events Sends Nothing ✅ 104ms
Batch Format.Multiple Events Batched Together ✅ 110ms
Error Handling.Does Not Retry On 403 ✅ 2110ms
Error Handling.Does Not Retry On 413 ✅ 2109ms
Error Handling.Retries On 408 ✅ 6114ms

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 129ms
Request Payload.Flags Request Uses V2 Query Param ✅ 106ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 107ms
Request Payload.Flags Request Omits Authorization Header ✅ 107ms
Request Payload.Token In Flags Body Matches Init ✅ 107ms
Request Payload.Groups Round Trip ✅ 109ms
Request Payload.Groups Default To Empty Object ✅ 106ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 107ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 107ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 107ms
Request Lifecycle.No Flags Request On Init Alone ✅ 3ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 105ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 209ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 109ms
Retry Behavior.Retries Flags On 502 ✅ 1109ms
Retry Behavior.Retries Flags On 504 ✅ 1108ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 207ms

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 9497641: 1 Should fix. This pull request is large, so the review ran in 4 parts with less depth than usual.

Comment thread benchmarks/Workload.cs Outdated
if (request.Method == HttpMethod.Get && request.RequestUri?.AbsolutePath.TrimEnd('/') == "/flags/definitions")
{
Interlocked.Increment(ref DefinitionRequests);
return Response(Fixtures.Definitions(spec.FlagCount, Generation));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Prepare definition responses outside the measured reloads

During flags.refresh.1000, every measured reload calls Fixtures.Definitions here, constructing the 1,000-flag object graph and serializing JSON inside the worker. These mock-server allocations inflate measured RSS and GC pressure, obscuring changes in the SDK's reload memory usage. Precompute generation responses before measurement and serve those fixtures.

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 4bc67e1: no findings. This pull request is large, so the review ran in 4 parts with less depth than usual.

@marandaneto marandaneto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good; no qualifying findings. Verdict: correct.

@dustinbyrne
dustinbyrne merged commit 14f7e3c into main Nov 4, 2026
22 checks passed
@dustinbyrne
dustinbyrne deleted the feat/dotnet-performance-benchmarks branch November 4, 2026 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants