Skip to content

Flaky under load: the handoff ticket test published a null sidecar in 97 ms (net11) #314

Open

Description

@Blind-Striker

What

In Surface's full gate at 264e13b5 (step 3 of #176, on master 5ccbe6da; main pass, net11.0, 2026-10-15 13:44–13:55, under full gate load), this test failed once in 97 ms:
ServicePtyHandoffTests.PrepareAsync_Should_Publish_The_Ticket_Exactly_As_The_Route_Answered_It

[Test Failure] InvalidOperationException: Cannot read the Value property of a Nullable object that has no value.
  at System.Nullable`1.get_Value()
  at ...ServicePtyHandoffTests.PrepareAsync_Should_Publish_The_Ticket_Exactly_As_The_Route_Answered_It() in ...\ServicePtyHandoffTests.cs:421

Line 421 is ReadSidecar()!.Handoff!.Value.GetRawText(). So a sidecar was published, with Handoff null. The class passed 5/5 on net11.0 in isolation afterwards. No standard output was captured.

Not #312

The test publishes through TestablyServiceFileSystem over an in-memory MockFileSystem (Handoff(), _fileSystem). The shipped ServiceFileSystem and the Windows POSIX-semantics rename #312 changed are not on its path, and the publish itself succeeded.

How PrepareAsync reaches a null sidecar (ServicePtyHandoff.cs)

  1. RequestTicketAsync fails with an OpenCodeException, or an OperationCanceledException that is not the caller's. The catch shuts the daemon's terminals down (the loopback double answers 200 to every path) and publishes NewSidecar(handoff: null).
  2. Parse throws OpenCodeServerException (no handoff member, or a ticket that fails IsHandoff, expiresAt, or the expiry check). That is an OpenCodeException, so it takes the same catch.
  3. Parse reads handoff: null, which this test's double never answers.

The test cannot tell which one ran. Branches 1 and 2 also send the shutdown request (server.RequestPaths contains the shutdown path); branch 3 does not. A loopback exchange failing fast under gate load (branch 1) fits the 97 ms best, but it is not proven.

Proposed (not taken during the planned stop)

  • Make the test say what happened: assert on the double's recorded requests (the handoff route only, no shutdown) before reading the sidecar, so the next failure names the branch, with the failed exchange's exception where possible.
  • Then fix the cause the next failure names: the loopback double under load, the request pipeline's fast failure, or the parse.

Recorded before any rerun, per the flake rule. Surface's gate rerun at 6f5334f4 will report this test's result here.

Activity

  1. added this to the 1.0 milestone on Oct 15, 2026
  2. Blind-Striker commented on Oct 15, 2026

    @Blind-Striker
    CollaboratorAuthor

    Result from the next full gate on the same machine: PrepareAsync_Should_Publish_The_Ticket_Exactly_As_The_Route_Answered_It passed on net10.0, net11.0 and net472. That was the step 3 branch at 6f5334f, on master 5ccbe6d with #312 included, run 14:01–14:10, bun 1.4.2. The main pass was 8763/8763; ServiceElection was 24/24 and NativeAot 3/3. In isolation, the class also passed 5/5 on net11 after the failing run. So one failure in three full-gate runs of this test on this branch, and no reproduction yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions