Skip to content

stream: fail and back off on refused streams - #360

Open
meling wants to merge 2 commits into
feat/325/peer-identityfrom
fix/325/refused-stream-retry
Open

meling wants to merge 2 commits into
feat/325/peer-identityfrom
fix/325/refused-stream-retry

Conversation

@meling

@meling meling commented Oct 11, 2026 •

Copy link
Copy Markdown
Member

Stacked on #359. Merge #359 first; this PR's base will then switch to master.

Peer identity setup is defined by #359.
This PR changes how the dialer handles refused streams, not the peer identity API.

Problem

When a server refused a dialed stream at accept time (for example, a peer whose TLS identity fails verification), the dialer requeued the pending call and opened a new stream at once.
One pending call caused about 15,000 refused streams per second; the call failed only at its deadline, with the context's error instead of the refusal's code.
Each attempt also counted as up until the refusal arrived, so the refused side's ConnectedPeers and WithPeerChange flapped, and under WithStreamDedup its WaitForAll could return although every stream was refused.

Change

A dialed stream is refused when it ends with Unauthenticated, PermissionDenied, InvalidArgument, or Unimplemented before the server sent anything.

  • Pending calls, including streaming calls, fail with the refusal's status rather than being resent.
  • The channel backs off before opening another stream: 50 ms, doubling to 2 s, with or without eager reconnect. Calls made during the wait fail with the refusal. A failed dial-side check is paced the same way.
  • The send/receive handoff leaves refusal disposition to the receive loop, which alone learns the server's status. Cancellation cannot bypass the ownership CAS.
  • After a refusal, successful sends do not clear Node.LastErr until a stream proves viable: it delivers a message, or a send succeeds on a stream that has lived 2 s. A newer error, such as Unavailable, may replace the refusal.

Transient failures (a broken connection, a server restart, GOAWAY, Unknown, or any status after the server has sent a frame) retain the existing retry and reconnect behavior.

Readiness and health limits

The timer-based readiness change was removed during review.
Connection reporting remains optimistic: each new stream counts as up when opened and verified locally, before the server accepts or refuses it.
ConnectedPeers and WithPeerChange can therefore report a refused peer once per backoff period, and WaitForAll is not confirmation of remote acceptance.
Idle accepted streams become ready without a two-second probation, including under repeated GOAWAY.
An explicit acceptance signal is deferred to a separate issue.

With one-way-only traffic and GOAWAY rotation that keeps every stream shorter than 2 s, LastErr can remain stale after recovery even while messages reach the server.
This limit was reproduced and documented; health behavior was not changed.

Earlier measurements

These measurements predate the rebase and were not rerun on the rebased branch.
The runtime refusal and retry implementation is unchanged by the rebase.

  • One refused pending call: 1 stream instead of about 15,600; the call fails in under 1 ms with the refusal's code.
  • End to end with mTLS: the refusing server runs its verifier 4 to 8 times per second instead of about 13,000.

Testing

  • Session and channel tests cover classification, handoff, pacing, fail-fast, LastErr, recovery, optimistic stream reporting, and transient retries.
  • A deterministic regression covers cancellation between the sender's handoff load and CAS for two-way and streaming calls.
  • An mTLS regression covers refusal followed by idle acceptance and repeated GOAWAY under stream deduplication.
  • After the rebase, the TLS tests retain automatic peer identity checks from gorums: authenticate peers using node TLS identities #359. gRPC stream interceptors count stream attempts and inject temporary refusals for recovery coverage, without restoring the removed public verifier API.

Validation passed on the rebased branch:

make test
make integrationtest
go test -race ./internal/stream ./internal/conn ./internal/tests/tls .
go test -race -tags=integration ./internal/tests/tls -run '^TestPeerIdentity' -count=5
make modernize
make goplscheck
git diff --check

make test includes benchmark validation.

Docs: WithPeers, the user guide's Authenticating Peers section, and the developer guide's stream layer describe the behavior and its limits.

Copilot AI balanced review requested due to automatic review settings October 11, 2026 19:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@meling
meling force-pushed the fix/325/refused-stream-retry branch from 1978abc to 66dc711 Compare October 11, 2026 20:40
@meling
meling added this pull request to stack #362 October 11, 2026 20:40
@meling
meling requested a lite review from Copilot October 12, 2026 07:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

internal/stream/outbound.go has unresolved LastErr preservation and clearing issues.

1 open finding

🧠 Review effort: Lite

Comment thread internal/stream/outbound.go
@meling
meling force-pushed the fix/325/refused-stream-retry branch from 66dc711 to 6b82aa3 Compare October 12, 2026 20:10
@meling
meling force-pushed the fix/325/refused-stream-retry branch from 6b82aa3 to 0acb890 Compare October 12, 2026 20:22
@meling
meling force-pushed the fix/325/refused-stream-retry branch from 0acb890 to fec7394 Compare October 17, 2026 16:32

@meling meling left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed against the stacked base feat/325/peer-identity, not master. Four findings are attached, with reproductions. The existing LastErr-preservation discussion is not duplicated here.

Simplifications

  • In sendFailed, removing the s.ended() escape from the ownership decision both simplifies the state machine and fixes the handoff interleaving described below. Cancellation should not bypass the CAS that decides who disposes of pending calls.
  • Extract one received-frame-or-age predicate and use one time origin for health eligibility, readiness promotion, and backoff reset. These currently use opened, a separate probation timer, and the run loop's start. Keep the timer as a wakeup mechanism. Do not reuse viable() unchanged for backoff reset: its !suspect shortcut has different semantics.
  • pace and backoff duplicate timer creation and capped doubling. pace can retain its non-eager early return and otherwise delegate to backoff, discarding the deadline.
  • Give readiness probation its own named duration instead of coupling it to the maximum reconnect delay. More broadly, consider separating the heuristic readiness change from the refusal/backoff fix until its GOAWAY and slow-refusal contracts are settled.

Validation

Existing tests pass for ./internal/stream ./internal/conn ./internal/tests/tls . in default, -race, and -tags=integration modes. Scratch Go-overlay tests reproduce the four findings without changing tracked files. The handoff failure uses a deliberately forced legal interleaving; an additional ungated 300,000-iteration race stress run passed.

The wider go test ./... ./examples/... ./benchkit/... run failed in the unchanged benchkit/cmd/sweep test TestWriteReplayScript: replay_test.go:57: mode = -rwx------, want 0755. This was not investigated or changed as part of the PR review.

Comment thread internal/stream/session.go Outdated
Comment thread internal/stream/outbound.go Outdated
Comment thread internal/stream/session.go Outdated
Comment thread doc/user-guide.md Outdated
@meling
meling force-pushed the fix/325/refused-stream-retry branch 2 times, most recently from 54bc386 to be90084 Compare October 20, 2026 21:44
@meling
meling force-pushed the fix/325/refused-stream-retry branch from be90084 to e7e337c Compare October 20, 2026 23:02
A server that refuses a stream, for example because the dialer fails
its identity check, ends the stream with a status before it sends any
frame. The outbound channel treated that as a lost stream: it requeued
the pending calls and opened a new stream at once, because a request
was waiting. The calls failed only when their contexts ended, never
with the refusal's code, and both servers spun at thousands of streams
per second.

A stream now counts as refused when it ends with Unauthenticated,
PermissionDenied, InvalidArgument, or Unimplemented before it
delivered a frame. Only the receive loop learns the stream's status.
So a send that fails with io.EOF before the first frame only cancels
the stream and leaves the pending calls to the receive loop, and an
atomic handoff decides which goroutine disposes of them. A one-way
send that fails this way waits for the receive loop's verdict, so it
fails with the refusal too. If a frame arrives after the failed send,
the server accepted the stream: the buffered frames are delivered,
and the calls still pending are retried.

Calls on a refused stream fail with the refusal, streaming calls
included. The channel then backs off (50 ms, doubling to 2 s) before
it opens the next stream, with or without eager reconnect, and fails
each request at once with the refusal until the backoff ends. A failed
dial-side verification backs off the same way. Lost streams are still
requeued and reconnected. Connection state stays optimistic: a new
stream counts as up when it opens, before the server's verdict.

The user guide gains a Refused Streams section, since the behavior
applies to every client, not only to peers.
A send completes locally before the server's verdict on a new stream
arrives. So after a refusal, each new stream's first send cleared
LastErr, and LastErr flipped between the refusal and nil for about one
round trip per attempt. Callers that sort nodes with ByLastError then
saw a refusing node as healthy.

A session opened after a refusal is now suspect until a stream proves
viable: the stream delivers a frame, or it lives 2 s. A send on a
suspect session clears LastErr only once its stream has proven
viable. A received frame clears LastErr as before, and a newer error
may replace the refusal. The run loop uses the same test to start its
backoff over.
@meling
meling force-pushed the fix/325/refused-stream-retry branch from e7e337c to 23264be Compare October 20, 2026 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants