Skip to content

fix(net)!: stop one silent dial from stalling the reconnect loop - #6540

Closed
mattshep wants to merge 1 commit into
moq-dev:mainfrom
daily-co:net-attempt-deadline
Closed

mattshep wants to merge 1 commit into
moq-dev:mainfrom
daily-co:net-attempt-deadline

Conversation

@mattshep

Copy link
Copy Markdown
Contributor

Problem

connect() has no deadline, and Reload hands it only its teardown signal. So a dial the network never answers holds the reconnect loop until the browser or OS gives up. delay.timeout is only checked after an attempt fails, so it doesn't help.

This happens when a client reconnects through a layer-4 load balancer.

  • We opened fresh TCP and QUIC flows every 200 ms to each address of an AWS NLB while one of its two targets was force-killed. The NLB had a TCP_UDP listener, IP targets, a health check every 10 s, a 300 s deregistration delay, and connection termination off.
  • For about 12 s after the target started draining, about one new flow in two landed on the dead target.
  • Every packet of a flow goes to the target it was placed on, so retransmits never escape. Only a new flow does.

In Chrome 152 on macOS, against an address that never answers:

  • WebTransport gives up after 4 s on its own (quiche's 5 s idle-before-handshake timer, less 1 s on the client).
  • A WebSocket waits for the OS TCP connect timeout: 75 s on macOS.

Promise.any waits until both fail, so an attempt whose dials all land on the dead target waits 75 s. Where WebSocket is the only transport (Safari, Firefox before 153, Node, Bun, Deno), one dead dial is enough. A browser page in this situation took 80 s to reconnect.

Approach

Four parts, all in connect(), so Reload, the pool's shared and private loops, GOAWAY migration and a one-shot connect() all get them.

  • A timeout per attempt. ConnectProps.timeout and ConnectionProps.timeout (default 30 s; Milli(0) for none) cover the dial and the MoQ handshake.
  • A second WebTransport dial at 500 ms, if the first hasn't opened by then.
    • 500 ms is the default WebSocket head start, so a fresh WebTransport flow races the fallback instead of losing to it. Nothing upgrades a WebSocket session back.
    • Chrome opens each WebTransport on its own UDP socket (allowPooling: false), so each dial is a new flow.
    • Neither dial is cut on a timer of its own. The browser's handshake timer can tell a silent dial from a slow one; script can't. And Chrome penalises every failed or abandoned handshake in a per-page throttle.
    • A first dial that fails outright isn't hedged.
    • If both dials fail, the error names only the first dial's reason, so no transport connected: … keeps one reason per transport. A browser words every handshake failure the same way.
  • A WebSocket redial at 10 s, doubling.
    • A browser allows one connecting WebSocket per address (RFC 6455 section 4.1), so a second dial queues behind a silent one. Closing the silent one frees the address.
    • Checked in Chrome 152, for both WebSocketStream and WebSocket:
      • Against a local forwarder that accepts the TCP connection and never answers, a second WebSocket opened no connection until the first was closed, then connected at once.
      • Against an address that drops SYNs (WebSocket), a second WebSocket dialed 1.8 s in failed 75.0 s after the first was closed (at 5.8 s), not 75 s after its own dial. So it had been queued too.
    • Safari and Firefox are not checked.
  • Losers are closed. A contender that opened in the same turn as the winner had already returned its transport, so cancelling couldn't close it. That leak already existed between WebTransport and WebSocket. Every contender that wasn't chosen is now closed.

Consumers: <moq-watch> and <moq-publish> use the shared pool, so they get all of this through the defaults. The timeout options are the escape hatch described in MIGRATE.md.

Impact

  • New ConnectProps.timeout and ConnectionProps.timeout. share: true refuses timeout, as it does delay.
  • New public Error.ConnectTimeout, with timeout in milliseconds.
  • Breaking (MIGRATE.md): connect() and each Connection attempt reject with Error.ConnectTimeout after 30 s. A deadline set through signal that is longer than that no longer takes effect.
  • An invalid timeout (negative, NaN, infinite) is a RangeError: connect() rejects, and a Connection throws when it is built.
  • Servers may see a second WebTransport session with the same URL and token, closed straight away.
  • Slow WebSocket opens: with the 30 s default, the slowest WebSocket open an attempt accepts is 20 s (10 s, then a 20 s redial), or 19.5 s after WebTransport's head start. Firefox already fails WebSocket opens at 20 s, so in practice this affects Safari, and Chrome on networks that block UDP.
  • Chrome's WebTransport throttle: each failed or abandoned handshake costs a 5-minute entry.
    • A client whose opens regularly take over 500 ms (round trips above about 250 ms) hedges on most connects.
    • In a long outage where every dial goes unanswered, hedging reaches the throttle's cap in about half as many attempts. At the cap, each new handshake is delayed 30 to 90 s while 14 or more entries are live.
    • The WebSocket fallback still recovers.
  • No wire change.

Alternatives

  • Racing whole connect() calls inside Reload: a near-tie wires both sessions into the origins and announces on both, an auth rejection gets lost in the AggregateError, and the WebSocket gains nothing because of the per-address limit.
  • A cancelling deadline on WebTransport, fixed or escalating: it either cuts slow handshakes that would have worked or waits on dead ones, and it cancels dials the browser judges better. The WebSocket redial is such a deadline, because the WebSocket can't be hedged.
  • Hedging at 1 s: fewer extra handshakes on slow links, but the WebSocket, already dialing at 500 ms, usually wins first and the session stays on WebSocket.
  • Deriving the hedge delay from a previous session's RTT, which RFC 8305 allows: a first connect has no RTT, and the outage is dominated by detection rather than by the 500 ms.
  • Skipping the hedge for a URL after an attempt in which every dial failed, remembered the way websocketWon is: it would halve the throttle cost in a long outage, but slow recovery when both of an attempt's dials happen to land on dead targets.

Follow-ups

  • accept() has no SETUP deadline on the server side either. That's a different problem (what a silent peer can make a server hold); happy to open a quest.
  • moq-tokio has the 30 s timeout but no hedge or redial. Native clients have no per-address WebSocket limit, so a parallel redial would work there; a quest if you want it.

Decisions for you, with my recommendation

  • The 30 s default: keep it, to match Rust.
  • The 500 ms hedge and the 10 s redial as constants: keep them until something needs to tune them. If something does, they'd go on ConnectProps next to timeout.
  • The throttle cost: accept it as documented. The per-URL memory is the option if recovery with WebSocket disabled matters.

(Written by Claude Opus 5.5)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator

Thanks for this, Matt. The approach holds up, and it is continuing in moq-dev/moq#6608.

This branch lives on an organization-owned fork, and GitHub does not let maintainers push to those even with "Allow edits by maintainers" on. So #6608 carries your commit unchanged (your authorship kept, no rebase), plus a merge of main and one test fix: main's @moq/qmux (moq-dev/moq#6528) now refuses an empty negotiated subprotocol, so the new WebSocketStream stubs negotiate qmux-01. Your three decisions are settled there as you recommended.

This PR stays open for now.

(Written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator

Superseded by #6608, which carried this change through review and has merged. Closing as agreed with the maintainer.

(Written by Claude Opus 5.5)

@kixelated kixelated closed this Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants