From b7af477ea35de79e02648cf456ecbf226b3b43ef Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 13:50:45 -0800 Subject: [PATCH 01/11] chore(quest): claim quest/m1/cluster-routing/routes/wire.md Quest-Claim: 04df79eb6704a210 From 6ffb0756633b254960e2870c08d9093a244be783 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 21:52:57 -0800 Subject: [PATCH 02/11] quest: plan per-certificate chain rules for moq auth serve Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 + quest/m1/auth-cert-rules.md | 66 +++++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+) create mode 100644 quest/m1/auth-cert-rules.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 23dc7549706..b10000bd4d8 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -179,6 +179,7 @@ their end report, so an embedder's drain alert pages on real cuts. - [Poll-first async](/quest/m1/poll-first/README.md) - library crates compile no coroutines by default: async wrappers are generated from `poll_*` and gated behind an `async` feature - [Ordered reads can skip owed holes, and snapshot latest() does](/quest/m1/ordered-owed-holes.md) - moq-net's `Ordered` gains a read that skips past an owed hole to the newest buffered group, while keeping the reader's bounds - [The relay reports native sessions it force-closes at the drain deadline](/quest/m1/relay-drain-cuts.md) - the drain's public outcome and live tally separate sessions cut while open from sessions only finishing their end report, so an embedder's drain alert pages on real cuts +- [moq auth serve scopes each client certificate by its chain](/quest/m1/auth-cert-rules.md) - per-certificate and per-intermediate rules in a reloaded TOML config, keyed by the chain hashes the relay sends - [A Rust relay never replays from group 0 to a lite-03/04 subscriber](/quest/m1/relay-inflight-replay.md) - A Rust relay or publisher serving a lite-03 or lite-04 SUBSCRIBE from a track copy that is still resolving starts at the newest group once the copy resolves - [Rename moq-rtc to moq-webrtc](/quest/m1/webrtc-rename.md) - one published `moq-webrtc` crate holds the gateway, the ICE driver, and the data-channel transport - [Vendor and trim the WebRTC stack](/quest/m1/webrtc-vendor.md) - str0m, dimpl, `is`, and sctp-proto live in moq-webrtc with unused code removed, so moq-rtc can take rind diff --git a/quest/m1/auth-cert-rules.md b/quest/m1/auth-cert-rules.md new file mode 100644 index 00000000000..0c3a8601e68 --- /dev/null +++ b/quest/m1/auth-cert-rules.md @@ -0,0 +1,66 @@ +# [M] moq auth serve scopes each client certificate by its chain + +## Goal + +An operator running `moq auth serve` can give different client certificates +different access. Without a restart, a config edit provisions, narrows, or +revokes one certificate, or every certificate under one intermediate CA. +Today every verified certificate gets the same `--mtls-publish` and +`--mtls-subscribe` rules, so two affiliates holding certificates from one CA +get identical access, and scoping them means writing your own auth endpoint. + +Not in scope: changing how the relay verifies certificates, or the JWT and +anonymous paths. + +## Plan + +Decided 2026-11-05 (maintainer, /quest-import of +[#7921](https://github.com/moq-dev/moq/issues/7921)): + +- ✅ **Plan it.** Not chosen: won't fix and leave per-certificate scope to + custom endpoints. The auth contract already carries enough for a leaf + match (`Peer::fingerprint`, `rs/moq-auth/src/request.rs`), and the + reporter showed a per-certificate endpoint working, including revocation + within one re-check. Only the reference server lacks it. +- ✅ **A TOML config like moq-relay's.** `moq auth serve` takes an optional + config file merged with its CLI flags, the way `moq_relay::Config::load` + does (`rs/moq-relay/src/config.rs`), so every serve setting can live in + the file. Not chosen: a `--mtls-dir` of one file per certificate, or a + side file holding only the per-certificate rules. +- ✅ **Rules key on certificate hashes across the verified chain.** The + relay sends the SHA-256 of every certificate in the verified chain (leaf, + intermediates, root) in the auth request's `tls` facts, as an additive + field beside the existing leaf `fingerprint`. A rule names one of those + hashes, so it can scope a single leaf or everything under one + intermediate. The maintainer's words: "per-intermediate, relay sends the + hash of the entire chain"; this quest reads that as one hash per + certificate, so any link can be matched. When several rules match, the one + naming the certificate closest to the leaf wins. Not chosen: per-leaf + only, `Peer::name` (it folds SAN, CN, and fingerprint into one string), or + the issuer DN (two CAs can share one). +- ✅ **A certificate no rule matches falls back to the `--mtls-*` rules.** + Not chosen: refusing it. +- ✅ **A rule can set everything a grant has**: publish and subscribe + patterns, the cluster-link role, tier, and publish/subscribe rates. +- ✅ **A rule's patterns replace the shared mTLS patterns**, never union, so + a rule can narrow. Fields it omits inherit the server-wide value. Not + chosen: each rule being the whole grant. +- ✅ **Reload on change.** The server watches its config file. It swaps in a + valid edit, and on an invalid one keeps the last valid config and logs an + error, as moq-tokio's reloaded address map does + (`rs/moq-tokio/src/listen.rs`). An edit takes effect at each session's + next re-check (`--revalidate`). Not chosen: re-reading per request, + SIGHUP, or requiring a restart. +- ✅ m1. + +Public API: additive. The CLI flags keep working beside the file, and the +auth request gains one `tls` field. Update the auth +contract docs (`doc/bin/relay/`) and `doc/bin/cli.md`. + +Tests: a leaf rule, an intermediate rule, leaf-over-intermediate +precedence, the fallback, an edit narrowing and a delete revoking at the next +re-check, and an invalid edit keeping the last valid config. + +## Closes + +- [#7921](https://github.com/moq-dev/moq/issues/7921) - `moq auth serve`: scope a client certificate by its identity From 33452f467e095c814a2f1396cf60fa7769cd7ad5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 22:04:45 -0800 Subject: [PATCH 03/11] quest: plan the datagram answer race and clear resolved waits Co-Authored-By: Claude Opus 5.5 --- quest/m1/bench-ci.md | 22 ++++++--- quest/m1/crates-trusted-publisher.md | 2 +- quest/m1/datagram-tracks/README.md | 1 + quest/m1/datagram-tracks/answer-race.md | 63 +++++++++++++++++++++++++ 4 files changed, 80 insertions(+), 8 deletions(-) create mode 100644 quest/m1/datagram-tracks/answer-race.md diff --git a/quest/m1/bench-ci.md b/quest/m1/bench-ci.md index d2d41824766..c9fb20bbb1e 100644 --- a/quest/m1/bench-ci.md +++ b/quest/m1/bench-ci.md @@ -11,16 +11,24 @@ runs on the hosted ARM runners, and a deliberately slowed bench fails the job. - Maintainer: dispatch the slowed-bench Nightly run below. Agents never dispatch Nightly; the 2026-11-04 authorization covered the two A/A runs only. -- moq-dev/moq: once A/A run - [41921320690](https://github.com/moq-dev/moq/actions/runs/41921320690) - finishes, dispatch the second A/A run under the same authorization - (`gh workflow run nightly.yml --ref main -f bench-base=`; as of - 2026-11-04 it is not dispatched yet), then read both runs' `paired.json` (the - `bench-rust*` artifacts). Find the second with `gh run list --repo - moq-dev/moq --workflow nightly.yml --event workflow_dispatch`. +- Maintainer: authorize one more A/A Nightly run for the moq-net + (`bench-rust`) and `bench-rust-relay` groups once + [moq-net's benches run to completion](/quest/m1/bench-test-mode.md) lands. + Neither group produced a `paired.json` in either 2026-11-04 run, because + their benches panic. Check: both groups' `paired.json` exist in that run. ## Plan +A/A results, 2026-11-04 (/quest-unblock): runs +[41921320690](https://github.com/moq-dev/moq/actions/runs/41921320690) and +[41933525520](https://github.com/moq-dev/moq/actions/runs/41933525520) both +produced `paired.json` for the shared, extra, and uring groups. The first was +cancelled by Nightly's concurrency group after its bench artifacts uploaded. +At a 10% threshold over 3 rounds, neither run confirmed a regression, and +every case over 10% came back as noise. Single-pass crossings: shared 18 and +26 of 716, extra 10 and 15 of 460, uring 1 and 1 of 41. |delta| p95 was +3.7-8.3%, p99 10-23%. Tune the threshold for those groups from these. + Decided 2026-11-04 (maintainer, /quest-unblock): ✅ authorize one-time agent dispatch of the two A/A Nightly runs on `main`, with `bench-base` set to each run's own commit. Not chosen: the maintainer dispatching them, or skipping. diff --git a/quest/m1/crates-trusted-publisher.md b/quest/m1/crates-trusted-publisher.md index 0ae39050f41..1681a8309b3 100644 --- a/quest/m1/crates-trusted-publisher.md +++ b/quest/m1/crates-trusted-publisher.md @@ -9,7 +9,7 @@ Delete this quest once every crate has one. ## Waiting -- Maintainer: after the next release publishes kio-macros, moq-c, +- Maintainer: after the next `main` to `release` cut publishes kio-macros, moq-c, moq-congestion, moq-crypto, moq-ice, moq-quic, and moq-rtsp by token, add their trusted publishers the same way. Settings pages need a crates.io login, so check with a `release-rs.yml` run that publishes through OIDC. diff --git a/quest/m1/datagram-tracks/README.md b/quest/m1/datagram-tracks/README.md index 2de1a7452ae..743e3b8b81f 100644 --- a/quest/m1/datagram-tracks/README.md +++ b/quest/m1/datagram-tracks/README.md @@ -122,6 +122,7 @@ datagram track on lite-07, direct and through a relay. - [JS datagram tracks](/quest/m1/datagram-tracks/js.md) - `@moq/net` datagram types and the lite-07 datagram request - [moq-transport datagram tracks](/quest/m1/datagram-tracks/ietf.md) - Rust and JS map a datagram track to SUBSCRIBE and OBJECT_DATAGRAM with object ID 0, keeping the requested kind +- [A datagram sent right behind its answer reaches the subscriber](/quest/m1/datagram-tracks/answer-race.md) - receivers decode answers before routing datagrams, and count the ones they still can't route ## Related diff --git a/quest/m1/datagram-tracks/answer-race.md b/quest/m1/datagram-tracks/answer-race.md new file mode 100644 index 00000000000..0fac692d68e --- /dev/null +++ b/quest/m1/datagram-tracks/answer-race.md @@ -0,0 +1,63 @@ +# [M] A datagram sent right behind its answer reaches the subscriber + +## Goal + +On lite-05, 06, and 07, in Rust and JS, a datagram that arrives in the same +receive pass as its subscription's answer (TRACK_INFO on lite-05/06, +DATAGRAMS_INFO on lite-07) is delivered, not dropped. A datagram the +receiver still cannot route is logged at debug and counted, never dropped +silently. + +Not in scope: a datagram whose answer is lost and retransmitted. That is +ordinary datagram loss, and the receiver keeps no buffer for it. + +## Plan + +Decided 2026-11-05 (maintainer, /quest-import of +[#7882](https://github.com/moq-dev/moq/issues/7882)): + +- ✅ **Decode answers before routing datagrams.** The receiver handles its + pending stream answers before it routes datagrams, so a datagram that + shares a packet with its answer finds its route. Not chosen: buffering raw + datagrams per unanswered subscription and replaying them (state a peer + influences, needing a bound), the publisher holding datagrams until its + answer is acknowledged (about one RTT of datagrams lost per subscription + instead), or accepting the loss with only a log. +- ✅ **One quest for Rust and JS.** The JS part builds on the paths + [JS datagram tracks](/quest/m1/datagram-tracks/js.md) introduces + ([moq-dev/moq#7846](https://github.com/moq-dev/moq/pull/7846)), so it + waits for that quest. +- ✅ m1, in the Datagram tracks epic. + +Verified on `main` at 246916845a, from the issue's report: + +- The issue's second question, the `newest - budget` start rule, no longer + applies. Every version and both languages start datagrams live from the + request (`rs/moq-net/src/model/datagram.rs`, `js/net/src/track.ts`), so + this quest leaves the start rule alone. +- Rust: `route_datagram` returns `Ok(())` when `datagram_target` finds no + route (`rs/moq-net/src/lite/subscriber.rs`). Routes register only once the + answer decodes: `register` after DATAGRAMS_INFO on lite-07, and + `legacy_datagrams` after TRACK_INFO on lite-05/06. The driver drains up to + 32 datagrams before polling the source tasks that decode answers, so even a + datagram sharing its answer's packet is dropped. +- The publisher sends datagrams as soon as the answer is written + (`rs/moq-net/src/lite/publisher.rs`), so the first live datagram is the one + at risk. +- JS: `#routeDatagram` returns silently on a miss + (`js/net/src/lite/subscriber.ts`), and #7846 keeps that. +- No stats count the drop; `datagrams` in `rs/moq-net/src/stats.rs` counts + deliveries only. + +Tests: per version, a datagram written in the same packet as its answer +arrives, and an unroutable one bumps the counter. The integration tests in +`rs/moq-net/tests/integration/datagram.rs` all wait for `consumer.info()` +before appending, so add a case that doesn't. + +## Required + +- [JS datagram tracks](/quest/m1/datagram-tracks/js.md) - the JS paths this fix builds on + +## Closes + +- [#7882](https://github.com/moq-dev/moq/issues/7882) - the datagram served at subscribe time is dropped by the receiver when TRACK and SUBSCRIBE are pipelined From 17484e230868f2e6524f6bd18befae5f6408be62 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 22:23:21 -0800 Subject: [PATCH 04/11] feat(net)!: specify the lite-07 route wire and decode its messages The draft specifies lite-07's route layer on the announce stream: ROUTE_START, ROUTE_UPDATE, ROUTE_END, and ANNOUNCE_FLOOR toward the subscriber, ROUTE_REQUEST and the ROUTE_END echo back, path-less claims naming a Route ID, and a Node ID on every request. Rust encodes and decodes the new messages; the subscriber refuses them until session integration drives the router and switches the live ANNOUNCE layout. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 585 +++++++++++++-------- quest/m1/announce-takeover.md | 5 +- quest/m1/api-audit/route/handle.md | 2 +- quest/m1/cluster-routing/routes/README.md | 6 +- quest/m1/cluster-routing/routes/js.md | 1 - quest/m1/cluster-routing/routes/proof.md | 1 - quest/m1/cluster-routing/routes/session.md | 15 +- quest/m1/cluster-routing/routes/wire.md | 91 ---- rs/moq-net/src/fuzz.rs | 3 +- rs/moq-net/src/lite/announce.rs | 464 +++++++++++++++- rs/moq-net/src/lite/subscriber.rs | 6 + rs/moq-net/src/lite/version.rs | 10 + 12 files changed, 872 insertions(+), 317 deletions(-) delete mode 100644 quest/m1/cluster-routing/routes/wire.md diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 029f98648a6..47e48a2ad45 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -35,6 +35,7 @@ normative: informative: I-D.lcurley-moq-cluster: + RFC8966: moqflate: title: "DEFLATE Compressed Tracks for MoQ" target: https://datatracker.ietf.org/doc/draft-lcurley-moq-flate/ @@ -131,16 +132,16 @@ A Broadcast is a collection of Tracks named by a path. This corresponds to a MoqTransport's "track namespace". A path is a sequence of non-empty segments joined by `/` (0x2F), so it never starts or ends with `/` or contains two in a row; the empty path has no segments and prefixes every path. A `\` (0x5C) escapes the `\` or `/` that follows it, as a MoqTransport namespace field holding either is spelled; a path with any other `\` is a PROTOCOL_VIOLATION. -A continuous stream is named by an Instance, carried on its routes (see [ANNOUNCE_START](#announce-start)): a path and an Instance name one Broadcast, and a publisher MUST NOT serve different content under the same pair. -Within one Instance, group sequences never restart and never name two different groups, so routes sharing it serve the same bytes; two Instances are independent streams, never stitched together. +A continuous stream is named by an Instance, carried on its claims (see [ANNOUNCE_START](#announce-start)): a path and an Instance name one Broadcast, and a publisher MUST NOT serve different content under the same pair. +Within one Instance, group sequences never restart and never name two different groups, so claims sharing it serve the same bytes; two Instances are independent streams, never stitched together. A publisher that starts its group sequences over, such as a restarted process, uses a new Instance, while replicas of the same content share one. A publisher MAY keep an Instance across restarts only if it continues the sequence space and announces a higher Epoch each time. -The Epoch is a counter within one Instance that picks its live incarnation while a stale route lingers; it is never compared across Instances. +The Epoch is a counter within one Instance that picks its live incarnation while a stale claim lingers; it is never compared across Instances. -A publisher advertises what it can serve via ANNOUNCE_START messages, each carrying a path prefix: a route covering every broadcast path beneath it. -A route is the only shape an advertisement takes: a publisher that serves only some of the paths beneath a prefix, such as a transcoder for any broadcast's derivative, advertises the covering prefix and refuses the requests it will not serve (see [Resolution](#resolution)); no message narrows a route. -The subscriber uses the ANNOUNCE_REQUEST message to discover these routes, then subscribes to specific paths under them; which paths name broadcasts is an application convention. -The common convention is that a publisher announces each broadcast's exact path as its own route, so subscribers can enumerate broadcasts, while a service announces one short prefix and serves whatever is requested beneath it. +A publisher advertises what it can serve via ANNOUNCE_START messages, each carrying a path prefix: a claim covering every broadcast path beneath it. +A claim is the only shape an advertisement takes: a publisher that serves only some of the paths beneath a prefix, such as a transcoder for any broadcast's derivative, advertises the covering prefix and refuses the requests it will not serve (see [Resolution](#resolution)); no message narrows a claim. +The subscriber uses the ANNOUNCE_REQUEST message to discover these claims, then subscribes to specific paths under them; which paths name broadcasts is an application convention. +The common convention is that a publisher announces each broadcast's exact path as its own claim, so subscribers can enumerate broadcasts, while a service announces one short prefix and serves whatever is requested beneath it. Announcements are live and can change over time, allowing for dynamic origin discovery. A broadcast consists of any number of Tracks. @@ -389,113 +390,154 @@ There's a 1-byte STREAM_TYPE at the beginning of each stream. | ------- | ------------- | ----------- | ### Announce -A subscriber can open an Announce Stream to discover routes matching a prefix. -A route is an advertisement that paths under its prefix can be served; it claims capability, not inventory, so it never asserts that any specific broadcast exists. +A subscriber can open an Announce Stream to discover claims matching a prefix. +A claim is an announcement that paths under its prefix can be served by one origin node; it claims capability, not inventory, so it never asserts that any specific broadcast exists. +A route is how the publisher reaches an origin node, and every claim on the stream travels under the route to its origin (see [Routing](#routing)). The subscriber creates the stream with an ANNOUNCE_REQUEST message. -The publisher replies with a single ANNOUNCE_OK message followed by announcements for any matching routes and any future changes: - -- ANNOUNCE_START: a matching route over a path prefix is available. -- ANNOUNCE_END: a previously started route is no longer advertised. -- ANNOUNCE_UPDATE: a previously started advertisement was atomically updated (new hops or cost). -- ANNOUNCE_RESTART: a previously started advertisement now serves another Instance, or a higher Epoch of the same one. -- ANNOUNCE_TAKEOVER: a matching route is available and continues what a previously started advertisement served beneath it. +The publisher replies with a single ANNOUNCE_OK message followed by the matching routes and claims and any future changes: + +- ROUTE_START: the publisher reaches an origin node. +- ROUTE_UPDATE: a route's seqno, metric, or down-only bit changed, or the route is held. +- ROUTE_END: the publisher let go of a route and every claim under it. +- ANNOUNCE_START: a matching claim of a route's origin node. +- ANNOUNCE_END: the origin withdrew a claim. +- ANNOUNCE_UPDATE: a claim's cost changed. +- ANNOUNCE_RESTART: a claim now serves another Instance, or a higher Epoch of the same one. +- ANNOUNCE_TAKEOVER: a matching claim continues what a live claim served beneath it. +- ANNOUNCE_FLOOR: a batch of a route's claims is complete. - SYNCED: the initial set is complete. -ANNOUNCE_OK carries metadata that applies to every announcement on the stream: the publisher's own `Hop ID` (the implicit trailing entry of every announcement's path) (see [ANNOUNCE_OK](#announce-ok)). +In the other direction, the subscriber sends ROUTE_REQUEST and echoes every ROUTE_END (see [Routing](#routing)). + The publisher MUST send [SYNCED](#synced) exactly once, after the initial set. -The subscriber MAY batch announcements until SYNCED; subsequent announcements are live updates. +The subscriber MAY batch messages until SYNCED; subsequent ones are live updates. The initial set is relative to this hop: a relay answers with what it holds without waiting for upstreams. -The publisher MAY coalesce changes to a route it has not yet sent, and sends no more events before SYNCED than the initial set holds routes. -A retraction of a route already sent MAY precede SYNCED. +The publisher MAY coalesce changes to a claim it has not yet sent, and sends no more claims before SYNCED than the initial set holds. +A retraction of a claim or route already sent MAY precede SYNCED. -Each ANNOUNCE_START or ANNOUNCE_TAKEOVER implicitly assigns the next Announce ID on the stream: a counter starting at 0 that increments by 1 per advertisement. -The id never appears on the wire; both endpoints derive it from the message order on the (reliable, ordered) stream. -ANNOUNCE_END, ANNOUNCE_UPDATE, ANNOUNCE_RESTART, and ANNOUNCE_TAKEOVER reference an Announce ID instead of repeating the route's prefix. +Each ROUTE_START implicitly assigns the next Route ID on the stream, and each ANNOUNCE_START or ANNOUNCE_TAKEOVER the next Announce ID: two counters starting at 0 that increment by 1 per assignment. +Neither id appears on the wire where it is assigned; both endpoints derive it from the message order on the (reliable, ordered) stream, and other messages reference it instead of repeating the origin node or the claim's prefix. +Ids are never reused within a stream. +A ROUTE_END retires its Route ID and the Announce ID of every claim under the route; an ANNOUNCE_END retires one Announce ID. -Each route has at most one current advertisement per stream. -A second ANNOUNCE_START for an already-advertised route is a protocol violation; an ANNOUNCE_UPDATE or ANNOUNCE_RESTART atomically replaces the current advertisement while keeping its id live, and an ANNOUNCE_TAKEOVER at its predecessor's own prefix replaces it under a new id. +A stream has at most one live route per origin node, and each route at most one live claim per prefix. +Another origin claiming the same prefix is another claim, under its own route. +An ANNOUNCE_UPDATE or ANNOUNCE_RESTART atomically replaces a claim while keeping its id live, and an ANNOUNCE_TAKEOVER at its predecessor's own prefix replaces it under a new id; a claim never moves to another route. -The subscriber MUST close the session with a PROTOCOL_VIOLATION if it receives an ANNOUNCE_END, ANNOUNCE_UPDATE, ANNOUNCE_RESTART, or ANNOUNCE_TAKEOVER referencing an Announce ID that was never assigned or already retired, an ANNOUNCE_START or ANNOUNCE_TAKEOVER for a route that is already advertised other than as its predecessor, or any announcement before ANNOUNCE_OK. -When the stream is closed, the subscriber MUST assume that all routes are now unavailable. +The subscriber MUST close the session with a PROTOCOL_VIOLATION if it receives a message referencing a Route ID or Announce ID that was never assigned or already retired, a ROUTE_START for an origin node that already has a live route on the stream, an ANNOUNCE_START or ANNOUNCE_TAKEOVER for a prefix its route already claims other than as its predecessor, or any message before ANNOUNCE_OK. +When the stream is closed, the subscriber MUST assume that every route and claim on it is gone. A broadcast path has at most 32 segments and at most 4,096 bytes; a receiver MUST close the session with a PROTOCOL_VIOLATION on a longer one, including one resolved by [Compression](#announce-compression). -A route covers a path when its prefix is a leading run of the path's segments; matching is per path segment, so a prefix never matches half a segment, and equality is byte-by-byte within each segment. -A publisher answering a request stream presents each of its routes clamped to the intersection with the requested prefix: a route above the request's prefix appears as the request prefix itself (an empty suffix), which is exactly the covered set the subscriber may see. -There MAY be multiple Announce Streams, potentially containing overlapping prefixes, that get their own ANNOUNCE_OK + announcements. +A claim covers a path when its prefix is a leading run of the path's segments; matching is per path segment, so a prefix never matches half a segment, and equality is byte-by-byte within each segment. +A publisher answering a request stream presents each of its claims clamped to the intersection with the requested prefix: a claim above the request's prefix appears as the request prefix itself (an empty suffix), which is exactly the covered set the subscriber may see. +There MAY be multiple Announce Streams, potentially containing overlapping prefixes, that get their own ANNOUNCE_OK and messages. + +#### Floors {#floors} +Each origin node numbers the changes to its own claims with a Revision, starting at 0 for each Node ID. +A stream carries a route's claims in batches: the ANNOUNCE messages under the route since its last ANNOUNCE_FLOOR, closed by the next one. +The publisher sends a ROUTE_START before any claim under it, and SYNCED only once every route it sent has a floor. + +The floor lists every region the publisher certifies for the route: a suffix of the requested prefix, and the Revision as of which the stream's claims for the route within that region are complete. +Each list replaces the previous one, so a region left out is no longer certified. +A floor certifies the claims overlapping its region: those inside it, and those above it. +A publisher that certifies anything for a route sends a floor for it on every stream, one with no regions when the request meets none of its regions. + +The subscriber applies a batch whole, once its floor arrives. +A receiver that hears one origin's claims on several streams merges them claim by claim: whichever stream's floor certifying the claim carries the higher Revision decides whether it exists. +An older floor never revives a claim whose withdrawal a newer one delivered. +While a receiver has a usable route to the origin and every peer whose last finite route to it is feasible here has answered the current request, it lets go of a region none of those peers certifies: its claims go offline, which is not an ANNOUNCE_END. #### Compression {#announce-compression} -An ANNOUNCE_START, ANNOUNCE_UPDATE, ANNOUNCE_RESTART, or ANNOUNCE_TAKEOVER MAY copy the head of a path or the tail of a Hop ID list from a live advertisement on the same stream, its base. -A `Base` field names it by distance: the base's Announce ID is the next unassigned Announce ID minus `Base`, so 1 names the latest ANNOUNCE_START or ANNOUNCE_TAKEOVER, and 0 names no base. -The receiver resolves every base against the advertisements live when the message arrives, and the result is identical to the literal encoding: a base ending later changes nothing, including a predecessor that the ANNOUNCE_TAKEOVER naming it as a base retires. -A publisher MAY always send a `Base` of 0. +An ANNOUNCE_START or ANNOUNCE_TAKEOVER MAY copy the head of its path from a live claim on the same stream, its base. +`Path Base` names it by distance: the base's Announce ID is the next unassigned Announce ID minus `Path Base`, so 1 names the latest ANNOUNCE_START or ANNOUNCE_TAKEOVER, and 0 names no base. +The receiver resolves the base against the claims live when the message arrives, and the result is identical to the literal encoding: a base ending later changes nothing, including a predecessor that the ANNOUNCE_TAKEOVER naming it as a base retires. +The base need not share the new claim's route. +A publisher MAY always send a `Path Base` of 0. -The subscriber MUST close the session with a PROTOCOL_VIOLATION if a non-zero `Base` names an Announce ID that was never assigned or already retired, a `Keep` is non-zero with a `Base` of 0 or exceeds what the base has, or the resolved path or Hop ID list breaks its own rules. +The subscriber MUST close the session with a PROTOCOL_VIOLATION if a non-zero `Path Base` names an Announce ID that was never assigned or already retired, a `Path Keep` is non-zero with a `Path Base` of 0 or exceeds the base's segments, or the resolved path breaks its own rules. #### Hidden Paths {#hidden} -A route is hidden from a request when a segment of its path below the requested prefix starts with `.` (0x2E). -A segment inside the prefix never hides anything, so a request that names the hidden segment itself (`.stats`) lists what is under it, and a route at or above the prefix has no segment below it. +A claim is hidden from a request when a segment of its path below the requested prefix starts with `.` (0x2E). +A segment inside the prefix never hides anything, so a request that names the hidden segment itself (`.stats`) lists what is under it, and a claim at or above the prefix has no segment below it. Only the first byte counts: `catalog.v2` is not hidden. -A publisher SHOULD NOT announce a hidden route unless the ANNOUNCE_REQUEST set `Hidden`. +A publisher SHOULD NOT announce a hidden claim unless the ANNOUNCE_REQUEST set `Hidden`. Hiding is a convenience for discovery, not access control: a publisher MAY treat a subscriber it trusts, such as another relay in its own cluster, as opted in, and MUST authorize hidden paths like any other. SUBSCRIBE, FETCH, and TRACK resolve a hidden path exactly as any other. #### Routing {#routing} -Each advertisement carries the path of Hop IDs it traversed and an accumulated Route Cost (see [ANNOUNCE_START](#announce-start)), which relays use to build a loop-free mesh. - -A receiver MUST discard an announcement whose reconstructed path contains its own Hop ID: it has looped back, so forwarding it would extend the loop and subscribing through it would route the receiver back to itself. -This is the only loop defense moq-lite requires, and it catches loops of any length. -A conforming sender never sends one (see below), so a receiver MAY instead close the session with a protocol violation; discarding is what keeps a mesh working when one member does not conform. -A Hop ID of 0 means unknown and never matches anything; withholding an ID trades loop detection for privacy. -A Hop ID spans 64 bits. -A receiver MAY assign an identity of its own to a peer that declared 0, as local selection state for filtering that session; it MUST NOT forward that identity as a Hop ID. - -A publisher MUST NOT advertise a path whose entries contain the Hop ID the subscriber declared in its SETUP (see [Hop Parameter](#hop-parameter)). -The receiver can only discard it, and acting on it would form a loop, so sending one is never useful. -Of the paths that remain a publisher SHOULD advertise the best, and nothing when every known path contains that Hop ID. -Selection is per subscriber, so a subscriber that the serving path flows through still receives the best standby path, which is what lets it fail over if its own copy dies. -The per-subscriber winner changing travels as an ANNOUNCE_UPDATE when the new winner serves the same Instance at the same Epoch, and as an ANNOUNCE_RESTART when it serves another Instance or a higher Epoch of the same one; the last qualifying path appearing or disappearing travels as an ANNOUNCE_START or ANNOUNCE_END. -A route continuing what the subscriber was sent for its predecessor travels as an ANNOUNCE_TAKEOVER (see [ANNOUNCE_TAKEOVER](#announce-takeover)); where continuity does not hold, it travels as an ANNOUNCE_START, or an ANNOUNCE_RESTART at the predecessor's own prefix. - -When serving a subscription, a publisher MUST select the source by that same exclusion; if only excluded sources remain, the subscription is unroutable. -Applying one rule to both advertisement and dispatch keeps advertised paths truthful, which is what prevents subscription cycles of any length. - -When resolving a path covered by several routes (across any number of streams), the subscriber SHOULD prefer the most specific covering route (see [Resolution](#resolution)). -Among those, a route is ineligible while another route of the same prefix carries its Instance at a higher Epoch, however cheap it is. -Among the eligible routes, the subscriber SHOULD prefer a path that contains no 0 Hop ID over one that does, then the lowest Route Cost after adding each arriving link's cost (see [Cost Parameter](#cost-parameter)), then a broadcast the receiver publishes itself, breaking ties toward the shortest path, then toward the lowest Instance Hash, then, among routes of one Instance, toward the lowest Chain Hash, and then toward the most recently received, so a reconnecting publisher is not outranked by the stale session it replaced. - -Both hashes are the 64-bit FNV-1a hash, with offset basis `0x420C0DECB00B` and the standard FNV-64 prime, of the requested path's UTF-8 bytes followed by more bytes: for the Instance Hash, the route's Instance; for the Chain Hash, each Hop ID of the route's path, oldest first, as 8 little-endian bytes. -They are keyed on the requested path rather than the route's prefix, so equal-cost advertisers of one prefix, each under its own Instance, share its paths instead of the first one taking them all. -The Instance reads the same on every relay, so one path resolves to the same stream on every relay that holds the same routes. -When choosing which route to advertise for a prefix, the requested path is the prefix itself. -A relay skips both hashes for a route a publisher announced to it directly at the requested path, which no other relay ranks, so a restarted publisher under a new Instance wins at once over its lingering session. - -Routes with the same Instance serve the same Broadcast, so a relay MAY move a live subscription between them, including onto a higher Epoch, continuing from the first frame the subscriber lacks, and a route change (a reconnect, a cheaper path, a draining session) is invisible to the subscriber. -A relay MUST NOT move a subscription to a route with a different Instance: it stays on its Instance's routes and ends with the last of them. -When another Broadcast wins the path, subscriptions already open stay on theirs until the subscriber drops them or the route goes; new requests resolve the winner, and the ANNOUNCE_RESTART tells subscribers to request again. -A receiver of an ANNOUNCE_RESTART (or of the ANNOUNCE_END and ANNOUNCE_START standing in for one) MUST NOT serve a new request under its prefix from anything resolved through the replaced advertisement. -An ANNOUNCE_TAKEOVER is the one exception to staying on an Instance: what was resolved through its predecessor beneath its prefix is the Broadcast the new route serves, so those subscriptions continue on the new route and new requests MAY join them, whatever either route's Instance. -A version before this one carries no Instance, so a receiver of such a session mints one for each ANNOUNCE_START it receives, with Epoch 0, and forwards it as the route's Instance. +Routing follows the Babel routing protocol [RFC8966], keyed by origin node: each node selects one route per origin node and advertises it to its neighbours. + +An origin node names itself with a Node ID: a non-zero 64-bit value, fresh for each process restart and shared by every session of that process. +An origin MUST NOT declare a Node ID with the most significant bit set. +That bit marks a qualified Node ID: a relay files every Node ID a peer with a scoped publish grant sends (see [AUTH_OK](#auth-ok)), the peer's own included, under a qualified Node ID of its own derivation, collision-resistant at 64 bits, and translates it back when it forwards a request on that session. +A relay MUST ignore a qualified Node ID received from such a peer, and claims under its qualified Node IDs outside that peer's grant. +A Node ID is otherwise declared, not proven. + +An origin advertises its own route with a Metric of 0 and a Seqno that only it advances. +A receiver adds the session's link cost (see [Cost Parameter](#cost-parameter)) to the Metric of every route the peer sends; the sum is its metric through that peer. +Each link adds at least 1, so a route's metric strictly increases away from its origin. +A metric that reaches 2^64-1 makes the route unusable. + +A node keeps a feasibility distance per Node ID: the Seqno and metric of the last finite route it advertised. +A route is feasible when its Seqno is greater than the distance's, or equal with a lower metric. +A node selects, per Node ID, among its feasible routes that are not held: a session that has not received a GOAWAY, then the highest local priority, then the lowest metric. +It MUST NOT select an infeasible route, and never raises a distance to accept one. +A node MAY mark a session upstream. +A route is down-only when it arrived on an upstream session or with its Down Only bit set; a node sends a down-only route with the bit set, and never on an upstream session. + +A node sends its selected route to every origin node it knows on each Announce Stream it answers, and MAY omit routes with no claim matching the stream's request. +A node that does not forward, such as a client, sends only its own route, and holds no feasibility distance for any other. +An origin advances its Seqno whenever its own set of sessions changes. +It never advertises a Seqno or Revision of 2^64-1: a change that would reach it instead rotates its Node ID, ending the old route everywhere and announcing its claims again under the new one. + +A node with no usable route to an origin node holds it while it can still recover: while every route it has to the node is infeasible, or a peer holds a route whose last finite advertisement is feasible here. +A held route travels as a ROUTE_UPDATE with an infinite Metric (2^64-1), and its claims stay shown but unavailable. +Otherwise the node lets go, sending ROUTE_END wherever it advertised the route, and its claims go offline. +Neither proves that the origin withdrew its claims. + +A node whose routes to an origin node are all infeasible sends a ROUTE_REQUEST with its distance's Seqno plus 1 to the peer whose route it would select were it feasible. +A node holding a selected route at least that new answers by advertising it, and the origin answers by raising its Seqno to at least the requested one; a request for 2^64-1 rotates the origin's Node ID. +Any other node forwards the request to the peer whose route it would select, even the requester. +Every node keeps one pending request per origin node, its own or forwarded, dropping duplicates, until it selects a route at least that new or has no route left, and sends it again whenever that peer changes. + +The receiver of a ROUTE_END echoes it back on the same stream once it dropped the route. +A node drops a feasibility distance once it neither selects nor advertises the route and every ROUTE_END it sent for it was echoed; a closed session counts as an echo. + +A SUBSCRIBE, FETCH, TRACK, TRACKS, or DATAGRAMS request names the origin node that serves it. +A subscriber that names 0 lets the publisher select one (see [Resolution](#resolution)), and a relay forwarding a request names the origin node it selected or was given. +A relay forwards a request naming an origin node only on its selected route to that node, and resets it with UNROUTABLE when it has no usable route to it, so the request never reaches another origin. + +Claims with the same Instance serve the same Broadcast, so a relay MAY move a live subscription between them, including onto a higher Epoch, continuing from the first frame the subscriber lacks, and a route change (a reconnect, a cheaper path, a draining session) is invisible to the subscriber. +A relay MUST NOT move a subscription to a claim with a different Instance: it stays on its Instance's claims and ends with the last of them. +When another Broadcast wins the path, subscriptions already open stay on theirs until the subscriber drops them or the claim goes; new requests resolve the winner, and the ANNOUNCE_RESTART tells subscribers to request again. +A receiver of an ANNOUNCE_RESTART (or of the ANNOUNCE_END and ANNOUNCE_START standing in for one) MUST NOT serve a new request under its prefix from anything resolved through the replaced claim. +An ANNOUNCE_TAKEOVER is the one exception to staying on an Instance: what was resolved through its predecessor beneath its prefix is the Broadcast the new claim serves, so those subscriptions continue on the new claim and new requests MAY join them, whatever either claim's Instance. +A version before this one carries no Instance, so a receiver of such a session mints one for each ANNOUNCE_START it receives, with Epoch 0, and forwards it as the claim's Instance. A minted Instance SHOULD start with `~` (0x7E), so a reader can tell it names an announcement rather than the publisher's stream; a publisher SHOULD NOT name its own Instance with that prefix. A minted Instance is new for every announcement, so a publisher reconnecting or re-announcing through such a hop is another stream. #### Resolution {#resolution} -A SUBSCRIBE, FETCH, or TRACK request names a path, and the receiver resolves it against the routes covering that path, after the per-subscriber exclusion above. +A SUBSCRIBE, FETCH, or TRACK request naming no origin node names a path, and the receiver resolves it against the claims covering that path whose route it selected and is usable. -Only the most specific covering routes are consulted: those with the longest prefix. -A route strictly inside another's paths always ranks above it, and a broadcast's exact path is the most specific route there is, so a concrete announcement shadows every broader route covering it at any cost; routes over the same prefix form one tier. -The winning tier is the whole answer: a refusal from it never falls through to a less specific route, so a service refusing a path does not leak the request to a catch-all, and one unserved path costs one round trip rather than a walk down the candidates. -Nor is a refusal retried at another route of the same prefix. +Only the most specific covering claims are consulted: those with the longest prefix. +A claim strictly inside another's paths always ranks above it, and a broadcast's exact path is the most specific claim there is, so a concrete announcement shadows every broader claim covering it at any cost; claims over the same prefix form one tier. +The winning tier is the whole answer: a refusal from it never falls through to a less specific claim, so a service refusing a path does not leak the request to a catch-all, and one unserved path costs one round trip rather than a walk down the candidates. +Nor is a refusal retried at another claim of the same prefix. -Within the tier, cost and the tie-breaks of [Routing](#routing) order the routes. +Within the tier, a claim is ineligible while another claim of the same prefix carries its Instance at a higher Epoch, however cheap it is. +Among the eligible claims, the receiver SHOULD prefer the highest local priority of the route's session, then the lowest cost, the claim's Announce Cost plus its route's metric, then the lowest Rendezvous Hash, and then the most recently received, so a reconnecting publisher is not outranked by the stale session it replaced. +The Rendezvous Hash is the 64-bit FNV-1a hash, with offset basis `0x420C0DECB00B` and the standard FNV-64 prime, of the requested path's UTF-8 bytes followed by the origin's Node ID as 8 little-endian bytes. +Every node that knows the same claims ranks them the same, so equal-cost origins of one prefix share its paths and every relay sends one path to the same origin. +When choosing which claim to advertise for a prefix, the requested path is the prefix itself. -A route resolved this way serves the request like any other, and the route is not consumed by it. -A relay MUST NOT announce a path merely because it resolved it: the covering route stays the only advertisement until the advertiser announces the concrete path, which it SHOULD do once it is producing, so a later request finds the running broadcast by its exact path instead of resolving a second producer. +A claim resolved this way serves the request like any other, and the claim is not consumed by it. +A relay MUST NOT announce a path merely because it resolved it: the covering claim stays the only advertisement until the advertiser announces the concrete path, which it SHOULD do once it is producing, so a later request finds the running broadcast by its exact path instead of resolving a second producer. An advertiser that will not serve a resolved request resets the request stream with a typed code (see [Error Codes](#error-codes)). Every refusal is terminal and propagates without re-resolution, so probing unserved paths costs one round trip per path. -An advertiser signals capacity through its route alone, withdrawing or re-pricing it before it runs out. +An advertiser signals capacity through its claim alone, withdrawing or re-pricing it before it runs out. ### Subscribe A subscriber opens Subscribe Streams to request a Track. @@ -508,7 +550,7 @@ A Track that finished before producing any Group is the one case with nothing to A requested start past the publisher's largest position is the exception: the publisher MUST send SUBSCRIBE_OK without waiting, so a subscriber resuming just past what it holds learns where the live feed is from the answer alone (a quiet track may not reach the start for a long time). A relay knows that position only from its upstream's answer, never from its cache: a group it cached earlier says nothing about where the live feed is now. A rejection is a stream reset: a publisher that cannot serve the subscription (no such track, an ended broadcast, or any other refusal) MUST promptly reset the stream rather than leave it pending, so a subscriber distinguishes "pending" from "refused" by the reset, not by a timeout. -A route claims capability rather than inventory, so a subscription for a covered path that names nothing is refused this way too. +A claim asserts capability rather than inventory, so a subscription for a covered path that names nothing is refused this way too. The track's immutable publisher properties are not carried here; they are fetched once via a [Track Stream](#track-stream). The subscriber needs the track's TRACK_INFO (notably its timescale) to interpret FRAME messages, and MAY open the Track and Subscribe streams concurrently, buffering frames until it arrives. @@ -566,7 +608,7 @@ The publisher replies with listing events, then FINs the stream once the Broadca A Broadcast lists the Tracks its publisher produces, except datagram Tracks, which are never listed: TRACK_INFO does not say which kind a Track is, so a subscriber could not tell whether to SUBSCRIBE or send DATAGRAMS. A Track the publisher serves only on request, such as any name a transcoder answers for, is never listed, and listing never creates one. A removal, either TRACKS_REMOVE or TRACKS_FINISH, promises only that the Track is no longer listed; a SUBSCRIBE for its name MAY still succeed. -TRACKS_FINISH says the Track's publisher declared its end, so its Groups are all the Track will hold; TRACKS_REMOVE says it left unfinished, such as a publisher that went away or a route that was withdrawn, so it may go on elsewhere. +TRACKS_FINISH says the Track's publisher declared its end, so its Groups are all the Track will hold; TRACKS_REMOVE says it left unfinished, such as a publisher that went away or a claim that was withdrawn, so it may go on elsewhere. A relay forwards each removal as its upstream reported it, never as its own cache would suggest. A Track listed again under a removed name is the same Track: within one Broadcast a name always means the same content. @@ -580,7 +622,7 @@ A removal of a Track already sent MAY precede SYNCED. The subscriber MUST close the session with a PROTOCOL_VIOLATION on a TRACKS_ADD for a name already listed on the stream, a TRACKS_REMOVE or TRACKS_FINISH for a name not listed, a second SYNCED, or an unknown event type. A FIN before SYNCED means the Broadcast ended. -A publisher MUST reset a Tracks Stream with NOT_FOUND when the route it would serve the path from covers it by a shorter prefix: a route claims capability, not inventory, so it names no Broadcast to list. +A publisher MUST reset a Tracks Stream with NOT_FOUND when the claim it would serve the path from covers it by a shorter prefix: a claim asserts capability, not inventory, so it names no Broadcast to list. A subscriber MAY skip the Track Stream for a Track it learned from TRACKS_ADD, since the properties are the same immutable ones. ### Datagrams {#datagrams-stream} @@ -913,8 +955,6 @@ The following Setup Parameters are defined: |------|----------------|-------------| | 0x4 | Cost | Cost (i) | |------|----------------|-------------| -| 0x5 | Hop | Hop ID (i) | -|------|----------------|-------------| | 0x6 | Implementation | Tokens (s) | |------|----------------|-------------| | 0x7 | Token | Token (b) | @@ -959,23 +999,15 @@ The role is a hint that only ever narrows the session: a server MUST still enfor Only the client sends it; a client that receives one MUST close the session with a PROTOCOL_VIOLATION. A relay MUST NOT forward it. ### Cost Parameter {#cost-parameter} -The Cost Parameter declares what subscribing from this endpoint costs: a receiver adds the value the sender declared to the Route Cost of every announcement that sender forwards (see [Routing](#routing)). +The Cost Parameter declares what reaching this endpoint costs: a receiver adds the value the sender declared to the Metric of every route that sender sends (see [Routing](#routing)). -The Parameter Value is a variable-length integer in deployment-chosen units, the same units as the Route Cost. -An absent parameter means the default cost of 1, under which the accumulated Route Cost equals the hop count and routing degenerates to shortest-path. -A value of 0 is meaningful and distinct from absent: it makes that direction free, e.g. between two relays in the same datacenter. +The Parameter Value is a variable-length integer in deployment-chosen units, the same units as the Metric and the Announce Cost. +An absent parameter means the default cost of 1, under which a route's metric is its hop count. +A value of 0 is a PROTOCOL_VIOLATION: a free link would leave a route as close as the one it came from, which the feasibility condition relies on never happening. Both endpoints send it and the two values need not match: the parameter prices the sender's own egress. A relay MUST NOT forward it. -A declared cost is an assertion, not an instruction: a receiver MAY charge a locally configured value instead, so a peer cannot reprice its neighbours by declaring itself cheap. - -### Hop Parameter {#hop-parameter} -The Hop Parameter declares the sender's Hop ID: the identity that names it in the paths of announcements it forwards. -The Parameter Value is a variable-length integer; a value of 0 carries no identity and is equivalent to omitting the parameter. - -Declaring it at setup gives the receiver the peer's identity before any other stream arrives, so route selection applies the same exclusion to the peer's subscriptions as to its announcements (see [Routing](#routing)), even on a session that never opens an Announce Stream. -Either endpoint MAY send it; a subscriber-only endpoint with no identity MAY omit it, but a publisher SHOULD have a Hop ID regardless (see [ANNOUNCE_OK](#announce-ok)). -A relay MUST NOT forward it. +A declared cost is an assertion, not an instruction: a receiver MAY charge a locally configured value of at least 1 instead, so a peer cannot reprice its neighbours by declaring itself cheap. ### Implementation Parameter {#implementation-parameter} The Implementation Parameter names the sender's software, so a peer can attribute a session to its build. @@ -996,7 +1028,7 @@ Only the client sends it; a client that receives one MUST close the session with ## ANNOUNCE_REQUEST {#announce-request} -A subscriber sends an ANNOUNCE_REQUEST message to indicate it wants to receive announcements for any broadcasts with a path that starts with the requested prefix. +A subscriber sends an ANNOUNCE_REQUEST message to indicate it wants to receive the claims, and the routes they travel under, for any broadcasts with a path that starts with the requested prefix. ~~~ ANNOUNCE_REQUEST Message { @@ -1010,31 +1042,23 @@ ANNOUNCE_REQUEST Message { Indicate interest for any broadcasts with a path that starts with this prefix. **Hidden**: -1 to also receive hidden routes (see [Hidden Paths](#hidden)), 0 otherwise. +1 to also receive hidden claims (see [Hidden Paths](#hidden)), 0 otherwise. Any other value is a PROTOCOL_VIOLATION. -The publisher MUST respond with an ANNOUNCE_OK message followed by ANNOUNCE_START messages for any matching routes, followed by ANNOUNCE_START, ANNOUNCE_END, ANNOUNCE_UPDATE, ANNOUNCE_RESTART, and ANNOUNCE_TAKEOVER messages for any future updates, subject to [Routing](#routing). -Implementations SHOULD consider reasonable limits on the number of matching broadcasts to prevent resource exhaustion. +The publisher MUST respond with an ANNOUNCE_OK message followed by the matching routes and claims and any future changes, subject to [Routing](#routing). +The subscriber then sends only [ROUTE_REQUEST](#route-request) and [ROUTE_END](#route-end) echoes on the stream. +Implementations SHOULD consider reasonable limits on the number of matching claims and routes to prevent resource exhaustion. ## ANNOUNCE_OK {#announce-ok} A publisher sends an ANNOUNCE_OK message exactly once, as the first message on the response side of an Announce Stream. -It carries metadata that is constant for the lifetime of the stream and applies to every announcement that follows. ~~~ ANNOUNCE_OK Message { - Message Length (i) - Hop ID (i) + Message Length (i) = 0 } ~~~ -**Hop ID**: -The publisher's own Hop ID. -This is treated as the implicit trailing entry of every ANNOUNCE_START, ANNOUNCE_UPDATE, ANNOUNCE_RESTART, and ANNOUNCE_TAKEOVER Hop ID list on this stream; those messages MUST NOT repeat this value as the last entry of their `Hop ID` list. -The value 0 is reserved to mean "unknown": either no Hop ID was assigned (e.g. when bridging from an older protocol version) or the endpoint deliberately withholds it to obscure the underlying routing. -A publisher that assigns a Hop ID MUST choose a non-zero value, and SHOULD assign itself one (a fresh random value per session suffices), so downstream receivers can detect loops through it. -Receivers reconstruct the full path as `Hop IDs ++ [ANNOUNCE_OK.Hop ID]`. - ## SYNCED {#synced} The initial set on a discovery stream has been delivered. @@ -1045,16 +1069,128 @@ SYNCED Message { } ~~~ -SYNCED has no body and assigns no Announce ID. +SYNCED has no body and assigns no id. A second SYNCED on the same stream is a PROTOCOL_VIOLATION. An Announce Stream ending before SYNCED is a PROTOCOL_VIOLATION; a Tracks Stream FIN before SYNCED means the Broadcast ended (see [Tracks](#tracks-stream)). +## ROUTE_START {#route-start} +A publisher sends a ROUTE_START message when it reaches an origin node (see [Routing](#routing)). +Each ROUTE_START implicitly assigns the next Route ID on the stream, later referenced by ROUTE_UPDATE, ROUTE_END, ANNOUNCE_FLOOR, ANNOUNCE_START, and ANNOUNCE_TAKEOVER (see [Announce](#announce)). + +~~~ +ROUTE_START Message { + Type (i) = 0x6 + Message Length (i) + Node ID (i), + Seqno (i), + Metric (i), + Down Only (8), +} +~~~ + +**Type**: +Set to 0x6 to indicate a ROUTE_START message. + +**Node ID**: +The origin node, as the publisher names it on this session. +0 is a PROTOCOL_VIOLATION. + +**Seqno**: +The origin's sequence number of the route, which only the origin advances. +2^64-1 is a PROTOCOL_VIOLATION. + +**Metric**: +The publisher's metric to the origin node, 0 when it is the origin. +2^64-1 is a PROTOCOL_VIOLATION here: a route starts usable. + +**Down Only**: +1 if the route is down-only (see [Routing](#routing)), 0 otherwise. +Any other value is a PROTOCOL_VIOLATION. + + +## ROUTE_UPDATE {#route-update} +A publisher sends a ROUTE_UPDATE message when a route it sent changes, referencing its Route ID. + +~~~ +ROUTE_UPDATE Message { + Type (i) = 0x7 + Message Length (i) + Route ID (i), + Seqno (i), + Metric (i), + Down Only (8), +} +~~~ + +**Type**: +Set to 0x7 to indicate a ROUTE_UPDATE message. + +**Route ID**: +The ordinal implicitly assigned by a prior ROUTE_START on this stream. + +**Seqno**, **Metric**, and **Down Only**: +As defined for [ROUTE_START](#route-start), replacing the route's. +A Metric of 2^64-1 means the route is held: the publisher has no usable route to the origin node but still holds its claims while it recovers. +A held route carries a Seqno and Down Only of 0, and the receiver keeps the last finite values; any other value is a PROTOCOL_VIOLATION. +A later finite ROUTE_UPDATE makes the route usable again. + + +## ROUTE_END {#route-end} +A publisher sends a ROUTE_END message when it lets go of a route, referencing its Route ID. +It retires the Route ID and every claim under the route, whose ANNOUNCE_END the publisher does not send. +It claims nothing about the origin's claims, which may still be served elsewhere. + +The subscriber sends the same message back on the stream once it dropped the route, its echo (see [Routing](#routing)). +A subscriber MUST send exactly one echo for each ROUTE_END, and a publisher MUST close the session with a PROTOCOL_VIOLATION on an echo naming a Route ID it has not ended or already saw echoed. + +~~~ +ROUTE_END Message { + Type (i) = 0x8 + Message Length (i) + Route ID (i) +} +~~~ + +**Type**: +Set to 0x8 to indicate a ROUTE_END message. + +**Route ID**: +The ordinal implicitly assigned by a prior ROUTE_START on this stream. + + +## ROUTE_REQUEST {#route-request} +A subscriber sends a ROUTE_REQUEST message to ask for a route to an origin node at least as new as a Seqno (see [Routing](#routing)). +It names the origin node rather than a Route ID, so it stays valid while a ROUTE_END for that route is in flight. + +~~~ +ROUTE_REQUEST Message { + Type (i) = 0xA + Message Length (i) + Node ID (i), + Seqno (i), +} +~~~ + +**Type**: +Set to 0xA to indicate a ROUTE_REQUEST message. + +**Node ID**: +The origin node, as the publisher named it on this stream. +0 is a PROTOCOL_VIOLATION. + +**Seqno**: +The Seqno the subscriber needs. +A publisher with no route to the origin node drops the request. + +A publisher MUST close the session with a PROTOCOL_VIOLATION if it receives any message type other than ROUTE_REQUEST or ROUTE_END after the ANNOUNCE_REQUEST. + + ## ANNOUNCE_START {#announce-start} -A publisher sends an ANNOUNCE_START message to advertise a route: a claim that paths under a prefix can be served. +A publisher sends an ANNOUNCE_START message to advertise a claim of a route's origin node: that paths under a prefix can be served. Each ANNOUNCE_START implicitly assigns the next Announce ID on the stream, later referenced by ANNOUNCE_END, ANNOUNCE_UPDATE, ANNOUNCE_RESTART, and ANNOUNCE_TAKEOVER (see [Announce](#announce)). -Only the suffix is encoded on the wire, as the full route prefix can be constructed by prepending the requested prefix. +Only the suffix is encoded on the wire, as the full claim prefix can be constructed by prepending the requested prefix. ~~~ ANNOUNCE_START Message { @@ -1062,18 +1198,11 @@ ANNOUNCE_START Message { Message Length (i) Path Base (i), Path Keep (i), - Route Prefix Suffix (s), + Claim Prefix Suffix (s), + Route ID (i), Instance (b), Epoch (i), - Hops (..), - Route Cost (i), -} - -Hops { - Hop Base (i), - Hop Count (i), - Hop ID (i) ..., - Hop Keep (i), + Announce Cost (i), } ~~~ @@ -1081,53 +1210,34 @@ Hops { Set to 0x0 to indicate an ANNOUNCE_START message. **Path Base** and **Path Keep**: -The suffix begins with the first `Path Keep` segments of the suffix advertised by the base that `Path Base` names (see [Compression](#announce-compression)). +The suffix begins with the first `Path Keep` segments of the suffix claimed by the base that `Path Base` names (see [Compression](#announce-compression)). + +**Claim Prefix Suffix**: +The remaining segments of the suffix, which is combined with the requested prefix to form the claim's full prefix. +An empty suffix claims the requested prefix itself, which is how a claim covering more than the request presents (see [Announce](#announce)). -**Route Prefix Suffix**: -The remaining segments of the suffix, which is combined with the requested prefix to form the route's full prefix. -An empty suffix advertises the requested prefix itself, which is how a route covering more than the request presents (see [Announce](#announce)). +**Route ID**: +The route the claim travels under: its origin node is the claim's. +A claim never changes its Route ID. **Instance**: -The stream the route serves (see [Broadcast](#broadcast)): non-empty UTF-8 text. +The stream the claim serves (see [Broadcast](#broadcast)): non-empty UTF-8 text. An empty or non-UTF-8 Instance is a PROTOCOL_VIOLATION. A relay forwards the Instance unchanged. **Epoch**: -Which incarnation of the Instance the route serves: a counter the publisher raises within one Instance, 0 unless it keeps an Instance across restarts. +Which incarnation of the Instance the claim serves: a counter the publisher raises within one Instance, 0 unless it keeps an Instance across restarts. A relay forwards the Epoch unchanged. Only an ANNOUNCE_RESTART or ANNOUNCE_TAKEOVER changes the Instance or the Epoch; ANNOUNCE_UPDATE never does. -**Hop Base** and **Hop Keep**: -The Hop ID list ends with the last `Hop Keep` entries of the list advertised by the base that `Hop Base` names (see [Compression](#announce-compression)), following the literal `Hop ID` entries. -The two bases are independent. - -**Hop Count**: -The number of literal Hop ID entries that follow. -The resolved list does NOT include the publisher's own `Hop ID` from ANNOUNCE_OK. -An empty list indicates either that the announcement originated locally on the publisher (the publisher itself is the origin) or that the upstream peer does not support hop tracking. -A receiver MUST close the session with a PROTOCOL_VIOLATION if the Hop Count does not match the number of subsequent Hop ID entries. - -**Hop ID**: -A unique identifier for each relay in the path from the origin publisher, ordered from origin to the upstream of the responding publisher. -The responding publisher's own Hop ID is NOT included in the resolved list; it is carried once in ANNOUNCE_OK, so the total path length is the resolved list's length plus 1 (`Hop Count + Hop Keep + 1`). -When forwarding an announcement received from an upstream peer, a relay MUST append the upstream peer's ANNOUNCE_OK `Hop ID` to the resolved list, since that ID is no longer implicit downstream. -A Hop ID value of 0 means the hop is unknown: either it was never assigned or a relay deliberately withholds it (see [Routing](#routing)). -A received 0 is forwarded unchanged. -When bridging an announcement from an upstream that sent no hop list, a relay writes 0 for that hop. -An identity a receiver assigned that upstream is local selection state and MUST NOT be forwarded as a Hop ID. - -A receiver MUST close the session with a PROTOCOL_VIOLATION if a non-zero Hop ID appears twice in the resolved list. -Duplicate values of 0 are not a violation, since 0 identifies nothing and any number of hops may be unknown. - -**Route Cost**: -The static production and link price, in units chosen by the deployment. -The publisher seeds its production cost (0 for content it already produces, larger for a standby that starts work on demand). Each forwarding relay adds the upstream peer's declared link cost (see [Cost Parameter](#cost-parameter)). -Each sum MUST saturate at 2^62-1. +**Announce Cost**: +The origin's price for this prefix, in the units of the [Cost Parameter](#cost-parameter): 0 for content it already produces, larger for a standby that starts work on demand. +A relay forwards it unchanged; the cost of a path through the claim is the Announce Cost plus the route's metric. ## ANNOUNCE_END {#announce-end} -A publisher sends an ANNOUNCE_END message to retract a previously started route, referencing its Announce ID. +A publisher sends an ANNOUNCE_END message when the origin withdrew a claim, referencing its Announce ID. The id is retired and MUST NOT be referenced again. Retraction claims nothing about the content: a broadcast whose publisher stops advertising may remain readable by exact path, serving its stored groups over FETCH. Retraction does not disturb subscriptions already in flight, which conclude normally with SUBSCRIBE_FINISH. @@ -1145,23 +1255,21 @@ ANNOUNCE_END Message { Set to 0x1 to indicate an ANNOUNCE_END message. **Announce ID**: -The ordinal implicitly assigned by a prior ANNOUNCE_START on this stream. -Referencing an id that was never assigned, or one already retired, is a protocol violation. -Announce IDs are never reused within a stream; a route that is announced again after an ANNOUNCE_END gets a fresh id from its next ANNOUNCE_START. +The ordinal implicitly assigned by a prior ANNOUNCE_START or ANNOUNCE_TAKEOVER on this stream. +A claim that is announced again after an ANNOUNCE_END gets a fresh id from its next ANNOUNCE_START. ## ANNOUNCE_UPDATE {#announce-update} -A publisher sends an ANNOUNCE_UPDATE message to atomically update a previously started advertisement's metadata, referencing its Announce ID. -The route's prefix is unchanged and the id stays live; the Hop ID list MAY differ from the original (e.g. after a relay failover or upstream restart). -An update carries no content claim: in-flight subscriptions under the route are undisturbed. +A publisher sends an ANNOUNCE_UPDATE message when the origin re-prices a claim, referencing its Announce ID. +The claim's prefix and route are unchanged and the id stays live. +An update carries no content claim: in-flight subscriptions under the claim are undisturbed. ~~~ ANNOUNCE_UPDATE Message { Type (i) = 0x2 Message Length (i) Announce ID (i), - Hops (..), - Route Cost (i), + Announce Cost (i), } ~~~ @@ -1169,19 +1277,16 @@ ANNOUNCE_UPDATE Message { Set to 0x2 to indicate an ANNOUNCE_UPDATE message. **Announce ID**: -The ordinal implicitly assigned by a prior ANNOUNCE_START on this stream. -Referencing an id that was never assigned, or one already retired by an ANNOUNCE_END, is a protocol violation. +As defined for [ANNOUNCE_END](#announce-end). -**Hops** and **Route Cost**: -As defined for [ANNOUNCE_START](#announce-start). -`Hop Base` may name this advertisement itself, which resolves against the list being replaced. -An update whose only change is a Route Cost is valid: it is how a relay re-prices a route without disturbing it. +**Announce Cost**: +As defined for [ANNOUNCE_START](#announce-start), replacing the claim's. ## ANNOUNCE_RESTART {#announce-restart} -A publisher sends an ANNOUNCE_RESTART message when a previously started advertisement now serves another Instance, or a higher Epoch of the same one (see [Routing](#routing)), referencing its Announce ID. -The route's prefix is unchanged and the id stays live. -Under another Instance, whatever the subscriber resolved under the prefix through the advertisement is the old stream: in-flight subscriptions on it are undisturbed, and a new request resolves afresh. +A publisher sends an ANNOUNCE_RESTART message when a claim now serves another Instance, or a higher Epoch of the same one (see [Routing](#routing)), referencing its Announce ID. +The claim's prefix and route are unchanged and the id stays live. +Under another Instance, whatever the subscriber resolved under the prefix through the claim is the old stream: in-flight subscriptions on it are undisturbed, and a new request resolves afresh. Under a higher Epoch of the same Instance the stream continues, and so do its subscriptions. ~~~ @@ -1191,8 +1296,7 @@ ANNOUNCE_RESTART Message { Announce ID (i), Instance (b), Epoch (i), - Hops (..), - Route Cost (i), + Announce Cost (i), } ~~~ @@ -1200,15 +1304,14 @@ ANNOUNCE_RESTART Message { Set to 0x3 to indicate an ANNOUNCE_RESTART message. **Announce ID**: -As defined for [ANNOUNCE_UPDATE](#announce-update). +As defined for [ANNOUNCE_END](#announce-end). -**Instance**, **Epoch**, **Hops**, and **Route Cost**: -As defined for [ANNOUNCE_START](#announce-start), replacing the advertisement's. -`Hop Base` may name this advertisement itself, which resolves against the list being replaced. +**Instance**, **Epoch**, and **Announce Cost**: +As defined for [ANNOUNCE_START](#announce-start), replacing the claim's. ## ANNOUNCE_TAKEOVER {#announce-takeover} -A publisher sends an ANNOUNCE_TAKEOVER message to advertise a route that continues what a live advertisement, its predecessor, served beneath the new route's prefix. -A worker advertising a prefix under its own Instance answers a request under it by publishing that path under a fresh Instance; the ANNOUNCE_TAKEOVER keeps subscriptions already resolved through the prefix rather than resetting them to the more specific route. +A publisher sends an ANNOUNCE_TAKEOVER message to advertise a claim that continues what a live claim, its predecessor, served beneath the new claim's prefix. +A worker advertising a prefix under its own Instance answers a request under it by publishing that path under a fresh Instance; the ANNOUNCE_TAKEOVER keeps subscriptions already resolved through the prefix rather than resetting them to the more specific claim. It is an ANNOUNCE_START in every other respect and assigns the next Announce ID. ~~~ @@ -1218,11 +1321,11 @@ ANNOUNCE_TAKEOVER Message { Predecessor (i), Path Base (i), Path Keep (i), - Route Prefix Suffix (s), + Claim Prefix Suffix (s), + Route ID (i), Instance (b), Epoch (i), - Hops (..), - Route Cost (i), + Announce Cost (i), } ~~~ @@ -1230,19 +1333,54 @@ ANNOUNCE_TAKEOVER Message { Set to 0x5 to indicate an ANNOUNCE_TAKEOVER message. **Predecessor**: -The Announce ID of a live advertisement whose prefix covers the new route's prefix. -One at the predecessor's own prefix retires the predecessor's id, which this advertisement replaces; otherwise the predecessor stays live. -The receiver MUST close the session with a PROTOCOL_VIOLATION if the predecessor was never assigned, is already retired, or does not cover the new prefix. +The Announce ID of a live claim under the same route whose prefix covers the new claim's prefix. +One at the predecessor's own prefix retires the predecessor's id, which this claim replaces; otherwise the predecessor stays live. +The receiver MUST close the session with a PROTOCOL_VIOLATION if the predecessor was never assigned, is already retired, is under another route, or does not cover the new prefix. -**Path Base**, **Path Keep**, **Route Prefix Suffix**, **Instance**, **Epoch**, **Hops**, and **Route Cost**: +**Path Base**, **Path Keep**, **Claim Prefix Suffix**, **Route ID**, **Instance**, **Epoch**, and **Announce Cost**: As defined for [ANNOUNCE_START](#announce-start), resolved before the predecessor retires. -The receiver treats whatever it resolved through the predecessor beneath the new prefix as served by the new route: subscriptions on it continue, and a new request for those paths MAY join them. -Only what the predecessor's current advertisement resolved continues; anything resolved before an earlier ANNOUNCE_RESTART of it does not. +The receiver treats whatever it resolved through the predecessor beneath the new prefix as served by the new claim: subscriptions on it continue, and a new request for those paths MAY join them. +Only what the predecessor's current claim resolved continues; anything resolved before an earlier ANNOUNCE_RESTART of it does not. + +A publisher MUST send an ANNOUNCE_TAKEOVER only when the new claim serves exactly what the subscriber resolved through the predecessor beneath its prefix: the publisher that answered requests through the predecessor now advertises their paths. +A relay forwards one only when the subscriber was last sent that predecessor, and every claim the relay sent at its prefix serves the same Instance; otherwise it sends an ANNOUNCE_START, or an ANNOUNCE_RESTART at the predecessor's own prefix. -A publisher MUST send an ANNOUNCE_TAKEOVER only when the new route serves exactly what the subscriber resolved through the predecessor beneath its prefix: the publisher that answered requests through the predecessor now advertises their paths. -A relay forwards one only when the subscriber was last sent that predecessor, and every route the relay presented there serves the same Instance; otherwise it sends an ANNOUNCE_START, or an ANNOUNCE_RESTART at the predecessor's own prefix. -Before moq-lite-07, a publisher sends an ANNOUNCE_START instead, or an ANNOUNCE_END and then an ANNOUNCE_START at the predecessor's own prefix. + +## ANNOUNCE_FLOOR {#announce-floor} +A publisher sends an ANNOUNCE_FLOOR message to close a batch of a route's claims, listing every region it certifies for the route (see [Floors](#floors)). + +~~~ +ANNOUNCE_FLOOR Message { + Type (i) = 0x9 + Message Length (i) + Route ID (i), + Region Count (i), + Region (..) ..., +} + +Region { + Region Suffix (s), + Revision (i), +} +~~~ + +**Type**: +Set to 0x9 to indicate an ANNOUNCE_FLOOR message. + +**Route ID**: +The ordinal implicitly assigned by a prior ROUTE_START on this stream. + +**Region Count**: +The number of regions that follow, 0 when the publisher certifies nothing within the requested prefix. + +**Region Suffix**: +The region, combined with the requested prefix like a Claim Prefix Suffix; empty for the whole requested prefix. +Regions are in no particular order, and a repeated one is a PROTOCOL_VIOLATION. + +**Revision**: +The origin's Revision as of which the stream's claims for the route within the region are complete. +2^64-1 is a PROTOCOL_VIOLATION. ## SUBSCRIBE @@ -1254,6 +1392,7 @@ SUBSCRIBE Message { Subscribe ID (i) Broadcast Path (s) Instance (b) + Node ID (i) Track Name (s) Subscriber Priority (8) Group Order (8) @@ -1281,6 +1420,9 @@ Anchor { A unique identifier chosen by the subscriber. A Subscribe ID MUST NOT be reused within the same session, even if the prior subscription has been closed. +**Instance** and **Node ID**: +As in [TRACK](#track). + **Subscriber Priority**: The priority of the subscription within the session, represented as a u8 with 128 as the midpoint. The publisher SHOULD transmit *higher* values first during congestion. @@ -1386,6 +1528,7 @@ TRACK Message { Message Length (i) Broadcast Path (s) Instance (b) + Node ID (i) Track Name (s) } ~~~ @@ -1395,8 +1538,12 @@ The broadcast path of the track. **Instance**: The stream the subscriber expects, encoded as in [ANNOUNCE_START](#announce-start), or empty for whichever wins the path. -A publisher MUST refuse a request with UNROUTABLE when no route of that Instance serves the path, so a request is never answered by a different stream than the one it named. -A relay fills in the Instance of the route it resolved when forwarding a request upstream; TRACKS, SUBSCRIBE and FETCH carry it the same way. +A publisher MUST refuse a request with UNROUTABLE when no claim of that Instance serves the path, so a request is never answered by a different stream than the one it named. +A relay fills in the Instance of the claim it resolved when forwarding a request upstream; TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS carry it the same way. + +**Node ID**: +The origin node that serves the request, as the publisher names it on this session, or 0 for whichever the publisher selects (see [Routing](#routing)). +TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS carry it the same way. **Track Name**: The name of the track. @@ -1463,14 +1610,15 @@ TRACKS Message { Message Length (i) Broadcast Path (s) Instance (b) + Node ID (i) } ~~~ **Broadcast Path**: The path of the Broadcast to list. -**Instance**: -The stream the subscriber expects, as in [TRACK](#track). +**Instance** and **Node ID**: +As in [TRACK](#track). The listing is pinned to the stream that answers it and ends with that stream. ## TRACKS_ADD {#tracks-add} @@ -1529,6 +1677,7 @@ DATAGRAMS Message { Subscribe ID (i) Broadcast Path (s) Instance (b) + Node ID (i) Track Name (s) Subscriber Priority (8) } @@ -1681,6 +1830,7 @@ FETCH Message { Message Length (i) Broadcast Path (s) Instance (b) + Node ID (i) Track Name (s) Subscriber Priority (8) Subscriber Max Age (i) @@ -1693,6 +1843,9 @@ FETCH Message { **Broadcast Path**: The broadcast path of the track to fetch from. +**Instance** and **Node ID**: +As in [TRACK](#track). + **Track Name**: The name of the track to fetch from. @@ -1771,7 +1924,7 @@ An endpoint MUST close the session with a protocol violation if it receives more A peer that reconnects to a provided URI SHOULD keep using that URI for subsequent reconnects rather than reverting to the original. -Routes that arrived on a session that received a GOAWAY stay usable until the session ends. +Routes that arrived on a session that received a GOAWAY stay usable until the session ends, ranked below every other (see [Routing](#routing)). ## AUTH {#auth-message} AUTH is the first message on an Auth Stream, presenting a token. @@ -1928,7 +2081,7 @@ The `Message Length` describes the payload size on the wire. - Switched every variable-length integer, including SETUP parameter values, from QUIC's two-bit length prefix to moq-transport's leading-ones encoding, widening the range to 64 bits. - Capped the Message Length of every message except FRAME at 65,535 bytes. SETUP parameter counts are bounded by Message Length, without a separate count cap. -- Any Setup Stream or bidirectional stream message that fails to decode closes the session with PROTOCOL_VIOLATION; previously only an unexpected Message Length did, as a SHOULD. This includes a frame bound without its group bound, and a Hop Count that disagrees with its Hop ID entries, each of which previously reset only the stream. A stream ending mid-message outside the Setup and Auth Streams, a malformed GROUP or FRAME on a Group or Fetch Stream, and an unknown stream type still affect only that stream. +- Any Setup Stream or bidirectional stream message that fails to decode closes the session with PROTOCOL_VIOLATION; previously only an unexpected Message Length did, as a SHOULD. This includes a frame bound without its group bound, which previously reset only the stream. A stream ending mid-message outside the Setup and Auth Streams, a malformed GROUP or FRAME on a Group or Fetch Stream, and an unknown stream type still affect only that stream. - A session without a negotiated subprotocol negotiates the version in a draft-14 CLIENT_SETUP and SERVER_SETUP exchange. - Added the reliable ordered RTCDataChannel binding, with one Qmux Record per binary message and no datagrams. - Added the SETUP `Implementation` parameter (0x6): the sender's software as `name/version` product tokens. @@ -1959,16 +2112,21 @@ The `Message Length` describes the payload size on the wire. - Limited a broadcast path to 32 segments and 4,096 bytes. - A path never has an empty segment, so a leading, trailing, or doubled `/` is a PROTOCOL_VIOLATION rather than a second spelling of the same path. - A `\` in a path escapes only `\` or `/`; any other `\` is a PROTOCOL_VIOLATION rather than a second spelling of the same namespace field. -- Hid routes with a `.`-prefixed segment below the requested prefix from announce discovery, and added the ANNOUNCE_REQUEST `Hidden` field to opt in. -- Replaced ANNOUNCE_OK Active Count with an ordered, empty SYNCED event (0x4), marking the initial set without retaining routes that end before delivery. -- Added `Instance` and `Epoch` to ANNOUNCE_START and ANNOUNCE_RESTART, and `Instance` to TRACK, TRACKS, SUBSCRIBE, and FETCH. The Instance is non-empty UTF-8 text naming one continuous stream, and a path and an Instance name one Broadcast; the Epoch is a counter within one Instance. A relay MAY move a subscription between routes with the same Instance, continuing from the first frame the subscriber lacks instead of at a group boundary, and never between routes with different Instances. Within one Instance only its highest Epoch is eligible. A request's Instance is empty when it pins none. Replaces the first-hop identity. +- Hid claims with a `.`-prefixed segment below the requested prefix from announce discovery, and added the ANNOUNCE_REQUEST `Hidden` field to opt in. +- Replaced ANNOUNCE_OK Active Count with an ordered, empty SYNCED event (0x4), marking the initial set without retaining claims that end before delivery. +- Added `Instance` and `Epoch` to ANNOUNCE_START and ANNOUNCE_RESTART, and `Instance` to TRACK, TRACKS, SUBSCRIBE, and FETCH. The Instance is non-empty UTF-8 text naming one continuous stream, and a path and an Instance name one Broadcast; the Epoch is a counter within one Instance. A relay MAY move a subscription between claims with the same Instance, continuing from the first frame the subscriber lacks instead of at a group boundary, and never between claims with different Instances. Within one Instance only its highest Epoch is eligible. A request's Instance is empty when it pins none. Replaces the first-hop identity. - Added ANNOUNCE_RESTART (0x3): a live advertisement now serves another Instance, or a higher Epoch of the same one. -- Added ANNOUNCE_TAKEOVER (0x5): a route that continues what a live advertisement served beneath it, such as a job answering a request through a worker's prefix at its exact path, so subscriptions resolved through the prefix are not reset. +- Added ANNOUNCE_TAKEOVER (0x5): a claim that continues what a live claim served beneath it, such as a job answering a request through a worker's prefix at its exact path, so subscriptions resolved through the prefix are not reset. - A receiver mints an Instance, starting with `~`, for each announcement a session of an earlier version delivers. -- Added announce compression: ANNOUNCE_START gains `Path Base` and `Path Keep` to copy the head of a live advertisement's suffix, and ANNOUNCE_START and ANNOUNCE_UPDATE gain `Hop Base` and `Hop Keep` to copy the tail of a live advertisement's Hop ID list. -- Hop IDs span 64 bits. -- Replaced Warm and Cold Route Cost with one static Route Cost. -- Added the Instance Hash and Chain Hash tie-breaks after the shortest path: hashes of the requested path with the route's Instance, then with its Hop IDs, so equal-cost advertisers of one prefix share its paths. Both are skipped for a route a publisher announced directly at the requested path. +- Added announce compression: ANNOUNCE_START and ANNOUNCE_TAKEOVER gain `Path Base` and `Path Keep` to copy the head of a live claim's suffix. +- Split routing into two layers on the Announce Stream. ROUTE_START (0x6), ROUTE_UPDATE (0x7), and ROUTE_END (0x8) advertise how the publisher reaches an origin node, named by a Node ID, with a seqno, an additive metric, and a down-only bit, and loops are prevented by Babel's feasibility condition. Every ANNOUNCE_START and ANNOUNCE_TAKEOVER names its route's Route ID in place of the Hop ID list, so a link change sends one ROUTE_UPDATE per origin rather than an update per claim. Removed the Hop ID list, ANNOUNCE_OK's Hop ID, and the SETUP Hop parameter (0x5). +- A ROUTE_UPDATE with a Metric of 2^64-1 holds a route while it recovers; ROUTE_END lets it go. Neither withdraws the origin's claims, and the subscriber echoes every ROUTE_END back. +- Added ROUTE_REQUEST (0xA), sent by the subscriber on the Announce Stream to ask for a route with a newer seqno when every route it has is infeasible. +- Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the regions it certifies and the origin Revision each is complete as of, so a stale copy never revives a claim whose withdrawal a newer one delivered. +- Added `Node ID` to TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS: the origin node the first hop selected, followed by every later hop. +- Replaced Warm and Cold Route Cost with the origin's Announce Cost, unchanged in transit. ANNOUNCE_UPDATE carries only the Announce Cost. +- The Cost parameter must be at least 1. +- Replaced the shortest-path, Hop ID, Instance Hash, and Chain Hash tie-breaks with the local priority of the route's session, the Announce Cost plus the route's metric, and a Rendezvous Hash of the requested path and the origin's Node ID. - Removed the standby-ordering deployment requirements on Route Cost seeds and bounds. ### Tracks {#changelog-07-tracks} @@ -2186,10 +2344,11 @@ A token presented on an Auth Stream is as sensitive as one in the request URI, a GOAWAY carries an optional New Session URI that asks the peer to reconnect elsewhere. A malicious or compromised peer could use this to redirect a client to an attacker-controlled server. A recipient MUST validate the URI against local policy (scheme, authority, and port) before reconnecting, and MUST NOT reconnect if validation fails (see [GOAWAY](#goaway)). Migrated subscriptions carry no implicit trust from the prior session; the new session is authenticated independently. ## Routing Metadata and Privacy -Hop IDs (see [ANNOUNCE_OK](#announce-ok) and [ANNOUNCE_START](#announce-start)) expose the relay path of a broadcast, which may reveal internal topology. A relay that does not wish to disclose its position MAY use the reserved value 0 ("unknown") instead of a stable identifier, at the cost of losing loop detection through itself (see [Routing](#routing)). The Hop ID announcement filter (see [Hop Parameter](#hop-parameter)) exists for loop avoidance, not access control: a subscriber cannot verify that a publisher honored it, so it MUST NOT be relied upon to hide a broadcast from a peer that declared its Hop ID. +A Node ID (see [Routing](#routing)) is opaque and fresh for each restart: it tells a peer which claims share an origin, and a route's metric how far that origin is, but not the relays in between. +A Node ID is declared, not proven, so a peer can forge another origin's Node ID or advertise a better route than it has. Qualifying the Node IDs of a peer with a scoped publish grant confines such a forgery to the claims inside that grant; an unscoped peer is trusted with every origin. ## Resource Exhaustion -A peer can open many streams (subscriptions, announcements, fetches), request large announce prefixes, or advertise broad routes. Implementations SHOULD bound the number of concurrent subscriptions, announce matches, and cached groups; an endpoint whose peer goes past its bound on subscriptions or announcements SHOULD close the session with TOO_MANY_REQUESTS, and SHOULD rely on QUIC flow control and stream limits to backpressure a misbehaving peer (see [ANNOUNCE_REQUEST](#announce-request)). Expiration (see [Expiration](#expiration)) bounds how long stale groups consume memory and flow control. A broad route invites a request for any covered path, each of which may start work: an advertiser SHOULD bound the work it starts, withdrawing its route before it runs out, and every refusal is terminal, so a flood of requests costs the mesh one round trip each rather than a search (see [Resolution](#resolution)). +A peer can open many streams (subscriptions, announcements, fetches), request large announce prefixes, or advertise broad claims. Implementations SHOULD bound the number of concurrent subscriptions, announce matches, routes, floor regions, and cached groups; an endpoint whose peer goes past its bound on subscriptions or announcements SHOULD close the session with TOO_MANY_REQUESTS, and SHOULD rely on QUIC flow control and stream limits to backpressure a misbehaving peer (see [ANNOUNCE_REQUEST](#announce-request)). Expiration (see [Expiration](#expiration)) bounds how long stale groups consume memory and flow control. A broad claim invites a request for any covered path, each of which may start work: an advertiser SHOULD bound the work it starts, withdrawing its claim before it runs out, and every refusal is terminal, so a flood of requests costs the mesh one round trip each rather than a search (see [Resolution](#resolution)). ## Datagram Injection Datagrams are routed to a subscription solely by Subscribe ID and carry no per-group authentication beyond that of the QUIC connection. On an unmodified QUIC/WebTransport connection this is sufficient, since datagrams are protected by the transport. A subscriber MUST silently drop any datagram with an unknown Subscribe ID (see [Datagrams](#datagrams)). diff --git a/quest/m1/announce-takeover.md b/quest/m1/announce-takeover.md index c9db540f92d..41b437577b9 100644 --- a/quest/m1/announce-takeover.md +++ b/quest/m1/announce-takeover.md @@ -131,9 +131,8 @@ separate from the identity rewrite of this Plan, which landed with - Reword "takeover" as an event or operation, in terms of identity continuity, in [moq-bench pins the instances it measures](/quest/m2/bench-epoch-pin.md) - (Goal and Plan), [the HLS server follows each path once](/quest/m2/hls-follow-once.md) - (Goal and Tests), and [lite-07 route wire](/quest/m1/cluster-routing/routes/wire.md) - (the wire sketch, "the separately proven Takeover operation"). + (Goal and Plan) and [the HLS server follows each path once](/quest/m2/hls-follow-once.md) + (Goal and Tests). - Rewrite the EPOCH references in `drafts/draft-lcurley-moq-cluster.md` and `drafts/draft-lcurley-moq-epoch.md` in terms of IDENTITY. - Add a regression test where a publisher restart's new announcement diff --git a/quest/m1/api-audit/route/handle.md b/quest/m1/api-audit/route/handle.md index adf63a37a88..e8c1c317fda 100644 --- a/quest/m1/api-audit/route/handle.md +++ b/quest/m1/api-audit/route/handle.md @@ -42,7 +42,7 @@ read-only facts on a received announcement, and `Hop`, `Hops` and Non-goals: the instance model ([Instance and epoch](https://github.com/moq-dev/moq/pull/7489)), announce events ([Listing and follow](/quest/m1/api-audit/route/events.md)), the read side, which instance owns, and the route layer's wire metric -([lite-07 route wire](/quest/m1/cluster-routing/routes/wire.md)). +(`drafts/draft-lcurley-moq-lite.md`). ## Plan diff --git a/quest/m1/cluster-routing/routes/README.md b/quest/m1/cluster-routing/routes/README.md index 7793203f1fe..7942b49293e 100644 --- a/quest/m1/cluster-routing/routes/README.md +++ b/quest/m1/cluster-routing/routes/README.md @@ -51,7 +51,10 @@ contract below. The routing contract's remaining model fixtures live in port each topology to the implementation as acceptance, then delete the model. The handoff contract below, decided 2026-10-16, settles claim freshness: -The wire sketch is in [wire](/quest/m1/cluster-routing/routes/wire.md); the bullets below are the contract it carries. +The lite-07 wire is specified in `drafts/draft-lcurley-moq-lite.md` (Announce, +Floors, and Routing), and `rs/moq-net/src/lite/announce.rs` encodes and decodes +its new messages ([moq-dev/moq#7764](https://github.com/moq-dev/moq/pull/7764)); +the bullets below are the contract it carries. - The announce cost is the origin's per-prefix seed (`transcode/**` at 10 and `transcode/foobar` at 1 from one node, decided 2026-10-01) and nobody @@ -498,7 +501,6 @@ benchmarks. ## Required -- [lite-07 route wire](/quest/m1/cluster-routing/routes/wire.md) - lite-07 and the draft carry ROUTE, ANNOUNCE_FLOOR, ROUTE_REQUEST, and path-less ANNOUNCE - [Route session integration](/quest/m1/cluster-routing/routes/session.md) - one router per origin node drives the origin and every lite-07 announce stream - [Scoped peers prove a route key](/quest/m1/cluster-routing/routes/proof.md) - a peer proves a route key once per announce stream, and its forwarded identities are qualified by it - [Announcement availability](/quest/m1/cluster-routing/routes/availability.md) - announcements report availability, and a let-go is offline rather than END, in Rust and JS diff --git a/quest/m1/cluster-routing/routes/js.md b/quest/m1/cluster-routing/routes/js.md index 8a97d15c8f4..6fff2f20d4b 100644 --- a/quest/m1/cluster-routing/routes/js.md +++ b/quest/m1/cluster-routing/routes/js.md @@ -21,7 +21,6 @@ the cheapest one; fix it with the route-selection rework here. ## Required -- [lite-07 route wire](/quest/m1/cluster-routing/routes/wire.md) - the messages - [Announcement availability](/quest/m1/cluster-routing/routes/availability.md) - the API this feeds ## Related diff --git a/quest/m1/cluster-routing/routes/proof.md b/quest/m1/cluster-routing/routes/proof.md index 3d39fee6050..f18eb0c3d9b 100644 --- a/quest/m1/cluster-routing/routes/proof.md +++ b/quest/m1/cluster-routing/routes/proof.md @@ -70,5 +70,4 @@ change in lite-07 and its draft. ## Related - [Route session integration](/quest/m1/cluster-routing/routes/session.md) - files identities under the key this proves -- [lite-07 route wire](/quest/m1/cluster-routing/routes/wire.md) - the announce messages this extends - [P2P](/quest/m2/p2p/README.md) - its roster id is already planned as a public key diff --git a/quest/m1/cluster-routing/routes/session.md b/quest/m1/cluster-routing/routes/session.md index c006ea8c030..82130f19d9a 100644 --- a/quest/m1/cluster-routing/routes/session.md +++ b/quest/m1/cluster-routing/routes/session.md @@ -10,6 +10,18 @@ to the identity it selected. A lite-06 peer is translated at the relay. A peer can make the router hold only bounded state. The configuration changes the contract decided ship with it. +lite-07's live wire switches to the route layer that +`drafts/draft-lcurley-moq-lite.md` specifies, in Rust: ANNOUNCE_START, ANNOUNCE_UPDATE, ANNOUNCE_RESTART, and +ANNOUNCE_TAKEOVER name a Route ID and carry the Announce Cost in place of +the Hop ID list (`HopsRef`, `Hop Base`/`Hop Keep`, and `cluster.max_hops` +go from lite-07); ANNOUNCE_OK loses its Hop ID and SETUP its Hop parameter; +TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS carry the Node ID; a Cost +parameter of 0 is refused; the subscriber sends ROUTE_REQUEST and echoes +ROUTE_END. The new messages' codecs already exist (`AnnounceBroadcast`'s +route variants and `AnnounceReply`); until this lands the subscriber +refuses route messages. A hop-less ANNOUNCE cannot run before the router +drives sessions, which is why the switch lands here. + ## Plan - Scope route state to interested origins, not the CDN's whole node table. @@ -76,7 +88,7 @@ Update/Restart split as the only takeover rule. Once each origin is its own ANNOUNCE and the first hop pins the request to its identity, the first hop's route is the instance, so the per-hop renewal has nothing left to protect. An epochless change of serving route stays a Restart, including the -same-origin next-hop change in the [wire sketch](/quest/m1/cluster-routing/routes/wire.md), and the routing +same-origin next-hop change with no epoch (test it), and the routing contract's epochless fixtures keep passing. Minting an epoch for every route instead was tried and rejected ([moq-dev/moq#6948](https://github.com/moq-dev/moq/pull/6948)): each relay @@ -120,7 +132,6 @@ admission. ## Required -- [lite-07 route wire](/quest/m1/cluster-routing/routes/wire.md) - the messages this drives - [One Route handle](/quest/m1/api-audit/route/handle.md) - the origin's route surface this rewrites - [Listing and follow](/quest/m1/api-audit/route/events.md) - the announcement events this feeds ([moq-dev/moq#7678](https://github.com/moq-dev/moq/pull/7678)) diff --git a/quest/m1/cluster-routing/routes/wire.md b/quest/m1/cluster-routing/routes/wire.md deleted file mode 100644 index 581fd5b6857..00000000000 --- a/quest/m1/cluster-routing/routes/wire.md +++ /dev/null @@ -1,91 +0,0 @@ -# [L] lite-07 route wire - -## Goal - -lite-07 carries the route layer of [Routes and announces](/quest/m1/cluster-routing/routes/README.md) on -the announce stream: ROUTE_START, ROUTE_UPDATE, ROUTE_END, and -ANNOUNCE_FLOOR toward the subscriber, and ROUTE_REQUEST and the ROUTE_END -echo back toward the publisher. ANNOUNCE_START and ANNOUNCE_UPDATE carry a -Route ID and the origin's seed in place of the hop list. Rust encodes and -decodes it, and `drafts/draft-lcurley-moq-lite.md` specifies it. Driving the -router with it is [session integration](/quest/m1/cluster-routing/routes/session.md). - -## Plan - -- Wire sketch, flexible in detail. ROUTE_START carries the node id, a seqno, - the metric, and the down-only bit (the epic's one-selection decision), and implicitly takes the next - stream-local Route ID, as ANNOUNCE_START takes the next Announce ID today. - ROUTE_UPDATE (Route ID, seqno, metric, down-only) and ROUTE_END (Route - ID) reference it; the bit rides UPDATE so a route moving between a - mesh path and a CDN fallback changes it without retracting its announces. - ANNOUNCE_START carries the prefix (keeping today's Path Base/Keep - compression), a Route ID, the origin's cost for that prefix, and the epoch, - and lite-07's restart message from - Restart keeps its meaning. On a - route without an epoch, the source is the origin node and its ANNOUNCE: a - new one at the same path is a restart. This is routing provenance, not - permission to stitch. A concrete serving-route change without an epoch - remains Restart even when the origin node is unchanged; only a matching - explicit epoch or the separately proven Takeover operation preserves an - instance (superseded 2026-10-30: a shared route `instance` preserves it, per - [Instance and epoch](https://github.com/moq-dev/moq/pull/7489)). Test a same-origin next-hop change with no epoch. - ANNOUNCE_UPDATE re-prices it and ANNOUNCE_END withdraws this origin's claim - while the route stays up. A wildcard claim is capability, not an inventory - of broadcasts; another origin may still serve the same epoch. An announce - never changes its Route ID: another origin - serving the same path is another ANNOUNCE. An ANNOUNCE naming an unknown - Route ID is a protocol violation. A ROUTE_UPDATE with an infinite metric - (2^64-1, the largest lite-07 varint; a sum reaching it is unreachable) - means held: the sender has no usable route but still holds the origin's - announcements while it recovers. ROUTE_END means the sender let go of the - origin on this stream; neither is proof that the origin withdrew its - claims (see the epic's handoff contract). Each batch of ANNOUNCEs for a route - closes with one ANNOUNCE_FLOOR listing every region the sender certifies - for that route: a count, then per region a suffix of the stream's interest - prefix (empty for the whole interest) and the origin revision as of which - the stream's set for that route in that region is complete. The list - replaces the previous one, so a region left out is no longer certified, - and a count of zero answers a stream whose interest none of the sender's - regions meets. - ROUTE_REQUEST (node id, seqno) travels - the other way on the same stream, toward the neighbour that sent the route, - and so does the echo of every ROUTE_END: the receiver sends the same - ROUTE_END (Route ID) back once it has dropped the route, which tells the - sender no neighbour still holds its advertisement (see the epic's - handoff contract). Every receiver echoes, clients included; a closed session - counts as the echo. - The hop list leaves lite-07, and with it the - `Hop Base`/`Hop Keep` compression; `cluster.max_hops` then stops applying - to lite-07. Most of the 62% saving measured in - [moq-dev/moq#4196](https://github.com/moq-dev/moq/pull/4196) came from hop - chains, so rerun its announce bench on path-less ANNOUNCE and drop - `Path Base`/`Path Keep` too unless it still pays for its rules. - - ```text - ROUTE_START node=0x7a3f seqno=41 metric=12 -> route 0 - ANNOUNCE_START prefix=transcode/ route=0 cost=10 -> announce 0 - ANNOUNCE_START prefix=transcode/foobar route=0 cost=1 -> announce 1 - ANNOUNCE_FLOOR route=0 count=1 ""@7 # complete as of revision 7 - ROUTE_UPDATE route=0 seqno=42 metric=15 - ANNOUNCE_END announce=1 # this origin withdraws a claim - ANNOUNCE_FLOOR route=0 count=1 ""@8 - ANNOUNCE_FLOOR route=1 count=2 "live/a/"@3 "live/b/"@5 # a scoped key - ANNOUNCE_FLOOR route=1 count=1 "live/b/"@6 # live/a/ is no longer certified - ROUTE_UPDATE route=0 seqno=42 metric=inf # held: unusable, still recovering - ROUTE_END route=0 # let go: forget it on this stream - ROUTE_END route=0 # <- the receiver echoes it back - ``` -- Requests forwarded between routing nodes name the serving identity - (decided 2026-10-13). ROUTE carries one down-only bit and no export metric. -- The region list's order and any other wire detail of it are settled here - with Rust and JS interop, not in the proof. The core keeps regions sorted - by path. -- A hop-less ANNOUNCE_START cannot run before the router drives sessions, so - the new messages may land first and the ANNOUNCE layout switch with - [session integration](/quest/m1/cluster-routing/routes/session.md), as long as every lite-07 build - between them still interoperates with itself. - -## Related - -- [Routes and announces](/quest/m1/cluster-routing/routes/README.md) - the contract the wire carries -- [A job's exact route continues its claim's readers](/quest/m1/announce-takeover.md) - also edits lite-07's ANNOUNCE messages diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index 4734539e11e..aee0fb64fbf 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -137,7 +137,7 @@ const LITE_VERSIONS: &[lite::Version] = &[ const IETF_VERSIONS: &[ietf::Version] = &[ietf::Version::Draft18, ietf::Version::Draft20, ietf::Version::Draft22]; /// How many types [`lite_wire`] dispatches over. -const LITE_KINDS: u8 = 33; +const LITE_KINDS: u8 = 34; /// How many types [`ietf_wire`] dispatches over. const IETF_KINDS: u8 = 27; @@ -305,6 +305,7 @@ pub fn lite_wire(data: &[u8]) -> bool { 30 => roundtrip::, _>(rest, &mut Vec::new(), version), 31 => roundtrip::(rest, &mut Vec::new(), version), 32 => roundtrip::(rest, &mut Vec::new(), version), + 33 => roundtrip::(rest, &mut Vec::new(), version), _ => unreachable!("kind is taken modulo LITE_KINDS"), } } diff --git a/rs/moq-net/src/lite/announce.rs b/rs/moq-net/src/lite/announce.rs index 83c962eb072..0a42c8ec1fa 100644 --- a/rs/moq-net/src/lite/announce.rs +++ b/rs/moq-net/src/lite/announce.rs @@ -1,6 +1,13 @@ +use std::collections::BTreeMap; + use num_enum::{IntoPrimitive, TryFromPrimitive}; -use crate::{Hop, Hops, Instance, Path, coding::*, origin::Cost}; +use crate::{ + Hop, Hops, Instance, Path, + coding::*, + origin::Cost, + route::{self, Advert, Reach}, +}; use super::{ Message, Version, @@ -18,6 +25,12 @@ const ANNOUNCE_RESTART: u64 = 3; const SYNCED: u64 = 4; // lite-07: a new advertisement continues what a live one served beneath its prefix. const ANNOUNCE_TAKEOVER: u64 = 5; +// lite-07: the route layer. ROUTE_END also travels back as the echo. +const ROUTE_START: u64 = 6; +const ROUTE_UPDATE: u64 = 7; +const ROUTE_END: u64 = 8; +const ANNOUNCE_FLOOR: u64 = 9; +const ROUTE_REQUEST: u64 = 10; /// Whether the negotiated version can update a live advertisement's metadata. On lite-05 an /// update travels as a duplicate ANNOUNCE (a second `active` for an already-announced path); on @@ -95,6 +108,108 @@ pub enum AnnounceBroadcast { Skipped, /// The initial set has been delivered (lite-07+). Synced, + /// ROUTE_START (lite-07+): the sender has a route to the origin node `node`, and + /// assigns it the next Route ID. + RouteStart { node: route::Id, advert: Advert }, + /// ROUTE_UPDATE (lite-07+): the route with this id changed, or is held. + RouteUpdate { id: u64, reach: Reach }, + /// ROUTE_END (lite-07+): the sender let go of the route with this id, and of every + /// advertisement under it. The receiver echoes it back. + RouteEnd { id: u64 }, + /// ANNOUNCE_FLOOR (lite-07+): closes the batch of advertisements under the route + /// with this id, listing every region the sender certifies for it: a suffix of the + /// requested prefix and the origin revision as of which it is complete. + Floor { id: u64, regions: BTreeMap }, +} + +/// Refuse what no node advertises: node 0, and the last varint as a seqno or metric, +/// which mean nothing and held. +fn valid_start(node: route::Id, advert: &Advert) -> bool { + node != 0 && advert.seqno != route::MAX && advert.metric != route::MAX +} + +fn encode_advert(w: &mut Encoder<'_>, advert: &Advert) -> Result<(), EncodeError> { + w.varint(advert.seqno)?; + w.varint(advert.metric)?; + w.bool(advert.down_only); + Ok(()) +} + +/// Encode a ROUTE_UPDATE body after its id. A held route carries a zero seqno and bit. +fn encode_reach(w: &mut Encoder<'_>, reach: &Reach) -> Result<(), EncodeError> { + match reach { + Reach::Finite(advert) if advert.seqno == route::MAX || advert.metric == route::MAX => { + Err(EncodeError::InvalidState) + } + Reach::Finite(advert) => encode_advert(w, advert), + Reach::Held => encode_advert( + w, + &Advert { + seqno: 0, + metric: route::MAX, + down_only: false, + }, + ), + } +} + +fn decode_advert(r: &mut Decoder<'_>) -> Result { + Ok(Advert { + seqno: r.varint()?, + metric: r.varint()?, + down_only: r.bool()?, + }) +} + +fn decode_reach(r: &mut Decoder<'_>) -> Result { + let advert = decode_advert(r)?; + if advert.metric != route::MAX { + if advert.seqno == route::MAX { + return Err(DecodeError::InvalidValue); + } + return Ok(Reach::Finite(advert)); + } + if advert.seqno != 0 || advert.down_only { + return Err(DecodeError::InvalidValue); + } + Ok(Reach::Held) +} + +fn encode_floor( + w: &mut Encoder<'_>, + version: Version, + id: u64, + regions: &BTreeMap, +) -> Result<(), EncodeError> { + w.varint(id)?; + w.varint(regions.len() as u64)?; + for (region, rev) in regions { + if *rev == route::MAX { + return Err(EncodeError::InvalidState); + } + region.encode(w, version)?; + w.varint(*rev)?; + } + Ok(()) +} + +fn decode_floor(r: &mut Decoder<'_>, version: Version) -> Result { + let id = r.varint()?; + let count = r.varint()?; + // Each region takes at least a length byte and a revision byte, so the body bounds + // the count before anything is read. + if count > (r.remaining() / 2) as u64 { + return Err(DecodeError::Short); + } + let mut regions = BTreeMap::new(); + for _ in 0..count { + let region = Path::decode(r, version)?; + let rev = r.varint()?; + if rev == route::MAX || regions.insert(region, rev).is_some() { + return Err(DecodeError::InvalidValue); + } + } + Ok(AnnounceBroadcast::Floor { id, regions }) } /// A path suffix as it travels: the first `keep` segments of a base announcement's @@ -244,6 +359,13 @@ impl Encode for AnnounceBroadcast { Self::Takeover { .. } if version.has_announce_takeover() => ANNOUNCE_TAKEOVER, // Older versions send a start instead. Self::Takeover { .. } => return Err(EncodeError::Version), + Self::RouteStart { .. } if version.has_routes() => ROUTE_START, + Self::RouteUpdate { .. } if version.has_routes() => ROUTE_UPDATE, + Self::RouteEnd { .. } if version.has_routes() => ROUTE_END, + Self::Floor { .. } if version.has_routes() => ANNOUNCE_FLOOR, + Self::RouteStart { .. } | Self::RouteUpdate { .. } | Self::RouteEnd { .. } | Self::Floor { .. } => { + return Err(EncodeError::Version); + } // The pre-lite-06 path-form retraction has no place on lite-06. Self::Ended { .. } => return Err(EncodeError::Version), // Decode-only: an unknown type is never sent. @@ -298,6 +420,19 @@ impl Encode for AnnounceBroadcast { cost.encode(w, version)?; } Self::Synced => {} + Self::RouteStart { node, advert } => { + if !valid_start(*node, advert) { + return Err(EncodeError::InvalidState); + } + w.varint(*node)?; + encode_advert(w, advert)?; + } + Self::RouteUpdate { id, reach } => { + w.varint(*id)?; + encode_reach(w, reach)?; + } + Self::RouteEnd { id } => w.varint(*id)?, + Self::Floor { id, regions } => encode_floor(w, version, *id, regions)?, Self::Ended { .. } | Self::Skipped => unreachable!("refused above"), } if w.since(&prefix) > MAX_MESSAGE_SIZE { @@ -332,7 +467,11 @@ impl Encode for AnnounceBroadcast { | Self::Restart { .. } | Self::Takeover { .. } | Self::Skipped - | Self::Synced => { + | Self::Synced + | Self::RouteStart { .. } + | Self::RouteUpdate { .. } + | Self::RouteEnd { .. } + | Self::Floor { .. } => { return Err(EncodeError::Version); } } @@ -396,6 +535,20 @@ impl<'a> Decode<'a, Version> for AnnounceBroadcast { cost: Cost::decode(&mut body, version)?, } } + ROUTE_START if version.has_routes() => { + let node = body.varint()?; + let advert = decode_advert(&mut body)?; + if !valid_start(node, &advert) { + return Err(DecodeError::InvalidValue); + } + Self::RouteStart { node, advert } + } + ROUTE_UPDATE if version.has_routes() => Self::RouteUpdate { + id: body.varint()?, + reach: decode_reach(&mut body)?, + }, + ROUTE_END if version.has_routes() => Self::RouteEnd { id: body.varint()? }, + ANNOUNCE_FLOOR if version.has_routes() => decode_floor(&mut body, version)?, // Unknown types are skipped by length so an earlier Lite06 build // negotiating the same ALPN does not kill the announce stream. _ => { @@ -529,6 +682,71 @@ impl<'a> Message<'a> for AnnounceRequest { } } +/// A message the subscriber sends on the Announce Stream after its ANNOUNCE_REQUEST +/// (lite-07+), toward the publisher that sent the routes. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum AnnounceReply { + /// ROUTE_REQUEST: asks for a route to `node`, as the publisher named it, at least + /// as new as `seqno`. + RouteRequest { node: route::Id, seqno: route::Seqno }, + /// The echo of the ROUTE_END that retired this Route ID, sent once the subscriber + /// dropped the route. + RouteEnd { id: u64 }, +} + +impl Encode for AnnounceReply { + fn encode(&self, w: &mut Encoder<'_>, version: Version) -> Result<(), EncodeError> { + w.control = Some(crate::stats::Message::Announce); + if !version.has_routes() { + return Err(EncodeError::Version); + } + w.varint(match self { + Self::RouteRequest { .. } => ROUTE_REQUEST, + Self::RouteEnd { .. } => ROUTE_END, + })?; + let prefix = w.prefix_varint(); + match self { + Self::RouteRequest { node: 0, .. } => return Err(EncodeError::InvalidState), + Self::RouteRequest { node, seqno } => { + w.varint(*node)?; + w.varint(*seqno)?; + } + Self::RouteEnd { id } => w.varint(*id)?, + } + w.fill(prefix) + } +} + +impl<'a> Decode<'a, Version> for AnnounceReply { + fn decode(buf: &mut Decoder<'a>, version: Version) -> Result { + buf.control = Some(crate::stats::Message::Announce); + if !version.has_routes() { + return Err(DecodeError::Version); + } + let typ = buf.varint()?; + let size = decode_size(buf, MAX_MESSAGE_SIZE)?; + let mut body = buf.sub(size)?; + let msg = (|| { + Ok(match typ { + ROUTE_REQUEST => match body.varint()? { + 0 => return Err(DecodeError::InvalidValue), + node => Self::RouteRequest { + node, + seqno: body.varint()?, + }, + }, + ROUTE_END => Self::RouteEnd { id: body.varint()? }, + typ => return Err(DecodeError::InvalidMessage(typ)), + }) + })() + .map_err(DecodeError::complete)?; + if !body.is_empty() { + return Err(DecodeError::Long); + } + Ok(msg) + } +} + /// Send by the publisher, used to determine the message that follows. #[derive(Clone, Copy, Debug, IntoPrimitive, TryFromPrimitive)] #[repr(u8)] @@ -1279,4 +1497,246 @@ mod tests { assert!(input.is_empty()); } } + + fn advert(seqno: u64, metric: u64, down_only: bool) -> Advert { + Advert { + seqno, + metric, + down_only, + } + } + + /// Encode a typed lite-07 announce message from its raw body, bypassing the encoder's checks. + fn forge(typ: u64, body: impl FnOnce(&mut Encoder<'_>)) -> Vec { + let version = Version::Lite07; + let mut inner = Vec::new(); + body(&mut Encoder::new(&mut inner, version.into())); + let mut buf = Vec::new(); + let mut w = Encoder::new(&mut buf, version.into()); + w.varint(typ).unwrap(); + w.varint(inner.len() as u64).unwrap(); + buf.extend_from_slice(&inner); + buf + } + + fn decode_broadcast(buf: &[u8], version: Version) -> Result { + let mut slice = buf; + crate::coding::decode_buf(&mut slice, version, |r, v| AnnounceBroadcast::decode(r, v)) + } + + fn decode_reply(buf: &[u8], version: Version) -> Result { + let mut slice = buf; + crate::coding::decode_buf(&mut slice, version, |r, v| AnnounceReply::decode(r, v)) + } + + #[test] + fn route_messages_round_trip_on_lite07() { + let regions = |list: &[(&'static str, u64)]| { + list.iter() + .map(|(path, rev)| (Path::from_static(path), *rev)) + .collect::>() + }; + for msg in [ + AnnounceBroadcast::RouteStart { + node: 0x7a3f, + advert: advert(41, 12, false), + }, + AnnounceBroadcast::RouteStart { + node: route::QUALIFIED | 9, + advert: advert(route::MAX - 1, route::MAX - 1, true), + }, + AnnounceBroadcast::RouteUpdate { + id: 0, + reach: Reach::Finite(advert(42, 15, true)), + }, + AnnounceBroadcast::RouteUpdate { id: 3, reach: Reach::Held }, + AnnounceBroadcast::RouteEnd { id: 0 }, + AnnounceBroadcast::Floor { + id: 0, + regions: regions(&[]), + }, + AnnounceBroadcast::Floor { + id: 1, + regions: regions(&[("live/a", 3), ("live/b", 5), ("", 0)]), + }, + ] { + assert_eq!(broadcast_round_trip(&msg, Version::Lite07), msg); + } + } + + /// The common floor certifies the whole interest: one byte over (Route ID, revision). + #[test] + fn a_whole_interest_floor_is_one_byte_over_its_revision() { + let mut buf = Vec::new(); + AnnounceBroadcast::Floor { + id: 0, + regions: [(Path::from_static(""), 7)].into(), + } + .encode(&mut Encoder::new(&mut buf, Version::Lite07.into()), Version::Lite07) + .unwrap(); + assert_eq!(buf, vec![9, 4, 0, 1, 0, 7]); + } + + #[test] + fn route_messages_need_lite07() { + for msg in [ + AnnounceBroadcast::RouteStart { + node: 1, + advert: advert(0, 1, false), + }, + AnnounceBroadcast::RouteUpdate { id: 0, reach: Reach::Held }, + AnnounceBroadcast::RouteEnd { id: 0 }, + AnnounceBroadcast::Floor { + id: 0, + regions: BTreeMap::new(), + }, + ] { + for version in [Version::Lite05, Version::Lite06] { + let mut buf = Vec::new(); + let res = msg.encode(&mut Encoder::new(&mut buf, version.into()), version); + assert!(matches!(res, Err(EncodeError::Version)), "{msg:?} on {version:?}"); + } + } + // A lite-06 receiver skips the type like any other it does not know. + let wire = forge(ROUTE_END, |w| w.varint(0u64).unwrap()); + assert_eq!(decode_broadcast(&wire, Version::Lite06).unwrap(), AnnounceBroadcast::Skipped); + + for reply in [ + AnnounceReply::RouteRequest { node: 1, seqno: 0 }, + AnnounceReply::RouteEnd { id: 0 }, + ] { + let mut buf = Vec::new(); + let res = reply.encode(&mut Encoder::new(&mut buf, Version::Lite06.into()), Version::Lite06); + assert!(matches!(res, Err(EncodeError::Version))); + } + assert!(matches!(decode_reply(&wire, Version::Lite06), Err(DecodeError::Version))); + } + + /// What no node advertises is refused on both ends: node 0, the last varint as a + /// seqno or a starting metric, and a held route carrying a seqno or the bit. + #[test] + fn route_messages_refuse_what_no_node_advertises() { + let refused = [ + AnnounceBroadcast::RouteStart { + node: 0, + advert: advert(0, 1, false), + }, + AnnounceBroadcast::RouteStart { + node: 1, + advert: advert(route::MAX, 1, false), + }, + AnnounceBroadcast::RouteStart { + node: 1, + advert: advert(0, route::MAX, false), + }, + AnnounceBroadcast::RouteUpdate { + id: 0, + reach: Reach::Finite(advert(route::MAX, 1, false)), + }, + AnnounceBroadcast::RouteUpdate { + id: 0, + reach: Reach::Finite(advert(0, route::MAX, false)), + }, + AnnounceBroadcast::Floor { + id: 0, + regions: [(Path::from_static(""), route::MAX)].into(), + }, + ]; + for msg in refused { + let mut buf = Vec::new(); + let res = msg.encode(&mut Encoder::new(&mut buf, Version::Lite07.into()), Version::Lite07); + assert!(matches!(res, Err(EncodeError::InvalidState)), "{msg:?}"); + } + + let advert = |node: u64, seqno: u64, metric: u64, down_only: u8| { + move |w: &mut Encoder<'_>| { + w.varint(node).unwrap(); + w.varint(seqno).unwrap(); + w.varint(metric).unwrap(); + w.u8(down_only); + } + }; + let malformed = [ + forge(ROUTE_START, advert(0, 0, 1, 0)), + forge(ROUTE_START, advert(1, route::MAX, 1, 0)), + forge(ROUTE_START, advert(1, 0, route::MAX, 0)), + forge(ROUTE_START, advert(1, 0, 1, 2)), + // ROUTE_UPDATE's leading field is the Route ID, so 0 is a valid id here. + forge(ROUTE_UPDATE, advert(0, route::MAX, 1, 0)), + forge(ROUTE_UPDATE, advert(0, 1, route::MAX, 0)), + forge(ROUTE_UPDATE, advert(0, 0, route::MAX, 1)), + forge(ANNOUNCE_FLOOR, |w| { + w.varint(0u64).unwrap(); + w.varint(2u64).unwrap(); + for _ in 0..2 { + Path::from_static("live").encode(w, Version::Lite07).unwrap(); + w.varint(1u64).unwrap(); + } + }), + forge(ANNOUNCE_FLOOR, |w| { + w.varint(0u64).unwrap(); + w.varint(1u64).unwrap(); + Path::from_static("").encode(w, Version::Lite07).unwrap(); + w.varint(route::MAX).unwrap(); + }), + // A count the body cannot hold fails before any region is read. + forge(ANNOUNCE_FLOOR, |w| { + w.varint(0u64).unwrap(); + w.varint(u32::MAX.into()).unwrap(); + }), + // Trailing bytes after a complete body. + forge(ROUTE_END, |w| { + w.varint(0u64).unwrap(); + w.u8(0); + }), + ]; + for wire in malformed { + assert!(decode_broadcast(&wire, Version::Lite07).is_err(), "{wire:?}"); + } + } + + #[test] + fn announce_reply_round_trips_on_lite07() { + for reply in [ + AnnounceReply::RouteRequest { + node: 0x7a3f, + seqno: 42, + }, + // A request may name the last seqno: only a rotation answers it. + AnnounceReply::RouteRequest { + node: route::QUALIFIED | 1, + seqno: route::MAX, + }, + AnnounceReply::RouteEnd { id: 0 }, + ] { + let mut buf = Vec::new(); + reply + .encode(&mut Encoder::new(&mut buf, Version::Lite07.into()), Version::Lite07) + .unwrap(); + assert_eq!(decode_reply(&buf, Version::Lite07).unwrap(), reply); + } + } + + #[test] + fn announce_reply_refuses_node_zero_and_unknown_types() { + let mut buf = Vec::new(); + let res = AnnounceReply::RouteRequest { node: 0, seqno: 1 } + .encode(&mut Encoder::new(&mut buf, Version::Lite07.into()), Version::Lite07); + assert!(matches!(res, Err(EncodeError::InvalidState))); + + let zero = forge(ROUTE_REQUEST, |w| { + w.varint(0u64).unwrap(); + w.varint(1u64).unwrap(); + }); + assert!(matches!( + decode_reply(&zero, Version::Lite07), + Err(DecodeError::InvalidValue) + )); + // Only routes travel back: an announcement type is not a reply. + let start = forge(ROUTE_START, |w| w.varint(1u64).unwrap()); + assert!(matches!( + decode_reply(&start, Version::Lite07), + Err(DecodeError::InvalidMessage(ROUTE_START)) + )); + } } diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 58f76ed899d..42798cf109c 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -597,6 +597,12 @@ impl Subscriber { )?; } lite::AnnounceBroadcast::Skipped | lite::AnnounceBroadcast::Synced => {} + // Nothing drives the route layer on this stream yet, so a peer relying on it + // cannot be served. + lite::AnnounceBroadcast::RouteStart { .. } + | lite::AnnounceBroadcast::RouteUpdate { .. } + | lite::AnnounceBroadcast::RouteEnd { .. } + | lite::AnnounceBroadcast::Floor { .. } => return Err(Error::ProtocolViolation), } Ok(()) } diff --git a/rs/moq-net/src/lite/version.rs b/rs/moq-net/src/lite/version.rs index 4732ac9958d..880dcda3a51 100644 --- a/rs/moq-net/src/lite/version.rs +++ b/rs/moq-net/src/lite/version.rs @@ -337,6 +337,16 @@ impl Version { ) } + /// Whether the announce stream carries the route layer: ROUTE_START, ROUTE_UPDATE, + /// ROUTE_END, and ANNOUNCE_FLOOR toward the subscriber, and ROUTE_REQUEST and the + /// ROUTE_END echo back. Added in lite-07. + pub(crate) fn has_routes(self) -> bool { + !matches!( + self, + Self::Lite01 | Self::Lite02 | Self::Lite03 | Self::Lite04 | Self::Lite05 | Self::Lite06 + ) + } + /// Whether the announce stream has ANNOUNCE_TAKEOVER: a new advertisement that /// continues what a live one served beneath its prefix. Added in lite-07. Older /// versions send an ANNOUNCE_START, which the receiver treats as another instance. From 527f6bcbcd6094ed62986f8101d5d42ec71f8c23 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 22:56:41 -0800 Subject: [PATCH 05/11] fix(net): format, seed the reply fuzz kind, and allow the undriven reply codec Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/fuzz.rs | 5 +++-- rs/moq-net/src/lite/announce.rs | 33 ++++++++++++++++++++++++++++----- 2 files changed, 31 insertions(+), 7 deletions(-) diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index aee0fb64fbf..3b05901eaa0 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -1265,8 +1265,9 @@ pub fn seeds() -> Vec { } } - // A typed empty TRACKS_SYNCED and AUTH_OK reply need nonuniform envelope bytes. - for (kind, body) in [(23, &[4, 0][..]), (27, &[0, 3, 0, 0, 0][..])] { + // A typed empty TRACKS_SYNCED, AUTH_OK reply, and ROUTE_END echo need nonuniform + // envelope bytes. + for (kind, body) in [(23, &[4, 0][..]), (27, &[0, 3, 0, 0, 0][..]), (33, &[8, 1, 0][..])] { let mut data = vec![(LITE_VERSIONS.len() - 1) as u8, kind]; data.extend_from_slice(body); seeds.push(Seed { diff --git a/rs/moq-net/src/lite/announce.rs b/rs/moq-net/src/lite/announce.rs index 0a42c8ec1fa..4300bae4163 100644 --- a/rs/moq-net/src/lite/announce.rs +++ b/rs/moq-net/src/lite/announce.rs @@ -30,6 +30,10 @@ const ROUTE_START: u64 = 6; const ROUTE_UPDATE: u64 = 7; const ROUTE_END: u64 = 8; const ANNOUNCE_FLOOR: u64 = 9; +#[cfg_attr( + not(any(test, feature = "fuzz")), + allow(dead_code, reason = "the lite-07 session does not drive the route layer yet") +)] const ROUTE_REQUEST: u64 = 10; /// Whether the negotiated version can update a live advertisement's metadata. On lite-05 an @@ -119,7 +123,10 @@ pub enum AnnounceBroadcast { /// ANNOUNCE_FLOOR (lite-07+): closes the batch of advertisements under the route /// with this id, listing every region the sender certifies for it: a suffix of the /// requested prefix and the origin revision as of which it is complete. - Floor { id: u64, regions: BTreeMap }, + Floor { + id: u64, + regions: BTreeMap, + }, } /// Refuse what no node advertises: node 0, and the last varint as a seqno or metric, @@ -684,6 +691,10 @@ impl<'a> Message<'a> for AnnounceRequest { /// A message the subscriber sends on the Announce Stream after its ANNOUNCE_REQUEST /// (lite-07+), toward the publisher that sent the routes. +#[cfg_attr( + not(any(test, feature = "fuzz")), + allow(dead_code, reason = "the lite-07 session does not drive the route layer yet") +)] #[derive(Clone, Debug, PartialEq, Eq)] pub enum AnnounceReply { /// ROUTE_REQUEST: asks for a route to `node`, as the publisher named it, at least @@ -1549,7 +1560,10 @@ mod tests { id: 0, reach: Reach::Finite(advert(42, 15, true)), }, - AnnounceBroadcast::RouteUpdate { id: 3, reach: Reach::Held }, + AnnounceBroadcast::RouteUpdate { + id: 3, + reach: Reach::Held, + }, AnnounceBroadcast::RouteEnd { id: 0 }, AnnounceBroadcast::Floor { id: 0, @@ -1584,7 +1598,10 @@ mod tests { node: 1, advert: advert(0, 1, false), }, - AnnounceBroadcast::RouteUpdate { id: 0, reach: Reach::Held }, + AnnounceBroadcast::RouteUpdate { + id: 0, + reach: Reach::Held, + }, AnnounceBroadcast::RouteEnd { id: 0 }, AnnounceBroadcast::Floor { id: 0, @@ -1599,7 +1616,10 @@ mod tests { } // A lite-06 receiver skips the type like any other it does not know. let wire = forge(ROUTE_END, |w| w.varint(0u64).unwrap()); - assert_eq!(decode_broadcast(&wire, Version::Lite06).unwrap(), AnnounceBroadcast::Skipped); + assert_eq!( + decode_broadcast(&wire, Version::Lite06).unwrap(), + AnnounceBroadcast::Skipped + ); for reply in [ AnnounceReply::RouteRequest { node: 1, seqno: 0 }, @@ -1609,7 +1629,10 @@ mod tests { let res = reply.encode(&mut Encoder::new(&mut buf, Version::Lite06.into()), Version::Lite06); assert!(matches!(res, Err(EncodeError::Version))); } - assert!(matches!(decode_reply(&wire, Version::Lite06), Err(DecodeError::Version))); + assert!(matches!( + decode_reply(&wire, Version::Lite06), + Err(DecodeError::Version) + )); } /// What no node advertises is refused on both ends: node 0, the last varint as a From 944842925e3fde169640ce96b5089bc8407b9910 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 23:00:01 -0800 Subject: [PATCH 06/11] fix(drafts): the feasibility distance keeps the lowest metric within a seqno Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 47e48a2ad45..ed895d4c8e3 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -480,10 +480,10 @@ A receiver adds the session's link cost (see [Cost Parameter](#cost-parameter)) Each link adds at least 1, so a route's metric strictly increases away from its origin. A metric that reaches 2^64-1 makes the route unusable. -A node keeps a feasibility distance per Node ID: the Seqno and metric of the last finite route it advertised. +A node keeps a feasibility distance per Node ID: the highest Seqno of any finite route it advertised, and the lowest metric it advertised at that Seqno, as in [RFC8966] Section 3.5.1. A route is feasible when its Seqno is greater than the distance's, or equal with a lower metric. A node selects, per Node ID, among its feasible routes that are not held: a session that has not received a GOAWAY, then the highest local priority, then the lowest metric. -It MUST NOT select an infeasible route, and never raises a distance to accept one. +It MUST NOT select an infeasible route, and a distance only ever falls within a Seqno, so advertising a longer route never raises it. A node MAY mark a session upstream. A route is down-only when it arrived on an upstream session or with its Down Only bit set; a node sends a down-only route with the bit set, and never on an upstream session. From c5d21a32c4a070c8363180b2738c0bde61bb254c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 08:55:50 -0800 Subject: [PATCH 07/11] docs(drafts): align route request node scope and record floor limit decision Co-Authored-By: GPT-6.1 Sol --- drafts/draft-lcurley-moq-lite.md | 2 +- quest/m1/cluster-routing/routes/README.md | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index fe1a3bfc06e..9ad72cd5d7b 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -1178,7 +1178,7 @@ ROUTE_REQUEST Message { Set to 0xA to indicate a ROUTE_REQUEST message. **Node ID**: -The origin node, as the publisher named it on this stream. +The origin node, as the publisher names it on this session. 0 is a PROTOCOL_VIOLATION. **Seqno**: diff --git a/quest/m1/cluster-routing/routes/README.md b/quest/m1/cluster-routing/routes/README.md index 8e7f24ed5b1..28d2a0e42c1 100644 --- a/quest/m1/cluster-routing/routes/README.md +++ b/quest/m1/cluster-routing/routes/README.md @@ -512,6 +512,7 @@ benchmarks. ## Waiting +- Maintainer: a floor replaces the whole certified-region list, but that list can exceed the 65,535-byte message limit (for example, sixteen 4,096-byte regions). Options: make a floor batch span messages with an atomic completion marker, or constrain certification to a representable projection. Recommend multipart atomic floors rather than an arbitrary region cap; the continuation wire shape needs a decision before implementation. - Maintainer: two original claims (`live` and `live/a`) clamp to the same prefix for interest `live/a/x`, violating the one-claim-per-route/prefix rule and losing the key needed by cross-stream floors. Options: carry the original claim prefix (including ancestors of the interest), or define a projection and reconciliation rule that selects one covering claim. Recommend preserving original claim identity, with a grant-safe ancestor encoding; this changes the wire shape and needs your choice. - Maintainer: a Hidden=0 floor currently certifies absence for hidden claims delivered on a Hidden=1 stream. Options: make certification depend on each stream's requested filter, or isolate differently filtered views entirely. Recommend using the existing request filter to restrict negative evidence, without another field on the wire. Confirm this certification rule alongside the claim identity decision. From 9f49958f0222a6ecd36e5d1b97d94f8471d06587 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 10:49:03 -0800 Subject: [PATCH 08/11] feat(net)!: make ANNOUNCE_FLOOR a marker and collapse clamped claims ANNOUNCE_FLOOR is now {Route ID, Revision}: it certifies the stream's claims for the route across the whole requested prefix, or nothing when the Revision is 2^64-1. A route presents only its most specific claim covering the request, and a floor certifies only what its own request shows. Restores the reload close-drain test. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 38 ++---- js/net/src/connection/reload.test.ts | 3 +- quest/m1/cluster-routing/routes/README.md | 34 +++--- quest/m1/cluster-routing/routes/acceptance.md | 4 +- quest/m1/cluster-routing/routes/js.md | 2 +- quest/m1/cluster-routing/routes/session.md | 10 ++ quest/m2/route-interest-prefixes.md | 14 +-- rs/moq-net/src/lite/announce.rs | 113 ++++++------------ 8 files changed, 89 insertions(+), 129 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 9ad72cd5d7b..d7efee3e27b 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -430,7 +430,8 @@ When the stream is closed, the subscriber MUST assume that every route and claim A broadcast path has at most 32 segments and at most 4,096 bytes; a receiver MUST close the session with a PROTOCOL_VIOLATION on a longer one, including one resolved by [Compression](#announce-compression). A claim covers a path when its prefix is a leading run of the path's segments; matching is per path segment, so a prefix never matches half a segment, and equality is byte-by-byte within each segment. -A publisher answering a request stream presents each of its claims clamped to the intersection with the requested prefix: a claim above the request's prefix appears as the request prefix itself (an empty suffix), which is exactly the covered set the subscriber may see. +A publisher answering a request stream presents, per route, each claim inside the requested prefix and only the most specific claim at or above it, at the requested prefix itself (an empty suffix), which is exactly the covered set the subscriber may see. +When that most specific covering claim changes, such as when it is withdrawn and a broader one still covers the request, the publisher moves the presented claim to the new one with an ANNOUNCE_UPDATE or ANNOUNCE_RESTART, and ends it only when no claim of the route covers the requested prefix. There MAY be multiple Announce Streams, potentially containing overlapping prefixes, that get their own ANNOUNCE_OK and messages. #### Floors {#floors} @@ -438,15 +439,15 @@ Each origin node numbers the changes to its own claims with a Revision, starting A stream carries a route's claims in batches: the ANNOUNCE messages under the route since its last ANNOUNCE_FLOOR, closed by the next one. The publisher sends a ROUTE_START before any claim under it, and SYNCED only once every route it sent has a floor. -The floor lists every region the publisher certifies for the route: a suffix of the requested prefix, and the Revision as of which the stream's claims for the route within that region are complete. -Each list replaces the previous one, so a region left out is no longer certified. -A floor certifies the claims overlapping its region: those inside it, and those above it. -A publisher that certifies anything for a route sends a floor for it on every stream, one with no regions when the request meets none of its regions. +A floor names a Revision and certifies that the stream's claims for the route, across the whole requested prefix, are the origin's claims at that Revision as the request presents them (see [Announce](#announce)). +It certifies only what its own request shows: a stream whose ANNOUNCE_REQUEST did not set `Hidden` certifies nothing about hidden claims. +A publisher that does not know the route's claims across the whole requested prefix at one Revision, such as a relay that knows them only beneath part of it, still closes each batch, with a floor that certifies nothing. The subscriber applies a batch whole, once its floor arrives. -A receiver that hears one origin's claims on several streams merges them claim by claim: whichever stream's floor certifying the claim carries the higher Revision decides whether it exists. +A receiver that hears one origin's claims on several streams merges them claim by claim, each at the prefix its stream presents it: of the streams whose floor certifies the claim, the one with the highest Revision decides whether it exists, and a claim no floor certifies exists while some stream carries it. An older floor never revives a claim whose withdrawal a newer one delivered. -While a receiver has a usable route to the origin and every peer whose last finite route to it is feasible here has answered the current request, it lets go of a region none of those peers certifies: its claims go offline, which is not an ANNOUNCE_END. +A floor that certifies nothing proves no claim absent: the stream's claims stay until its ANNOUNCE_END or ROUTE_END. +While a receiver has a usable route to the origin and every peer whose last finite route to it is feasible here has answered the current request, it lets go of a claim none of those peers certifies or carries: it goes offline, which is not an ANNOUNCE_END. #### Compression {#announce-compression} An ANNOUNCE_START or ANNOUNCE_TAKEOVER MAY copy the head of its path from a live claim on the same stream, its base. @@ -1350,19 +1351,13 @@ A relay forwards one only when the subscriber was last sent that predecessor, an ## ANNOUNCE_FLOOR {#announce-floor} -A publisher sends an ANNOUNCE_FLOOR message to close a batch of a route's claims, listing every region it certifies for the route (see [Floors](#floors)). +A publisher sends an ANNOUNCE_FLOOR message to close a batch of a route's claims (see [Floors](#floors)). ~~~ ANNOUNCE_FLOOR Message { Type (i) = 0x9 Message Length (i) Route ID (i), - Region Count (i), - Region (..) ..., -} - -Region { - Region Suffix (s), Revision (i), } ~~~ @@ -1373,16 +1368,8 @@ Set to 0x9 to indicate an ANNOUNCE_FLOOR message. **Route ID**: The ordinal implicitly assigned by a prior ROUTE_START on this stream. -**Region Count**: -The number of regions that follow, 0 when the publisher certifies nothing within the requested prefix. - -**Region Suffix**: -The region, combined with the requested prefix like a Claim Prefix Suffix; empty for the whole requested prefix. -Regions are in no particular order, and a repeated one is a PROTOCOL_VIOLATION. - **Revision**: -The origin's Revision as of which the stream's claims for the route within the region are complete. -2^64-1 is a PROTOCOL_VIOLATION. +The origin's Revision at which the floor certifies the stream's claims for the route, or 2^64-1, which no origin advertises (see [Routing](#routing)), when it certifies nothing. ## SUBSCRIBE @@ -2124,7 +2111,8 @@ The `Message Length` describes the payload size on the wire. - Split routing into two layers on the Announce Stream. ROUTE_START (0x6), ROUTE_UPDATE (0x7), and ROUTE_END (0x8) advertise how the publisher reaches an origin node, named by a Node ID, with a seqno, an additive metric, and a down-only bit, and loops are prevented by Babel's feasibility condition. Every ANNOUNCE_START and ANNOUNCE_TAKEOVER names its route's Route ID in place of the Hop ID list, so a link change sends one ROUTE_UPDATE per origin rather than an update per claim. Removed the Hop ID list, ANNOUNCE_OK's Hop ID, and the SETUP Hop parameter (0x5). - A ROUTE_UPDATE with a Metric of 2^64-1 holds a route while it recovers; ROUTE_END lets it go. Neither withdraws the origin's claims, and the subscriber echoes every ROUTE_END back. - Added ROUTE_REQUEST (0xA), sent by the subscriber on the Announce Stream to ask for a route with a newer seqno when every route it has is infeasible. -- Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the regions it certifies and the origin Revision each is complete as of, so a stale copy never revives a claim whose withdrawal a newer one delivered. +- Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the origin Revision at which the stream's claims across the requested prefix are certified, or 2^64-1 to certify nothing, so a stale copy never revives a claim whose withdrawal a newer one delivered. A floor certifies only the claims its own request shows. +- A route presents only its most specific claim covering the requested prefix, moved to the next covering claim when it is withdrawn rather than ended. - Added `Node ID` to TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS: the origin node the first hop selected, followed by every later hop. - Replaced Warm and Cold Route Cost with the origin's Announce Cost, unchanged in transit. ANNOUNCE_UPDATE carries only the Announce Cost. - The Cost parameter must be at least 1. @@ -2350,7 +2338,7 @@ A Node ID (see [Routing](#routing)) is opaque and fresh for each restart: it tel A Node ID is declared, not proven, so a peer can forge another origin's Node ID or advertise a better route than it has. Qualifying the Node IDs of a peer with a scoped publish grant confines such a forgery to the claims inside that grant; an unscoped peer is trusted with every origin. ## Resource Exhaustion -A peer can open many streams (subscriptions, announcements, fetches), request large announce prefixes, or advertise broad claims. Implementations SHOULD bound the number of concurrent subscriptions, announce matches, routes, floor regions, and cached groups; an endpoint whose peer goes past its bound on subscriptions or announcements SHOULD close the session with TOO_MANY_REQUESTS, and SHOULD rely on QUIC flow control and stream limits to backpressure a misbehaving peer (see [ANNOUNCE_REQUEST](#announce-request)). Expiration (see [Expiration](#expiration)) bounds how long stale groups consume memory and flow control. A broad claim invites a request for any covered path, each of which may start work: an advertiser SHOULD bound the work it starts, withdrawing its claim before it runs out, and every refusal is terminal, so a flood of requests costs the mesh one round trip each rather than a search (see [Resolution](#resolution)). +A peer can open many streams (subscriptions, announcements, fetches), request large announce prefixes, or advertise broad claims. Implementations SHOULD bound the number of concurrent subscriptions, announce matches, routes, and cached groups; an endpoint whose peer goes past its bound on subscriptions or announcements SHOULD close the session with TOO_MANY_REQUESTS, and SHOULD rely on QUIC flow control and stream limits to backpressure a misbehaving peer (see [ANNOUNCE_REQUEST](#announce-request)). Expiration (see [Expiration](#expiration)) bounds how long stale groups consume memory and flow control. A broad claim invites a request for any covered path, each of which may start work: an advertiser SHOULD bound the work it starts, withdrawing its claim before it runs out, and every refusal is terminal, so a flood of requests costs the mesh one round trip each rather than a search (see [Resolution](#resolution)). ## Datagram Injection Datagrams are routed to a subscription solely by Subscribe ID and carry no per-group authentication beyond that of the QUIC connection. On an unmodified QUIC/WebTransport connection this is sufficient, since datagrams are protected by the transport. A subscriber MUST silently drop any datagram with an unknown Subscribe ID (see [Datagrams](#datagrams)). diff --git a/js/net/src/connection/reload.test.ts b/js/net/src/connection/reload.test.ts index 3ff29e8961e..80e37e1e321 100644 --- a/js/net/src/connection/reload.test.ts +++ b/js/net/src/connection/reload.test.ts @@ -945,8 +945,7 @@ test("a retired GOAWAY predecessor still closes by its handover cap", async () = // A publication only a token's grant covers keeps that grant while the session drains, so // closing the loop ends the tokens after the drain rather than cutting the final group. -// Disabled: load-sensitive fin.closed assertion; investigate drain scheduling before restoring. -test.skip("closing the loop keeps the tokens until the session drains", async () => { +test("closing the loop keeps the tokens until the session drains", async () => { const original = globalThis.WebTransport; const clock = new Manual(); const installed = clock.install(); diff --git a/quest/m1/cluster-routing/routes/README.md b/quest/m1/cluster-routing/routes/README.md index 28d2a0e42c1..49ce72fd0fa 100644 --- a/quest/m1/cluster-routing/routes/README.md +++ b/quest/m1/cluster-routing/routes/README.md @@ -17,11 +17,14 @@ claim whose withdrawal is already known. 2. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **The request field is `Node ID (i)`, with 0 meaning "the publisher selects".** It is added to every request that resolves a path: TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS. Node ID 0 is reserved everywhere. 3. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **ROUTE_REQUEST names a Node ID, not a Route ID.** A request can then never race a ROUTE_END into a protocol violation. 4. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **A held ROUTE_UPDATE carries a seqno and down-only bit of 0.** Any other value is refused, rather than letting the fields carry no meaning. -5. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **Floor regions have no order, and a repeated region is a violation.** A sorted order would need Rust's `Path` ordering reproduced exactly in JS. Decoding into a map rejects duplicates at the same cost. +5. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **Floor regions have no order, and a repeated region is a violation.** A sorted order would need Rust's `Path` ordering reproduced exactly in JS. Decoding into a map rejects duplicates at the same cost. Superseded the same day by item 10, which removes regions. 6. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **The Hop setup parameter and ANNOUNCE_OK's Hop ID are dropped from lite-07.** Their only roles were path-vector exclusion and the implicit trailing hop. ANNOUNCE_OK reserves no body for a route-key proof: [moq-dev/moq#7937](https://github.com/moq-dev/moq/pull/7937), accepted the same day, proves the route key once per session in SETUP. The live hop-based codec still switches in session integration, as choice 1 requires. 7. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **An echo naming a Route ID that was never ended, or was already echoed, is a violation at the publisher.** The publisher can check this from its live and unacknowledged ids alone, with no growing state. 8. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **ANNOUNCE_TAKEOVER keeps its shape, with a Route ID in place of hops, and its predecessor must be under the same route.** [quest/m1/announce-takeover.md](https://github.com/moq-dev/moq/blob/main/quest/m1/announce-takeover.md) deletes the message, so this keeps the change minimal. 9. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **Unknown announce types stay skipped by length on lite-06+**, as before. An unknown reply type is refused. +10. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **ANNOUNCE_FLOOR is a marker, not a region list:** `{Route ID, Revision}`. It closes a batch and certifies the stream's claims for that route across the whole requested prefix as the origin's at that Revision. A publisher that cannot vouch for the whole requested prefix at one Revision still closes the batch, with a Revision of 2^64-1 that certifies nothing: the one Revision no origin advertises, as the infinite Metric marks a held route, so the common floor carries no flag and no offset. Regions, the replacing list, and the 65,535-byte overflow go away. Accepted cost: a relay with partial knowledge of an origin certifies nothing for that route, so a receiver lets its stale claims go only on ROUTE_END. Rejected: one region per message; a multipart atomic list. +11. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **Clamped claims collapse to the best one.** When several of a route's claims cover a request's prefix (`live` at cost 10 and `live/a` at cost 1, request `live/a/x`), the publisher presents only the most specific, and when it is withdrawn moves the presented claim to the next covering one with ANNOUNCE_UPDATE or ANNOUNCE_RESTART instead of ending it. No wire change; one live claim per prefix per route holds, and a receiver merges claims at the prefix each stream presents them. Rejected: carrying each claim's ancestor depth. +12. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **A floor certifies only what its own request shows.** A stream whose request did not set `Hidden` certifies nothing about hidden claims. No new wire field. Rejected: isolating differently filtered views. The route layer's core landed in @@ -50,7 +53,10 @@ Settled by the core, for review: interest overlaps it (`a_claim_above_an_interest_overlaps_it`). - Letting go of a region keeps the proof's rule: keep every still-certified region, or rebuild them all if a lost floor is newer than a kept one, a - collection whose replay is bounded like any other. + collection whose replay is bounded like any other. The core and the proof + keep per-region floors internally; the wire carries only item 10's + whole-interest marker, which [session integration](/quest/m1/cluster-routing/routes/session.md) + projects the view onto. - A rotated identity ends the old one with ROUTE_END on every stream rather than closing sessions, since the own identity is no longer proven per session. @@ -258,11 +264,13 @@ loss never reports an END: seqno leaves room for the successor a request names (`exhausted_counters_rotate_the_identity`). A session sends an origin's announcements in batches, each closed by its - floors: per region, the revision as of which the stream's announcements - for that route within the region are complete. A bare identity's region is - the stream's interest. A qualified key certifies only within the - grant scopes it arrived through, and one key can arrive through grants - whose pattern unions differ, so its view carries one floor per scope. + floor: the revision at which the stream's announcements for that route, + across the whole interest, are the origin's, or nothing when the sender + does not know them across the interest at one revision (Plan item 10). A + qualified key is known only within the grant scopes it arrived through, + and one key can arrive through grants whose pattern unions differ, so the + view inside a node carries one floor per scope, and a stream whose + interest is wider than those scopes certifies nothing. - A receiver keeps each session's last view, to apply the next batch, and merges every session's view into its own claim by claim: whichever view covers the claim with the newer floor decides. A floor certifies only the @@ -448,7 +456,8 @@ Decided with the maintainer 2026-10-16, from the handoff proof: which grows with publisher churn times uptime on every relay that saw an identity, about 16 bytes each, until a restart; and Babel's source GC timer, which assumes stale echoes are gone by then rather than proving it. -7. Scope coverage under one key: ✅ per-region floors. ANNOUNCE_FLOOR +7. Scope coverage under one key: ✅ per-region floors (superseded on the + wire 2026-11-05 by Plan item 10; the core keeps regions internally). ANNOUNCE_FLOOR names the region it certifies, a suffix of the stream's interest prefix (the scope met with the interest), each node tracks which regions each feasible neighbour's last view certifies, and a region none certifies @@ -461,7 +470,8 @@ Decided with the maintainer 2026-10-16, from the handoff proof: implementation, which leaves the proof's freshness claim false for scoped delivery. -8. Region-set framing: ✅ one ANNOUNCE_FLOOR per batch lists every region +8. Region-set framing (superseded 2026-11-05 by Plan item 10, a + whole-interest marker): ✅ one ANNOUNCE_FLOOR per batch lists every region the sender certifies for the route: (Route ID, count, then a region suffix and revision per region). The list replaces the last, so a sender stops certifying a region it lost by leaving it out, and a count of zero answers @@ -510,12 +520,6 @@ benchmarks. subscriptions through the radio mesh and the split refuses, and after a partition each island reaches the other through its uplink. -## Waiting - -- Maintainer: a floor replaces the whole certified-region list, but that list can exceed the 65,535-byte message limit (for example, sixteen 4,096-byte regions). Options: make a floor batch span messages with an atomic completion marker, or constrain certification to a representable projection. Recommend multipart atomic floors rather than an arbitrary region cap; the continuation wire shape needs a decision before implementation. -- Maintainer: two original claims (`live` and `live/a`) clamp to the same prefix for interest `live/a/x`, violating the one-claim-per-route/prefix rule and losing the key needed by cross-stream floors. Options: carry the original claim prefix (including ancestors of the interest), or define a projection and reconciliation rule that selects one covering claim. Recommend preserving original claim identity, with a grant-safe ancestor encoding; this changes the wire shape and needs your choice. -- Maintainer: a Hidden=0 floor currently certifies absence for hidden claims delivered on a Hidden=1 stream. Options: make certification depend on each stream's requested filter, or isolate differently filtered views entirely. Recommend using the existing request filter to restrict negative evidence, without another field on the wire. Confirm this certification rule alongside the claim identity decision. - ## Required - [Route session integration](/quest/m1/cluster-routing/routes/session.md) - one router per origin node drives the origin and every lite-07 announce stream diff --git a/quest/m1/cluster-routing/routes/acceptance.md b/quest/m1/cluster-routing/routes/acceptance.md index cec1981c155..b4eda63a89f 100644 --- a/quest/m1/cluster-routing/routes/acceptance.md +++ b/quest/m1/cluster-routing/routes/acceptance.md @@ -28,8 +28,8 @@ replace are retired. go out to a stream whose interest meets none of the origin's claims: - a transit node forwarding requests for paths outside its own interest; - a node holding or forwarding a seqno request for that identity; - - a sender whose certified regions miss the interest, which must still - answer with an empty floor list + - a sender that cannot certify the interest, which must still close the + batch with a floor that certifies nothing (`a_disjoint_interest_is_answered_with_nothing`). The counterexamples from #4349 (closed; this epic still owns the fix) are the `residual_*` tests in diff --git a/quest/m1/cluster-routing/routes/js.md b/quest/m1/cluster-routing/routes/js.md index 6fff2f20d4b..ddc67171a1c 100644 --- a/quest/m1/cluster-routing/routes/js.md +++ b/quest/m1/cluster-routing/routes/js.md @@ -17,7 +17,7 @@ Decided 2026-10-27 (maintainer), from `origin.ts` a detached request stays on its own route instead of following the cheapest one; fix it with the route-selection rework here. - Feed the availability API from [Announcement availability](/quest/m1/cluster-routing/routes/availability.md). -- Rust/JS interop covers the route messages, the floor list, and the echo. +- Rust/JS interop covers the route messages, the floor marker, and the echo. ## Required diff --git a/quest/m1/cluster-routing/routes/session.md b/quest/m1/cluster-routing/routes/session.md index b0d10f335ac..2bca160dd1d 100644 --- a/quest/m1/cluster-routing/routes/session.md +++ b/quest/m1/cluster-routing/routes/session.md @@ -68,6 +68,16 @@ drives sessions, which is why the switch lands here. - A batch carries changes against the last batch on its stream. Restricting a view to an interest is still linear in that origin's claims per change; make it incremental here. +- Decided 2026-11-05 (maintainer, /quest-plan, the epic's Plan items 10 to + 12): each stream's floor is a whole-interest marker. Project the router's + per-region view onto it, certifying nothing unless the view knows the + route's claims across the stream's interest at one revision; present per + route only the most specific claim covering the interest, moved rather + than ended when it is withdrawn; and certify absence only for what the + stream's `Hidden` filter shows. Rerun the handoff sweeps through that + projection: a scope a relay lost now goes offline downstream only on + ROUTE_END, the accepted cost, so update the scoped fixtures' expectations + to it, while no sweep may revive a known withdrawal or report a false END. - Bound what a peer makes the router hold (routes, claims, views per session) by design, as the session caps announcements today. - Public API: the route-change surface on `broadcast::Route` and its diff --git a/quest/m2/route-interest-prefixes.md b/quest/m2/route-interest-prefixes.md index b1024bbec41..7607934fe9c 100644 --- a/quest/m2/route-interest-prefixes.md +++ b/quest/m2/route-interest-prefixes.md @@ -4,9 +4,9 @@ A route-layer Announce Stream's interest is a set of prefixes instead of one, so a subscriber that wants several disjoint subtrees opens one stream -rather than one per prefix. ANNOUNCE_FLOOR regions and certification hold -across the set, and the proof's floor and certification sweeps pass over -multi-prefix interests. +rather than one per prefix. An ANNOUNCE_FLOOR certifies the whole set at +one revision or nothing, and the proof's floor and certification sweeps pass +over multi-prefix interests. ## Plan @@ -21,15 +21,15 @@ Split out of [Routes and announces](/quest/m1/cluster-routing/routes/README.md) the route layer's core with a single interest prefix per stream (its regions are suffixes of that prefix) and leaves this as a separate m2 quest. -- Define how ANNOUNCE_FLOOR names a region under a set (which prefix it - extends), and what a count of zero means when no prefix meets the sender's - regions. +- Define what a whole-set floor certifies (lite-07's floor is a marker over + its one prefix, decided 2026-11-05), and how a sender that knows only some + of the set's prefixes answers. - Redo the proof's floor and certification sweeps with multi-prefix interests, including overlapping prefixes and a set that changes. - Update the lite draft and both implementations together. Public API: the route layer's interest type. Wire: an Announce Stream's -interest and ANNOUNCE_FLOOR regions on the next unpublished lite version. +interest and what ANNOUNCE_FLOOR certifies on the next unpublished lite version. ## Related diff --git a/rs/moq-net/src/lite/announce.rs b/rs/moq-net/src/lite/announce.rs index 4300bae4163..3596392a581 100644 --- a/rs/moq-net/src/lite/announce.rs +++ b/rs/moq-net/src/lite/announce.rs @@ -1,5 +1,3 @@ -use std::collections::BTreeMap; - use num_enum::{IntoPrimitive, TryFromPrimitive}; use crate::{ @@ -121,12 +119,9 @@ pub enum AnnounceBroadcast { /// advertisement under it. The receiver echoes it back. RouteEnd { id: u64 }, /// ANNOUNCE_FLOOR (lite-07+): closes the batch of advertisements under the route - /// with this id, listing every region the sender certifies for it: a suffix of the - /// requested prefix and the origin revision as of which it is complete. - Floor { - id: u64, - regions: BTreeMap, - }, + /// with this id, certifying them across the whole requested prefix at the origin's + /// revision, or certifying nothing (`None`, the last varint on the wire). + Floor { id: u64, revision: Option }, } /// Refuse what no node advertises: node 0, and the last varint as a seqno or metric, @@ -182,41 +177,21 @@ fn decode_reach(r: &mut Decoder<'_>) -> Result { Ok(Reach::Held) } -fn encode_floor( - w: &mut Encoder<'_>, - version: Version, - id: u64, - regions: &BTreeMap, -) -> Result<(), EncodeError> { +/// Encode an ANNOUNCE_FLOOR body. No origin advertises the last revision, so it +/// stands for a floor that certifies nothing. +fn encode_floor(w: &mut Encoder<'_>, id: u64, revision: Option) -> Result<(), EncodeError> { w.varint(id)?; - w.varint(regions.len() as u64)?; - for (region, rev) in regions { - if *rev == route::MAX { - return Err(EncodeError::InvalidState); - } - region.encode(w, version)?; - w.varint(*rev)?; + match revision { + Some(route::MAX) => Err(EncodeError::InvalidState), + Some(rev) => w.varint(rev), + None => w.varint(route::MAX), } - Ok(()) } -fn decode_floor(r: &mut Decoder<'_>, version: Version) -> Result { +fn decode_floor(r: &mut Decoder<'_>) -> Result { let id = r.varint()?; - let count = r.varint()?; - // Each region takes at least a length byte and a revision byte, so the body bounds - // the count before anything is read. - if count > (r.remaining() / 2) as u64 { - return Err(DecodeError::Short); - } - let mut regions = BTreeMap::new(); - for _ in 0..count { - let region = Path::decode(r, version)?; - let rev = r.varint()?; - if rev == route::MAX || regions.insert(region, rev).is_some() { - return Err(DecodeError::InvalidValue); - } - } - Ok(AnnounceBroadcast::Floor { id, regions }) + let revision = Some(r.varint()?).filter(|&rev| rev != route::MAX); + Ok(AnnounceBroadcast::Floor { id, revision }) } /// A path suffix as it travels: the first `keep` segments of a base announcement's @@ -439,7 +414,7 @@ impl Encode for AnnounceBroadcast { encode_reach(w, reach)?; } Self::RouteEnd { id } => w.varint(*id)?, - Self::Floor { id, regions } => encode_floor(w, version, *id, regions)?, + Self::Floor { id, revision } => encode_floor(w, *id, *revision)?, Self::Ended { .. } | Self::Skipped => unreachable!("refused above"), } if w.since(&prefix) > MAX_MESSAGE_SIZE { @@ -555,7 +530,7 @@ impl<'a> Decode<'a, Version> for AnnounceBroadcast { reach: decode_reach(&mut body)?, }, ROUTE_END if version.has_routes() => Self::RouteEnd { id: body.varint()? }, - ANNOUNCE_FLOOR if version.has_routes() => decode_floor(&mut body, version)?, + ANNOUNCE_FLOOR if version.has_routes() => decode_floor(&mut body)?, // Unknown types are skipped by length so an earlier Lite06 build // negotiating the same ALPN does not kill the announce stream. _ => { @@ -1542,11 +1517,6 @@ mod tests { #[test] fn route_messages_round_trip_on_lite07() { - let regions = |list: &[(&'static str, u64)]| { - list.iter() - .map(|(path, rev)| (Path::from_static(path), *rev)) - .collect::>() - }; for msg in [ AnnounceBroadcast::RouteStart { node: 0x7a3f, @@ -1567,28 +1537,31 @@ mod tests { AnnounceBroadcast::RouteEnd { id: 0 }, AnnounceBroadcast::Floor { id: 0, - regions: regions(&[]), + revision: Some(0), }, AnnounceBroadcast::Floor { id: 1, - regions: regions(&[("live/a", 3), ("live/b", 5), ("", 0)]), + revision: Some(route::MAX - 1), }, + AnnounceBroadcast::Floor { id: 2, revision: None }, ] { assert_eq!(broadcast_round_trip(&msg, Version::Lite07), msg); } } - /// The common floor certifies the whole interest: one byte over (Route ID, revision). + /// A floor is a marker: the Route ID and one revision, the last varint when it + /// certifies nothing. #[test] - fn a_whole_interest_floor_is_one_byte_over_its_revision() { - let mut buf = Vec::new(); - AnnounceBroadcast::Floor { - id: 0, - regions: [(Path::from_static(""), 7)].into(), - } - .encode(&mut Encoder::new(&mut buf, Version::Lite07.into()), Version::Lite07) - .unwrap(); - assert_eq!(buf, vec![9, 4, 0, 1, 0, 7]); + fn a_floor_is_its_route_and_revision() { + let encode = |revision| { + let mut buf = Vec::new(); + AnnounceBroadcast::Floor { id: 0, revision } + .encode(&mut Encoder::new(&mut buf, Version::Lite07.into()), Version::Lite07) + .unwrap(); + buf + }; + assert_eq!(encode(Some(7)), vec![9, 2, 0, 7]); + assert_eq!(encode(None), [&[9, 10, 0][..], &[0xff; 9]].concat()); } #[test] @@ -1603,10 +1576,7 @@ mod tests { reach: Reach::Held, }, AnnounceBroadcast::RouteEnd { id: 0 }, - AnnounceBroadcast::Floor { - id: 0, - regions: BTreeMap::new(), - }, + AnnounceBroadcast::Floor { id: 0, revision: None }, ] { for version in [Version::Lite05, Version::Lite06] { let mut buf = Vec::new(); @@ -1660,9 +1630,10 @@ mod tests { id: 0, reach: Reach::Finite(advert(0, route::MAX, false)), }, + // The last revision is the floor that certifies nothing, never a revision. AnnounceBroadcast::Floor { id: 0, - regions: [(Path::from_static(""), route::MAX)].into(), + revision: Some(route::MAX), }, ]; for msg in refused { @@ -1688,24 +1659,12 @@ mod tests { forge(ROUTE_UPDATE, advert(0, route::MAX, 1, 0)), forge(ROUTE_UPDATE, advert(0, 1, route::MAX, 0)), forge(ROUTE_UPDATE, advert(0, 0, route::MAX, 1)), - forge(ANNOUNCE_FLOOR, |w| { - w.varint(0u64).unwrap(); - w.varint(2u64).unwrap(); - for _ in 0..2 { - Path::from_static("live").encode(w, Version::Lite07).unwrap(); - w.varint(1u64).unwrap(); - } - }), + // A floor without its revision, and one with trailing bytes. + forge(ANNOUNCE_FLOOR, |w| w.varint(0u64).unwrap()), forge(ANNOUNCE_FLOOR, |w| { w.varint(0u64).unwrap(); w.varint(1u64).unwrap(); - Path::from_static("").encode(w, Version::Lite07).unwrap(); - w.varint(route::MAX).unwrap(); - }), - // A count the body cannot hold fails before any region is read. - forge(ANNOUNCE_FLOOR, |w| { - w.varint(0u64).unwrap(); - w.varint(u32::MAX.into()).unwrap(); + w.u8(0); }), // Trailing bytes after a complete body. forge(ROUTE_END, |w| { From 42c479b888304ef4554644c99473fba122710561 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 12:05:39 -0800 Subject: [PATCH 09/11] feat(net)!: hold routes with ROUTE_HOLD and offset the floor revision A held route is now its own ROUTE_HOLD (0xB) message carrying only the Route ID, replacing a ROUTE_UPDATE with a Metric of 2^64-1. ANNOUNCE_FLOOR encodes Revision + 1, with 0 certifying nothing, replacing the 2^64-1 sentinel. A metric is bounded only by overflow, so the wire refuses none. The draft's feasibility condition now covers a missing distance and the advertised metric. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 47 ++++--- quest/m1/bench-ci.md | 5 +- quest/m1/cluster-routing/routes/README.md | 25 ++-- quest/m1/cluster-routing/routes/session.md | 5 + rs/moq-net/src/fuzz.rs | 11 +- rs/moq-net/src/lite/announce.rs | 139 +++++++++------------ rs/moq-net/src/lite/subscriber.rs | 1 + rs/moq-net/src/route/mod.rs | 16 +-- rs/moq-net/src/route/tests/routing.rs | 2 +- 9 files changed, 128 insertions(+), 123 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 9038dc593f2..59b673abe75 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -401,7 +401,8 @@ The subscriber creates the stream with an ANNOUNCE_REQUEST message. The publisher replies with a single ANNOUNCE_OK message followed by the matching routes and claims and any future changes: - ROUTE_START: the publisher reaches an origin node. -- ROUTE_UPDATE: a route's seqno, metric, or down-only bit changed, or the route is held. +- ROUTE_UPDATE: a route's seqno, metric, or down-only bit changed. +- ROUTE_HOLD: the publisher has no usable route but holds it while it recovers. - ROUTE_END: the publisher let go of a route and every claim under it. - ANNOUNCE_START: a matching claim of a route's origin node. - ANNOUNCE_END: the origin withdrew a claim. @@ -450,7 +451,7 @@ The subscriber applies a batch whole, once its floor arrives. A receiver that hears one origin's claims on several streams merges them claim by claim, each at the prefix its stream presents it: of the streams whose floor certifies the claim, the one with the highest Revision decides whether it exists, and a claim no floor certifies exists while some stream carries it. An older floor never revives a claim whose withdrawal a newer one delivered. A floor that certifies nothing proves no claim absent: the stream's claims stay until its ANNOUNCE_END or ROUTE_END. -While a receiver has a usable route to the origin and every peer whose last finite route to it is feasible here has answered the current request, it lets go of a claim none of those peers certifies or carries: it goes offline, which is not an ANNOUNCE_END. +While a receiver has a usable route to the origin and every peer whose last advertised route to it is feasible here has answered the current request, it lets go of a claim none of those peers certifies or carries: it goes offline, which is not an ANNOUNCE_END. #### Compression {#announce-compression} An ANNOUNCE_START or ANNOUNCE_TAKEOVER MAY copy the head of its path from a live claim on the same stream, its base. @@ -482,10 +483,10 @@ A Node ID is otherwise declared, not proven. An origin advertises its own route with a Metric of 0 and a Seqno that only it advances. A receiver adds the session's link cost (see [Cost Parameter](#cost-parameter)) to the Metric of every route the peer sends; the sum is its metric through that peer. Each link adds at least 1, so a route's metric strictly increases away from its origin. -A metric that reaches 2^64-1 makes the route unusable. +A metric that overflows 64 bits makes the route unusable. -A node keeps a feasibility distance per Node ID: the highest Seqno of any finite route it advertised, and the lowest metric it advertised at that Seqno, as in [RFC8966] Section 3.5.1. -A route is feasible when its Seqno is greater than the distance's, or equal with a lower metric. +A node keeps a feasibility distance per Node ID: the highest Seqno of any route it advertised, and the lowest metric it advertised at that Seqno, as in [RFC8966] Section 3.5.1. +A route is feasible when no feasibility distance is kept for its Node ID yet, when its Seqno is greater than the distance's, or when its Seqno is equal and its advertised Metric is lower. A node selects, per Node ID, among its feasible routes that are not held: a session that has not received a GOAWAY, then the highest local priority, then the lowest metric. It MUST NOT select an infeasible route, and a distance only ever falls within a Seqno, so advertising a longer route never raises it. A node MAY mark a session upstream. @@ -496,8 +497,8 @@ A node that does not forward, such as a client, sends only its own route, and ho An origin advances its Seqno whenever its own set of sessions changes. It never advertises a Seqno or Revision of 2^64-1: a change that would reach it instead rotates its Node ID, ending the old route everywhere and announcing its claims again under the new one. -A node with no usable route to an origin node holds it while it can still recover: while every route it has to the node is infeasible, or a peer holds a route whose last finite advertisement is feasible here. -A held route travels as a ROUTE_UPDATE with an infinite Metric (2^64-1), and its claims stay shown but unavailable. +A node with no usable route to an origin node holds it while it can still recover: while every route it has to the node is infeasible, or a peer holds a route whose last advertisement is feasible here. +A held route travels as a ROUTE_HOLD, and its claims stay shown but unavailable until a ROUTE_UPDATE makes it usable again. Otherwise the node lets go, sending ROUTE_END wherever it advertised the route, and its claims go offline. Neither proves that the origin withdrew its claims. @@ -1082,7 +1083,7 @@ An Announce Stream ending before SYNCED is a PROTOCOL_VIOLATION; a Tracks Stream ## ROUTE_START {#route-start} A publisher sends a ROUTE_START message when it reaches an origin node (see [Routing](#routing)). -Each ROUTE_START implicitly assigns the next Route ID on the stream, later referenced by ROUTE_UPDATE, ROUTE_END, ANNOUNCE_FLOOR, ANNOUNCE_START, and ANNOUNCE_TAKEOVER (see [Announce](#announce)). +Each ROUTE_START implicitly assigns the next Route ID on the stream, later referenced by ROUTE_UPDATE, ROUTE_HOLD, ROUTE_END, ANNOUNCE_FLOOR, ANNOUNCE_START, and ANNOUNCE_TAKEOVER (see [Announce](#announce)). ~~~ ROUTE_START Message { @@ -1108,7 +1109,6 @@ The origin's sequence number of the route, which only the origin advances. **Metric**: The publisher's metric to the origin node, 0 when it is the origin. -2^64-1 is a PROTOCOL_VIOLATION here: a route starts usable. **Down Only**: 1 if the route is down-only (see [Routing](#routing)), 0 otherwise. @@ -1137,9 +1137,26 @@ The ordinal implicitly assigned by a prior ROUTE_START on this stream. **Seqno**, **Metric**, and **Down Only**: As defined for [ROUTE_START](#route-start), replacing the route's. -A Metric of 2^64-1 means the route is held: the publisher has no usable route to the origin node but still holds its claims while it recovers. -A held route carries a Seqno and Down Only of 0, and the receiver keeps the last finite values; any other value is a PROTOCOL_VIOLATION. -A later finite ROUTE_UPDATE makes the route usable again. +A ROUTE_UPDATE for a held route makes it usable again. + + +## ROUTE_HOLD {#route-hold} +A publisher sends a ROUTE_HOLD message when it has no usable route to the origin node but still holds the route and its claims while it recovers (see [Routing](#routing)), referencing its Route ID. +The receiver keeps the route's last Seqno, Metric, and Down Only, and does not select it until a ROUTE_UPDATE. + +~~~ +ROUTE_HOLD Message { + Type (i) = 0xB + Message Length (i) + Route ID (i) +} +~~~ + +**Type**: +Set to 0xB to indicate a ROUTE_HOLD message. + +**Route ID**: +The ordinal implicitly assigned by a prior ROUTE_START on this stream. ## ROUTE_END {#route-end} @@ -1372,7 +1389,7 @@ Set to 0x9 to indicate an ANNOUNCE_FLOOR message. The ordinal implicitly assigned by a prior ROUTE_START on this stream. **Revision**: -The origin's Revision at which the floor certifies the stream's claims for the route, or 2^64-1, which no origin advertises (see [Routing](#routing)), when it certifies nothing. +The origin's Revision at which the floor certifies the stream's claims for the route, plus one, or 0 when it certifies nothing. ## SUBSCRIBE @@ -2112,9 +2129,9 @@ The `Message Length` describes the payload size on the wire. - A receiver mints an Instance, starting with `~`, for each announcement a session of an earlier version delivers. - Added announce compression: ANNOUNCE_START and ANNOUNCE_TAKEOVER gain `Path Base` and `Path Keep` to copy the head of a live claim's suffix. - Split routing into two layers on the Announce Stream. ROUTE_START (0x6), ROUTE_UPDATE (0x7), and ROUTE_END (0x8) advertise how the publisher reaches an origin node, named by a Node ID, with a seqno, an additive metric, and a down-only bit, and loops are prevented by Babel's feasibility condition. Every ANNOUNCE_START and ANNOUNCE_TAKEOVER names its route's Route ID in place of the Hop ID list, so a link change sends one ROUTE_UPDATE per origin rather than an update per claim. Removed the Hop ID list, ANNOUNCE_OK's Hop ID, and the SETUP Hop parameter (0x5). -- A ROUTE_UPDATE with a Metric of 2^64-1 holds a route while it recovers; ROUTE_END lets it go. Neither withdraws the origin's claims, and the subscriber echoes every ROUTE_END back. +- Added ROUTE_HOLD (0xB), which holds a route while it recovers; ROUTE_END lets it go. Neither withdraws the origin's claims, and the subscriber echoes every ROUTE_END back. - Added ROUTE_REQUEST (0xA), sent by the subscriber on the Announce Stream to ask for a route with a newer seqno when every route it has is infeasible. -- Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the origin Revision at which the stream's claims across the requested prefix are certified, or 2^64-1 to certify nothing, so a stale copy never revives a claim whose withdrawal a newer one delivered. A floor certifies only the claims its own request shows. +- Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the origin Revision at which the stream's claims across the requested prefix are certified plus one, or 0 to certify nothing, so a stale copy never revives a claim whose withdrawal a newer one delivered. A floor certifies only the claims its own request shows. - A route presents only its most specific claim covering the requested prefix, moved to the next covering claim when it is withdrawn rather than ended. - Added `Node ID` to TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS: the origin node the first hop selected, followed by every later hop. - Replaced Warm and Cold Route Cost with the origin's Announce Cost, unchanged in transit. ANNOUNCE_UPDATE carries only the Announce Cost. diff --git a/quest/m1/bench-ci.md b/quest/m1/bench-ci.md index c9fb20bbb1e..5d6f2a777c8 100644 --- a/quest/m1/bench-ci.md +++ b/quest/m1/bench-ci.md @@ -12,8 +12,9 @@ runs on the hosted ARM runners, and a deliberately slowed bench fails the job. - Maintainer: dispatch the slowed-bench Nightly run below. Agents never dispatch Nightly; the 2026-11-04 authorization covered the two A/A runs only. - Maintainer: authorize one more A/A Nightly run for the moq-net - (`bench-rust`) and `bench-rust-relay` groups once - [moq-net's benches run to completion](/quest/m1/bench-test-mode.md) lands. + (`bench-rust`) and `bench-rust-relay` groups, now that + [moq-dev/moq#7940](https://github.com/moq-dev/moq/pull/7940) lets moq-net's + benches run to completion. Neither group produced a `paired.json` in either 2026-11-04 run, because their benches panic. Check: both groups' `paired.json` exist in that run. diff --git a/quest/m1/cluster-routing/routes/README.md b/quest/m1/cluster-routing/routes/README.md index 49ce72fd0fa..f68fc3e93ee 100644 --- a/quest/m1/cluster-routing/routes/README.md +++ b/quest/m1/cluster-routing/routes/README.md @@ -16,15 +16,17 @@ claim whose withdrawal is already known. 1. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **The draft describes the target before the code switches.** A hop-less ANNOUNCE cannot run before the router drives sessions; the quest's own Plan says so. So the draft now specifies the whole lite-07 route wire, while main's lite-07 session keeps sending hop-based ANNOUNCEs until [Route session integration](https://github.com/moq-dev/moq/blob/main/quest/m1/cluster-routing/routes/session.md) lands. lite-07 is unpublished `-wip`, off by default, and the epic lands in one release before the cut. The other option was to put only the new message sections in the draft and leave ANNOUNCE hop-based. I rejected it because ANNOUNCE_FLOOR's "for that route" and the Route ID would then be undefined in the draft. 2. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **The request field is `Node ID (i)`, with 0 meaning "the publisher selects".** It is added to every request that resolves a path: TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS. Node ID 0 is reserved everywhere. 3. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **ROUTE_REQUEST names a Node ID, not a Route ID.** A request can then never race a ROUTE_END into a protocol violation. -4. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **A held ROUTE_UPDATE carries a seqno and down-only bit of 0.** Any other value is refused, rather than letting the fields carry no meaning. +4. ~~A held ROUTE_UPDATE carries a seqno and down-only bit of 0.~~ Superseded by item 14: a held route is its own message and carries no seqno or bit. 5. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **Floor regions have no order, and a repeated region is a violation.** A sorted order would need Rust's `Path` ordering reproduced exactly in JS. Decoding into a map rejects duplicates at the same cost. Superseded the same day by item 10, which removes regions. 6. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **The Hop setup parameter and ANNOUNCE_OK's Hop ID are dropped from lite-07.** Their only roles were path-vector exclusion and the implicit trailing hop. ANNOUNCE_OK reserves no body for a route-key proof: [moq-dev/moq#7937](https://github.com/moq-dev/moq/pull/7937), accepted the same day, proves the route key once per session in SETUP. The live hop-based codec still switches in session integration, as choice 1 requires. 7. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **An echo naming a Route ID that was never ended, or was already echoed, is a violation at the publisher.** The publisher can check this from its live and unacknowledged ids alone, with no growing state. 8. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **ANNOUNCE_TAKEOVER keeps its shape, with a Route ID in place of hops, and its predecessor must be under the same route.** [quest/m1/announce-takeover.md](https://github.com/moq-dev/moq/blob/main/quest/m1/announce-takeover.md) deletes the message, so this keeps the change minimal. 9. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **Unknown announce types stay skipped by length on lite-06+**, as before. An unknown reply type is refused. -10. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **ANNOUNCE_FLOOR is a marker, not a region list:** `{Route ID, Revision}`. It closes a batch and certifies the stream's claims for that route across the whole requested prefix as the origin's at that Revision. A publisher that cannot vouch for the whole requested prefix at one Revision still closes the batch, with a Revision of 2^64-1 that certifies nothing: the one Revision no origin advertises, as the infinite Metric marks a held route, so the common floor carries no flag and no offset. Regions, the replacing list, and the 65,535-byte overflow go away. Accepted cost: a relay with partial knowledge of an origin certifies nothing for that route, so a receiver lets its stale claims go only on ROUTE_END. Rejected: one region per message; a multipart atomic list. +10. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **ANNOUNCE_FLOOR is a marker, not a region list:** `{Route ID, Revision}`. It closes a batch and certifies the stream's claims for that route across the whole requested prefix as the origin's at that Revision. A publisher that cannot vouch for the whole requested prefix at one Revision still closes the batch, with a floor that certifies nothing, encoded per item 13. Regions, the replacing list, and the 65,535-byte overflow go away. Accepted cost: a relay with partial knowledge of an origin certifies nothing for that route, so a receiver lets its stale claims go only on ROUTE_END. Rejected: one region per message; a multipart atomic list. 11. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **Clamped claims collapse to the best one.** When several of a route's claims cover a request's prefix (`live` at cost 10 and `live/a` at cost 1, request `live/a/x`), the publisher presents only the most specific, and when it is withdrawn moves the presented claim to the next covering one with ANNOUNCE_UPDATE or ANNOUNCE_RESTART instead of ending it. No wire change; one live claim per prefix per route holds, and a receiver merges claims at the prefix each stream presents them. Rejected: carrying each claim's ancestor depth. 12. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **A floor certifies only what its own request shows.** A stream whose request did not set `Hidden` certifies nothing about hidden claims. No new wire field. Rejected: isolating differently filtered views. +13. ✅ Decided 2026-11-05 (maintainer): **ANNOUNCE_FLOOR encodes `Revision + 1`, with 0 meaning the floor certifies nothing,** the same offset as `Max Age`. Rust decodes it to `Option`. It replaces a Revision of 2^64-1 as the marker: a huge sentinel is a bad pattern and costs 9 bytes as a lite-07 varint. Rejected: a separate message; a boolean field. +14. ✅ Decided 2026-11-05 (maintainer: "use 0 as a sentinel, or make a new message type"; the agent picked the message): **A held route is ROUTE_HOLD (0xB), `{Route ID}`,** replacing a ROUTE_UPDATE with a Metric of 2^64-1. A metric of 0 is real (the origin's own route), so 0 cannot mark held, and offsetting every metric by one would still leave a held update's seqno and down-only bit meaningless and refused. The message carries only what a hold means, item 4's rule goes away, and a metric is now only bounded by overflow, so the wire refuses no metric. The 2^64-1 Seqno and Revision bounds stay: they are bounds, not markers, leaving room for a ROUTE_REQUEST's successor and for the floor's offset. The route layer's core landed in @@ -167,8 +169,8 @@ the bullets below are the contract it carries. by metric overflow. - Loop freedom is Babel's feasibility condition (RFC 8966) keyed by node: accept a route if its seqno is newer, or equal with a metric below the - feasibility distance. Only the origin advances its seqno. Retraction is an - infinite metric or ROUTE_END; there is no count to infinity. + feasibility distance. Only the origin advances its seqno. Retraction is + ROUTE_HOLD or ROUTE_END; there is no count to infinity. - Starvation recovery is Babel's seqno request (RFC 8966 s3.8, decided 2026-10-12). A node whose routes to an origin are all infeasible sends ROUTE_REQUEST for that origin, with its feasibility seqno plus one, to the @@ -257,7 +259,7 @@ loss never reports an END: - Each origin numbers the changes to its own claims with a revision that starts at zero for each routing identity, separate from the route seqno. Both are varints. An origin never advertises the last value, 2^64-1 (the - largest lite-07 varint, as for the infinite metric): a + largest lite-07 varint): a seqno advance, claim change, or seqno request that would reach it rotates the identity instead, ending the old one everywhere and announcing the claims again under a fresh one, so nothing wraps and every advertised @@ -310,7 +312,7 @@ loss never reports an END: withdrawing one origin's wildcard leaves another origin's copy (`wildcard_claims_end_per_origin_and_are_certified_below`). - A node tracks the regions each neighbour's last view certifies. Only a - neighbour whose last finite advertisement is feasible here counts, held or + neighbour whose last advertisement is feasible here counts, held or not: feasible edges strictly decrease the metric toward the origin, so two nodes never keep a region alive through each other. While the node has a usable route and every such neighbour has answered the current interest, a @@ -341,9 +343,8 @@ loss never reports an END: withdrawal. - Per identity, a node is usable (a feasible selected route: announcements shown available), held, or let go. It is held while starved, or while a - neighbor's held route has a last finite advertisement feasible here; it - keeps showing the announcements as unavailable and sends ROUTE_UPDATE with - an infinite metric. Holds therefore follow strictly decreasing metrics + neighbor's held route has a last advertisement feasible here; it + keeps showing the announcements as unavailable and sends ROUTE_HOLD. Holds therefore follow strictly decreasing metrics down to a starved node and cannot prop each other up. Otherwise the node lets go: ROUTE_END to every neighbor it advertised to, and its reader sees the announcements go offline, a distinct event from an END the origin @@ -413,8 +414,10 @@ Decided with the maintainer 2026-10-16, from the handoff proof: which adds bytes and removes no state, since a receiver still needs each session's last view to apply a batch from a sender whose own view jumped; and the floor on ROUTE_UPDATE, which ties claim churn to route processing. -2. Held encoding: ✅ an infinite metric (2^64-1) means held and ROUTE_END - means let go, with no new field. Rejected: an explicit held bit. +2. Held encoding: ~~an infinite metric (2^64-1) means held and ROUTE_END + means let go, with no new field. Rejected: an explicit held bit.~~ + Superseded 2026-11-05 by Plan item 14: ROUTE_HOLD means held, and + ROUTE_END still means let go. 3. Reader API: ✅ an announcement carries availability, and letting go is a distinct offline event, separate from the origin's END, in Rust and JS. Rejected: presence alone, which gives a reader nothing to re-request on diff --git a/quest/m1/cluster-routing/routes/session.md b/quest/m1/cluster-routing/routes/session.md index 2bca160dd1d..a7ad353569b 100644 --- a/quest/m1/cluster-routing/routes/session.md +++ b/quest/m1/cluster-routing/routes/session.md @@ -78,6 +78,11 @@ drives sessions, which is why the switch lands here. projection: a scope a relay lost now goes offline downstream only on ROUTE_END, the accepted cost, so update the scoped fixtures' expectations to it, while no sweep may revive a known withdrawal or report a false END. + Also cover a fresh claim that arrives only on an uncertified stream while + an older certified floor on another stream proves it absent + ([review](https://github.com/moq-dev/moq/pull/7764#discussion_r4485081975)): + the merge rule hides it until the certifying peer refreshes or lets go, + and the test pins that bound. - Bound what a peer makes the router hold (routes, claims, views per session) by design, as the session caps announcements today. - Public API: the route-change surface on `broadcast::Route` and its diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index 940dfd8c555..2565a67b293 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -1313,9 +1313,14 @@ pub fn seeds() -> Vec { } } - // A typed empty TRACKS_SYNCED, AUTH_OK reply, and ROUTE_END echo need nonuniform - // envelope bytes. - for (kind, body) in [(23, &[4, 0][..]), (27, &[0, 3, 0, 0, 0][..]), (33, &[8, 1, 0][..])] { + // A typed empty TRACKS_SYNCED, AUTH_OK reply, ROUTE_END echo, and ROUTE_REQUEST need + // nonuniform envelope bytes. + for (kind, body) in [ + (23, &[4, 0][..]), + (27, &[0, 3, 0, 0, 0][..]), + (33, &[8, 1, 0][..]), + (33, &[10, 2, 1, 0][..]), + ] { let mut data = vec![(LITE_VERSIONS.len() - 1) as u8, kind]; data.extend_from_slice(body); seeds.push(Seed { diff --git a/rs/moq-net/src/lite/announce.rs b/rs/moq-net/src/lite/announce.rs index 3596392a581..8e9d42770b4 100644 --- a/rs/moq-net/src/lite/announce.rs +++ b/rs/moq-net/src/lite/announce.rs @@ -4,7 +4,7 @@ use crate::{ Hop, Hops, Instance, Path, coding::*, origin::Cost, - route::{self, Advert, Reach}, + route::{self, Advert}, }; use super::{ @@ -33,6 +33,7 @@ const ANNOUNCE_FLOOR: u64 = 9; allow(dead_code, reason = "the lite-07 session does not drive the route layer yet") )] const ROUTE_REQUEST: u64 = 10; +const ROUTE_HOLD: u64 = 11; /// Whether the negotiated version can update a live advertisement's metadata. On lite-05 an /// update travels as a duplicate ANNOUNCE (a second `active` for an already-announced path); on @@ -113,84 +114,58 @@ pub enum AnnounceBroadcast { /// ROUTE_START (lite-07+): the sender has a route to the origin node `node`, and /// assigns it the next Route ID. RouteStart { node: route::Id, advert: Advert }, - /// ROUTE_UPDATE (lite-07+): the route with this id changed, or is held. - RouteUpdate { id: u64, reach: Reach }, + /// ROUTE_UPDATE (lite-07+): the route with this id changed, and is usable. + RouteUpdate { id: u64, advert: Advert }, + /// ROUTE_HOLD (lite-07+): the sender has no usable route with this id, but still + /// holds its advertisements while it recovers. + RouteHold { id: u64 }, /// ROUTE_END (lite-07+): the sender let go of the route with this id, and of every /// advertisement under it. The receiver echoes it back. RouteEnd { id: u64 }, /// ANNOUNCE_FLOOR (lite-07+): closes the batch of advertisements under the route /// with this id, certifying them across the whole requested prefix at the origin's - /// revision, or certifying nothing (`None`, the last varint on the wire). + /// revision, or certifying nothing (`None`). Floor { id: u64, revision: Option }, } -/// Refuse what no node advertises: node 0, and the last varint as a seqno or metric, -/// which mean nothing and held. -fn valid_start(node: route::Id, advert: &Advert) -> bool { - node != 0 && advert.seqno != route::MAX && advert.metric != route::MAX -} - +/// Encode a seqno, metric, and down-only bit. No origin advertises the last seqno, +/// which would leave no room for the successor a ROUTE_REQUEST names. fn encode_advert(w: &mut Encoder<'_>, advert: &Advert) -> Result<(), EncodeError> { + if advert.seqno == route::MAX { + return Err(EncodeError::InvalidState); + } w.varint(advert.seqno)?; w.varint(advert.metric)?; w.bool(advert.down_only); Ok(()) } -/// Encode a ROUTE_UPDATE body after its id. A held route carries a zero seqno and bit. -fn encode_reach(w: &mut Encoder<'_>, reach: &Reach) -> Result<(), EncodeError> { - match reach { - Reach::Finite(advert) if advert.seqno == route::MAX || advert.metric == route::MAX => { - Err(EncodeError::InvalidState) - } - Reach::Finite(advert) => encode_advert(w, advert), - Reach::Held => encode_advert( - w, - &Advert { - seqno: 0, - metric: route::MAX, - down_only: false, - }, - ), - } -} - fn decode_advert(r: &mut Decoder<'_>) -> Result { + let seqno = r.varint()?; + if seqno == route::MAX { + return Err(DecodeError::InvalidValue); + } Ok(Advert { - seqno: r.varint()?, + seqno, metric: r.varint()?, down_only: r.bool()?, }) } -fn decode_reach(r: &mut Decoder<'_>) -> Result { - let advert = decode_advert(r)?; - if advert.metric != route::MAX { - if advert.seqno == route::MAX { - return Err(DecodeError::InvalidValue); - } - return Ok(Reach::Finite(advert)); - } - if advert.seqno != 0 || advert.down_only { - return Err(DecodeError::InvalidValue); - } - Ok(Reach::Held) -} - -/// Encode an ANNOUNCE_FLOOR body. No origin advertises the last revision, so it -/// stands for a floor that certifies nothing. +/// Encode an ANNOUNCE_FLOOR body: the revision plus one, or 0 to certify nothing. fn encode_floor(w: &mut Encoder<'_>, id: u64, revision: Option) -> Result<(), EncodeError> { w.varint(id)?; match revision { + // No origin advertises the last revision, so it never needs the offset. Some(route::MAX) => Err(EncodeError::InvalidState), - Some(rev) => w.varint(rev), - None => w.varint(route::MAX), + Some(rev) => w.varint(rev + 1), + None => w.varint(0), } } fn decode_floor(r: &mut Decoder<'_>) -> Result { let id = r.varint()?; - let revision = Some(r.varint()?).filter(|&rev| rev != route::MAX); + let revision = r.varint()?.checked_sub(1); Ok(AnnounceBroadcast::Floor { id, revision }) } @@ -343,11 +318,14 @@ impl Encode for AnnounceBroadcast { Self::Takeover { .. } => return Err(EncodeError::Version), Self::RouteStart { .. } if version.has_routes() => ROUTE_START, Self::RouteUpdate { .. } if version.has_routes() => ROUTE_UPDATE, + Self::RouteHold { .. } if version.has_routes() => ROUTE_HOLD, Self::RouteEnd { .. } if version.has_routes() => ROUTE_END, Self::Floor { .. } if version.has_routes() => ANNOUNCE_FLOOR, - Self::RouteStart { .. } | Self::RouteUpdate { .. } | Self::RouteEnd { .. } | Self::Floor { .. } => { - return Err(EncodeError::Version); - } + Self::RouteStart { .. } + | Self::RouteUpdate { .. } + | Self::RouteHold { .. } + | Self::RouteEnd { .. } + | Self::Floor { .. } => return Err(EncodeError::Version), // The pre-lite-06 path-form retraction has no place on lite-06. Self::Ended { .. } => return Err(EncodeError::Version), // Decode-only: an unknown type is never sent. @@ -403,17 +381,17 @@ impl Encode for AnnounceBroadcast { } Self::Synced => {} Self::RouteStart { node, advert } => { - if !valid_start(*node, advert) { + if *node == 0 { return Err(EncodeError::InvalidState); } w.varint(*node)?; encode_advert(w, advert)?; } - Self::RouteUpdate { id, reach } => { + Self::RouteUpdate { id, advert } => { w.varint(*id)?; - encode_reach(w, reach)?; + encode_advert(w, advert)?; } - Self::RouteEnd { id } => w.varint(*id)?, + Self::RouteHold { id } | Self::RouteEnd { id } => w.varint(*id)?, Self::Floor { id, revision } => encode_floor(w, *id, *revision)?, Self::Ended { .. } | Self::Skipped => unreachable!("refused above"), } @@ -452,6 +430,7 @@ impl Encode for AnnounceBroadcast { | Self::Synced | Self::RouteStart { .. } | Self::RouteUpdate { .. } + | Self::RouteHold { .. } | Self::RouteEnd { .. } | Self::Floor { .. } => { return Err(EncodeError::Version); @@ -520,15 +499,16 @@ impl<'a> Decode<'a, Version> for AnnounceBroadcast { ROUTE_START if version.has_routes() => { let node = body.varint()?; let advert = decode_advert(&mut body)?; - if !valid_start(node, &advert) { + if node == 0 { return Err(DecodeError::InvalidValue); } Self::RouteStart { node, advert } } ROUTE_UPDATE if version.has_routes() => Self::RouteUpdate { id: body.varint()?, - reach: decode_reach(&mut body)?, + advert: decode_advert(&mut body)?, }, + ROUTE_HOLD if version.has_routes() => Self::RouteHold { id: body.varint()? }, ROUTE_END if version.has_routes() => Self::RouteEnd { id: body.varint()? }, ANNOUNCE_FLOOR if version.has_routes() => decode_floor(&mut body)?, // Unknown types are skipped by length so an earlier Lite06 build @@ -1524,16 +1504,13 @@ mod tests { }, AnnounceBroadcast::RouteStart { node: route::QUALIFIED | 9, - advert: advert(route::MAX - 1, route::MAX - 1, true), + advert: advert(route::MAX - 1, route::MAX, true), }, AnnounceBroadcast::RouteUpdate { id: 0, - reach: Reach::Finite(advert(42, 15, true)), - }, - AnnounceBroadcast::RouteUpdate { - id: 3, - reach: Reach::Held, + advert: advert(42, 0, true), }, + AnnounceBroadcast::RouteHold { id: 3 }, AnnounceBroadcast::RouteEnd { id: 0 }, AnnounceBroadcast::Floor { id: 0, @@ -1549,8 +1526,8 @@ mod tests { } } - /// A floor is a marker: the Route ID and one revision, the last varint when it - /// certifies nothing. + /// A floor is a marker: the Route ID and the revision plus one, 0 when it certifies + /// nothing. #[test] fn a_floor_is_its_route_and_revision() { let encode = |revision| { @@ -1560,8 +1537,9 @@ mod tests { .unwrap(); buf }; - assert_eq!(encode(Some(7)), vec![9, 2, 0, 7]); - assert_eq!(encode(None), [&[9, 10, 0][..], &[0xff; 9]].concat()); + assert_eq!(encode(Some(0)), vec![9, 2, 0, 1]); + assert_eq!(encode(Some(7)), vec![9, 2, 0, 8]); + assert_eq!(encode(None), vec![9, 2, 0, 0]); } #[test] @@ -1573,8 +1551,9 @@ mod tests { }, AnnounceBroadcast::RouteUpdate { id: 0, - reach: Reach::Held, + advert: advert(0, 1, false), }, + AnnounceBroadcast::RouteHold { id: 0 }, AnnounceBroadcast::RouteEnd { id: 0 }, AnnounceBroadcast::Floor { id: 0, revision: None }, ] { @@ -1605,8 +1584,8 @@ mod tests { )); } - /// What no node advertises is refused on both ends: node 0, the last varint as a - /// seqno or a starting metric, and a held route carrying a seqno or the bit. + /// What no node advertises is refused on both ends: node 0, and the last seqno or + /// revision. #[test] fn route_messages_refuse_what_no_node_advertises() { let refused = [ @@ -1618,19 +1597,11 @@ mod tests { node: 1, advert: advert(route::MAX, 1, false), }, - AnnounceBroadcast::RouteStart { - node: 1, - advert: advert(0, route::MAX, false), - }, - AnnounceBroadcast::RouteUpdate { - id: 0, - reach: Reach::Finite(advert(route::MAX, 1, false)), - }, AnnounceBroadcast::RouteUpdate { id: 0, - reach: Reach::Finite(advert(0, route::MAX, false)), + advert: advert(route::MAX, 1, false), }, - // The last revision is the floor that certifies nothing, never a revision. + // The last revision has no room for the offset. AnnounceBroadcast::Floor { id: 0, revision: Some(route::MAX), @@ -1653,12 +1624,14 @@ mod tests { let malformed = [ forge(ROUTE_START, advert(0, 0, 1, 0)), forge(ROUTE_START, advert(1, route::MAX, 1, 0)), - forge(ROUTE_START, advert(1, 0, route::MAX, 0)), forge(ROUTE_START, advert(1, 0, 1, 2)), // ROUTE_UPDATE's leading field is the Route ID, so 0 is a valid id here. forge(ROUTE_UPDATE, advert(0, route::MAX, 1, 0)), - forge(ROUTE_UPDATE, advert(0, 1, route::MAX, 0)), - forge(ROUTE_UPDATE, advert(0, 0, route::MAX, 1)), + // A hold carries only its Route ID. + forge(ROUTE_HOLD, |w| { + w.varint(0u64).unwrap(); + w.varint(0u64).unwrap(); + }), // A floor without its revision, and one with trailing bytes. forge(ANNOUNCE_FLOOR, |w| w.varint(0u64).unwrap()), forge(ANNOUNCE_FLOOR, |w| { diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 7ac011268eb..10fdf5df2a2 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -601,6 +601,7 @@ impl Subscriber { // cannot be served. lite::AnnounceBroadcast::RouteStart { .. } | lite::AnnounceBroadcast::RouteUpdate { .. } + | lite::AnnounceBroadcast::RouteHold { .. } | lite::AnnounceBroadcast::RouteEnd { .. } | lite::AnnounceBroadcast::Floor { .. } => return Err(Error::ProtocolViolation), } diff --git a/rs/moq-net/src/route/mod.rs b/rs/moq-net/src/route/mod.rs index b14b9ffa373..6899ba6aa0b 100644 --- a/rs/moq-net/src/route/mod.rs +++ b/rs/moq-net/src/route/mod.rs @@ -45,9 +45,9 @@ pub(crate) type Seqno = u64; /// An origin's claim revision, advancing on every change to its own claims. pub(crate) type Rev = u64; -/// The largest lite-07 varint. As a metric it means held; no origin ever -/// advertises it as a seqno or revision, rotating its identity instead, so every -/// advertised seqno leaves room for the successor a request names. +/// The largest lite-07 varint. No origin ever advertises it as a seqno or revision, +/// rotating its identity instead, so every advertised seqno leaves room for the +/// successor a request names, and every revision for ANNOUNCE_FLOOR's offset. pub(crate) const MAX: u64 = u64::MAX; /// A neighbor key, bounded so per-identity state can be read as one range. @@ -128,8 +128,8 @@ pub(crate) struct Advert { pub(crate) enum Reach { /// A usable route. Finite(Advert), - /// An infinite metric: unusable, but the sender still holds the identity's - /// announcements while it recovers. Only follows a finite route. + /// Unusable, but the sender still holds the identity's announcements while it + /// recovers. Only follows a finite route. Held, } @@ -1346,8 +1346,8 @@ impl Router { } } -/// `metric` plus a strictly positive link `cost`, or `None` when it reaches the -/// infinite metric: overflow is unreachable, never saturated. +/// `metric` plus a strictly positive link `cost`, or `None` when it overflows: an +/// overflowed route is unreachable, never saturated. fn add(metric: Metric, cost: Metric) -> Option { - metric.checked_add(cost).filter(|&metric| metric < MAX) + metric.checked_add(cost) } diff --git a/rs/moq-net/src/route/tests/routing.rs b/rs/moq-net/src/route/tests/routing.rs index 2397704c8c7..8318649faba 100644 --- a/rs/moq-net/src/route/tests/routing.rs +++ b/rs/moq-net/src/route/tests/routing.rs @@ -802,7 +802,7 @@ fn an_overflowing_route_neither_certifies_nor_holds() { }; let (near, far) = (1, 2); let far_link = Link { - cost: MAX - 1, + cost: MAX, ..Link::default() }; From 26b07729ff9404e7752192c373e57fe43e66308c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 12:51:45 -0800 Subject: [PATCH 10/11] fix(net): seed every route message type, rank draining claims last, name ROUTE_HOLD Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 2 +- quest/m1/bench-ci.md | 7 ++++--- rs/moq-net/src/fuzz.rs | 9 +++++++-- rs/moq-net/src/lite/version.rs | 4 ++-- rs/moq-net/src/route/mod.rs | 2 +- 5 files changed, 15 insertions(+), 9 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 59b673abe75..6d60dc5ebc1 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -532,7 +532,7 @@ The winning tier is the whole answer: a refusal from it never falls through to a Nor is a refusal retried at another claim of the same prefix. Within the tier, a claim is ineligible while another claim of the same prefix carries its Instance at a higher Epoch, however cheap it is. -Among the eligible claims, the receiver SHOULD prefer the highest local priority of the route's session, then the lowest cost, the claim's Announce Cost plus its route's metric, then the lowest Rendezvous Hash, and then the most recently received, so a reconnecting publisher is not outranked by the stale session it replaced. +Among the eligible claims, the receiver SHOULD prefer a route whose session has not received a GOAWAY, then the highest local priority of the route's session, then the lowest cost, the claim's Announce Cost plus its route's metric, then the lowest Rendezvous Hash, and then the most recently received, so a reconnecting publisher is not outranked by the stale session it replaced. The Rendezvous Hash is the 64-bit FNV-1a hash, with offset basis `0x420C0DECB00B` and the standard FNV-64 prime, of the requested path's UTF-8 bytes followed by the origin's Node ID as 8 little-endian bytes. Every node that knows the same claims ranks them the same, so equal-cost origins of one prefix share its paths and every relay sends one path to the same origin. When choosing which claim to advertise for a prefix, the requested path is the prefix itself. diff --git a/quest/m1/bench-ci.md b/quest/m1/bench-ci.md index 5d6f2a777c8..79a1e73bfb6 100644 --- a/quest/m1/bench-ci.md +++ b/quest/m1/bench-ci.md @@ -25,9 +25,10 @@ A/A results, 2026-11-04 (/quest-unblock): runs [41933525520](https://github.com/moq-dev/moq/actions/runs/41933525520) both produced `paired.json` for the shared, extra, and uring groups. The first was cancelled by Nightly's concurrency group after its bench artifacts uploaded. -At a 10% threshold over 3 rounds, neither run confirmed a regression, and -every case over 10% came back as noise. Single-pass crossings: shared 18 and -26 of 716, extra 10 and 15 of 460, uring 1 and 1 of 41. |delta| p95 was +At a 10% threshold over 3 rounds, neither run confirmed a regression: every +case the 20-case confirmation budget (`CONFIRMATIONS`) reached came back as +noise. Single-pass crossings: shared 18 and 26 of 716, extra 10 and 15 of 460, +uring 1 and 1 of 41, so the second run left 6 shared cases `unconfirmed`. |delta| p95 was 3.7-8.3%, p99 10-23%. Tune the threshold for those groups from these. Decided 2026-11-04 (maintainer, /quest-unblock): ✅ authorize one-time agent diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index 2565a67b293..ec181cd1e1f 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -1313,11 +1313,16 @@ pub fn seeds() -> Vec { } } - // A typed empty TRACKS_SYNCED, AUTH_OK reply, ROUTE_END echo, and ROUTE_REQUEST need - // nonuniform envelope bytes. + // A typed empty TRACKS_SYNCED, AUTH_OK reply, the route messages past ROUTE_START + // (whose type bytes the sweep's lengths never reach), the ROUTE_END echo, and + // ROUTE_REQUEST need nonuniform envelope bytes. for (kind, body) in [ (23, &[4, 0][..]), (27, &[0, 3, 0, 0, 0][..]), + (5, &[7, 4, 0, 1, 2, 0][..]), + (5, &[8, 1, 0][..]), + (5, &[9, 2, 0, 1][..]), + (5, &[11, 1, 0][..]), (33, &[8, 1, 0][..]), (33, &[10, 2, 1, 0][..]), ] { diff --git a/rs/moq-net/src/lite/version.rs b/rs/moq-net/src/lite/version.rs index 880dcda3a51..507ba7d304a 100644 --- a/rs/moq-net/src/lite/version.rs +++ b/rs/moq-net/src/lite/version.rs @@ -338,8 +338,8 @@ impl Version { } /// Whether the announce stream carries the route layer: ROUTE_START, ROUTE_UPDATE, - /// ROUTE_END, and ANNOUNCE_FLOOR toward the subscriber, and ROUTE_REQUEST and the - /// ROUTE_END echo back. Added in lite-07. + /// ROUTE_HOLD, ROUTE_END, and ANNOUNCE_FLOOR toward the subscriber, and ROUTE_REQUEST + /// and the ROUTE_END echo back. Added in lite-07. pub(crate) fn has_routes(self) -> bool { !matches!( self, diff --git a/rs/moq-net/src/route/mod.rs b/rs/moq-net/src/route/mod.rs index 6899ba6aa0b..c0dece43947 100644 --- a/rs/moq-net/src/route/mod.rs +++ b/rs/moq-net/src/route/mod.rs @@ -136,7 +136,7 @@ pub(crate) enum Reach { /// A message for one neighbor. #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) enum Out { - /// ROUTE_START or ROUTE_UPDATE, or ROUTE_END for `None`. + /// ROUTE_START or ROUTE_UPDATE, ROUTE_HOLD for a held reach, or ROUTE_END for `None`. Route { to: N, id: Id, reach: Option }, /// A batch of announcement changes for `id`, closed by its floor list, which /// replaces the last one sent for that route. From ea55b4826d0fd8a0f42f52855b63d2e55dc85f69 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 13:38:45 -0800 Subject: [PATCH 11/11] fix(net): seed ROUTE_START and keep the highest pending ROUTE_REQUEST in the draft The fuzz sweep never formed a whole ROUTE_START, so add a typed seed and check that every typed seed decodes. The draft's pending-request rule now says a newer request raises the pending one and is sent again, as the router already does; a router test pins it. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 2 +- rs/moq-net/src/fuzz.rs | 40 ++++++++++++++++++--------- rs/moq-net/src/route/tests/routing.rs | 39 ++++++++++++++++++++++++++ 3 files changed, 67 insertions(+), 14 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index ca52c095157..246d26ddd5a 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -509,7 +509,7 @@ Neither proves that the origin withdrew its claims. A node whose routes to an origin node are all infeasible sends a ROUTE_REQUEST with its distance's Seqno plus 1 to the peer whose route it would select were it feasible. A node holding a selected route at least that new answers by advertising it, and the origin answers by raising its Seqno to at least the requested one; a request for 2^64-1 rotates the origin's Node ID. Any other node forwards the request to the peer whose route it would select, even the requester. -Every node keeps one pending request per origin node, its own or forwarded, dropping duplicates, until it selects a route at least that new or has no route left, and sends it again whenever that peer changes. +Every node keeps one pending request per origin node, its own or forwarded, at the highest Seqno requested, dropping requests no newer, until it selects a route at least that new or has no route left, and sends it again whenever its Seqno rises or that peer changes. The receiver of a ROUTE_END echoes it back on the same stream once it dropped the route. A node drops a feasibility distance once it neither selects nor advertises the route and every ROUTE_END it sent for it was echoed; a closed session counts as an echo. diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index ec181cd1e1f..ce956c6edeb 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -1269,6 +1269,22 @@ pub fn pattern(data: &[u8]) -> bool { true } +/// Lite seeds whose envelope bytes the uniform sweep never forms, on the newest +/// version: a typed empty TRACKS_SYNCED, an AUTH_OK reply, every route message +/// (the sweep sets ROUTE_START's type byte but fills its size byte with the body +/// pattern, so none decodes whole), the ROUTE_END echo, and ROUTE_REQUEST. +const TYPED_LITE_SEEDS: &[(u8, &[u8])] = &[ + (23, &[4, 0]), + (27, &[0, 3, 0, 0, 0]), + (5, &[6, 4, 1, 0, 1, 0]), + (5, &[7, 4, 0, 1, 2, 0]), + (5, &[8, 1, 0]), + (5, &[9, 2, 0, 1]), + (5, &[11, 1, 0]), + (33, &[8, 1, 0]), + (33, &[10, 2, 1, 0]), +]; + /// The inputs the fuzzer starts from: every (version, type) pair the dispatch knows, /// bodied with byte patterns that decode as small valid fields. /// @@ -1313,19 +1329,7 @@ pub fn seeds() -> Vec { } } - // A typed empty TRACKS_SYNCED, AUTH_OK reply, the route messages past ROUTE_START - // (whose type bytes the sweep's lengths never reach), the ROUTE_END echo, and - // ROUTE_REQUEST need nonuniform envelope bytes. - for (kind, body) in [ - (23, &[4, 0][..]), - (27, &[0, 3, 0, 0, 0][..]), - (5, &[7, 4, 0, 1, 2, 0][..]), - (5, &[8, 1, 0][..]), - (5, &[9, 2, 0, 1][..]), - (5, &[11, 1, 0][..]), - (33, &[8, 1, 0][..]), - (33, &[10, 2, 1, 0][..]), - ] { + for &(kind, body) in TYPED_LITE_SEEDS { let mut data = vec![(LITE_VERSIONS.len() - 1) as u8, kind]; data.extend_from_slice(body); seeds.push(Seed { @@ -1965,6 +1969,16 @@ mod tests { assert!(missing.is_empty(), "no seed decodes for {missing:?}"); } + /// Each typed seed exists to reach one message, so each must decode whole. + #[test] + fn typed_lite_seeds_decode() { + for (kind, body) in TYPED_LITE_SEEDS { + let mut data = vec![(LITE_VERSIONS.len() - 1) as u8, *kind]; + data.extend_from_slice(body); + assert!(lite_wire(&data), "kind {kind} body {body:?} does not decode"); + } + } + /// The stream seeds are built from our own framing, one per version and protocol, so /// each must decode whole: one that does not leaves its version's path unseeded. Those /// at a cap must not, or they sit beside no rejection path. diff --git a/rs/moq-net/src/route/tests/routing.rs b/rs/moq-net/src/route/tests/routing.rs index b16956640e5..01316c3c763 100644 --- a/rs/moq-net/src/route/tests/routing.rs +++ b/rs/moq-net/src/route/tests/routing.rs @@ -762,6 +762,45 @@ fn an_unanswerable_request_stops() { assert_eq!(sim.retained().pending, 0); } +/// A request newer than the one pending raises it and is sent on again, so the +/// newer requester is not left waiting on an answer that only meets the older +/// one; a repeat of the pending Seqno, or an older one, is a duplicate. +#[test] +fn a_newer_request_raises_the_pending_one() { + let (up, a, b) = (0, 1, 2); + let mut relay = Router::::new(9); + for neighbor in [up, a, b] { + relay.connect(neighbor, Link::default()).unwrap(); + } + let advert = Advert { + seqno: 0, + metric: 1, + down_only: false, + }; + relay.route(up, 7, Some(Reach::Finite(advert))); + relay.drain(); + + let requests = |relay: &mut Router| -> Vec<(Node, Seqno)> { + relay + .drain() + .into_iter() + .filter_map(|out| match out { + Out::Request { to, id: 7, seqno } => Some((to, seqno)), + _ => None, + }) + .collect() + }; + + relay.request(a, 7, 1); + assert_eq!(requests(&mut relay), [(up, 1)]); + relay.request(b, 7, 1); + assert_eq!(requests(&mut relay), [], "a duplicate"); + relay.request(b, 7, 2); + assert_eq!(requests(&mut relay), [(up, 2)], "raised and sent again"); + relay.request(a, 7, 1); + assert_eq!(requests(&mut relay), [], "older than the pending one"); +} + /// Each message recomputes only the identity it names, so the work a message /// costs does not grow with how many other origins a node routes to. #[test]