diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index c26aa3e6551..aeb31c65c40 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -111,4 +111,4 @@ Concurrent disjoint regions of one permit need the [interest set](/quest/m2/cluster-routes/interest-set.md), including per-prefix SYNCED; neither this model nor the current scoped sweep covers them. -One publisher, so routes for several publishers are left out, and so are the copies and copy rank of cache-aware routing, which `rs/moq-net/src/route/tests/paths.rs` prototypes; the DEAD message's per-publisher batching reduces to one route per prefix. Two prefixes, one nested in the other, and a scoped relay asks every neighbour for the narrow one: no wildcards, interest edits, stream generations or cancellation, grants, qualified keys, or hidden claims. Leaves out restarts, priorities other than 0 and 1, a priority that differs between a link's two ends, costs other than 1 and 2, RTT, upstream/down-only links, and the hop limit. Messages come only from honest routers: validation of an unrelated DEAD or its far node, and empty or overlong replacement paths, is outside the model's input domain. Rust fixtures cover those guards and exportable covering-source selection, including keeping a covering hold when every live source is unexportable. Interest-edit fixtures also cover a cancelled stream retaining its pending child replies; the model has no stream cancellation. A claims message carries the sender's whole view for its prefix, equal to applying its changes on a FIFO stream. A route is sent on every stream whether or not the publisher has a claim there. The publisher ignores what it hears. `Overlap.cfg` also simulates five relays with every session up. No exhaustive pass has more than four relays or two session changes; `Sim`'s sweeps on the path rule (`sweep_handoff`, `sweep_scoped_handoff`, `sweep_scoped_meshes_show_what_the_rule_says`, and `compare.rs`'s churn) cover larger meshes, grants, qualified keys, interest edits, restarts, and churn. +One publisher, so routes for several publishers are left out, and so are the copies and copy rank of cache-aware routing, which `rs/moq-net/src/route/tests/cache.rs` sweeps; the DEAD message's per-publisher batching reduces to one route per prefix. Two prefixes, one nested in the other, and a scoped relay asks every neighbour for the narrow one: no wildcards, interest edits, stream generations or cancellation, grants, qualified keys, or hidden claims. Leaves out restarts, priorities other than 0 and 1, a priority that differs between a link's two ends, costs other than 1 and 2, RTT, upstream/down-only links, and the hop limit. Messages come only from honest routers: validation of an unrelated DEAD or its far node, and empty or overlong replacement paths, is outside the model's input domain. Rust fixtures cover those guards and exportable covering-source selection, including keeping a covering hold when every live source is unexportable. Interest-edit fixtures also cover a cancelled stream retaining its pending child replies; the model has no stream cancellation. A claims message carries the sender's whole view for its prefix, equal to applying its changes on a FIFO stream. A route is sent on every stream whether or not the publisher has a claim there. The publisher ignores what it hears. `Overlap.cfg` also simulates five relays with every session up. No exhaustive pass has more than four relays or two session changes; `Sim`'s sweeps on the path rule (`sweep_handoff`, `sweep_scoped_handoff`, `sweep_scoped_meshes_show_what_the_rule_says`, and `compare.rs`'s churn) cover larger meshes, grants, qualified keys, interest edits, restarts, and churn. diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index cb55d9cdd17..193765b2ae2 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -412,8 +412,8 @@ The publisher replies with a single ANNOUNCE_OK message followed by the matching - ANNOUNCE_UPDATE: a claim's cost changed. - ANNOUNCE_RESTART: a claim now serves another Instance, or a higher Epoch of the same one. - ANNOUNCE_FLOOR: a batch of a route's claims is complete. -- WARM_START: the publisher keeps a copy of a broadcast whether or not the subscriber pulls through it (see [Warm Routes](#warm-routes)). -- WARM_UPDATE: a warm route's Warm Cost, carrier, or Hops changed. +- WARM_START: the publisher offers the copy of a broadcast the subscriber gets by pulling through it (see [Warm Routes](#warm-routes)). +- WARM_UPDATE: a warm route's Warm Cost, drain, priority, or Hops changed. - WARM_END: the publisher withdrew a warm route. - SYNCED: the initial set is complete. @@ -470,7 +470,7 @@ The subscriber MUST close the session with a PROTOCOL_VIOLATION if a non-zero `P A resolved path that breaks its own rules, or a base whose path the subscriber refused, refuses the claim (see [Refused Content](#refused-content)). #### Refused Content {#refused-content} -Route messages carry content the publisher may only have forwarded: a route's Node ID, Seqno, Flags, and Hops, a claim's path, Instance, and Lite-06 Chain, and a warm route's path, Instance, Carrier Index, and Hops. +Route messages carry content the publisher may only have forwarded: a route's Node ID, Seqno, Flags, and Hops, a claim's path, Instance, and Lite-06 Chain, and a warm route's path, Instance, Flags, and Hops. A newer version or release may send such content that an older receiver cannot use, so a receiver that refuses it withdraws only the route or claim it describes and keeps the session, as BGP treats a malformed attribute as a withdrawal ({{?RFC7606}}). This specification marks each such value "refused"; the receiver skips the rest of the message's body, and SHOULD log and count each refusal so a refused route is never silently missing. @@ -588,32 +588,32 @@ An advertiser signals capacity through its claim alone, withdrawing or re-pricin #### Warm Routes {#warm-routes} A relay MAY pull each broadcast through a neighbor of its choosing rather than along its selected route to the origin node, so a broadcast crosses fewer relays and neighboring relays share one copy. -A warm route offers a neighbor a copy of one broadcast: the Hops it crossed, its `Warm Cost`, and its carrier. -An endpoint that does not choose per broadcast ignores every WARM_START, WARM_UPDATE, and WARM_END, sends none, and pulls each broadcast along its selected route. +A warm route offers a neighbor a copy of one broadcast: the Hops it crossed, its `Warm Cost`, whether it drains, and its priority. +An endpoint that does not choose per broadcast ignores every WARM_START, WARM_UPDATE, and WARM_END, sends none, pulls each broadcast along its selected route, and says so in each SUBSCRIBE. A relay's **source** for a broadcast is where it pulls it from: its next hop on its selected route to the origin node, or a neighbor whose warm route it joins. -Pulling through a neighbor gets whatever that neighbor pulls, so pulling through the next hop joins its copy when it offers one. -A copy's **carrier** is the relay with a local reader where the copy left the origin node's route: a relay with a reader whose next hop offers it no copy. -A copy's Hops are those of the route as far as its carrier, then the Hop ID of each relay that joined it, the sender last, as for a route's Hops (see [Routing](#routing)). -Its `Warm Cost` is 0 at its carrier, and what joining it cost every relay after: the `Warm Cost` it joined plus the cost of the link it joined over, so a copy's cost rises with every link from where it left the route. - -A relay with a source offers its copy, if it has one, on every Announce Stream it answers whose request covers the broadcast's path, under its route to the origin node: the copy it joined, or, as a carrier, the one it pulls. -It offers the copy even to a neighbor its Hops name, which tells that neighbor that pulling through the relay leads back to it. -A relay with no demand left offers the copy it would join, so a neighbor can price it before it pulls, and keeps its pull only for as long as it keeps any copy whose demand is gone. -It sends a WARM_UPDATE when the copy it offers changes, and a WARM_END once it offers none, so a broadcast no relay carries costs no message of its own. - -A relay MUST NOT join a copy whose Hops name it or a neighbor whose subscription to the broadcast it serves, or whose Hops it would refuse for a route (see [Routing](#routing)), and MUST NOT join a copy of another Instance. -Among its next hop and the copies it may join, it SHOULD pull through the cheapest: a copy costs its `Warm Cost` plus the link's cost, and the next hop without a copy the relay's metric to the origin node. -A tie stays on the current source, then prefers a copy. -It SHOULD NOT pull through a copy offered on a session that received a GOAWAY, unless it is its next hop's, and moves a pull already on one. - -Two relays that start carrying in the same instant would each join the other's copy for good, so copies are ranked by the Rendezvous Hash of the broadcast's path and their carrier's Hop ID, as defined in [Resolution](#resolution). -A relay on a copy, as its carrier or by joining one still offered, MUST NOT move to another copy, its next hop's included, unless that copy's carrier is the same or ranks lower. -A relay that pulls along its route with no reader of its own hands whatever it joins to the neighbors that pull through it, or would on their selected routes, so it is bound by their carriers instead: each such neighbor offers back its copy with Hops that begin with the relay's own route's. -It MUST NOT join a copy other than its next hop's unless its carrier ranks at or below those of all of these offers, nor while a neighbor whose subscription it serves has offered none. -An offer whose Hops do not begin with the relay's route was sent before the relay took that route, and counts as none: it bounds no carrier, and from a neighbor whose subscription the relay serves it forbids the join. - -The Hops keep the pulls of a broadcast free of loops once the copies offered are delivered: no relay joins a copy that passes through itself or a neighbor it serves. +A source ranks as a route does (see [Routing](#routing)): one crossing no draining session first, then the one whose lowest link priority is highest, then the cheapest. +The next hop ranks as the relay's selected route; a warm route ranks by its own drain, priority, and `Warm Cost`, combined with the link it arrives on as a route's are. +Among sources of equal rank a relay prefers a warm route, then the one whose carrier has the lower Rendezvous Hash, then any fixed order. + +A relay with a local reader of a broadcast is that copy's **carrier**: it offers every neighbor its copy, with its own Hop ID as the only Hop, a `Warm Cost` of 0, no drain, and a priority of 127, whatever it pulls. +A relay with no local reader offers each neighbor the best of its sources whose Hops name neither relay, when that is a warm route: its Hops with the relay's own Hop ID appended, and its rank as the relay sees it. +What a relay offers so depends only on its readers, its route, and the warm routes offered to it, never on who pulls what, and a warm route ranks no better at each relay it crosses, so the warm routes of a broadcast settle once its readers and the routes do, as any path vector whose rank only worsens along a path does. +A relay offers a warm route only while it has a selected route to the origin node and the Hops fit in 32, on the Announce Stream it answers whose request covers the broadcast's path, under that route. +It sends a WARM_UPDATE when the copy it offers a neighbor changes, and a WARM_END once it offers none, so a broadcast no relay reads costs no message of its own. + +A carrier pulls through its best source, so what it pulls depends only on the warm routes offered to it and its route, never on who pulls from it. +A relay with no local reader that has a neighbor's SUBSCRIBE pulls through its best source whose Hops name no neighbor whose subscription it serves. +A carrier MUST NOT join a warm route whose carrier ranks above its own by the Rendezvous Hash of the broadcast's path and the carrier's Hop ID, as defined in [Resolution](#resolution), so two carriers never join each other. +Each SUBSCRIBE says in `Warm` whether it joins the copy the publisher offered or follows the subscriber's route. +A relay that serves a SUBSCRIBE from a relay following its route MUST pull along its own route, so a pull that follows routes reaches the origin node. +A relay with no local reader gets the copy it pulls to each neighbor whose subscription it serves, so it MUST pull along its own route unless, for each of them it offered a warm route, what it pulls is that warm route or a copy whose carrier ranks below that warm route's carrier; a neighbor it offered none, such as one whose SUBSCRIBE crossed a WARM_END, bounds nothing. +A relay MUST NOT join a copy of another Instance. + +Once the warm routes and subscriptions in flight are delivered, the pulls of a broadcast are free of loops. +A relay following its route forwards along the routes' tree to the origin node. +A relay that joins a warm route gets a copy whose carrier, then whose warm route, ranks strictly better than what it gives each relay joined to it: a carrier gives its own copy and joins only a lower-ranked carrier's, and a relay with no local reader joins either the warm route it offered, which ranks strictly better one hop closer to its carrier, or a lower-ranked carrier's copy. +So no chain of joins returns to a relay already on it, and a join that reaches a relay following its route continues along the routes' tree. Two relays acting on each other's stale offers can still join each other until the WARM_END or WARM_UPDATE crossing their link arrives, and the pull then moves on. ### Subscribe @@ -1529,7 +1529,8 @@ WARM_START Message { Instance (b), Minted (8), Warm Cost (i), - Carrier Index (i), + Flags (8), + Priority (8), Hop Count (i), Hop ID (i) ..., } @@ -1549,18 +1550,21 @@ Combined with the requested prefix to form the broadcast's path. The Instance of the copy the publisher carries, as defined for [ANNOUNCE_START](#announce-start). **Warm Cost**: -What pulling the copy cost the publisher, 0 when it is the copy's carrier (see [Warm Routes](#warm-routes)), in the units of the route's Metric. +What pulling the copy costs the publisher, 0 when it is the copy's carrier (see [Warm Routes](#warm-routes)), in the units of the route's Metric. + +**Flags**: +A bit field: 0x1 when a session on the copy's Hops drains (see [Routing](#routing)). +Any other bit set is refused (see [Refused Content](#refused-content)). -**Carrier Index**: -The position of the copy's carrier among the Hop IDs that follow, 0 for the first. -A value not below `Hop Count` is refused (see [Refused Content](#refused-content)). +**Priority**: +The lowest local priority of a link on the copy's Hops as a signed 8-bit integer, 127 at its carrier (see [Routing](#routing)). **Hop Count**: The number of Hop IDs that follow. -More than 32 is refused. +0, which names no carrier, or more than 32 is refused. **Hop ID**: -The copy's Hops (see [Warm Routes](#warm-routes)), in order, the publisher's own last. +The copy's Hops (see [Warm Routes](#warm-routes)), in order, its carrier first and the publisher's own last. A stream has at most one live warm route per route and Broadcast: a relay still pulling an Instance for readers pinned to it offers that copy beside the one of the Instance that replaced it. A subscriber that uses warm routes MUST close the session with a PROTOCOL_VIOLATION on a WARM_START naming a Route ID that was never assigned or already retired, or a path and Instance its route already has a live warm route for. @@ -1576,7 +1580,8 @@ WARM_UPDATE Message { Message Length (i) Warm ID (i), Warm Cost (i), - Carrier Index (i), + Flags (8), + Priority (8), Hop Count (i), Hop ID (i) ..., } @@ -1588,7 +1593,7 @@ Set to 0xD to indicate a WARM_UPDATE message. **Warm ID**: The ordinal implicitly assigned by a prior WARM_START on this stream. -**Warm Cost**, **Carrier Index**, **Hop Count**, and **Hop ID**: +**Warm Cost**, **Flags**, **Priority**, **Hop Count**, and **Hop ID**: As defined for [WARM_START](#warm-start), replacing the warm route's. @@ -1632,6 +1637,7 @@ SUBSCRIBE Message { Range Count (i) Range (..) ... Update (i) + Warm (8) } Range { @@ -1704,6 +1710,10 @@ A Range resolves once: an update that leaves it live does not resolve it again. The number of the subscription's terms this message sets: 0 on SUBSCRIBE, and on each SUBSCRIBE_UPDATE a larger value than the one before it. A [SUBSCRIBE_DROP](#subscribe-drop) echoes the number in effect when its groups were given up, so the subscriber can tell a stale verdict on its current terms from one an update has since replaced. +**Warm**: +1 when the subscriber joins the copy the publisher offered it in a warm route, and 0 when it follows the subscriber's selected route, as an endpoint that does not choose per broadcast always does (see [Warm Routes](#warm-routes)). +Any other value is a PROTOCOL_VIOLATION. + A `Start` is a floor, not a request; `Subscriber Max Age` is the only thing that asks for data. The publisher SHOULD start at the oldest group at or above the floor that [Expiration](#expiration) has not expired, and MUST NOT deliver an older one. A `Subscriber Max Age` of 0 therefore starts at the latest group, since every older group is already stale. @@ -1737,6 +1747,7 @@ SUBSCRIBE_UPDATE Message { Range (..) ... Update (i) Wanted Group (i) + Warm (8) } ~~~ @@ -2437,7 +2448,7 @@ An application-specific payload. - The Cost parameter must be at least 1. - Replaced the shortest-path, Hop ID, Instance Hash, and Chain Hash tie-breaks with the local priority of the route's session, the Announce Cost plus the route's metric, and a Rendezvous Hash of the requested path and the origin's Node ID. - Removed the standby-ordering deployment requirements on Route Cost seeds and bounds. -- Added warm routes: WARM_START (0xC), WARM_UPDATE (0xD), and WARM_END (0xE) on the Announce Stream. A relay MAY pull each broadcast through a neighbor that offers a copy of it, carrying the copy's Hops, its Warm Cost, and its carrier. A relay never joins a copy whose Hops name it or a neighbor it serves, and moves between copies only toward a carrier with a lower Rendezvous Hash, so the pulls of a broadcast are free of loops once the copies offered are delivered. +- Added warm routes: WARM_START (0xC), WARM_UPDATE (0xD), and WARM_END (0xE) on the Announce Stream, and `Warm` on SUBSCRIBE and SUBSCRIBE_UPDATE. A relay MAY pull each broadcast through a neighbor that offers a copy of it, carrying the copy's Hops from its carrier, its Warm Cost, its drain, and its priority. A carrier offers its copy whatever it pulls and joins only a lower-ranked carrier's; a relay with no reader offers each neighbor its best source that avoids it, pulls it only while that is what it offered each relay it serves, and follows its route when a neighbor following its own pulls through it. The offers settle as a path vector does, and the pulls are then free of loops. - Added the temporary Lite-06 Chain Setup Parameter (0x9) and a trailing Lite-06 Chain on ANNOUNCE_START, ANNOUNCE_UPDATE, and ANNOUNCE_RESTART: the moq-lite-06 Hop ID list of a claim learned over moq-lite-06, which a node drops when it names that node. ### Tracks {#changelog-07-tracks} diff --git a/js/net/src/lite/subscribe.test.ts b/js/net/src/lite/subscribe.test.ts index 678dc05a089..2027cfd67c2 100644 --- a/js/net/src/lite/subscribe.test.ts +++ b/js/net/src/lite/subscribe.test.ts @@ -203,10 +203,11 @@ test("lite-07 carries the span as its only range, exclusive at the end", async ( test("a lite-07 list decodes as the span covering it", async () => { // Priority, order, max delay, and an unstated min delay as SubscribeUpdate writes them, - // then no Range ID to retire, the ranges it adds, the update number, and an absent wanted group. + // then no Range ID to retire, the ranges it adds, the update number, an absent wanted group, and + // a pull that follows the subscriber's route. const head = (await encodeMessage(Version.DRAFT_07, new SubscribeUpdate({ priority: 0, ranges: [] }))).slice(1, 5); const update = async (ranges: number[]) => { - const body = [...head, 0, ...ranges, 0, 0]; + const body = [...head, 0, ...ranges, 0, 0, 0]; const got = await SubscribeUpdate.decode( new Reader(undefined, new Uint8Array([body.length, ...body]), Version.DRAFT_07), Version.DRAFT_07, diff --git a/js/net/src/lite/subscribe.ts b/js/net/src/lite/subscribe.ts index 439b1097f0e..6c3bba03a76 100644 --- a/js/net/src/lite/subscribe.ts +++ b/js/net/src/lite/subscribe.ts @@ -28,6 +28,7 @@ import { hasRanges, hasRefresh, hasSkip, + hasWarm, resolvesStart, Version, } from "./version.ts"; @@ -42,6 +43,18 @@ async function encodeUpdate(w: Writer, version: Version, update: bigint) { if (hasSkip(version)) await w.u62(update); } +/** Encode `Warm`: the pull follows this endpoint's route (0), since it chooses no source per broadcast. */ +async function encodeWarm(w: Writer, version: Version) { + if (hasWarm(version)) await w.u8(0); +} + +/** Decode `Warm`, whether the pull joins this side's copy, and ignore it: only a relay choosing a source per broadcast acts on it. */ +async function decodeWarm(r: Reader, version: Version) { + if (!hasWarm(version)) return; + const warm = await r.u8(); + if (warm > 1) throw new ProtocolViolation(`unknown warm ${warm}`); +} + /** * Encode the `Group Start` field shared by SUBSCRIBE and SUBSCRIBE_UPDATE. * @@ -695,6 +708,7 @@ export class SubscribeUpdate { } await encodeUpdate(w, version, this.update); if (hasRefresh(version)) await w.u62(this.wantedGroup === undefined ? 0n : this.wantedGroup + 1n); + await encodeWarm(w, version); break; } } @@ -720,6 +734,7 @@ export class SubscribeUpdate { } const update = await decodeUpdate(r, version); const wanted = hasRefresh(version) ? await r.u62() : 0n; + await decodeWarm(r, version); return new SubscribeUpdate({ wantedGroup: wanted === 0n ? undefined : wanted - 1n, priority, @@ -853,6 +868,7 @@ export class Subscribe { break; } await encodeUpdate(w, version, this.update); + await encodeWarm(w, version); } static async #decode(r: Reader, version: Version): Promise { @@ -873,6 +889,7 @@ export class Subscribe { const minDelay = await decodeMinDelay(r, version); const span = await decodeSpan(r, version); const update = await decodeUpdate(r, version); + await decodeWarm(r, version); return new Subscribe({ id, broadcast, diff --git a/js/net/src/lite/version.ts b/js/net/src/lite/version.ts index 56dc6455bdc..b635e705bb4 100644 --- a/js/net/src/lite/version.ts +++ b/js/net/src/lite/version.ts @@ -488,6 +488,23 @@ export function hasAnnounceCompression(version: Version): boolean { } } +/** Whether the announce stream carries warm routes (WARM_START, WARM_UPDATE, and WARM_END), and + * SUBSCRIBE and SUBSCRIBE_UPDATE say whether each pull joins the publisher's copy. Added in lite-07. */ +export function hasWarm(version: Version): boolean { + // Explicitly list older versions so future versions keep the lite-07+ behavior. + switch (version) { + case Version.DRAFT_01: + case Version.DRAFT_02: + case Version.DRAFT_03: + case Version.DRAFT_04: + case Version.DRAFT_05: + case Version.DRAFT_06: + return false; + default: + return true; + } +} + /** Whether TRACK_INFO carries the refresh ratio and SUBSCRIBE_UPDATE a wanted group. Added in lite-07. * Older peers see a track that never refreshes. */ export function hasRefresh(version: Version): boolean { diff --git a/quest/m2/cache-routing-proof.md b/quest/m2/cache-routing-proof.md deleted file mode 100644 index 25bf364906d..00000000000 --- a/quest/m2/cache-routing-proof.md +++ /dev/null @@ -1,132 +0,0 @@ -# [L] Prove cache-aware routing - -## Goal - -Sweeps over the route layer show that the [epic](/quest/m3/cache-routing/README.md)'s -warm routes, which propagate per broadcast, stay loop-free once the offers in -flight are delivered, under stale, reordered, and in-flight messages. Once demand and topology stop -changing, every demanded broadcast is delivered, parent choices stop -changing, and a relay carries a broadcast only when it has a local reader, -feeds two or more downstream sessions, or is its consumers' cheapest route -given the copies the others hold, as the epic's Goal states. Lite-07's warm -message carries the proven rule. - -## Plan - -Decided 2026-11-09 (maintainer, /quest-plan from Kixel's audit of its fleet -plan): ✅ pull the proof into m2, ahead of the lite-07 cut, so a warm -message that ships in lite-07 is the proven one (rejected: a design-only -quest in m2 that sizes the message with no proof; leaving this in m3 and -taking an additive SETUP extension if it lands after the cut). This reverses -the 2026-11-08 choice of m3 for the proof alone; -[the implementation](/quest/m3/cache-routing/select.md) stays in m3. - -Reopened 2026-11-11 (maintainer, /quest-plan on the route redesign, epic -Plan item 13): a warm route propagates, and the rule that keeps it free of -loops and stale copies is the route's path -([routes epic](/quest/m2/cluster-routes/README.md) Plan item 27, -picked 2026-11-11 on -[moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533)), which also -removes the seqno, ROUTE_REQUEST, and the ROUTE_END echo. - -[moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457) proved the -one-hop warm route first decided (epic items 8 to 12). - -Found 2026-11-12 ([moq-dev/moq#8615](https://github.com/moq-dev/moq/pull/8615)), -rewriting `rs/moq-net/src/route/tests/cache.rs` on the path prototype in -`paths.rs` (Rules item 27 picked: causes, holds, a 4-hop doubt timer), with -demand, subscriptions on their own streams, lingers, GOAWAY, legacy peers, -and instances. A path alone does not settle copies that propagate; each rule -below was forced by a sweep scenario that otherwise never settles or loops -for good: - -- **Monotone copy costs.** Only a relay with a reader whose pull follows the - publisher's route offers its copy at no cost; a relay that joined a copy - offers it at what its own pull costs. Pricing every reader's copy at 0 - made a dispute wheel round a ring of five, two consistent states that a - lockstep schedule flips between forever (scenario 77). Joining a copy - still costs only the links from where it left the publisher's route. -- **Carrier rank.** Every copy names its carrier, the relay with a reader - where it left the publisher's route. A relay on a copy, its own included, - moves onto another, its next hop's included, only if that copy's carrier - ranks below by the Rendezvous Hash. A rank per pair of neighbours, or only - between readers, left longer crossing cycles trading places (scenarios 139 - and 72). -- **A relay off a copy is bound by the copies it feeds.** A relay pulling - along its route with no reader of its own, or with no demand at all, hands - whatever it joins to every neighbour that pulls through it or whose route's - next hop it is, since a next hop's copy is no choice. It joins a copy other - than its next hop's only if that copy's carrier ranks at or below the - carrier each such neighbour offers back along this relay's route, and none - while a neighbour that pulls through it has offered none. Free, a relay - with no reader joined a higher-ranked copy and relabelled the carrier - behind it, and a copy circulated round a ring of five (scenario 246 of the - long sweep); an offer back along another path is an echo of the relay's - own old copy, which kept two relays crossing every four rounds. -- **Truthful offers.** A relay offers every neighbour that reads copies its - current copy, even one whose path names that neighbour (poisoned reverse, - not split horizon), and pulling through a neighbour means its copy when it - offers one, its route only when it does not. With split horizon a request - that follows a route reaches a relay that forwards it into a copy leading - back, a cycle that outlives quiescence (scenario 1563 of the long sweep). - A copy whose path names a neighbour the relay serves is unusable. A relay - serving a legacy neighbour may still join a copy: the path check keeps out - one through that neighbour, and refusing every copy while its subscription - came and went kept a ring flipping. -- **GOAWAY** moves every pull off a draining neighbour's copy other than - the next hop's, rather than ranking it last ahead of cost, which let two - relays each prefer the other's propagation of the draining carrier's copy - (scenario 149 without holds). -- **A pull on another relay's behalf offers nothing new**, so `Serves` and - its roots are not needed: a relay with no reader offers only a copy it - joined, at its cost. - -Decided 2026-11-13 (maintainer, /quest-run): ✅ accept a cycle that lasts -until the crossing withdrawal is delivered, as routes item 27 accepted for -selected routes, and word the Goal "loop-free once the offers in flight are -delivered" (rejected: a per-broadcast distance, the carrier's rank then the -cost, with a reset by echo or timer, which brings back the seqno and echo -item 27 removed). Two relays acting on each other's stale offers both join -in the same instant, which only such a distance prevents. - -Measured (`sweep_cache_routing_long`, 3000 scenarios each with and without -held links, the gate's seed): every scenario settles on the steady state, an -idle broadcast sends no warm message, and about 3.2 warm messages go out per -reader change. A broadcast's pulls form a cycle after 0.39% of churn steps with -held links and 0.50% without, each ending when the withdrawal crossing its -link is delivered, at most 53 deliveries later in the sweep's random order. -Each subscription is a stream of its own, so a replacement pull can overtake -the cancel of the stream it replaces, and the steady state holds one stream -per neighbour fed. The -LAN fixture keeps one uplink copy per broadcast in a full mesh of 6, and 15 -copies for 8 broadcasts in a ring of 6. `settle` lets a hop of delay pass every -64 rounds of deliveries, so a doubt lifts during a long exchange as it would. - -Lite-07 carries the rule: WARM_START and WARM_UPDATE carry the copy's Warm -Cost, its carrier's position in its Hops, and the Hops, and `Serves` left -SUBSCRIBE and SUBSCRIBE_UPDATE -([moq-dev/moq#8615](https://github.com/moq-dev/moq/pull/8615)). - -Remaining: - -- Under other seeds the long sweep still finds scenarios that never settle, - 2 in 48,000 across 8 seeds (the rules first proven here failed every seed - within 1,200 scenarios): `MOQ_ROUTE_SEED=305419896` scenarios 196 and - 1391, both without held links. Each rule above closed one such class and - exposed a rarer one, and binding a - relay by the copy a neighbour routed through it sits on now, wherever it - came from, made a third seed fail, so the next step is an argument that - the rules converge, such as a TLA+ model of the copy choice alongside - `verify-route`'s, rather than another rule per scenario. Replay one with - `MOQ_ROUTE_SEED= cargo test -p moq-net --release --lib - route::tests::cache::sweep_cache_routing_long -- --ignored`. -- Once fresh seeds settle, run the long sweep nightly from - `test/route-sweep.sh` with a fresh `MOQ_ROUTE_SEED`, and drop its line from - `sh/gh/test-inventory.allow`. -- The [epic](/quest/m3/cache-routing/README.md)'s items 2, 3, 9, and 10 - describe the one-hop warm route these rules replace; its item 14 records - the replacement. - -## Related - -- [Routes and announces](/quest/m2/cluster-routes/README.md) - the path rule this extends to warm routes diff --git a/quest/m2/cluster-routes/route-exhaustive-sim.md b/quest/m2/cluster-routes/route-exhaustive-sim.md index bf96557f709..ee2fa8c346f 100644 --- a/quest/m2/cluster-routes/route-exhaustive-sim.md +++ b/quest/m2/cluster-routes/route-exhaustive-sim.md @@ -54,7 +54,7 @@ Decided 2026-11-12 (maintainer, pick round): the search runs on the `Router` after [the router quest](https://github.com/moq-dev/moq/pull/8622)'s rewrite to the path rule (`route::path::Router`), so it checks the code that ships and nothing is built twice (rejected: today's Babel `Router`, which the rewrite throws away; the test-only -`paths.rs` prototype). The storm self-check above then runs against a router +`paths.rs` prototype, since deleted). The storm self-check above then runs against a router from before the rewrite. Public API: none. Wire: none. diff --git a/quest/m2/cluster-routes/switch.md b/quest/m2/cluster-routes/switch.md index 200dfee5a4b..4cc9ed91e02 100644 --- a/quest/m2/cluster-routes/switch.md +++ b/quest/m2/cluster-routes/switch.md @@ -128,10 +128,10 @@ Work: Test it with the two-stream, two-instance case from #8443's review. - [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)'s warm-route sweep (`route/tests/cache.rs`) and `Router::eligible` rest on the - feasibility distance. The reopened - [cache-aware routing proof](/quest/m2/cache-routing-proof.md) rewrites them - for the path; if it has not landed, move the sweep onto the new router's - query here and say what it no longer checks. + feasibility distance. + [moq-dev/moq#8732](https://github.com/moq-dev/moq/pull/8732) rewrites the + sweep on `route::path::Router`; if it has not landed, move the sweep onto the + new router's query here and say what it no longer checks. - Draft: Routing states the path rule, the withdrawal, holds, and the doubt. Floors says a floor certifies its route's prefix, and the condition "every peer whose last advertised route to it is feasible here has answered the diff --git a/quest/m2/lite07-finalize.md b/quest/m2/lite07-finalize.md index 683f4b4b002..620da2d333d 100644 --- a/quest/m2/lite07-finalize.md +++ b/quest/m2/lite07-finalize.md @@ -129,7 +129,6 @@ transcript freezes at the cut and a flaw in it can only be fixed on the wire. - [Lite-07 by start](/quest/m2/time-addressing/lite07.md) - lite-07 carries group starts, previous-start links, GROUP_END, and time bounds instead of sequence numbers - [@moq/net by start](/quest/m2/time-addressing/js.md) - JS speaks the same lite-07 wire - [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - may add SUBSCRIBE_DROP for a stopped group; leaves this list if it takes the no-wire path -- [Prove cache-aware routing](/quest/m2/cache-routing-proof.md) - rewrites lite-07's one-hop warm routes and `Serves` for routes that propagate ## Related diff --git a/quest/m3/cache-routing/README.md b/quest/m3/cache-routing/README.md index ec9dae6a653..f5eb60e30eb 100644 --- a/quest/m3/cache-routing/README.md +++ b/quest/m3/cache-routing/README.md @@ -30,9 +30,10 @@ Decided 2026-11-08 (maintainer, /quest-plan): cache-dependent route prices. (rejected: per-broadcast forwarding with a cache-only subscribe and no advertisement; keeping routing cache-agnostic) 2. ✅ **A warm route is per neighbor and travels one hop.** Reopened by - item 13; replaced by item 14. It goes only to - direct neighbors and is never forwarded. B is warm toward N only while B - has demand besides N's (a local reader or another downstream session), + item 13; replaced by item 14, where a warm route propagates and a relay + offers each neighbor the best copy whose Hops name neither. It went only to + direct neighbors and was never forwarded. B was warm toward N only while B + had demand besides N's (a local reader or another downstream session), and withdraws once that demand has been gone for the copy's existing linger, with no new timer. B sends it only to neighbors whose cold metric for the origin is at least its own, since no other neighbor may use it @@ -40,7 +41,9 @@ Decided 2026-11-08 (maintainer, /quest-plan): warm because it still feeds N, and N never skips it again. (rejected: broadcast-wide warmth, as the pre-#3225 design had; forwarding warmth k hops, which floods every warm transition) -3. ✅ **One cost per link, priced marginally.** Replaced by item 14. Pulling through a warm +3. ✅ **One cost per link, priced marginally.** Replaced by item 14: a copy + ranks as a route does, drain then priority then cost, and its carrier offers + it at no cost, so joining a copy costs only the links it crosses. Pulling through a warm neighbor costs the link alone, since that neighbor's cold metric is already paid; a cold neighbor costs the link plus its cold metric. A tie goes to the warm route, then to item 4's hash. A topology that wants both @@ -94,8 +97,9 @@ Decided 2026-11-10 (maintainer, /quest-iterate on [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)), from the proof's proposals: -9. ✅ **Each pull names whom it serves**, amending item 2; replaced by item 14: a trailing - `Serves` field on SUBSCRIBE and SUBSCRIBE_UPDATE. (rejected: no field) +9. ✅ **Each pull names whom it serves**, amending item 2; replaced by item 14, + whose SUBSCRIBE says only whether it joins a copy, and lite-07 no longer + carries `Serves`: a trailing `Serves` field on SUBSCRIBE and SUBSCRIBE_UPDATE. (rejected: no field) "Demand besides N's" can't be told from one hop: if B's only demand is a relay pulling on N's behalf, B counts warm toward N, N joins B, B's demand vanishes, and the cycle repeats every linger. `Serves` is alone (a local @@ -105,7 +109,9 @@ proof's proposals: narrowed warm route stands until the copy's linger runs out, as the withdrawal already does. 10. ✅ **Price a dependent copy at its cost, and move only on strict - improvement**, amending item 3; replaced by item 14; the Goal's "cheapest cold route" became + improvement**, amending item 3; replaced by item 14, whose offers do not + depend on who pulls what, and whose sweep settles every scenario with no + `UNSETTLED` list; the Goal's "cheapest cold route" became "cheapest route given the copies the others hold". (rejected: dropping mutual warmth so only a relay with a local reader advertises, which converges trivially but loses the LAN sharing that feeding two sessions @@ -119,9 +125,10 @@ proof's proposals: Under the sweep's churn, with the draft's Rendezvous Hash, 2999 of 3000 scenarios settle; scenario 2251 never does, as two carriers keep a narrowed route standing while two relays swap between them. The sweep names them in - `UNSETTLED` and checks each still never settles, and the - [proof's rewrite](/quest/m2/cache-routing-proof.md) settles them first. -11. ✅ **The wire is in moq-lite-07 itself** (its warm fields replaced by item 14), overturning the proof's choice + `UNSETTLED` and checks each still never settles; item 14's rule settles + them. +11. ✅ **The wire is in moq-lite-07 itself** (its warm fields and `Serves` + replaced by item 14's), overturning the proof's choice of a SETUP extension. (rejected: an additive Warm Routes SETUP extension) Lite-07 is unpublished, so its shape can still change until the cut, and a later change goes to the next version. The draft's @@ -180,19 +187,54 @@ Measured by the proof: [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457) proves items 2 and 4 as first decided. -14. **The proven rule** ([proof](/quest/m2/cache-routing-proof.md), - [moq-dev/moq#8615](https://github.com/moq-dev/moq/pull/8615)), on the - path prototype with item 27's routes. A warm route offers a copy's Hops, - its Warm Cost (0 at its carrier, the relay with a reader where it left the - route; else the cost of the sender's own pull), and its carrier. A relay - pulls through the cheapest of its next hop and the copies whose Hops name - neither it nor a neighbor it serves, offers its copy to every neighbor - even one the Hops name, moves between copies only toward a carrier with a - lower Rendezvous Hash, and, pulling along its route, is bound by the - carriers of the neighbors behind it. `Serves` is gone. The pulls are free - of loops once the copies offered are delivered, which the maintainer - accepted on 2026-11-13. The proof's remaining work is a few scenarios - under fresh seeds that never settle. +14. **The proven rule** ([moq-dev/moq#8732](https://github.com/moq-dev/moq/pull/8732)), + proven in `rs/moq-net/src/route/tests/cache.rs` over `route::path::Router` + ([moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622)), in three + layers that each settle before the next depends on them: + - **Routes** are item 27's, which copies never change. + - **Offers.** A relay with a local reader is its copy's carrier and offers + every neighbor its copy, its own Hop ID the only hop, at no cost. A relay + with no reader offers each neighbor the best of its route and the copies + offered to it whose Hops name neither relay, when that best is a copy. + Copies rank as routes do, drain then priority then cost, so an offer + depends only on readers, routes, and other offers, and settles as a + monotone path vector does. + - **Pulls.** A carrier pulls through its best source among its route and + the copies of carriers ranked below it by the Rendezvous Hash. A relay + with no reader pulls through its best source whose Hops name no + neighbor it serves, so long as that is, for each such neighbor, the copy + it offered it or a lower-ranked carrier's; otherwise it follows its + route. SUBSCRIBE's `Warm` says whether a pull joins the copy offered or + follows the route, and a relay pulled along its route follows its own. + + Each join gets a copy whose carrier, then whose rank, is strictly better + than what it gives the relays joined to it, so once the offers and + subscriptions in flight are delivered the pulls are free of loops, as the + maintainer accepted on 2026-11-13. Unranked, two carriers join each other + and neither is fed. This replaces the rules of + [moq-dev/moq#8615](https://github.com/moq-dev/moq/pull/8615) (monotone + copy costs, carrier rank on every move, a relay bound by the carriers + behind it), which failed a few scenarios per seed because what a relay + offered depended on who pulled through it. + + Measured by `sweep_cache_routing_long` (3000 scenarios each with and + without held links, 6 relays, 2 publishers, priorities and drains), under + 12 seeds (72,000 scenarios): every scenario settles on the offers computed + centrally, an idle broadcast sends no warm message, and about 3.1 warm + messages go out per reader change. A pull lingering with no demand can hold + a copy its parent promised another neighbour, so the pulls around it may + move until the linger (30 s) runs out; settling counts from there. During + churn a broadcast's pulls form a cycle after 3.3% of steps, in about half + the scenarios, each gone within 142 deliveries. The sweep runs nightly + from `test/route-sweep.sh`. + + The wire: WARM_START and WARM_UPDATE carry a copy's Warm Cost, a Flags + byte whose low bit marks a draining copy, its priority, and its Hops, + carrier first and the sender last. SUBSCRIBE and SUBSCRIBE_UPDATE end with + a `Warm` byte. Rust's codec reads and writes both, and `js/net` the `Warm` + byte, which both send as 0 (follow the route) and ignore when read, as + `js/net` skips the warm messages; [the implementation](/quest/m3/cache-routing/select.md) + acts on them. Prior art: warm/cold route prices ([moq-dev/moq#2925](https://github.com/moq-dev/moq/pull/2925)) were deleted diff --git a/quest/m3/cache-routing/select.md b/quest/m3/cache-routing/select.md index 96f4a3d4ba0..e45103b0cb9 100644 --- a/quest/m3/cache-routing/select.md +++ b/quest/m3/cache-routing/select.md @@ -12,14 +12,14 @@ and how many moves warm routes caused. ## Plan -- Implement what [Prove cache-aware routing](/quest/m2/cache-routing-proof.md) - settles for routes that propagate (the epic's item 13) and pass its - fixtures and sweep: the warm message as the proof shapes it, warm state - derived from each front's demand, and eligibility. Lite-07 today carries the - one-hop warm routes and `Serves` that `rs/moq-net/src/route/tests/cache.rs` - models, and `rs/moq-net/src/lite/` reads and ignores them; the proof - rewrites both first. Once every scenario settles, run the long sweep - nightly, which the merge gate skips as slow. +- Implement the rule the [epic](/quest/m3/cache-routing/README.md)'s item 14 + records ([moq-dev/moq#8732](https://github.com/moq-dev/moq/pull/8732)) and + pass the fixtures and sweep in `rs/moq-net/src/route/tests/cache.rs`: the + warm messages lite-07 carries, which `rs/moq-net/src/lite/` reads and + ignores today, and SUBSCRIBE's `Warm` byte, which it and `js/net` send as 0 + (follow the route) and ignore when read, warm state derived + from each front's demand, and the choice of what to pull. The long sweep + already runs nightly from `test/route-sweep.sh`. - Extend today's selection rather than adding a second selector. `route_order` in `rs/moq-net/src/model/origin.rs` already hashes equal routes per path, and a front already moves to a better route of its own @@ -44,15 +44,6 @@ change the implementation needs updates the draft's Warm Routes section in the same PR, in lite-07 while it is unpublished and in the next version after. -Noted 2026-11-12 (/quest-audit): this plan predates the path rule (epic Plan -item 8, [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533)). Its -one-hop warm routes and feasibility-distance eligibility are rewritten once -[moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457) settles. - -## Required - -- [Prove cache-aware routing](/quest/m2/cache-routing-proof.md) - the rule for routes that propagate, its fixtures, and the wire this implements - ## Related - [Routes and announces](/quest/m2/cluster-routes/README.md) - the route layer driving lite-07 sessions since [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023), whose feasibility distances eligibility reads diff --git a/quest/m3/route-trace-validation.md b/quest/m3/route-trace-validation.md index 5f2ec74099d..e4d12ed9fab 100644 --- a/quest/m3/route-trace-validation.md +++ b/quest/m3/route-trace-validation.md @@ -21,7 +21,7 @@ Decided 2026-11-10 (maintainer, a later /quest-plan the same day): - ✅ m1 (rejected: staying in m4; superseded 2026-11-13: m3, above). The mesh configs beside it, the inductive invariant (dropped 2026-11-13: no checker can step the model), and the - [cache-aware routing proof](/quest/m2/cache-routing-proof.md) all build on + cache-aware routing proof all build on `RouteHandoff.tla`, so drift there costs the most. - ✅ The other three models get trace validation too, each its own m4 quest behind this one (rejected: "route only for now; judge the other models diff --git a/rs/moq-net/src/lite/announce.rs b/rs/moq-net/src/lite/announce.rs index c224180eee3..c161c89e74a 100644 --- a/rs/moq-net/src/lite/announce.rs +++ b/rs/moq-net/src/lite/announce.rs @@ -30,7 +30,7 @@ const ROUTE_END: u64 = 8; const ANNOUNCE_FLOOR: u64 = 9; const ROUTE_REQUEST: u64 = 10; const ROUTE_HOLD: u64 = 11; -// lite-07: warm routes, one hop toward the subscriber. +// lite-07: warm routes, copies of one broadcast offered toward the subscriber. const WARM_START: u64 = 12; const WARM_UPDATE: u64 = 13; const WARM_END: u64 = 14; @@ -118,9 +118,9 @@ pub enum AnnounceBroadcast { /// with this id, certifying them across the whole requested prefix at the origin's /// revision, or certifying nothing (`None`). Floor { id: u64, revision: Option }, - /// WARM_START (lite-07+): under the route with Route ID `route`, the sender keeps its - /// copy of the broadcast at `suffix` whether or not the receiver pulls through it, and - /// assigns it the next Warm ID. See [`Warm`] for what the copy carries. + /// WARM_START (lite-07+): under the route with Route ID `route`, the sender offers the + /// copy of the broadcast at `suffix` that the receiver gets by joining it, and assigns + /// it the next Warm ID. See [`Warm`] for what the copy carries. WarmStart { route: u64, suffix: Path, @@ -167,7 +167,7 @@ pub(crate) enum Refusal { Flags(u8), #[error("{0} hops, past the limit")] Hops(u64), - #[error("a carrier not among its hops")] + #[error("a warm route with no hops, so no carrier")] Carrier, #[error("an invalid path")] Path, @@ -190,24 +190,30 @@ fn refuse(r: &mut Decoder<'_>, reason: Refusal) -> Content { /// A copy of a broadcast a warm route offers. #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) struct Warm { - /// What pulling the copy through the sender costs it: 0 at the copy's carrier, else - /// its own pull's cost, in the units of the route's metric. + /// What pulling the copy through the sender costs, in the units of the route's + /// metric: 0 at its carrier, the relay with a local reader that offers it. pub cost: u64, - /// The position in `hops` of the carrier, the relay with a reader where the copy left - /// its origin's route. - pub carrier: u64, - /// Every node the copy crossed, the origin node first and the sender last. + /// A node on `hops` sent GOAWAY, which ranks the copy below every source that avoids one. + pub draining: bool, + /// The lowest local priority of a link on `hops`, ranked before cost; the carrier's + /// own copy has no link to lower it, so `i8::MAX`. + pub priority: i8, + /// Every relay the copy crossed, the carrier first and the sender last. pub hops: Arc<[route::Id]>, } -/// Encode a warm route's cost, carrier, and hops. No relay offers a copy that crossed too -/// many hops, or whose carrier is not among them. +/// WARM_START and WARM_UPDATE flag: a node on the copy's hops drains. +const WARM_DRAINING: u8 = 0x1; + +/// Encode a warm route's cost, drain, priority, and hops. No relay offers a copy with no +/// carrier, or that crossed too many hops. fn encode_warm(w: &mut Encoder<'_>, warm: &Warm) -> Result<(), EncodeError> { - if warm.hops.len() > route::MAX_HOPS || warm.carrier >= warm.hops.len() as u64 { + if warm.hops.is_empty() || warm.hops.len() > route::MAX_HOPS { return Err(EncodeError::InvalidState); } w.varint(warm.cost)?; - w.varint(warm.carrier)?; + w.u8(if warm.draining { WARM_DRAINING } else { 0 }); + w.u8(warm.priority as u8); w.varint(warm.hops.len() as u64)?; for hop in warm.hops.iter() { w.varint(*hop)?; @@ -217,16 +223,26 @@ fn encode_warm(w: &mut Encoder<'_>, warm: &Warm) -> Result<(), EncodeError> { fn decode_warm(r: &mut Decoder<'_>) -> Content { let cost = r.varint()?; - let carrier = r.varint()?; + let draining = match r.u8()? { + 0 => false, + WARM_DRAINING => true, + flags => return refuse(r, Refusal::Flags(flags)), + }; + let priority = r.u8()? as i8; let count = r.varint()?; if count > route::MAX_HOPS as u64 { return refuse(r, Refusal::Hops(count)); } - if carrier >= count { + if count == 0 { return refuse(r, Refusal::Carrier); } let hops = (0..count).map(|_| r.varint()).collect::>()?; - Ok(Ok(Warm { cost, carrier, hops })) + Ok(Ok(Warm { + cost, + draining, + priority, + hops, + })) } /// Decode a WARM_START's body after its Route ID. @@ -1452,8 +1468,8 @@ mod tests { } /// The draft's warm route layouts, byte for byte: WARM_START is the Route ID, the path - /// suffix, the instance and its minted flag, then the copy's cost, its carrier's - /// position in its hops, and the hops; WARM_UPDATE the Warm ID and the same three; + /// suffix, the instance and its minted flag, then the copy's cost, its flags, its + /// priority as a signed byte, and its hops; WARM_UPDATE the Warm ID and the same four; /// WARM_END the Warm ID. Lite-06 skips the types as unknown. #[test] fn lite07_warm_route_layout() { @@ -1464,7 +1480,8 @@ mod tests { }; let warm = Warm { cost: 4, - carrier: 1, + draining: true, + priority: -1, hops: Arc::from([5, 9, 6]), }; let start = AnnounceBroadcast::WarmStart { @@ -1476,8 +1493,8 @@ mod tests { let update = AnnounceBroadcast::WarmUpdate { id: 7, warm }; let end = AnnounceBroadcast::WarmEnd { id: 7 }; let layouts: [(&AnnounceBroadcast, &[u8]); 3] = [ - (&start, &[12, 12, 3, 1, b'c', 1, b'a', 0, 4, 1, 3, 5, 9, 6]), - (&update, &[13, 7, 7, 4, 1, 3, 5, 9, 6]), + (&start, &[12, 13, 3, 1, b'c', 1, b'a', 0, 4, 1, 0xff, 3, 5, 9, 6]), + (&update, &[13, 8, 7, 4, 1, 0xff, 3, 5, 9, 6]), (&end, &[14, 1, 7]), ]; for (msg, wire) in layouts { @@ -1490,21 +1507,23 @@ mod tests { } } - /// A warm route's carrier is one of its hops, of which there are at most 32. A - /// receiver refuses one that breaks either, without failing the stream. + /// A warm route starts at its carrier, so it has at least one hop and at most 32, and + /// sets no flag but drain. A receiver refuses one that breaks any, without failing + /// the stream. #[test] - fn lite07_warm_route_names_its_carrier_among_its_hops() { - let outside = AnnounceBroadcast::WarmUpdate { + fn lite07_warm_route_hops_and_flags_are_bounded() { + let empty = AnnounceBroadcast::WarmUpdate { id: 7, warm: Warm { cost: 4, - carrier: 3, - hops: Arc::from([5, 9, 6]), + draining: false, + priority: 0, + hops: Arc::from([]), }, }; let mut buf = Vec::new(); assert!(matches!( - outside.encode(&mut Encoder::new(&mut buf, Version::Lite07.into()), Version::Lite07), + empty.encode(&mut Encoder::new(&mut buf, Version::Lite07.into()), Version::Lite07), Err(EncodeError::InvalidState) )); let decode = |wire: &[u8]| { @@ -1515,20 +1534,21 @@ mod tests { reason, }; assert_eq!( - decode(&[13, 4, 7, 4, 0, 0]).unwrap(), + decode(&[13, 5, 7, 4, 0, 0, 0]).unwrap(), refused(Refusal::Carrier), - "no hops" + "no hops, so no carrier" ); assert_eq!( - decode(&[13, 7, 7, 4, 3, 3, 5, 9, 6]).unwrap(), - refused(Refusal::Carrier), - "a carrier past the hops" + decode(&[13, 6, 7, 4, 2, 0, 1, 5]).unwrap(), + refused(Refusal::Flags(2)), + "an unknown flag" ); - let mut long = vec![13, 37, 7, 4, 0, 33]; + decode(&[13, 6, 7, 4, 0, 0, 1, 5]).unwrap(); + let mut long = vec![13, 38, 7, 4, 0, 0, 33]; long.extend([1; 33]); assert_eq!(decode(&long).unwrap(), refused(Refusal::Hops(33))); - long[1] = 36; - long[5] = 32; + long[1] = 37; + long[6] = 32; long.pop(); decode(&long).unwrap(); } diff --git a/rs/moq-net/src/lite/subscribe.rs b/rs/moq-net/src/lite/subscribe.rs index 5795dbce870..b8989127c9e 100644 --- a/rs/moq-net/src/lite/subscribe.rs +++ b/rs/moq-net/src/lite/subscribe.rs @@ -154,6 +154,7 @@ impl<'a> Message<'a> for Subscribe<'a> { } }; let update = decode_update(r, version)?; + decode_warm(r, version)?; Ok(Self { id, @@ -189,6 +190,7 @@ impl<'a> Message<'a> for Subscribe<'a> { } } encode_update(w, version, self.update)?; + encode_warm(w, version); Ok(()) } @@ -210,6 +212,24 @@ fn encode_update(w: &mut Encoder<'_>, version: Version, update: u64) -> Result<( } } +/// Encode whether the pull joins the copy the publisher offered (1) or follows the +/// subscriber's route (0). This endpoint chooses no source per broadcast, so it always +/// follows its route. +fn encode_warm(w: &mut Encoder<'_>, version: Version) { + if version.has_warm() { + w.u8(0); + } +} + +/// Decode whether the pull joins the publisher's copy, and ignore it: only a relay +/// choosing a source per broadcast acts on it, which this endpoint does not do. +fn decode_warm(r: &mut Decoder<'_>, version: Version) -> Result<(), DecodeError> { + if version.has_warm() && r.u8()? > 1 { + return Err(DecodeError::InvalidValue); + } + Ok(()) +} + /// Decode the `Subscriber Min Age` after the max age: milliseconds plus one, zero meaning not /// stated. A wire without the field decodes as not stated. fn decode_min_delay(r: &mut Decoder<'_>, version: Version) -> Result, DecodeError> { @@ -761,6 +781,7 @@ impl<'a> Message<'a> for SubscribeUpdate { true => r.varint()?.checked_sub(1), false => None, }; + decode_warm(r, version)?; Ok(Self { wanted_group, @@ -801,6 +822,7 @@ impl<'a> Message<'a> for SubscribeUpdate { if version.has_refresh() { w.varint_opt(self.wanted_group)?; } + encode_warm(w, version); Ok(()) } } @@ -1479,6 +1501,25 @@ mod test { )); } + /// Lite-07 ends SUBSCRIBE with whether the pull joins the publisher's copy: this + /// endpoint follows its route (0), and reads and ignores 0 or 1. Any other value is + /// malformed. + #[test] + fn warm_trails_subscribe_only_on_lite07() { + let msg = subscribe_sample(); + let lite07 = encode(&msg, Version::Lite07).unwrap(); + assert_eq!(lite07.last(), Some(&0)); + let head = &lite07[..lite07.len() - 1]; + let decode = |warm: &[u8]| { + let buf = [head, warm].concat(); + crate::coding::decode_from(&mut buf.as_slice(), Version::Lite07, Subscribe::decode_msg).map(|_| ()) + }; + decode(&[0]).unwrap(); + decode(&[1]).unwrap(); + assert!(decode(&[2]).is_err()); + assert!(decode(&[]).is_err()); + } + /// SUBSCRIBE and SUBSCRIBE_UPDATE carry their update number on lite-07 alone. #[test] fn update_number_roundtrips_only_on_lite07() { @@ -1923,10 +1964,10 @@ mod test { msg.ranges = vec![Range::OLDEST]; let mut buf = encode(&msg, version).unwrap(); - // The list is last but for the update number: a count of one, then two anchors of - // type 0. - let at = buf.len() - 4; - assert_eq!(&buf[at..], [1, 0, 0, 0]); + // The list is last but for the update number and whether the pull joins a copy: a + // count of one, then two anchors of type 0. + let at = buf.len() - 5; + assert_eq!(&buf[at..], [1, 0, 0, 0, 0]); buf[at] = 0; buf.truncate(at + 1); assert!(matches!( @@ -1935,7 +1976,7 @@ mod test { )); let mut buf = encode(&msg, version).unwrap(); - let last = buf.len() - 2; + let last = buf.len() - 3; buf[last] = 4; assert!(matches!( crate::coding::decode_from(&mut buf.as_slice(), version, Subscribe::decode_msg), diff --git a/rs/moq-net/src/lite/version.rs b/rs/moq-net/src/lite/version.rs index 03e35385345..c9d9133f208 100644 --- a/rs/moq-net/src/lite/version.rs +++ b/rs/moq-net/src/lite/version.rs @@ -398,7 +398,8 @@ impl Version { } /// Whether the announce stream carries warm routes (WARM_START, WARM_UPDATE, and - /// WARM_END). Added in lite-07. + /// WARM_END), and SUBSCRIBE and SUBSCRIBE_UPDATE say whether each pull joins the + /// publisher's copy. Added in lite-07. pub(crate) fn has_warm(self) -> bool { !matches!( self, diff --git a/rs/moq-net/src/route/tests/cache.rs b/rs/moq-net/src/route/tests/cache.rs index fb667831552..53bdc015200 100644 --- a/rs/moq-net/src/route/tests/cache.rs +++ b/rs/moq-net/src/route/tests/cache.rs @@ -1,48 +1,173 @@ -//! Cache-aware routing on the path prototype in `paths.rs`, as -//! `quest/m3/cache-routing/README.md` decides it: a warm route is per broadcast, -//! propagates, and is kept free of loops by its path, with what -//! `quest/m2/cache-routing-proof.md` found a path needs besides. +//! Cache-aware routing over [`path::Router`], as `quest/m3/cache-routing/README.md` +//! decides it (Plan item 14) and this module proves it: a warm route offers a +//! copy of one broadcast, propagates, and the pulls it draws settle free of loops. //! -//! A relay with a reader whose pull follows the publisher's route offers its copy -//! at no cost, as its carrier, with the path the copy took. Each relay pulls a -//! broadcast it has demand for (a reader, or a neighbour's subscription) through -//! the cheapest of its next hop toward the publisher and each copy offered, and -//! offers a copy it joined on at what its pull costs, so a copy's cost rises with -//! every link from where it left the publisher's route. Pulling through a -//! neighbour goes wherever that neighbour pulls, so each relay offers every -//! neighbour its copy, even one whose path names that neighbour, which can then -//! use neither. A relay on one copy moves onto another only if that copy's carrier -//! ranks below by the Rendezvous Hash, so copies crossing on their way never trade -//! places; a relay pulling along its route is held to the carriers of the -//! neighbours behind it, which those neighbours' offers back name. A copy with no -//! demand left lingers. +//! The rule has three layers, each settled before the next depends on it: //! -//! [`Mesh`] adds what a sweep checks to [`PathSim`]: after every step, whether a -//! broadcast's pulls form a cycle, which the path allows only while a withdrawal -//! crosses a link; once quiet, every reader is fed and nothing would change. -//! Subscriptions ride streams of their own, unordered against the routes and -//! copies. +//! 1. **Routes.** [`path::Router`] selects each relay's route to the publisher, which +//! the copies never change. +//! 2. **Offers.** A relay with a local reader is its copy's carrier: it offers every +//! neighbour its copy at no cost, whatever it pulls. A relay with no reader offers +//! each neighbour the best of its route and the copies offered to it whose hops +//! name neither relay, ranked as routes are (drain, then priority, then cost), when +//! that best is a copy. Carriers' offers are constant and extending one strictly +//! worsens its rank, so the offers settle as any monotone path vector does, with no +//! regard for who pulls what. +//! 3. **Pulls.** A carrier pulls through its best source among its route and the copies +//! whose carrier ranks below its own by the Rendezvous Hash, whoever pulls from it, +//! so its pull is a function of the settled offers. A relay with no reader pulls +//! through its best source whose hops name no neighbour it serves, so long as that +//! is, for each neighbour it serves, the copy it offered or one of a lower-ranked +//! carrier. A subscription says whether it joins the copy offered or follows the +//! subscriber's route, and a relay a neighbour pulls along its route pulls along its +//! own, so a pull that follows routes reaches the publisher. A relay with no reader +//! that cannot follows its route too. +//! +//! Once quiet, the pulls are free of loops: each relay that joins a copy gets one whose +//! carrier, then whose rank, is strictly better than what it gives each relay joined to +//! it (a carrier its own copy, a relay with no reader the copy it offered), and a pull +//! along routes follows the routes' tree. [`Mesh`] checks after every step whether a +//! broadcast's pulls form a cycle, which stale offers allow until the change crossing a +//! link is delivered, and once quiet that every reader is fed and the offers are the +//! fixed point computed centrally. + +use std::{ + cmp::Reverse, + collections::{BTreeMap, BTreeSet, VecDeque}, + sync::Arc, +}; -use super::paths::{Cast, PathSim, Rules, rendezvous}; use super::*; -/// The rule item 27 of the routes epic picked: holds, causes, and a doubt that lifts -/// after 4 hops of delay. -const PICK: Rules = Rules::waits(4); +/// One broadcast: its publisher, its path, and its instance. Two instances of one +/// path are two broadcasts, and a copy of one never serves the other. +pub(super) type Cast = (Id, Path, u64); + +/// The draft's Rendezvous Hash of a broadcast's path and a relay, which ranks carriers. +pub(super) fn rendezvous(path: &Path, node: Id) -> u64 { + let mut hash: u64 = 0x420C_0DEC_B00B; + for byte in path.as_str().bytes().chain(node.to_le_bytes()) { + hash = (hash ^ u64::from(byte)).wrapping_mul(0x0100_0000_01b3); + } + hash +} + +/// How a relay names itself in hops: no relay in a [`Mesh`] restarts. +fn id(node: Node) -> Id { + hop(node, 0) +} + +/// How a source ranks, lower first, as [`path::Offer`] ranks a route. +type Rank = (bool, Reverse, Metric); + +/// A copy one relay offers another: WARM_START and WARM_UPDATE. +#[derive(Clone, Debug, PartialEq, Eq)] +struct Copy { + /// The relays it crossed, its carrier first and the sender last. + hops: Arc<[Id]>, + draining: bool, + priority: i8, + cost: Metric, +} + +/// A source a relay can pull a broadcast from. +#[derive(Clone, Debug, PartialEq, Eq)] +struct Choice { + next: Node, + rank: Rank, + /// The relays the broadcast crossed before this one: the copy's hops, or the + /// route's path. + hops: Arc<[Id]>, + /// Whether it joins a copy, rather than follow the route. + warm: bool, +} + +impl Choice { + fn names(&self, node: Id) -> bool { + self.hops.contains(&node) + } + + fn carrier(&self) -> Option { + self.warm.then(|| self.hops[0]) + } + + /// What a relay pulling this offers onward. + fn copy(&self, me: Id) -> Copy { + Copy { + hops: self.hops.iter().copied().chain([me]).collect(), + draining: self.rank.0, + priority: self.rank.1.0, + cost: self.rank.2, + } + } +} + +/// A message about copies on a session's announce stream. +#[derive(Clone, Debug)] +enum Note { + Offer(Cast, Copy), + /// WARM_END. + End(Cast), +} + +/// A relay's demand for one broadcast, and where it pulls it from. +#[derive(Clone, Debug, Default)] +struct Carry { + reader: bool, + /// Each neighbour's subscription streams that arrived and were not cancelled, + /// with whether each joins this relay's copy. A neighbour that moves back here + /// opens a new stream, which can arrive before the old one's cancel. + downstream: BTreeMap>, + /// The neighbour it pulls through, and whether that joins its copy. + parent: Option<(Node, bool)>, + /// The stream of the pull through `parent`. + stream: u64, + /// Demand is gone since then, and the pull stays until the linger runs out. + lingering: Option, +} + +/// How long a pull with no demand stays: as long as an idle track does. +const LINGER: Duration = crate::track::IDLE_LINGER; + +/// One relay's copies, beside its [`path::Router`]. +#[derive(Default)] +struct Relay { + heard: BTreeMap<(Node, Cast), Copy>, + sent: BTreeMap<(Node, Cast), Copy>, + carry: BTreeMap, + /// Neighbours that sent GOAWAY. + draining: BTreeSet, +} /// A queue the mesh delivers from. #[derive(Clone, Debug)] enum Queue { Route(Node, Node), + Copy(Node, Node), /// One subscription stream: from, to, broadcast, stream. Sub(Node, Node, Cast, u64), } pub(super) struct Mesh { - pub sim: PathSim, + pub sim: Sim, + relays: Vec, + links: Vec>>, + legacy: Vec, /// Each publisher's node. origins: BTreeMap, casts: Vec, + held: BTreeSet<(Node, Node)>, + notes: BTreeMap<(Node, Node), VecDeque>, + /// Subscription streams, unordered against everything else: `Some` pulls or + /// updates whether it joins the copy, `None` cancels. + subs: BTreeMap<(Node, Node, Cast, u64), VecDeque>>, + streams: u64, + /// A carrier joins only a copy whose carrier ranks below its own. + pub ranked: bool, + /// Copy offers and their ends sent, per broadcast and in all. + warm: BTreeMap, + pub warm_msgs: usize, + pub subscribes: usize, /// Steps after which some broadcast's pulls formed a cycle. pub loops: usize, pub steps: usize, @@ -55,11 +180,22 @@ pub(super) struct Mesh { } impl Mesh { - pub fn new(nodes: usize, rules: Rules) -> Self { + pub fn new(nodes: usize) -> Self { Self { - sim: PathSim::new(nodes, rules), + sim: Sim::with(Rule::PATH, nodes), + relays: (0..nodes).map(|_| Relay::default()).collect(), + links: vec![vec![None; nodes]; nodes], + legacy: vec![false; nodes], origins: BTreeMap::new(), casts: Vec::new(), + held: BTreeSet::new(), + notes: BTreeMap::new(), + subs: BTreeMap::new(), + streams: 0, + ranked: true, + warm: BTreeMap::new(), + warm_msgs: 0, + subscribes: 0, loops: 0, steps: 0, starved: 0, @@ -70,21 +206,31 @@ impl Mesh { } fn nodes(&self) -> usize { - self.sim.nodes.len() + self.relays.len() } /// A publisher at `node`, a leaf that originates `id` and claims `live`. Called /// before its links. pub fn origin(&mut self, node: Node, id: Id) { self.sim.leaf(node); - self.sim.originate(node, id); - self.sim.announce(node, "live"); + self.sim.originate(node, id, 0); + self.sim.announce(node, "live", 0, None); self.origins.insert(id, node); } - /// `node` never sends or reads a copy. + fn publisher(&self, node: Node) -> bool { + self.origins.values().any(|&origin| origin == node) + } + + /// `node` never sends or reads a copy: it pulls along its route, as a relay that + /// does not choose per broadcast does. pub fn legacy(&mut self, node: Node) { - self.sim.nodes[node].legacy = true; + self.legacy[node] = true; + } + + /// Whether `node` reads copies. + fn reads(&self, node: Node) -> bool { + !self.publisher(node) && !self.legacy[node] } /// A broadcast at `path` of the publisher `id`, under `instance`. @@ -94,11 +240,17 @@ impl Mesh { cast } - pub fn join(&mut self, a: Node, b: Node, cost: Metric) { - self.sim.join(a, b, cost); + pub fn link(&mut self, a: Node, b: Node, ab: SimLink, ba: SimLink) { + self.links[a][b] = Some(ab); + self.links[b][a] = Some(ba); + self.sim.link(a, b, ab, ba); self.step(); } + pub fn join(&mut self, a: Node, b: Node, cost: Metric) { + self.link(a, b, SimLink::cost(cost), SimLink::cost(cost)); + } + pub fn connect(&mut self, a: Node, b: Node) { self.sim.connect(a, b); self.step(); @@ -106,96 +258,201 @@ impl Mesh { pub fn disconnect(&mut self, a: Node, b: Node) { self.sim.disconnect(a, b); + for (x, y) in [(a, b), (b, a)] { + self.notes.remove(&(x, y)); + self.subs.retain(|(from, to, ..), _| (*from, *to) != (x, y)); + self.held.remove(&(x, y)); + let relay = &mut self.relays[x]; + relay.heard.retain(|(peer, _), _| *peer != y); + relay.sent.retain(|(peer, _), _| *peer != y); + relay.draining.remove(&y); + for carry in relay.carry.values_mut() { + carry.downstream.remove(&y); + // The pull went with the session. + if carry.parent.is_some_and(|(next, _)| next == y) { + carry.parent = None; + } + } + } self.step(); } + pub fn connected(&self, a: Node, b: Node) -> bool { + self.sim.connected(a, b) + } + + fn peers(&self, node: Node) -> impl Iterator + '_ { + (0..self.nodes()).filter(move |&peer| self.links[node][peer].is_some() && self.connected(node, peer)) + } + + /// `node` sends GOAWAY on every session. pub fn goaway(&mut self, node: Node) { self.sim.goaway(node); + let peers: Vec = self.peers(node).collect(); + for peer in peers { + self.relays[peer].draining.insert(node); + } self.step(); } /// A reader at `node` starts or stops reading `cast`. pub fn read(&mut self, node: Node, cast: &Cast, reading: bool) { match reading { - true => self.sim.carry(node, cast), - false => self.sim.uncarry(node, cast), + true => self.relays[node].carry.entry(cast.clone()).or_default().reader = true, + false => { + if let Some(carry) = self.relays[node].carry.get_mut(cast) { + carry.reader = false; + } + } } self.step(); } pub fn reading(&self, node: Node, cast: &Cast) -> bool { - self.sim.nodes[node].reading(cast) + self.relays[node].carry.get(cast).is_some_and(|carry| carry.reader) } pub fn hold(&mut self, from: Node, to: Node) { + self.held.insert((from, to)); self.sim.hold(from, to); } pub fn release(&mut self, from: Node, to: Node) { + self.held.remove(&(from, to)); self.sim.release(from, to); } - /// The linger for `cast` at `node` runs out. + /// The linger for `cast` at `node` runs out: a pull with no demand left goes. pub fn expire(&mut self, node: Node, cast: &Cast) { - self.sim.expire(node, cast); + let carry = self.relays[node].carry.get_mut(cast); + if let Some(carry) = carry + && carry.lingering.is_some() + { + carry.lingering = None; + if let Some((parent, _)) = carry.parent.take() { + let stream = carry.stream; + self.send_sub(node, parent, cast, stream, None); + } + } self.step(); } /// One hop of delay passes everywhere. Whether a doubt still waits to lift. pub fn tick(&mut self) -> bool { - let waiting = self.sim.tick(); + self.sim.wait(HOP); self.step(); - waiting + self.sim.deadline().is_some() } - pub fn deliver_sub(&mut self, from: Node, to: Node, cast: &Cast) -> bool { - let delivered = self.sim.deliver_sub(from, to, cast); - self.step(); - delivered + /// The subscription streams from `from` to `to` with a change to deliver, oldest first. + pub fn streams(&self, from: Node, to: Node) -> Vec { + (self.subs.keys()) + .filter(|(f, t, ..)| (*f, *t) == (from, to)) + .map(|(.., stream)| *stream) + .collect() } + /// Deliver the next change on one subscription stream from `from` to `to`. pub fn deliver_stream(&mut self, from: Node, to: Node, cast: &Cast, stream: u64) -> bool { - let delivered = self.sim.deliver_stream(from, to, cast, stream); + let delivered = self.recv_sub(from, to, cast, stream); self.step(); delivered } - /// The subscription streams from `from` to `to` with a change to deliver, oldest first. - pub fn streams(&self, from: Node, to: Node) -> Vec { - (self.sim.pending_subs().into_iter()) - .filter(|(f, t, ..)| (*f, *t) == (from, to)) - .map(|(.., stream)| stream) - .collect() + /// Deliver the next change on the oldest stream of `cast` from `from` to `to`. + pub fn deliver_sub(&mut self, from: Node, to: Node, cast: &Cast) -> bool { + let stream = (self.subs.keys()).find(|key| (key.0, key.1, &key.2) == (from, to, cast)); + match stream.map(|key| key.3) { + Some(stream) => self.deliver_stream(from, to, cast, stream), + None => false, + } } /// Every unheld queue with something to deliver. fn busy(&self) -> Vec { - (self.sim.busy().into_iter().map(|(from, to)| Queue::Route(from, to))) - .chain( - (self.sim.busy_subs().into_iter()).map(|(from, to, cast, stream)| Queue::Sub(from, to, cast, stream)), - ) + let routes = (self.sim.busy().into_iter()).map(|(from, to)| Queue::Route(from, to)); + let notes = (self.notes.iter()) + .filter(|(_, queue)| !queue.is_empty()) + .map(|(&(from, to), _)| Queue::Copy(from, to)); + let subs = (self.subs.keys()).map(|(from, to, cast, stream)| Queue::Sub(*from, *to, cast.clone(), *stream)); + (routes.chain(notes).chain(subs)) + .filter(|queue| match queue { + Queue::Route(from, to) | Queue::Copy(from, to) | Queue::Sub(from, to, ..) => { + !self.held.contains(&(*from, *to)) + } + }) .collect() } fn deliver_one(&mut self, queue: &Queue) { self.delivered = true; match queue { - Queue::Route(from, to) => self.sim.deliver(*from, *to), - Queue::Sub(from, to, cast, stream) => self.sim.deliver_stream(*from, *to, cast, *stream), - }; + Queue::Route(from, to) => { + self.sim.deliver(*from, *to); + } + Queue::Copy(from, to) => { + if let Some(note) = self.notes.get_mut(&(*from, *to)).and_then(VecDeque::pop_front) { + self.recv_note(*from, *to, note); + } + } + Queue::Sub(from, to, cast, stream) => { + self.recv_sub(*from, *to, cast, *stream); + } + } self.step(); } - /// Deliver round-robin until every unheld queue is empty, letting no linger run - /// out and no time pass. False if it never empties. + fn recv_note(&mut self, from: Node, to: Node, note: Note) { + let heard = &mut self.relays[to].heard; + match note { + Note::Offer(cast, copy) => heard.insert((from, cast), copy), + Note::End(cast) => heard.remove(&(from, cast)), + }; + } + + fn recv_sub(&mut self, from: Node, to: Node, cast: &Cast, stream: u64) -> bool { + let key = (from, to, cast.clone(), stream); + let Some(change) = self.subs.get_mut(&key).and_then(VecDeque::pop_front) else { + return false; + }; + if self.subs[&key].is_empty() { + self.subs.remove(&key); + } + let carry = self.relays[to].carry.entry(cast.clone()).or_default(); + let streams = carry.downstream.entry(from).or_default(); + match change { + Some(warm) => streams.insert(stream, warm), + None => streams.remove(&stream), + }; + if streams.is_empty() { + carry.downstream.remove(&from); + } + true + } + + fn send_sub(&mut self, from: Node, to: Node, cast: &Cast, stream: u64, change: Option) { + if !self.connected(from, to) { + return; + } + self.subscribes += 1; + let key = (from, to, cast.clone(), stream); + self.subs.entry(key).or_default().push_back(change); + } + + /// Deliver round-robin until every unheld queue is empty, each round taking a hop + /// of delay, so a long exchange lets doubts lift. False if it never empties. pub fn run(&mut self) -> bool { self.rounds(10_000) } - /// Deliver up to `rounds` rounds, one message from each unheld queue per round. - /// Whether every unheld queue emptied. + /// Deliver until quiet, failing after `rounds` busy rounds. A pull lingering with no + /// demand can hold a copy its parent promised elsewhere, and the pulls around it + /// flip until the linger runs out, so a round with a linger pending is not counted, + /// up to four lingers' worth. fn rounds(&mut self, rounds: usize) -> bool { - for _ in 0..rounds { + let lingers = 4 * (LINGER.as_millis() / HOP.as_millis()) as usize; + let (mut counted, mut total) = (0, 0); + while counted < rounds && total < rounds + lingers { let ready = self.busy(); if ready.is_empty() { return true; @@ -203,26 +460,50 @@ impl Mesh { for queue in &ready { self.deliver_one(queue); } + self.sim.wait(HOP); + self.step(); + // A linger runs out on its timer, however busy the mesh is. + let now = self.sim.now(); + let lingering: Vec<(Node, Cast, Instant)> = (0..self.nodes()) + .flat_map(|node| { + (self.relays[node].carry.iter()) + .filter_map(move |(cast, carry)| Some((node, cast.clone(), carry.lingering?))) + }) + .collect(); + let mut pending = false; + for (node, cast, since) in lingering { + match now.saturating_duration_since(since) >= LINGER { + true => self.expire(node, &cast), + false => pending = true, + } + } + total += 1; + counted += usize::from(!pending); } false } /// Run, let time pass until every doubt lifts, and let every linger run out, - /// until nothing changes. A round of deliveries takes a hop of delay, so a long - /// exchange lets time pass too, if more slowly. False if it never settles. + /// until nothing changes. False if it never settles. pub fn settle(&mut self) -> bool { - for _ in 0..1000 { - if !self.rounds(64) { - self.tick(); - continue; + for _ in 0..100 { + if !self.rounds(300) { + return false; } - if self.tick() { + if let Some(at) = self.sim.deadline() { + let now = self.sim.now(); + self.sim.wait(at.saturating_duration_since(now)); + self.step(); continue; } let lingering: Vec<(Node, Cast)> = (0..self.nodes()) - .flat_map(|node| self.sim.nodes[node].lingers().into_iter().map(move |cast| (node, cast))) + .flat_map(|node| { + (self.relays[node].carry.iter()) + .filter(|(_, carry)| carry.lingering.is_some()) + .map(move |(cast, _)| (node, cast.clone())) + }) .collect(); - if lingering.is_empty() && self.busy().is_empty() { + if lingering.is_empty() { return true; } for (node, cast) in lingering { @@ -232,8 +513,254 @@ impl Mesh { false } + // --- The rule --- + + /// Where `node` pulls `cast` along its route, unless it publishes it. + fn cold(&self, node: Node, cast: &Cast) -> Option { + if self.publisher(node) { + return None; + } + let router = self.sim.path(node); + let path::Forward::Next { next, prefix, .. } = router.forward(cast.0, &cast.1, None)? else { + return None; + }; + let offer = router.route_for(cast.0, &prefix)?.offer()?; + let (_, before) = offer.path.split_last()?; + Some(Choice { + next, + rank: (offer.draining, Reverse(offer.priority), offer.metric), + hops: before.iter().map(|hop| hop.node).collect(), + warm: false, + }) + } + + /// Joining `copy`, heard from `peer`, at `node`: what `node` would pull, or + /// nothing if its hops name `node`. + fn extend(&self, node: Node, peer: Node, copy: &Copy) -> Option { + if copy.hops.contains(&id(node)) { + return None; + } + let link = self.links[node][peer]?; + let draining = copy.draining || self.relays[node].draining.contains(&peer); + Some(Choice { + next: peer, + rank: ( + draining, + Reverse(copy.priority.min(link.priority)), + copy.cost.checked_add(link.cost)?, + ), + hops: copy.hops.clone(), + warm: true, + }) + } + + /// Every source `node` has for `cast`: its route, and each copy offered to it. + fn options(&self, node: Node, cast: &Cast) -> Vec { + let mut options: Vec = self.cold(node, cast).into_iter().collect(); + if self.reads(node) { + for ((peer, heard), copy) in &self.relays[node].heard { + if heard == cast && self.connected(node, *peer) { + options.extend(self.extend(node, *peer, copy)); + } + } + } + options + } + + /// The best of `options`: by rank, then a copy over the route, then the lower + /// carrier, then the lower neighbour, so every relay that knows the same offers + /// picks the same. + fn pick(cast: &Cast, options: impl Iterator) -> Option { + options.min_by_key(|choice| { + let carrier = choice.carrier().map_or(0, |carrier| rendezvous(&cast.1, carrier)); + (choice.rank, !choice.warm, carrier, choice.next) + }) + } + + /// The best source of `options` at `node`: its route, or a copy whose hops name none + /// of `exclude` and, for a carrier, whose carrier ranks below it. The route stays + /// usable through a neighbour this relay serves, since following it makes that + /// neighbour follow its own. + fn best(&self, node: Node, cast: &Cast, options: &[Choice], exclude: &BTreeSet) -> Option { + let carrier = self.ranked && self.reading(node, cast); + let own = rendezvous(&cast.1, id(node)); + let usable = (options.iter()) + .filter(|choice| !choice.warm || !exclude.iter().any(|&node| choice.names(node))) + .filter(|choice| !carrier || choice.carrier().is_none_or(|by| rendezvous(&cast.1, by) < own)); + Self::pick(cast, usable.cloned()) + } + + /// The copy `node` offers `peer` for `cast`, from `options`. + fn offer_from(&self, node: Node, peer: Node, cast: &Cast, options: &[Choice]) -> Option { + if !self.reads(node) || !self.reads(peer) { + return None; + } + // A warm route travels under the route the relay sends. + self.cold(node, cast)?; + if self.reading(node, cast) { + return Some(Copy { + hops: Arc::from([id(node)]), + draining: false, + priority: i8::MAX, + cost: 0, + }); + } + let best = (options.iter()).filter(|choice| !choice.names(id(peer))); + Self::pick(cast, best.cloned()) + .filter(|choice| choice.warm) + .map(|choice| choice.copy(id(node))) + } + + fn offer_to(&self, node: Node, peer: Node, cast: &Cast) -> Option { + self.offer_from(node, peer, cast, &self.options(node, cast)) + } + + /// Where `node` would pull `cast` from, given who pulls through it. + fn decide(&self, node: Node, cast: &Cast) -> Option { + let carry = self.relays[node].carry.get(cast); + let downstream = carry.map(|carry| &carry.downstream); + let streams = || downstream.into_iter().flatten(); + // A neighbour that follows its route through this relay makes it follow its own. + let forced = streams().any(|(_, streams)| streams.values().any(|warm| !warm)); + if self.legacy[node] || forced { + return self.cold(node, cast); + } + let options = self.options(node, cast); + // A carrier's choice depends on the offers alone: a relay with no reader that it + // joins yields instead, so two neighbours never trade who pulls whom. + if self.reading(node, cast) { + return self.best(node, cast, &options, &BTreeSet::new()); + } + let serves: BTreeSet = streams().map(|(&peer, _)| id(peer)).collect(); + let choice = self.best(node, cast, &options, &serves); + // Each neighbour that joined this relay's copy gets what this relay pulls, so + // it pulls a copy only while, for each, that is the copy it offered or one whose + // carrier ranks below that copy's. Upstream along joined copies, the carrier's + // rank then the copy's rank strictly falls, so the joins form no cycle. + let pulled = choice + .as_ref() + .filter(|choice| choice.warm) + .map(|choice| choice.copy(id(node))); + let below = |offered: &Copy| { + let rank = |copy: &Copy| rendezvous(&cast.1, copy.hops[0]); + pulled.as_ref().is_some_and(|pulled| rank(pulled) < rank(offered)) + }; + let kept = streams().all(|(&peer, _)| match self.offer_from(node, peer, cast, &options) { + None => true, + Some(offered) => pulled.as_ref() == Some(&offered) || below(&offered), + }); + match kept { + true => choice, + false => self.cold(node, cast), + } + } + + /// Follow the demand for `cast` at `node`: pull from its source and move with it. + /// Once the demand is gone the pull stays where it is until the linger runs out. + fn pull(&mut self, node: Node, cast: &Cast) { + let choice = self.decide(node, cast).map(|choice| (choice.next, choice.warm)); + let now = self.sim.now(); + let Some(carry) = self.relays[node].carry.get_mut(cast) else { + return; + }; + let demand = carry.reader || !carry.downstream.is_empty(); + match !demand && carry.parent.is_some() { + true => { + carry.lingering.get_or_insert(now); + return; + } + false => carry.lingering = None, + } + let parent = choice.filter(|_| demand); + let old = std::mem::replace(&mut carry.parent, parent); + let stream = carry.stream; + if !demand && parent.is_none() { + self.relays[node].carry.remove(cast); + } + if old == parent { + return; + } + match (old, parent) { + (Some((was, _)), Some((next, warm))) if was == next => self.send_sub(node, next, cast, stream, Some(warm)), + (old, new) => { + if let Some((was, _)) = old { + self.send_sub(node, was, cast, stream, None); + } + if let Some((next, warm)) = new { + self.streams += 1; + let stream = self.streams; + self.relays[node].carry.get_mut(cast).unwrap().stream = stream; + self.send_sub(node, next, cast, stream, Some(warm)); + } + } + } + } + + /// Bring what `node` offers each neighbour for `cast` up to date. + fn offer(&mut self, node: Node, cast: &Cast) { + let options = self.options(node, cast); + let peers: Vec = self.peers(node).collect(); + for peer in peers { + let want = self.offer_from(node, peer, cast, &options); + let key = (peer, cast.clone()); + if self.relays[node].sent.get(&key) == want.as_ref() { + continue; + } + let note = match want { + Some(copy) => { + self.relays[node].sent.insert(key, copy.clone()); + Note::Offer(cast.clone(), copy) + } + None => { + self.relays[node].sent.remove(&key); + Note::End(cast.clone()) + } + }; + self.notes.entry((node, peer)).or_default().push_back(note); + self.warm_msgs += 1; + *self.warm.entry(cast.clone()).or_default() += 1; + } + } + + /// Each relay reacts to what changed, then the mesh counts a cycle. + fn step(&mut self) { + for node in 0..self.nodes() { + for cast in self.casts.clone() { + self.pull(node, &cast); + self.offer(node, &cast); + } + } + self.steps += 1; + let cycle = self.cycle().is_some(); + self.loops += usize::from(cycle); + self.run = match cycle { + true => self.run + usize::from(std::mem::take(&mut self.delivered)), + false => 0, + }; + self.delivered = false; + self.longest = self.longest.max(self.run); + self.starved += usize::from(cycle && self.starving()); + } + + // --- Queries --- + + /// What each relay pulls and offers, and what is in flight, for a failure report. + pub fn dump(&self) -> String { + let mut out = String::new(); + for (node, relay) in self.relays.iter().enumerate() { + for (cast, carry) in &relay.carry { + out += &format!(" {node} {cast:?}: {carry:?}\n"); + } + for ((peer, cast), copy) in &relay.sent { + out += &format!(" {node} offers {peer} {cast:?}: {copy:?}\n"); + } + } + out += &format!(" busy {:?}\n subs {:?}\n", self.busy(), self.subs); + out + } + pub fn parent(&self, node: Node, cast: &Cast) -> Option { - self.sim.nodes[node].parent(cast) + self.relays[node].carry.get(cast)?.parent.map(|(next, _)| next) } /// Whether `node` pulls `cast`. @@ -251,26 +778,76 @@ impl Mesh { /// Copy offers and their ends sent for `cast`. pub fn warm_sent(&self, cast: &Cast) -> usize { - self.sim.warm.get(cast).copied().unwrap_or(0) + self.warm.get(cast).copied().unwrap_or(0) } /// Whether `node` is offered a copy of `cast` by `from`. pub fn offered(&self, node: Node, from: Node, cast: &Cast) -> bool { - self.sim.nodes[node].offers(from, cast).1.is_some() + self.relays[node].heard.contains_key(&(from, cast.clone())) } - /// Count a step after which some broadcast's pulls form a cycle. - fn step(&mut self) { - self.steps += 1; - let cycle = self.cycle().is_some(); - self.loops += usize::from(cycle); - self.run = match cycle { - true => self.run + usize::from(std::mem::take(&mut self.delivered)), - false => 0, - }; - self.delivered = false; - self.longest = self.longest.max(self.run); - self.starved += usize::from(cycle && self.starving()); + /// The carrier of the copy `from` offers `node`. + pub fn carrier(&self, node: Node, from: Node, cast: &Cast) -> Option { + let copy = self.relays[node].heard.get(&(from, cast.clone()))?; + (0..self.nodes()).find(|&relay| id(relay) == copy.hops[0]) + } + + /// The pairs of relays and broadcasts where a relay with no reader serves + /// neighbours it offered different copies, and so follows its route. + pub fn split(&self) -> usize { + let mut split = 0; + for node in 0..self.nodes() { + for (cast, carry) in &self.relays[node].carry { + let offered: BTreeSet>> = (carry.downstream.keys()) + .map(|&peer| self.offer_to(node, peer, cast).map(|copy| copy.hops)) + .collect(); + split += usize::from(!carry.reader && offered.len() > 1); + } + } + split + } + + /// Deliver `rounds` more rounds, logging each delivery and the pulls it moved: what + /// keeps a mesh that never settles busy. + pub fn trace(&mut self, rounds: usize) -> String { + let mut out = String::new(); + for _ in 0..rounds { + for queue in self.busy() { + let before: Vec>> = (self.relays.iter()) + .map(|relay| { + self.casts + .iter() + .map(|cast| relay.carry.get(cast).and_then(|c| c.parent)) + .collect() + }) + .collect(); + let what = match &queue { + Queue::Copy(from, to) => format!("{:?}", self.notes.get(&(*from, *to)).and_then(|q| q.front())), + Queue::Sub(from, to, cast, stream) => { + format!( + "{:?}", + self.subs + .get(&(*from, *to, cast.clone(), *stream)) + .and_then(|q| q.front()) + ) + } + Queue::Route(..) => String::new(), + }; + out += &format!(" deliver {queue:?} {what}\n"); + self.deliver_one(&queue); + for (node, relay) in self.relays.iter().enumerate() { + for (i, cast) in self.casts.iter().enumerate() { + let after = relay.carry.get(cast).and_then(|c| c.parent); + if after != before[node][i] { + out += &format!(" {node} {cast:?}: {:?} -> {after:?}\n", before[node][i]); + } + } + } + } + self.sim.wait(HOP); + self.step(); + } + out } /// Whether some reader's pulls lead into a cycle. @@ -309,160 +886,203 @@ impl Mesh { None } + /// The offers for `cast` once they settle, computed centrally: every relay offers + /// from what the others offered it the round before, until a round changes nothing. + fn fixed_point(&self, cast: &Cast) -> BTreeMap<(Node, Node), Copy> { + let links: Vec<(Node, Node)> = (0..self.nodes()) + .flat_map(|node| self.peers(node).map(move |peer| (node, peer))) + .collect(); + let mut offers = BTreeMap::new(); + for _ in 0..4 * self.nodes() { + let mut next = BTreeMap::new(); + for &(node, peer) in &links { + let mut options: Vec = self.cold(node, cast).into_iter().collect(); + if self.reads(node) { + for &(from, to) in &links { + if let (true, Some(copy)) = (to == node, offers.get(&(from, to))) { + options.extend(self.extend(node, from, copy)); + } + } + } + if let Some(copy) = self.offer_from(node, peer, cast, &options) { + next.insert((node, peer), copy); + } + } + if next == offers { + return offers; + } + offers = next; + } + panic!("the offers for {cast:?} never settle"); + } + /// The steady state once demand and topology stop changing: every reader is fed /// from its publisher, subscriptions and copies offered match what their senders - /// hold, no copy lingers, and each relay pulls exactly what it has demand for, - /// from its cheapest source. So a relay carries a broadcast only for a reader, or - /// for neighbours whose cheapest source it is. + /// hold and what the rule computes centrally, no copy lingers, and each relay pulls + /// exactly what it has demand for, from the source the rule picks. pub fn steady(&mut self, label: &str) { assert!(self.busy().is_empty(), "{label}: still busy"); assert_eq!(self.cycle(), None, "{label}"); for cast in &self.casts { let origin = self.origins[&cast.0]; + let fixed = self.fixed_point(cast); for node in 0..self.nodes() { - let relay = &self.sim.nodes[node]; - if relay.reading(cast) { + let relay = &self.relays[node]; + if self.reading(node, cast) { let mut at = node; while let Some(next) = self.parent(at, cast) { at = next; } assert_eq!(at, origin, "{label}: {cast:?} reaches node {node}"); } - let feeds: BTreeSet = (0..self.nodes()) - .filter(|&n| self.parent(n, cast) == Some(node)) + let feeds: BTreeMap = (0..self.nodes()) + .filter_map(|n| { + let (next, warm) = self.relays[n].carry.get(cast)?.parent?; + (next == node).then_some((n, warm)) + }) .collect(); - assert_eq!(relay.downstream(cast), feeds, "{label}: {cast:?} at node {node}"); - assert_eq!( - relay.streams(cast), - feeds.len(), - "{label}: {cast:?} at node {node}: one stream each" + let carry = relay.carry.get(cast); + let downstream: BTreeMap = (carry.into_iter()) + .flat_map(|carry| &carry.downstream) + .map(|(&peer, streams)| { + assert_eq!(streams.len(), 1, "{label}: {cast:?} at node {node}: one stream each"); + (peer, *streams.values().next().unwrap()) + }) + .collect(); + assert_eq!(downstream, feeds, "{label}: {cast:?} at node {node}"); + assert!( + !carry.is_some_and(|carry| carry.lingering.is_some()), + "{label}: {cast:?} at node {node}" ); - assert!(!relay.lingering(cast), "{label}: {cast:?} at node {node}"); if node != origin { - let demand = relay.reading(cast) || !feeds.is_empty(); - let source = relay.source(cast).map(|source| source.next); - assert_eq!( - relay.parent(cast), - source.filter(|_| demand), - "{label}: {cast:?} at node {node}" - ); + let demand = self.reading(node, cast) || !feeds.is_empty(); + let source = self.decide(node, cast).map(|choice| (choice.next, choice.warm)); + let parent = carry.and_then(|carry| carry.parent); + assert_eq!(parent, source.filter(|_| demand), "{label}: {cast:?} at node {node}"); } - for peer in 0..self.nodes() { - if !self.sim.connected(node, peer) || self.sim.nodes[peer].legacy { - continue; - } - let sent = relay.offers(peer, cast).0; - let heard = self.sim.nodes[peer].offers(node, cast).1; + for peer in self.peers(node) { + let sent = relay.sent.get(&(peer, cast.clone())); + let heard = self.relays[peer].heard.get(&(node, cast.clone())); assert_eq!(sent, heard, "{label}: {cast:?} offered from {node} to {peer}"); + assert_eq!( + sent, + fixed.get(&(node, peer)), + "{label}: {cast:?} from {node} to {peer}, against the fixed point" + ); } } } - let stats = self.sim.stats; + let before = (self.sim.stats, self.warm_msgs, self.subscribes); self.tick(); - assert_eq!(self.sim.stats, stats, "{label}: something changed at quiescence"); + let after = (self.sim.stats, self.warm_msgs, self.subscribes); + assert_eq!(before, after, "{label}: something changed at quiescence"); } } +/// A broadcast path under which `low` ranks below `high` as a carrier. +fn ranking(low: Node, high: Node) -> String { + (0..) + .map(|i| format!("live/x{i}")) + .find(|path| rendezvous(&p(path), id(low)) < rendezvous(&p(path), id(high))) + .unwrap() +} + /// An origin behind relay `a`, then relays `a`, `b`, `c` in a chain priced 3, with a -/// skip from `a` to `c` priced `skip`, and one broadcast. -fn chain(skip: Metric) -> (Mesh, Cast) { +/// skip from `a` to `c` priced `skip`, and one broadcast at `path`. +fn chain(skip: Metric, path: &str) -> (Mesh, Cast) { let (o, a, b, c) = (0, 1, 2, 3); - let mut mesh = Mesh::new(4, PICK); + let mut mesh = Mesh::new(4); mesh.origin(o, 7); mesh.join(o, a, 1); mesh.join(a, b, 3); mesh.join(b, c, 3); mesh.join(a, c, skip); - let x = mesh.broadcast(7, "live/x", 0); - assert!(mesh.settle()); + let x = mesh.broadcast(7, path, 0); + assert!(mesh.settle(), "{}", mesh.dump()); (mesh, x) } /// A cold broadcast takes the skip. A reader on the skipped relay `b` makes `c` join -/// its copy, and when that reader leaves, `c` returns to the skip. Readers arriving -/// the other way round join the other way: `b` joins `c`'s copy, which is farther -/// from the publisher than `b` is. +/// its copy, and when that reader leaves, `c` returns to the skip. Where `c` ranks +/// lower, `b` joins `c`'s copy instead, though `c` is farther from the publisher, and +/// in whichever order the readers arrive. #[test] fn a_skipped_relay_draws_its_neighbour_while_it_has_a_reader() { let (a, b, c) = (1, 2, 3); - let (mut mesh, x) = chain(5); + let (mut mesh, x) = chain(5, &ranking(b, c)); mesh.read(c, &x, true); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); assert_eq!(mesh.parent(c, &x), Some(a), "a cold broadcast takes the skip"); assert!(!mesh.carries(b, &x)); mesh.steady("cold"); mesh.read(b, &x, true); - assert!(mesh.settle()); - assert_eq!(mesh.parent(b, &x), Some(c), "b joins c's copy, though c is farther"); - assert_eq!(mesh.parent(c, &x), Some(a)); - mesh.steady("b joins c"); - - let (mut mesh, x) = chain(5); - mesh.read(b, &x, true); - assert!(mesh.settle()); - assert_eq!(mesh.parent(b, &x), Some(a)); - mesh.read(c, &x, true); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); assert_eq!(mesh.parent(c, &x), Some(b), "c joins b's copy"); + assert_eq!(mesh.parent(b, &x), Some(a)); mesh.steady("c joins b"); mesh.read(b, &x, false); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); assert_eq!(mesh.parent(c, &x), Some(a), "back to the skip"); assert!(!mesh.carries(b, &x)); mesh.steady("cold again"); assert_eq!(mesh.loops, 0); + + for first in [b, c] { + let (mut mesh, x) = chain(5, &ranking(c, b)); + mesh.read(first, &x, true); + assert!(mesh.settle(), "{}", mesh.dump()); + mesh.read(b + c - first, &x, true); + assert!(mesh.settle(), "{}", mesh.dump()); + assert_eq!(mesh.parent(b, &x), Some(c), "b joins c's copy, though c is farther"); + assert_eq!(mesh.parent(c, &x), Some(a)); + mesh.steady(&format!("b joins c, {first} first")); + assert_eq!(mesh.loops, 0); + } } -/// A relay whose only consumer is `c` offers `c` nothing once its own reader goes. -/// With a cheaper skip, `c` bypasses it; when the relay is `c`'s cheapest route to -/// the publisher, `c` stays and the relay keeps pulling for it alone. +/// A relay whose only consumer is `c` keeps pulling for it once its own reader goes +/// only while it is `c`'s route: with a cheaper skip, `c` bypasses it. #[test] fn a_relay_feeding_one_consumer_stays_only_on_its_cheapest_route() { let (a, b, c) = (1, 2, 3); for (skip, stays) in [(5, false), (9, true)] { - let (mut mesh, x) = chain(skip); + let (mut mesh, x) = chain(skip, &ranking(b, c)); mesh.read(b, &x, true); mesh.read(c, &x, true); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); assert_eq!(mesh.parent(c, &x), Some(b), "skip {skip}"); mesh.read(b, &x, false); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); assert_eq!(mesh.parent(c, &x), Some(if stays { b } else { a }), "skip {skip}"); assert_eq!(mesh.carries(b, &x), stays, "skip {skip}"); mesh.steady(&format!("skip {skip}")); } } -/// `r2`'s cheapest route runs through `r1` to `r3`, so `r3` pulls only on `r2`'s -/// behalf, and offers `r2` nothing: were it to, `r2` would join it over the direct -/// link, `r1`'s pull would go, and with it the reason for `r3`'s copy. A reader of -/// `r1`'s own makes it offer a copy, which `r2` already pulls. +/// `r2` pulls along its route, through `r1` and `r3`. `r1` would rather join `r2`'s +/// own copy, which it offers `r3`, but a neighbour following its route through `r1` +/// makes `r1` follow its own, so the pull reaches the publisher. #[test] -fn a_pull_on_a_neighbours_behalf_offers_nothing() { +fn a_pull_along_the_route_stays_on_the_route() { let (o, r1, r2, r3) = (0, 1, 2, 3); - let mut mesh = Mesh::new(4, PICK); + let mut mesh = Mesh::new(4); mesh.origin(o, 7); mesh.join(o, r3, 1); mesh.join(r3, r1, 1); mesh.join(r1, r2, 1); mesh.join(r2, r3, 3); let x = mesh.broadcast(7, "live/x", 0); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); mesh.read(r2, &x, true); - assert!(mesh.settle()); - assert_eq!(mesh.parent(r2, &x), Some(r1)); - assert_eq!(mesh.parent(r1, &x), Some(r3)); - assert!(!mesh.offered(r2, r3, &x) && !mesh.offered(r2, r1, &x)); - assert_eq!(mesh.warm_sent(&x), 2, "r2's own copy, offered to r1 and r3"); - mesh.steady("behalf"); - - mesh.read(r1, &x, true); - assert!(mesh.settle()); - assert!(mesh.offered(r2, r1, &x)); + assert!(mesh.settle(), "{}", mesh.dump()); + assert_eq!(mesh.carrier(r3, r1, &x), Some(r2), "r1 offers r3 the copy of r2"); + assert!(!mesh.offered(r2, r1, &x), "and r2 nothing that leads back"); assert_eq!(mesh.parent(r2, &x), Some(r1)); - mesh.steady("own reader"); + assert_eq!(mesh.parent(r1, &x), Some(r3), "r1 follows its route"); + mesh.steady("route"); + assert_eq!(mesh.loops, 0); } const PATHS: [&str; 8] = [ @@ -472,28 +1092,30 @@ const PATHS: [&str; 8] = [ /// Two relays at equal cost, each with a reader of every broadcast, whose readers /// arrive in the same instant: per broadcast only the lower hash keeps its uplink and /// the other joins it, so the pair pulls one copy, and the owner differs across -/// broadcasts. Unranked, the two trade copies for good. +/// broadcasts. Unranked, each joins the other and neither is fed. #[test] fn equal_relays_pull_one_copy_owned_by_the_lower_hash() { let (o, x, y) = (0, 1, 2); - let build = |rules: Rules| { - let mut mesh = Mesh::new(3, rules); + let build = |ranked: bool| { + let mut mesh = Mesh::new(3); + mesh.ranked = ranked; mesh.origin(o, 7); mesh.join(o, x, 3); mesh.join(o, y, 3); mesh.join(x, y, 1); let casts: Vec = PATHS.iter().map(|path| mesh.broadcast(7, path, 0)).collect(); for cast in &casts { - mesh.sim.carry(x, cast); - mesh.sim.carry(y, cast); + mesh.relays[x].carry.entry(cast.clone()).or_default().reader = true; + mesh.relays[y].carry.entry(cast.clone()).or_default().reader = true; } + mesh.step(); (mesh, casts) }; - let (mut mesh, casts) = build(PICK); - assert!(mesh.settle()); + let (mut mesh, casts) = build(true); + assert!(mesh.settle(), "{}", mesh.dump()); let mut owners = BTreeSet::new(); for cast in &casts { - let rank = |node: Node| rendezvous(&cast.1, hop(node, 0)); + let rank = |node: Node| rendezvous(&cast.1, id(node)); let (low, high) = match rank(x) < rank(y) { true => (x, y), false => (y, x), @@ -507,14 +1129,23 @@ fn equal_relays_pull_one_copy_owned_by_the_lower_hash() { mesh.steady("pair"); assert_eq!(mesh.loops, 0); - let (mut mesh, _) = build(Rules { ranked: false, ..PICK }); - assert!(!mesh.run(), "unranked, the two trade copies for good"); + // Unranked, each takes the other's copy and the pair settles into a cycle that + // feeds neither: the rank is what breaks the tie. + let (mut mesh, casts) = build(false); + assert!(mesh.settle(), "{}", mesh.dump()); + for cast in &casts { + assert_eq!( + (mesh.parent(x, cast), mesh.parent(y, cast)), + (Some(y), Some(x)), + "{cast:?}" + ); + } } /// A ring of relays, each with its own link to the publisher and a reader, loses all /// but one of those links while every link of the ring is held, so each relay acts on /// what it last heard. Delivered in any order, once quiet every reader is fed through -/// the one link left, and the pulls formed a cycle only while a withdrawal was still +/// the one link left, and the pulls formed a cycle only while a change was still /// crossing a link. #[test] fn a_ring_of_stale_offers_settles_on_the_link_left() { @@ -523,7 +1154,7 @@ fn a_ring_of_stale_offers_settles_on_the_link_left() { let relays: Vec = (1..=RING).collect(); let mut loops = 0; for seed in 0..20u64 { - let mut mesh = Mesh::new(RING + 1, PICK); + let mut mesh = Mesh::new(RING + 1); mesh.origin(o, 7); for (i, &relay) in relays.iter().enumerate() { mesh.join(o, relay, 1); @@ -533,7 +1164,7 @@ fn a_ring_of_stale_offers_settles_on_the_link_left() { for &relay in &relays { mesh.read(relay, &x, true); } - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); for (i, &relay) in relays.iter().enumerate() { let next = relays[(i + 1) % RING]; mesh.hold(relay, next); @@ -555,24 +1186,24 @@ fn a_ring_of_stale_offers_settles_on_the_link_left() { } mesh.deliver_one(&busy[rng.next(busy.len() as u64) as usize]); } - assert!(mesh.settle(), "seed {seed}"); + assert!(mesh.settle(), "seed {seed}\n{}", mesh.dump()); mesh.steady(&format!("seed {seed}")); assert_eq!(mesh.copies(&x), 1, "seed {seed}"); assert_eq!(mesh.parent(relays[0], &x), Some(o), "seed {seed}"); loops += mesh.loops; } - assert!(loops <= 20, "{loops} steps with a cycle across 20 seeds"); + assert!(loops <= 40, "{loops} steps with a cycle across 20 seeds"); } /// `b` withdraws its copy while `c`'s subscription to it is on its way. The -/// subscription reaches `b` with no copy left, and `b` pulls along its own route, as -/// for any request; `c` then moves back to the skip. +/// subscription reaches `b` with no copy left, and `b`, which may not pull through the +/// neighbour it serves, follows its route; `c` then moves back to the skip. #[test] fn a_request_crossing_a_withdrawal_is_forwarded_without_a_loop() { let (a, b, c) = (1, 2, 3); - let (mut mesh, x) = chain(5); + let (mut mesh, x) = chain(5, &ranking(b, c)); mesh.read(b, &x, true); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); assert!(mesh.offered(c, b, &x)); mesh.hold(c, b); mesh.hold(b, c); @@ -594,7 +1225,7 @@ fn a_request_crossing_a_withdrawal_is_forwarded_without_a_loop() { ); mesh.release(c, b); mesh.release(b, c); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); assert_eq!(mesh.parent(c, &x), Some(a)); assert!(!mesh.carries(b, &x)); mesh.steady("crossed"); @@ -606,10 +1237,10 @@ fn a_request_crossing_a_withdrawal_is_forwarded_without_a_loop() { #[test] fn a_replacement_pull_overtaking_the_old_cancel_keeps_its_feed() { let (a, b, c) = (1, 2, 3); - let (mut mesh, x) = chain(5); + let (mut mesh, x) = chain(5, &ranking(b, c)); mesh.read(b, &x, true); mesh.read(c, &x, true); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); assert_eq!(mesh.parent(c, &x), Some(b)); mesh.hold(c, b); @@ -624,9 +1255,9 @@ fn a_replacement_pull_overtaking_the_old_cancel_keeps_its_feed() { assert_eq!(streams.len(), 2, "the old cancel and the new pull wait"); assert!(mesh.deliver_stream(c, b, &x, streams[1])); assert!(mesh.deliver_stream(c, b, &x, streams[0])); - assert_eq!(mesh.sim.nodes[b].downstream(&x), BTreeSet::from([c])); + assert_eq!(mesh.relays[b].carry[&x].downstream.len(), 1); mesh.release(c, b); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); assert_eq!(mesh.parent(c, &x), Some(b)); mesh.steady("overtaken"); assert_eq!(mesh.loops, 0); @@ -637,11 +1268,11 @@ fn a_replacement_pull_overtaking_the_old_cancel_keeps_its_feed() { #[test] fn a_copy_of_another_instance_is_never_joined() { let (a, b, c) = (1, 2, 3); - let (mut mesh, old) = chain(5); - let new = mesh.broadcast(7, "live/x", 1); + let (mut mesh, old) = chain(5, &ranking(b, c)); + let new = mesh.broadcast(7, old.1.as_str(), 1); mesh.read(b, &old, true); mesh.read(c, &new, true); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); assert!(mesh.offered(c, b, &old)); assert_eq!(mesh.parent(c, &new), Some(a)); assert!(!mesh.carries(b, &new)); @@ -649,15 +1280,15 @@ fn a_copy_of_another_instance_is_never_joined() { } /// `k` relays on a LAN, each with its own uplink and a reader of every broadcast. In -/// a full mesh each broadcast crosses one uplink. In a ring some relays keep an -/// uplink copy of their own, which the epic accepts. +/// a full mesh each broadcast crosses one uplink. In a ring each relay joins its +/// lower-ranked neighbour, so a relay ranked below both of its own keeps an uplink. #[test] fn lan_relays_share_uplink_copies() { const K: usize = 6; let o = 0; let relays: Vec = (1..=K).collect(); for full in [true, false] { - let mut mesh = Mesh::new(K + 1, PICK); + let mut mesh = Mesh::new(K + 1); mesh.origin(o, 7); for &relay in &relays { mesh.join(o, relay, 3); @@ -675,48 +1306,78 @@ fn lan_relays_share_uplink_copies() { mesh.read(relay, cast, true); } } - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); let label = if full { "full mesh" } else { "ring" }; mesh.steady(label); - let copies: Vec = casts.iter().map(|cast| mesh.copies(cast)).collect(); - // Recorded: one uplink copy per broadcast in the full mesh, and a few in the - // ring, where a relay joins only a copy whose carrier ranks below its own. - // Readers arriving everywhere at once cross their offers on the way, and the - // pulls form a cycle until those cross back. - let total: usize = copies.iter().sum(); - assert_eq!( - (total, mesh.loops), - if full { (8, 5) } else { (15, 0) }, - "{label}: {copies:?}" - ); + for cast in &casts { + let rank = |relay: Node| rendezvous(&cast.1, id(relay)); + let minima = (0..K) + .filter(|&i| { + rank(relays[i]) < rank(relays[(i + 1) % K]) && rank(relays[i]) < rank(relays[(i + K - 1) % K]) + }) + .count(); + assert_eq!(mesh.copies(cast), if full { 1 } else { minima }, "{label}: {cast:?}"); + } + let total: usize = casts.iter().map(|cast| mesh.copies(cast)).sum(); + // Recorded: readers arriving everywhere at once cross their offers on the way. + assert_eq!((total, mesh.loops), if full { (8, 0) } else { (16, 0) }, "{label}"); } } /// While a broadcast is pulled, a relay loses its link toward the publisher, and -/// later a relay drains (GOAWAY moves pulls off its copy). +/// later a relay drains: GOAWAY ranks every source through it last. #[test] fn a_lost_route_and_a_drain_move_pulls_without_a_loop() { let (a, b, c) = (1, 2, 3); - let (mut mesh, x) = chain(5); + let (mut mesh, x) = chain(5, &ranking(b, c)); mesh.read(b, &x, true); mesh.read(c, &x, true); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); assert_eq!(mesh.parent(c, &x), Some(b)); mesh.disconnect(a, b); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); assert_eq!(mesh.parent(c, &x), Some(a)); assert_eq!(mesh.parent(b, &x), Some(c), "b reaches the publisher only through c"); mesh.steady("route loss"); mesh.connect(a, b); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); mesh.steady("reconnected"); mesh.goaway(b); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); assert_eq!(mesh.parent(c, &x), Some(a), "c leaves the draining relay"); mesh.steady("drain"); - assert_eq!(mesh.loops, 0); + // The pull along b's new route crosses c's join of b's copy. + assert!(mesh.loops <= 2, "{} steps with a cycle", mesh.loops); +} + +/// A copy ranks by its links' priority before its cost, as a route does: `c` joins +/// `b`'s copy over a cheap link only while that link's priority is no lower than the +/// skip's. +#[test] +fn a_copy_ranks_by_priority_before_cost() { + let (o, a, b, c) = (0, 1, 2, 3); + for (low, joins) in [(false, true), (true, false)] { + let mut mesh = Mesh::new(4); + mesh.origin(o, 7); + mesh.join(o, a, 1); + mesh.join(a, b, 3); + let priority = if low { -1 } else { 0 }; + mesh.link( + b, + c, + SimLink::cost(1).priority(priority), + SimLink::cost(1).priority(priority), + ); + mesh.join(a, c, 5); + let x = mesh.broadcast(7, &ranking(b, c), 0); + mesh.read(b, &x, true); + mesh.read(c, &x, true); + assert!(mesh.settle(), "{}", mesh.dump()); + assert_eq!(mesh.parent(c, &x), Some(if joins { b } else { a }), "low {low}"); + mesh.steady(&format!("low {low}")); + } } const HOME: Node = 0; @@ -728,8 +1389,8 @@ const SIDE: Node = 5; /// Two relays with an uplink each to the publisher's relay, `NEAR`'s the cheaper, and /// a cheap link between them. `TAIL` hangs off `FAR`, and `SIDE` off `NEAR`. -fn uplinks(rules: Rules) -> (Mesh, Cast) { - let mut mesh = Mesh::new(6, rules); +fn uplinks(path: &str) -> (Mesh, Cast) { + let mut mesh = Mesh::new(6); mesh.origin(ORIGIN, 7); mesh.join(HOME, NEAR, 10); mesh.join(HOME, FAR, 12); @@ -737,122 +1398,96 @@ fn uplinks(rules: Rules) -> (Mesh, Cast) { mesh.join(FAR, TAIL, 1); mesh.join(NEAR, SIDE, 1); mesh.join(ORIGIN, HOME, 1); - let x = mesh.broadcast(7, "live", 0); - assert!(mesh.settle()); + let x = mesh.broadcast(7, path, 0); + assert!(mesh.settle(), "{}", mesh.dump()); (mesh, x) } /// `FAR` carries the broadcast and is farther from the publisher than `NEAR`. A reader /// at `SIDE` pulls through `NEAR`, which joins `FAR`'s copy over the cheap link rather -/// than pull its own over its uplink. Babel's distance cannot allow this: `FAR`'s cost -/// to the publisher is not below `NEAR`'s. +/// than pull its own over its uplink, though it offers its other neighbours `SIDE`'s. +/// Babel's distance cannot allow this: `FAR`'s cost to the publisher is not below +/// `NEAR`'s. #[test] fn a_relay_joins_a_copy_farther_from_the_publisher() { - for rules in [PICK, Rules::EPOCHS] { - let (mut mesh, x) = uplinks(rules); - mesh.read(FAR, &x, true); - assert!(mesh.settle()); - let before = mesh.sim.stats; - mesh.read(SIDE, &x, true); - assert!(mesh.settle()); - assert_eq!(mesh.parent(SIDE, &x), Some(NEAR), "{rules:?}"); - assert_eq!( - mesh.parent(NEAR, &x), - Some(FAR), - "{rules:?}: NEAR joins the farther copy" - ); - assert_eq!(mesh.parent(FAR, &x), Some(HOME), "{rules:?}"); - assert_eq!(mesh.sim.nodes[NEAR].cost(7), Some(11)); - assert_eq!(mesh.sim.nodes[FAR].cost(7), Some(13)); - mesh.steady(&format!("{rules:?}")); - // SIDE's copy came through NEAR, its only neighbour, so it offers it to no one. - assert_eq!(mesh.sim.stats.warm_msgs - before.warm_msgs, 0, "{rules:?}"); + let (mut mesh, x) = uplinks(&ranking(FAR, SIDE)); + mesh.read(FAR, &x, true); + assert!(mesh.settle(), "{}", mesh.dump()); + mesh.read(SIDE, &x, true); + assert!(mesh.settle(), "{}", mesh.dump()); + assert_eq!(mesh.parent(SIDE, &x), Some(NEAR)); + assert_eq!(mesh.parent(NEAR, &x), Some(FAR), "NEAR joins the farther copy"); + assert_eq!(mesh.parent(FAR, &x), Some(HOME)); + assert_eq!(mesh.carrier(SIDE, NEAR, &x), Some(FAR)); + assert_eq!(mesh.carrier(HOME, NEAR, &x), Some(SIDE)); + assert_eq!( + mesh.sim + .path(NEAR) + .route_for(7, &Path::empty()) + .unwrap() + .offer() + .unwrap() + .metric, + 11 + ); + mesh.steady("joined"); - mesh.read(FAR, &x, false); - assert!(mesh.settle()); - assert_eq!(mesh.parent(NEAR, &x), Some(HOME), "{rules:?}"); - assert!(!mesh.carries(FAR, &x), "{rules:?}"); - mesh.steady(&format!("{rules:?}: FAR's reader left")); - assert_eq!(mesh.loops, 0, "{rules:?}"); - } + mesh.read(FAR, &x, false); + assert!(mesh.settle(), "{}", mesh.dump()); + assert_eq!(mesh.parent(NEAR, &x), Some(HOME)); + assert!(!mesh.carries(FAR, &x)); + mesh.steady("FAR's reader left"); + assert_eq!(mesh.loops, 0); } -/// Both relays carry a copy, `NEAR`'s pulled from `FAR`'s whenever `FAR` ranks below -/// it. When `FAR`'s uplink closes the two swap if they must, each still fed from the -/// publisher, and the pulls loop only while the news crosses their link. +/// Both relays carry a copy, the higher-ranked one pulling the other's. When `FAR`'s +/// uplink closes the two swap if they must, each still fed from the publisher. #[test] fn two_copies_never_feed_each_other_once_quiet() { - for rules in [PICK, Rules::EPOCHS] { - let (mut mesh, x) = uplinks(rules); + for path in [ranking(NEAR, FAR), ranking(FAR, NEAR)] { + let (mut mesh, x) = uplinks(&path); mesh.read(FAR, &x, true); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); mesh.read(NEAR, &x, true); mesh.read(TAIL, &x, true); - assert!(mesh.settle()); - mesh.steady(&format!("{rules:?}")); - assert_eq!(mesh.loops, 0, "{rules:?}"); + assert!(mesh.settle(), "{}", mesh.dump()); + mesh.steady(&path); + assert_eq!(mesh.loops, 0, "{path}"); mesh.disconnect(HOME, FAR); - assert!(mesh.settle()); - assert_eq!(mesh.parent(NEAR, &x), Some(HOME), "{rules:?}"); - assert_eq!(mesh.parent(FAR, &x), Some(NEAR), "{rules:?}"); - assert_eq!(mesh.parent(TAIL, &x), Some(FAR), "{rules:?}"); - mesh.steady(&format!("{rules:?}: FAR's uplink closed")); - assert!(mesh.loops <= 3, "{rules:?}: {} steps with a cycle", mesh.loops); + assert!(mesh.settle(), "{}", mesh.dump()); + assert_eq!(mesh.parent(NEAR, &x), Some(HOME), "{path}"); + assert_eq!(mesh.parent(FAR, &x), Some(NEAR), "{path}"); + assert_eq!(mesh.parent(TAIL, &x), Some(FAR), "{path}"); + mesh.steady(&format!("{path}: FAR's uplink closed")); + assert!(mesh.loops <= 3, "{path}: {} steps with a cycle", mesh.loops); } } -/// `NEAR` pulls `FAR`'s copy, whose path crosses `HOME`, and then loses its own -/// session to `HOME`, which casts doubt on `HOME`. Doubt keeps a reader from moving -/// to a copy, not on the one it pulls, so `NEAR` stays on `FAR`'s throughout. -#[test] -fn a_doubt_leaves_the_copy_in_use_alone() { - let (mut mesh, x) = uplinks(PICK); - mesh.read(FAR, &x, true); - mesh.read(SIDE, &x, true); - assert!(mesh.settle()); - assert_eq!(mesh.parent(NEAR, &x), Some(FAR)); - - mesh.disconnect(HOME, NEAR); - assert_eq!(mesh.parent(NEAR, &x), Some(FAR)); - for _ in 0..64 { - match mesh.busy().first().cloned() { - Some(queue) => mesh.deliver_one(&queue), - None => { - mesh.tick(); - } - }; - assert_eq!(mesh.parent(NEAR, &x), Some(FAR)); - } - assert!(mesh.settle()); - mesh.steady("doubt"); - assert_eq!(mesh.loops, 0); -} - /// The publisher leaves while two relays carry a copy. Every copy goes with its -/// route, nothing is left pulling, and nothing remembers a dead hop. +/// route, and nothing is left pulling or offered. #[test] fn a_copy_that_goes_takes_its_offers_along() { - let (mut mesh, x) = uplinks(PICK); + let (mut mesh, x) = uplinks("live/x"); mesh.read(FAR, &x, true); mesh.read(NEAR, &x, true); mesh.read(TAIL, &x, true); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); mesh.disconnect(ORIGIN, HOME); - assert!(mesh.settle()); + assert!(mesh.settle(), "{}", mesh.dump()); for relay in [HOME, NEAR, FAR, TAIL, SIDE] { assert!(!mesh.carries(relay, &x), "relay {relay}"); - assert_eq!(mesh.sim.nodes[relay].retained(), (0, 0), "relay {relay}"); + assert!(mesh.relays[relay].heard.is_empty(), "relay {relay}"); } } /// Random relay meshes with two publishers under churn: readers come and go, /// sessions drop and return, messages are held and delivered out of step, lingers -/// run out, time passes, and relays drain, with links held or not. Once churn stops -/// every mesh settles on the steady state, and a broadcast nobody reads has sent no -/// copy offer. The pulls form a cycle only while a withdrawal crosses a link, which -/// the sweep counts. +/// run out, time passes, and relays drain, with links held or not and some links at +/// a lower priority. Once churn stops every mesh settles on the steady state, and a +/// broadcast nobody reads has sent no copy offer. The pulls form a cycle only while a +/// change crosses a link, which the sweep counts. #[test] fn sweep_cache_routing() { for holds in [true, false] { @@ -893,6 +1528,9 @@ struct Sweep { /// Copy offers and their ends sent, and reader changes, during churn. warm: usize, reads: usize, + /// Relays left following their route because they offered the neighbours they + /// serve different copies, once quiet. + split: usize, } fn sweep(seed: u64, scenarios: usize, holds: bool) -> Sweep { @@ -902,7 +1540,7 @@ fn sweep(seed: u64, scenarios: usize, holds: bool) -> Sweep { for scenario in 0..scenarios { let label = format!("seed {seed} holds {holds} scenario {scenario}"); let origins = [RELAYS, RELAYS + 1]; - let mut mesh = Mesh::new(RELAYS + 2, PICK); + let mut mesh = Mesh::new(RELAYS + 2); mesh.origin(origins[0], 100); mesh.origin(origins[1], 200); if rng.chance(30) { @@ -921,7 +1559,10 @@ fn sweep(seed: u64, scenarios: usize, holds: bool) -> Sweep { links.push((origin, (a + 1 + rng.next(RELAYS as u64 - 1) as usize) % RELAYS)); } for &(a, b) in &links { - mesh.join(a, b, 1 + rng.next(5)); + let cost = 1 + rng.next(5); + let mut side = || SimLink::cost(cost).priority(if rng.chance(15) { -1 } else { 0 }); + let (ab, ba) = (side(), side()); + mesh.link(a, b, ab, ba); } let read = [ mesh.broadcast(100, "live/a", 0), @@ -931,9 +1572,9 @@ fn sweep(seed: u64, scenarios: usize, holds: bool) -> Sweep { mesh.broadcast(100, "live/a", 1), ]; let idle = mesh.broadcast(100, "live/idle", 0); - assert!(mesh.settle(), "{label}"); + assert!(mesh.settle(), "{label}\n{}", mesh.dump()); - let warm = mesh.sim.stats.warm_msgs; + let warm = mesh.warm_msgs; for _ in 0..60 { match rng.next(12) { 0..=2 => { @@ -944,7 +1585,7 @@ fn sweep(seed: u64, scenarios: usize, holds: bool) -> Sweep { } 3 => { let (a, b) = links[rng.next(links.len() as u64) as usize]; - match mesh.sim.connected(a, b) { + match mesh.connected(a, b) { true => mesh.disconnect(a, b), false => mesh.connect(a, b), } @@ -974,16 +1615,18 @@ fn sweep(seed: u64, scenarios: usize, holds: bool) -> Sweep { } } } - out.warm += mesh.sim.stats.warm_msgs - warm; + out.warm += mesh.warm_msgs - warm; for &(a, b) in &links { mesh.release(a, b); mesh.release(b, a); - if !mesh.sim.connected(a, b) { + if !mesh.connected(a, b) { mesh.connect(a, b); } } - assert!(mesh.settle(), "{label}"); + if !mesh.settle() { + panic!("{label}\n{}{}", mesh.dump(), mesh.trace(3)); + } mesh.steady(&label); assert_eq!(mesh.warm_sent(&idle), 0, "{label}"); out.loops += mesh.loops; @@ -991,6 +1634,7 @@ fn sweep(seed: u64, scenarios: usize, holds: bool) -> Sweep { out.looped += usize::from(mesh.loops > 0); out.starved += mesh.starved; out.longest = out.longest.max(mesh.longest); + out.split += mesh.split(); } out } diff --git a/rs/moq-net/src/route/tests/mod.rs b/rs/moq-net/src/route/tests/mod.rs index 85f7f5d9667..f829a4f6e58 100644 --- a/rs/moq-net/src/route/tests/mod.rs +++ b/rs/moq-net/src/route/tests/mod.rs @@ -12,7 +12,6 @@ mod cache; mod compare; mod handoff; -mod paths; mod residual; mod routing; mod rtt; @@ -875,6 +874,10 @@ impl Sim { false } + pub fn now(&self) -> Instant { + self.now + } + /// The next doubt timer on any node. pub fn deadline(&self) -> Option { self.nodes.iter().filter_map(Relay::deadline).min() diff --git a/rs/moq-net/src/route/tests/paths.rs b/rs/moq-net/src/route/tests/paths.rs deleted file mode 100644 index 042848be12b..00000000000 --- a/rs/moq-net/src/route/tests/paths.rs +++ /dev/null @@ -1,1719 +0,0 @@ -//! A prototype router that keeps a route free of loops by its path alone, carrying -//! the warm copies `cache.rs` proves. -//! -//! The router comparisons in `compare.rs` run on [`Sim`] and the real -//! `route::path::Router`, which has no warm copies yet; this prototype exists only -//! for the copies. Nothing outside the tests uses it. -//! -//! A route is one per publisher and names every relay it crossed, each with the -//! session that relay learned the route on. A relay uses the cheapest route whose -//! path does not name it, at once, with no distance, seqno, request, or echo. -//! [`Rules`] switches on what the comparison showed that leaves wanting: -//! -//! - **Causes.** When a session closes, the relay that learned routes on it names -//! that hop in a [`Msg::Dead`], and every relay drops every route through the hop, -//! whichever neighbour offered it. One message per session covers every publisher. -//! - **Holds.** A relay with no route left keeps its claims, unavailable, until the -//! neighbours that learned the route from it have answered: with another way, or -//! with nothing once those behind them have answered in turn. Then it lets go. -//! - **Epochs.** A closed session casts doubt on the node at its far end, since a -//! dead relay is reported one neighbour at a time. No relay moves to a route -//! through a node in doubt until that node advances its epoch. -//! - **Copies.** A relay with a reader for a broadcast offers a route to its copy, -//! and pulls each broadcast it has demand for from the cheapest of its publisher's -//! route and the copies offered, which `cache.rs` proves. -//! -//! Left out, since the copies do not need them: scoped grants, qualified -//! keys, interests, priorities, upstream marks, and RTT. GOAWAY only moves -//! pulls off a neighbour's copy. - -use std::{ - collections::{BTreeMap, BTreeSet, VecDeque}, - sync::Arc, -}; - -use super::*; - -/// One relay on a route's path, and the session it learned the route on. The -/// publisher names itself with session 0. -#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] -pub(super) struct Hop { - pub node: Id, - pub session: u32, -} - -/// Which parts of the prototype run. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub(super) struct Rules { - /// A relay that loses a session names the hop that died, and every relay drops - /// every route through it. Off, each neighbour's copy lives until that neighbour - /// withdraws it: a bare path vector. - pub causes: bool, - /// One [`Msg::Dead`] says what became of every route through its hop on that - /// session. Off, each route is ended, held, or moved by its own message. - pub batch: bool, - /// A relay offers a neighbour no route whose path names that neighbour. - pub horizon: bool, - /// A relay with no route left keeps its claims, unavailable, while something - /// may still bring the route back, and its neighbours keep theirs. - pub holds: bool, - /// A closed session casts doubt on the node at its far end: no relay moves to a - /// route through that node until the node says it is still there, by advancing - /// its epoch. - pub epochs: bool, - /// The same doubt on a timer: it lifts by itself after this many hops of delay, - /// with no epoch and so no message. 0 is off. - pub wait: u64, - /// A relay on a copy moves onto another only if that copy's carrier ranks below - /// the one it is on, so relays never trade copies. - pub ranked: bool, -} - -impl Rules { - /// Paths and nothing else: what lite-06 does per broadcast, here per publisher. - pub const BARE: Self = Self { - causes: false, - batch: false, - horizon: false, - holds: false, - epochs: false, - wait: 0, - ranked: true, - }; - - /// The quest's candidate: a withdrawal names the hop that died. - pub const CAUSES: Self = Self { - causes: true, - ..Self::BARE - }; - - /// Causes, one message per cause and session, and holds. - pub const HOLDS: Self = Self { - batch: true, - holds: true, - ..Self::CAUSES - }; - - /// Holds, and a doubt that lifts by itself after `wait` hops of delay. - pub const fn waits(wait: u64) -> Self { - Self { wait, ..Self::HOLDS } - } - - /// Holds, and epochs. - pub const EPOCHS: Self = Self { - epochs: true, - ..Self::HOLDS - }; -} - -/// A message on a session. -#[derive(Clone, Debug)] -pub(super) enum Msg { - /// Starts a route, or replaces its path and cost. - Offer { id: Id, path: Arc<[Hop]>, cost: Metric }, - /// The sender has no route left but keeps the claims: it may yet recover. - Hold { id: Id }, - /// Ends a route, and the claims that rode it. - End { id: Id }, - /// The session `hop` names closed, so every route through it is dead, whoever - /// offered it. Of the sender's own routes through it, each one `moved` lists - /// continues on its new path. The rest are ended if `gone`, and held if not. - /// - /// `far` is the node at the other end of that session, and its epoch as the - /// relay that saw the session close knew it. - Dead { - hop: Hop, - far: (Id, u64), - moved: Vec<(Id, Arc<[Hop]>, Metric)>, - gone: bool, - }, - /// `node` is at `epoch`: it was still there after every session it saw close - /// before that. - Epoch { node: Id, epoch: u64 }, - /// A copy of one broadcast is `cost` away through the sender, having come from - /// its publisher along `path`: 0 at its carrier, the relay with a reader where the - /// copy left the publisher's route. - Warm { - cast: Cast, - path: Arc<[Hop]>, - cost: Metric, - carrier: Id, - }, - /// The sender offers no copy of that broadcast any more. - Cold { cast: Cast }, - /// Changes to a route's claims, complete as of the publisher's revision `rev`. - Claims { - id: Id, - changes: Vec<(Path, Option)>, - rev: Rev, - }, -} - -/// A publisher's claims, each with the revision its life began at. -type Claims = BTreeMap; - -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -enum State { - Live, - /// Its path holds a hop this relay knows is dead, and the neighbour has not - /// said what became of it yet. - Dead, - /// The neighbour lost the route and holds its claims. - Held, -} - -/// A route a neighbour offered. -#[derive(Clone, Debug)] -struct Heard { - /// The last path it offered. - path: Arc<[Hop]>, - cost: Metric, - state: State, - claims: Claims, - rev: Rev, -} - -/// What this relay last sent a neighbour for a route. -#[derive(Clone, Debug)] -struct Sent { - /// The last path it was offered. - path: Arc<[Hop]>, - cost: Metric, - held: bool, - claims: Claims, - rev: Rev, -} - -struct Session { - id: u32, - /// How the neighbour names itself in a path. - hop: Id, - /// Whether the neighbour passes routes on, as a relay does and a client does not. - forwards: bool, - /// Whether the neighbour reads copies: a relay that chooses per broadcast. - reads: bool, - cost: Metric, - rx: BTreeMap, - tx: BTreeMap, - /// The dead hops this neighbour was told of, while a route sent to it names one. - told: BTreeSet, - /// The epoch this neighbour was told each node is at. - epochs: BTreeMap, - /// The copies this neighbour offers. - copies: BTreeMap, - /// The copies this neighbour was offered. - casts: BTreeMap, -} - -/// The route a relay uses for a publisher, and the claims it shows. -#[derive(Clone, Debug)] -struct Route { - /// The neighbour it goes through, or the one this relay holds under. `None` at - /// the publisher itself, and at a relay holding on its own account. - next: Option, - /// The path this relay offers: its next hop's, then itself. `None` while held. - path: Option>, - cost: Metric, - claims: Claims, - rev: Rev, -} - -/// One broadcast: its publisher, its path, and its instance. Two instances of one -/// path are two broadcasts, and a copy of one never serves the other. -pub(super) type Cast = (Id, Path, u64); - -/// A relay's demand for one broadcast, and where it pulls it from. -#[derive(Clone, Debug, Default)] -struct Carry { - /// A reader of its own, such as a session that forwards nothing: the relay - /// carries a copy, and offers it. - reader: bool, - /// Each neighbour's subscription streams that arrived and were not cancelled. A - /// neighbour that moves back here opens a new stream, which can arrive before the - /// old one's cancel. - downstream: BTreeMap>, - /// The neighbour it pulls through, and whether that joins a copy. - parent: Option<(Node, bool)>, - /// The stream of the pull through `parent`. - stream: u64, - /// Demand is gone, and the copy stays until the linger runs out. - lingering: bool, -} - -/// The draft's Rendezvous Hash of a broadcast's path and a relay, which ranks two -/// neighbours offering each other a copy. -pub(super) fn rendezvous(path: &Path, node: Id) -> u64 { - let mut hash: u64 = 0x420C_0DEC_B00B; - for byte in path.as_str().bytes().chain(node.to_le_bytes()) { - hash = (hash ^ u64::from(byte)).wrapping_mul(0x0100_0000_01b3); - } - hash -} - -/// A copy of a broadcast offered to a neighbour: the path it took from its -/// publisher, its cost from its carrier, and its carrier. -pub(super) type Offer = (Arc<[Hop]>, Metric, Id); - -/// A copy of a broadcast a neighbour offers. -#[derive(Clone, Debug)] -struct Copy { - /// The path the copy took from its publisher to that neighbour. - path: Arc<[Hop]>, - cost: Metric, - carrier: Id, - /// Its path holds a hop this relay knows is dead. - dead: bool, -} - -/// Where a relay would pull a broadcast from. -#[derive(Clone, Debug, PartialEq, Eq)] -pub(super) struct Source { - pub next: Node, - pub cost: Metric, - /// The path the broadcast then took to this relay. - pub path: Arc<[Hop]>, - /// Whether it joins a copy, rather than follow the publisher's route. - pub warm: bool, - /// That copy's carrier. - pub carrier: Id, -} - -/// What a relay sends one neighbour for a route. -enum Want { - Path(Arc<[Hop]>, Metric), - Hold, - End, - /// Whatever it was sent last: the neighbour is the next hop this relay holds - /// under, and waits for this relay's answer. - Keep, -} - -pub(super) struct Relay { - hop: Id, - rules: Rules, - /// Sends only its own routes, like a client. - pub leaf: bool, - opened: u32, - sessions: BTreeMap, - own: Option, - routes: BTreeMap, - /// Dead hops, each kept while a neighbour's route still names it. - dead: BTreeSet, - /// The newest epoch heard of each node a neighbour's route names, its own included. - epochs: BTreeMap, - /// The epoch each such node was at when a session of its was last seen to close. - doubt: BTreeMap, - /// With a timer in place of epochs: when the doubt on each node lifts. - until: BTreeMap, - now: u64, - /// Each broadcast this relay has demand for, or still pulls while its linger runs. - carry: BTreeMap, - /// Never sends or reads a copy: it pulls along its publisher's route alone, as - /// an endpoint that does not choose per broadcast. - pub legacy: bool, - /// Neighbours that sent GOAWAY: no pull joins, or stays on, their copies. - draining: BTreeSet, - /// The source last chosen for each broadcast, pulled or only offered on: its - /// neighbour, whether it is a copy, and that copy's carrier. - via: BTreeMap, - /// Subscription changes for the request streams: a pull, or a cancel, each on - /// its stream. - subs: Vec<(Node, Cast, u64, bool)>, - /// The last subscription stream opened. - streams: u64, - /// A pull moved from one neighbour to another. - pub moves: usize, - shown: BTreeMap<(Id, Path), (Rev, bool)>, - out: Vec<(Node, Msg)>, - events: Vec<(Id, Path, Kind)>, -} - -fn names(path: &[Hop], node: Id) -> bool { - path.iter().any(|hop| hop.node == node) -} - -/// Whether the relay that offered `path` learned it from `node`. -fn child(path: &[Hop], node: Id) -> bool { - path.len() >= 2 && path[path.len() - 2].node == node -} - -fn state(dead: &BTreeSet, path: &[Hop]) -> State { - match path.iter().any(|hop| dead.contains(hop)) { - true => State::Dead, - false => State::Live, - } -} - -impl Relay { - pub fn new(hop: Id, rules: Rules) -> Self { - Self { - hop, - rules, - leaf: false, - opened: 0, - sessions: BTreeMap::new(), - own: None, - routes: BTreeMap::new(), - dead: BTreeSet::new(), - epochs: BTreeMap::new(), - doubt: BTreeMap::new(), - until: BTreeMap::new(), - now: 0, - carry: BTreeMap::new(), - legacy: false, - draining: BTreeSet::new(), - via: BTreeMap::new(), - subs: Vec::new(), - streams: 0, - moves: 0, - shown: BTreeMap::new(), - out: Vec::new(), - events: Vec::new(), - } - } - - pub fn drain(&mut self) -> Vec<(Node, Msg)> { - std::mem::take(&mut self.out) - } - - pub fn events(&mut self) -> Vec<(Id, Path, Kind)> { - std::mem::take(&mut self.events) - } - - pub fn connect(&mut self, peer: Node, hop: Id, forwards: bool, reads: bool, cost: Metric) { - self.opened += 1; - let session = Session { - id: self.opened, - hop, - forwards, - reads, - cost, - rx: BTreeMap::new(), - tx: BTreeMap::new(), - told: BTreeSet::new(), - epochs: BTreeMap::new(), - copies: BTreeMap::new(), - casts: BTreeMap::new(), - }; - self.sessions.insert(peer, session); - let ids: Vec = self.routes.keys().copied().collect(); - for id in ids { - self.advertise(id); - } - self.warm(None); - } - - /// The session to `peer` closed: every route learned on it is dead. - pub fn disconnect(&mut self, peer: Node) { - let Some(session) = self.sessions.remove(&peer) else { - return; - }; - let hop = Hop { - node: self.hop, - session: session.id, - }; - let lost = session.rx.keys().copied().collect(); - let far = (session.hop, self.epochs.get(&session.hop).copied().unwrap_or(0)); - self.draining.remove(&peer); - for carry in self.carry.values_mut() { - carry.downstream.remove(&peer); - } - if self.rules.epochs && session.forwards { - // This relay is still here, whatever its neighbour tells the others. A - // neighbour that passes nothing on tells nobody. - *self.epochs.entry(self.hop).or_default() += 1; - } - self.fail(hop, far, None, Vec::new(), false, lost); - self.spread(self.hop); - } - - pub fn originate(&mut self, id: Id) { - self.own = Some(id); - let path = Arc::from([Hop { - node: self.hop, - session: 0, - }]); - let route = Route { - next: None, - path: Some(path), - cost: 0, - claims: Claims::new(), - rev: 0, - }; - self.routes.insert(id, route); - self.advertise(id); - } - - pub fn announce(&mut self, claim: Path) { - let id = self.own.expect("not a publisher"); - let route = self.routes.get_mut(&id).unwrap(); - route.rev += 1; - route.claims.insert(claim, route.rev); - self.show(id); - self.advertise(id); - } - - pub fn recv(&mut self, from: Node, msg: Msg) { - let Some(session) = self.sessions.get_mut(&from) else { - return; - }; - match msg { - Msg::Offer { id, path, cost } => { - let state = state(&self.dead, &path); - match session.rx.get_mut(&id) { - Some(heard) => { - heard.path = path; - heard.cost = cost; - heard.state = state; - } - None => { - let heard = Heard { - path, - cost, - state, - claims: Claims::new(), - rev: 0, - }; - session.rx.insert(id, heard); - } - } - self.update(id); - } - Msg::Hold { id } => { - if let Some(heard) = session.rx.get_mut(&id) { - heard.state = State::Held; - } - self.update(id); - } - Msg::End { id } => { - session.rx.remove(&id); - // A ROUTE_END retires the copies offered under the route. - session.copies.retain(|cast, _| cast.0 != id); - self.update(id); - } - Msg::Claims { id, changes, rev } => { - let Some(heard) = session.rx.get_mut(&id) else { - return; - }; - for (claim, since) in changes { - match since { - Some(since) => heard.claims.insert(claim, since), - None => heard.claims.remove(&claim), - }; - } - heard.rev = rev; - self.update(id); - } - Msg::Dead { hop, far, moved, gone } => self.fail(hop, far, Some(from), moved, gone, BTreeSet::new()), - Msg::Epoch { node, epoch } => { - if self.epochs.get(&node).copied().unwrap_or(0) >= epoch { - return; - } - self.epochs.insert(node, epoch); - self.spread(node); - for id in self.heard() { - self.update(id); - } - self.warm(None); - } - Msg::Warm { - cast, - path, - cost, - carrier, - } => { - if self.legacy { - return; - } - let dead = state(&self.dead, &path) == State::Dead; - let copy = Copy { - path, - cost, - carrier, - dead, - }; - let id = cast.0; - session.copies.insert(cast, copy); - self.warm(Some(id)); - self.collect(); - } - Msg::Cold { cast } => { - let id = cast.0; - session.copies.remove(&cast); - self.warm(Some(id)); - self.collect(); - } - } - } - - /// The session `hop` names closed. `from` reported it, with what became of its - /// own routes, or this relay saw it close and `lost` what it had learned there. - fn fail( - &mut self, - hop: Hop, - far: (Id, u64), - from: Option, - moved: Vec<(Id, Arc<[Hop]>, Metric)>, - gone: bool, - lost: BTreeSet, - ) { - let mut touched = lost; - let (me, rules) = (self.hop, self.rules); - for (&peer, session) in self.sessions.iter_mut() { - let reported = from == Some(peer); - let through: Vec = (session.rx.iter()) - .filter(|(_, heard)| heard.path.contains(&hop)) - .map(|(id, _)| *id) - .collect(); - for id in through { - touched.insert(id); - let heard = session.rx.get_mut(&id).unwrap(); - let moved = moved.iter().find(|(named, ..)| *named == id).filter(|_| reported); - if let Some((_, path, cost)) = moved { - heard.state = state(&self.dead, path); - heard.path = path.clone(); - heard.cost = *cost; - } else if reported && rules.batch { - if gone || !rules.holds { - session.rx.remove(&id); - session.copies.retain(|cast, _| cast.0 != id); - } else if !child(&heard.path, me) { - heard.state = State::Held; - } - // A neighbour that holds under this relay answers once those behind it have. - } else if heard.state == State::Live { - heard.state = State::Dead; - } - } - } - for session in self.sessions.values_mut() { - for copy in session.copies.values_mut() { - copy.dead |= copy.path.contains(&hop); - } - } - self.dead.insert(hop); - if rules.wait > 0 { - let until = self.until.entry(far.0).or_default(); - *until = (self.now + rules.wait).max(*until); - touched.extend(self.heard()); - } - if rules.epochs { - let doubt = self.doubt.entry(far.0).or_default(); - *doubt = far.1.max(*doubt); - // Any route through that node may have become unusable. - touched.extend(self.heard()); - } - for &id in &touched { - self.select(id); - self.show(id); - } - if rules.causes { - self.report(hop, far, from); - } - for &id in &touched { - self.advertise(id); - } - self.warm(None); - self.collect(); - } - - /// Tell each neighbour that was sent a route through `hop` what became of those - /// routes, in one message where the rules allow. - fn report(&mut self, hop: Hop, far: (Id, u64), from: Option) { - let peers: Vec = self.sessions.keys().copied().collect(); - for peer in peers { - let through: Vec = (self.sessions[&peer].tx.iter()) - .filter(|(_, sent)| sent.path.contains(&hop)) - .map(|(id, _)| *id) - .collect(); - if through.is_empty() { - let session = self.sessions.get_mut(&peer).unwrap(); - let copied = session.casts.values().any(|(path, ..)| path.contains(&hop)); - if copied && from != Some(peer) && session.told.insert(hop) { - let (moved, gone) = (Vec::new(), true); - self.out.push((peer, Msg::Dead { hop, far, moved, gone })); - } - continue; - } - let first = self.sessions.get_mut(&peer).unwrap().told.insert(hop); - if !self.rules.batch { - // Each route's own message follows, and the reporter knows. - if first && from != Some(peer) { - let (moved, gone) = (Vec::new(), false); - self.out.push((peer, Msg::Dead { hop, far, moved, gone })); - } - continue; - } - let (mut moved, mut held, mut ended, mut rest) = (Vec::new(), Vec::new(), Vec::new(), 0); - for id in through { - let want = self.want(id, peer); - let session = self.sessions.get_mut(&peer).unwrap(); - match want { - Want::Path(path, cost) => { - let sent = session.tx.get_mut(&id).unwrap(); - sent.path = path.clone(); - sent.cost = cost; - sent.held = false; - moved.push((id, path, cost)); - } - Want::Hold => { - let sent = session.tx.get_mut(&id).unwrap(); - if !sent.held { - sent.held = true; - // The neighbour it learned the route from takes silence to mean - // this relay holds under it, so it is told outright. - let parent = child(&sent.path, session.hop); - held.push((id, parent)); - } - rest += 1; - } - Want::Keep => rest += 1, - Want::End => { - session.tx.remove(&id); - session.casts.retain(|cast, _| cast.0 != id); - ended.push(id); - } - } - } - let gone = rest == 0; - let paths: Vec> = moved.iter().map(|(_, path, _)| path.clone()).collect(); - if first || (gone && !ended.is_empty()) { - self.out.push((peer, Msg::Dead { hop, far, moved, gone })); - if !gone { - let held = held.into_iter().filter(|(_, parent)| *parent); - self.out.extend(held.map(|(id, _)| (peer, Msg::Hold { id }))); - self.out.extend(ended.into_iter().map(|id| (peer, Msg::End { id }))); - } - } else { - self.out.extend( - moved - .into_iter() - .map(|(id, path, cost)| (peer, Msg::Offer { id, path, cost })), - ); - self.out - .extend(held.into_iter().map(|(id, _)| (peer, Msg::Hold { id }))); - self.out.extend(ended.into_iter().map(|id| (peer, Msg::End { id }))); - } - // An epoch follows the route that names its node, or it would name nothing - // the neighbour holds yet. - for path in paths { - self.tell(peer, &path); - } - } - } - - fn update(&mut self, id: Id) { - self.select(id); - self.show(id); - self.advertise(id); - self.warm(Some(id)); - self.collect(); - } - - /// A reader for `cast` arrives at this relay, which carries a copy from now on. - pub fn carry(&mut self, cast: Cast) { - let id = cast.0; - self.carry.entry(cast).or_default().reader = true; - self.warm(Some(id)); - } - - /// The reader for `cast` leaves. - pub fn uncarry(&mut self, cast: &Cast) { - if let Some(carry) = self.carry.get_mut(cast) { - carry.reader = false; - } - self.warm(Some(cast.0)); - } - - /// A subscription for `cast` from `from` arrives on `stream`, or its cancel. - pub fn subscribed(&mut self, from: Node, cast: Cast, stream: u64, pull: bool) { - if !self.sessions.contains_key(&from) { - return; - } - let id = cast.0; - let carry = self.carry.entry(cast).or_default(); - let streams = carry.downstream.entry(from).or_default(); - match pull { - true => streams.insert(stream), - false => streams.remove(&stream), - }; - if streams.is_empty() { - carry.downstream.remove(&from); - } - self.warm(Some(id)); - } - - /// `peer` sent GOAWAY: every pull on its copy moves off. - pub fn goaway(&mut self, peer: Node) { - if self.sessions.contains_key(&peer) && self.draining.insert(peer) { - self.warm(None); - } - } - - /// The linger for `cast` runs out: a copy with no demand left goes. - pub fn expire(&mut self, cast: &Cast) { - if let Some(carry) = self.carry.get_mut(cast) - && carry.lingering - { - carry.lingering = false; - if let Some((parent, _)) = carry.parent.take() - && self.sessions.contains_key(&parent) - { - self.subs.push((parent, cast.clone(), carry.stream, false)); - } - } - self.warm(Some(cast.0)); - } - - /// Where this relay pulls `cast` from: the cheapest of its next hop toward the - /// publisher and each copy offered. Pulling through the next hop goes wherever - /// that neighbour pulls: its copy when it offers one, else its own route. A copy - /// whose path names this relay, or a neighbour it serves, would loop. - /// - /// A relay already on a copy, its own included, joins another, its next hop's - /// included, only if that copy's carrier ranks below, so copies crossing on their - /// way never trade places round any cycle. Off a copy, it is bound the same way by - /// the carriers of the neighbours that pull, or would pull, along its route, since - /// what it joins they take with it. Left with no source and no such neighbour, it - /// chooses afresh. It neither joins nor stays on a copy of a neighbour that sent - /// GOAWAY, unless that is its next hop's. A tie stays on the current source, then - /// goes to the copy. - pub fn source(&self, cast: &Cast) -> Option { - let carry = self.carry.get(cast); - let reader = carry.is_some_and(|carry| carry.reader); - let downstream = carry.map(|carry| &carry.downstream); - let serves: BTreeSet = (downstream.into_iter().flat_map(BTreeMap::keys)) - .filter_map(|peer| self.sessions.get(peer)) - .map(|session| session.hop) - .collect(); - let via = self.via.get(cast).copied(); - let pull = via.map(|(next, warm, _)| (next, warm)); - let rank = |node: Id| rendezvous(&cast.1, node); - // The carrier of the copy this relay is on: the one it joins while that is still - // offered, or its own, pulled along the publisher's route for a reader. - let own = match via { - Some((next, true, carrier)) => (self.sessions.get(&next)) - .and_then(|session| session.copies.get(cast)) - .filter(|copy| copy.carrier == carrier) - .map(|_| carrier), - Some((_, false, _)) if reader => Some(self.hop), - _ => None, - }; - let next = self - .routes - .get(&cast.0) - .and_then(|route| route.next.zip(route.path.clone())); - // Off a copy, the copies it would hand on are those of each neighbour that pulls - // through it, or would on its route, since that neighbour's next hop's copy is no - // choice: each offers its copy back along this relay's route. Joining another - // would hand each of them another carrier, so it joins only one that ranks below - // them all, and none until each that pulls through it has offered one: an offer - // along any other path is from before it took the route, and a neighbour that - // offers none may feed copies further on. `None` joins nothing it chooses. - let chosen: Option> = match own { - Some(own) => Some(Some(own)), - None => (self.sessions.iter()) - .filter_map(|(peer, session)| { - let pulls = downstream.is_some_and(|downstream| downstream.contains_key(peer)); - let behind = (session.rx.get(&cast.0)).is_some_and(|heard| child(&heard.path, self.hop)); - let fresh = (session.copies.get(cast)) - .filter(|copy| next.as_ref().is_some_and(|(_, path)| copy.path.starts_with(path))) - .map(|copy| copy.carrier); - match fresh { - Some(carrier) if pulls || behind => Some(Some(carrier)), - None if pulls => Some(None), - _ => None, - } - }) - .try_fold(None, |low: Option, carrier| { - let carrier = carrier?; - Some(Some(low.filter(|&low| rank(low) < rank(carrier)).unwrap_or(carrier))) - }), - }; - let mut options = Vec::new(); - for (&peer, session) in &self.sessions { - let hop = next.as_ref().filter(|(next, _)| *next == peer); - let Some(copy) = session.copies.get(cast) else { - options.extend(hop.map(|(next, path)| Source { - next: *next, - cost: self.routes[&cast.0].cost, - path: path.clone(), - warm: false, - carrier: 0, - })); - continue; - }; - // As in `select`, doubt keeps a reader from moving to a copy, not on the one - // it pulls: that copy's own neighbour says when it goes. - let doubted = pull != Some((peer, true)) && self.doubted(©.path); - let loops = copy - .path - .iter() - .any(|hop| hop.node == self.hop || serves.contains(&hop.node)); - if copy.dead || loops || doubted { - continue; - } - let ranked = |on: Id| copy.carrier == on || rank(copy.carrier) < rank(on); - let on = match hop { - Some(_) => Some(own), - None => chosen, - }; - if self.rules.ranked && !on.is_some_and(|on| on.is_none_or(ranked)) { - continue; - } - // The next hop's copy is where its route leads; any other is a choice. - if hop.is_none() && self.draining.contains(&peer) { - continue; - } - let Some(cost) = copy.cost.checked_add(session.cost) else { - continue; - }; - let mut path = copy.path.to_vec(); - path.push(Hop { - node: self.hop, - session: session.id, - }); - options.push(Source { - next: peer, - cost, - path: path.into(), - warm: true, - carrier: copy.carrier, - }); - } - (options.into_iter()).min_by_key(|source| { - let stay = pull == Some((source.next, source.warm)); - (source.cost, !stay, !source.warm, source.next) - }) - } - - /// What this relay offers its neighbours for `cast`, under its route to the - /// publisher: a copy it would join, at what joining it costs, for as long as that - /// is cheaper than the publisher's route, or else, with a reader of its own, the - /// copy it pulls along the publisher's route at no cost. A copy's cost so rises - /// with every link from where it left the publisher's route. - fn copy(&self, cast: &Cast) -> Option { - if self.own == Some(cast.0) || self.leaf || self.legacy { - return None; - } - self.routes.get(&cast.0)?.path.as_ref()?; - let source = self.source(cast)?; - match self.carry.get(cast).is_some_and(|carry| carry.reader) { - _ if source.warm => Some((source.path, source.cost, source.carrier)), - true => Some((source.path, 0, self.hop)), - false => None, - } - } - - /// Follow the demand for `cast`: pull it from its source, move with the source, - /// and keep the pull while the linger runs once the demand is gone. - fn pull(&mut self, cast: &Cast) { - let source = self.source(cast); - match &source { - Some(source) => self - .via - .insert(cast.clone(), (source.next, source.warm, source.carrier)), - None => self.via.remove(cast), - }; - let source = source.map(|source| (source.next, source.warm)); - let Some(carry) = self.carry.get_mut(cast) else { - return; - }; - let demand = carry.reader || !carry.downstream.is_empty(); - if demand { - carry.lingering = false; - } else if carry.parent.is_some() { - carry.lingering = true; - } - let parent = source.filter(|_| demand || carry.lingering); - let old = std::mem::replace(&mut carry.parent, parent); - let (old, new) = (old.map(|(next, _)| next), parent.map(|(next, _)| next)); - let stream = carry.stream; - if new.is_some() && old != new { - self.streams += 1; - carry.stream = self.streams; - } - if !demand && parent.is_none() { - self.carry.remove(cast); - } - if old == new { - return; - } - if let Some(old) = old - && self.sessions.contains_key(&old) - { - self.subs.push((old, cast.clone(), stream, false)); - } - if let Some(new) = new { - self.subs.push((new, cast.clone(), self.streams, true)); - } - if old.is_some() && new.is_some() { - self.moves += 1; - } - } - - /// Bring each pull, and what each neighbour was offered, up to date, for the - /// copies of `id` or for every copy. A neighbour that reads copies is offered this - /// relay's, even one whose path names it, under the route it was sent. - fn warm(&mut self, id: Option) { - let mut casts: BTreeSet = self.carry.keys().cloned().collect(); - for session in self.sessions.values() { - casts.extend(session.copies.keys().cloned()); - casts.extend(session.casts.keys().cloned()); - } - for cast in casts { - if id.is_some_and(|id| id != cast.0) { - continue; - } - // What this relay offers decides which offers it may join, so the two settle - // together: joining a neighbour's copy only ends what it offers that neighbour. - for _ in 0..4 { - self.pull(&cast); - if !self.offer(&cast) { - break; - } - } - } - } - - /// Bring what each neighbour was offered for `cast` up to date. Whether anything changed. - fn offer(&mut self, cast: &Cast) -> bool { - let mut changed = false; - { - let cast = cast.clone(); - let copy = self.copy(&cast); - let peers: Vec = self.sessions.keys().copied().collect(); - for peer in peers { - let session = &self.sessions[&peer]; - // Even a copy whose path names the neighbour: it tells the neighbour that - // pulling through this relay leads back to it. - let routed = session.tx.contains_key(&cast.0) && session.reads; - let want = copy.clone().filter(|_| routed); - if session.casts.get(&cast) == want.as_ref() { - continue; - } - changed = true; - match want { - Some((path, cost, carrier)) => { - let session = self.sessions.get_mut(&peer).unwrap(); - session.casts.insert(cast.clone(), (path.clone(), cost, carrier)); - let msg = Msg::Warm { - cast: cast.clone(), - path: path.clone(), - cost, - carrier, - }; - self.out.push((peer, msg)); - self.tell(peer, &path); - } - None => { - self.sessions.get_mut(&peer).unwrap().casts.remove(&cast); - self.out.push((peer, Msg::Cold { cast: cast.clone() })); - } - } - } - } - changed - } - - pub fn drain_subs(&mut self) -> Vec<(Node, Cast, u64, bool)> { - std::mem::take(&mut self.subs) - } - - /// The neighbour this relay pulls `cast` through, if it pulls it. - pub fn parent(&self, cast: &Cast) -> Option { - self.carry.get(cast)?.parent.map(|(next, _)| next) - } - - pub fn reading(&self, cast: &Cast) -> bool { - self.carry.get(cast).is_some_and(|carry| carry.reader) - } - - pub fn downstream(&self, cast: &Cast) -> BTreeSet { - self.carry - .get(cast) - .map(|carry| carry.downstream.keys().copied().collect()) - .unwrap_or_default() - } - - /// The subscription streams for `cast` that arrived and were not cancelled. - pub fn streams(&self, cast: &Cast) -> usize { - (self.carry.get(cast)).map_or(0, |carry| carry.downstream.values().map(BTreeSet::len).sum()) - } - - /// Whether the copy of `cast` stays only while its linger runs. - pub fn lingering(&self, cast: &Cast) -> bool { - self.carry.get(cast).is_some_and(|carry| carry.lingering) - } - - /// The broadcasts this relay pulls while its linger runs. - pub fn lingers(&self) -> Vec { - (self.carry.iter()) - .filter(|(_, carry)| carry.lingering) - .map(|(cast, _)| cast.clone()) - .collect() - } - - /// The copy of `cast` this relay offers `peer`, and the one `peer` offers it. - pub fn offers(&self, peer: Node, cast: &Cast) -> (Option, Option) { - let Some(session) = self.sessions.get(&peer) else { - return (None, None); - }; - let sent = session.casts.get(cast).cloned(); - let heard = (session.copies.get(cast)).map(|copy| (copy.path.clone(), copy.cost, copy.carrier)); - (sent, heard) - } - - /// Forget each dead hop no neighbour's route names any more. - fn collect(&mut self) { - let sessions = &self.sessions; - self.dead.retain(|hop| { - (sessions.values()).any(|session| { - let routes = session.rx.values().any(|heard| heard.path.contains(hop)); - routes || session.copies.values().any(|copy| copy.path.contains(hop)) - }) - }); - let me = self.hop; - let named = |node: &Id| { - *node == me - || (sessions.values()).any(|session| { - let routes = session.rx.values().any(|heard| names(&heard.path, *node)); - routes || session.copies.values().any(|copy| names(©.path, *node)) - }) - }; - self.epochs.retain(|node, _| named(node)); - self.doubt.retain(|node, _| named(node)); - self.until.retain(|node, _| named(node)); - for session in self.sessions.values_mut() { - let tx = &session.tx; - let casts = &session.casts; - session.told.retain(|hop| { - tx.values().any(|sent| sent.path.contains(hop)) || casts.values().any(|(path, ..)| path.contains(hop)) - }); - session.epochs.retain(|node, _| { - tx.values().any(|sent| names(&sent.path, *node)) || casts.values().any(|(path, ..)| names(path, *node)) - }); - } - } - - /// Every publisher a neighbour offered a route to. - fn heard(&self) -> BTreeSet { - (self.sessions.values()) - .flat_map(|session| session.rx.keys().copied()) - .collect() - } - - /// Whether `path` crosses a node that a closed session cast doubt on, and that - /// has not advanced its epoch since. - fn doubted(&self, path: &[Hop]) -> bool { - path.iter().any(|hop| { - let behind = - (self.doubt.get(&hop.node)).is_some_and(|at| self.epochs.get(&hop.node).copied().unwrap_or(0) <= *at); - behind || self.until.contains_key(&hop.node) - }) - } - - /// One hop of delay passes. Whether a doubt is still waiting to lift. - pub fn tick(&mut self) -> bool { - self.now += 1; - let before = self.until.len(); - let now = self.now; - self.until.retain(|_, until| *until > now); - if self.until.len() < before { - for id in self.heard() { - self.update(id); - } - self.warm(None); - } - !self.until.is_empty() - } - - /// Tell `peer` the epoch of each node `path` names, where it has an older one. - fn tell(&mut self, peer: Node, path: &[Hop]) { - let session = self.sessions.get_mut(&peer).unwrap(); - for hop in path { - let epoch = self.epochs.get(&hop.node).copied().unwrap_or(0); - if epoch > session.epochs.get(&hop.node).copied().unwrap_or(0) { - session.epochs.insert(hop.node, epoch); - let node = hop.node; - self.out.push((peer, Msg::Epoch { node, epoch })); - } - } - } - - /// Pass `node`'s epoch on to each neighbour that was sent a route naming it. - fn spread(&mut self, node: Id) { - let epoch = self.epochs.get(&node).copied().unwrap_or(0); - for (&peer, session) in self.sessions.iter_mut() { - let named = session.tx.values().any(|sent| names(&sent.path, node)) - || session.casts.values().any(|(path, ..)| names(path, node)); - if named && epoch > session.epochs.get(&node).copied().unwrap_or(0) { - session.epochs.insert(node, epoch); - self.out.push((peer, Msg::Epoch { node, epoch })); - } - } - } - - /// Pick the cheapest live route whose path does not name this relay, and take - /// the newest claims any such route carries. With none left, hold or let go. - fn select(&mut self, id: Id) { - if self.own == Some(id) { - return; - } - let was = self.routes.remove(&id); - let current = was.as_ref().and_then(|route| route.next); - let using = current.filter(|_| was.as_ref().is_some_and(|route| route.path.is_some())); - // Stay on the next hop among equals. - type Rank = (Metric, bool, Node); - let mut best: Option<(Rank, Arc<[Hop]>)> = None; - let mut hope = false; - for (&peer, session) in &self.sessions { - let Some(heard) = session.rx.get(&id) else { - continue; - }; - if heard.state != State::Live || names(&heard.path, self.hop) { - continue; - } - // Doubt keeps a relay from moving to a route, not on the one it uses: that - // one's own neighbour says when it goes. - if Some(peer) != using && self.doubted(&heard.path) { - // It may come back: the node in doubt has yet to say it is there. - hope = true; - continue; - } - let Some(cost) = heard.cost.checked_add(session.cost) else { - continue; - }; - let rank = (cost, Some(peer) != current, peer); - if best.as_ref().is_none_or(|(best, _)| rank < *best) { - let mut path = heard.path.to_vec(); - path.push(Hop { - node: self.hop, - session: session.id, - }); - best = Some((rank, path.into())); - } - } - let had = was.is_some(); - let (mut claims, mut rev) = was.map_or((Claims::new(), 0), |route| (route.claims, route.rev)); - for session in self.sessions.values() { - if let Some(heard) = session.rx.get(&id) - && heard.state == State::Live - && !names(&heard.path, self.hop) - && heard.rev > rev - { - claims = heard.claims.clone(); - rev = heard.rev; - } - } - let route = match best { - Some(((cost, _, peer), path)) => Route { - next: Some(peer), - path: Some(path), - cost, - claims, - rev, - }, - None => { - // Only a relay that had the route holds it. - if !self.rules.holds || !had { - return; - } - // Its next hop holds, or has yet to say what became of a dead route. A next - // hop whose own route names this relay waits on this relay instead. - let under = current - .and_then(|next| self.sessions.get(&next)) - .and_then(|session| session.rx.get(&id)) - .is_some_and(|heard| heard.state != State::Live && !names(&heard.path, self.hop)); - if !under && !hope && !self.pending(id) { - return; - } - Route { - next: current.filter(|_| under), - path: None, - cost: 0, - claims, - rev, - } - } - }; - self.routes.insert(id, route); - } - - /// Whether a neighbour that learned `id` from this relay has yet to answer its loss. - fn pending(&self, id: Id) -> bool { - self.sessions.values().any(|session| { - (session.rx.get(&id)).is_some_and(|heard| heard.state != State::Held && child(&heard.path, self.hop)) - }) - } - - /// Whether a neighbour offers a route to `id` that only doubt keeps this relay off. - fn hopes(&self, id: Id) -> bool { - self.sessions.values().any(|session| { - (session.rx.get(&id)).is_some_and(|heard| { - heard.state == State::Live && !names(&heard.path, self.hop) && self.doubted(&heard.path) - }) - }) - } - - fn want(&self, id: Id, peer: Node) -> Want { - let Some(route) = self.routes.get(&id) else { - return Want::End; - }; - if self.leaf && self.own != Some(id) { - return Want::End; - } - match &route.path { - Some(path) if self.rules.horizon && names(path, self.sessions[&peer].hop) => Want::End, - Some(path) => Want::Path(path.clone(), route.cost), - // The next hop it holds under hears nothing until those behind this relay answered. - None if route.next == Some(peer) => match self.pending(id) || self.hopes(id) { - true => Want::Keep, - false => Want::End, - }, - None => Want::Hold, - } - } - - /// Bring what each neighbour was sent for `id` up to date. - fn advertise(&mut self, id: Id) { - let peers: Vec = self.sessions.keys().copied().collect(); - for peer in peers { - let want = self.want(id, peer); - let session = self.sessions.get_mut(&peer).unwrap(); - let (path, cost) = match want { - Want::Path(path, cost) => (path, cost), - Want::Keep => continue, - Want::Hold => { - // A hold follows a route: there is nothing to hold before one. - if let Some(sent) = session.tx.get_mut(&id) - && !sent.held - { - sent.held = true; - self.out.push((peer, Msg::Hold { id })); - } - continue; - } - Want::End => { - if session.tx.remove(&id).is_some() { - session.casts.retain(|cast, _| cast.0 != id); - self.out.push((peer, Msg::End { id })); - } - continue; - } - }; - let sent = session.tx.get(&id); - let offer = sent.is_none_or(|sent| sent.held || sent.path != path || sent.cost != cost); - let (was, at) = sent.map_or((Claims::new(), 0), |sent| (sent.claims.clone(), sent.rev)); - if offer { - let offered = path.clone(); - self.out.push(( - peer, - Msg::Offer { - id, - path: offered, - cost, - }, - )); - self.tell(peer, &path); - } - let route = &self.routes[&id]; - let session = self.sessions.get_mut(&peer).unwrap(); - if was != route.claims || at != route.rev { - let mut changes: Vec<(Path, Option)> = (route.claims.iter()) - .filter(|(claim, since)| was.get(*claim) != Some(*since)) - .map(|(claim, since)| (claim.clone(), Some(*since))) - .collect(); - let ended = was.keys().filter(|claim| !route.claims.contains_key(*claim)); - changes.extend(ended.map(|claim| (claim.clone(), None))); - let rev = route.rev; - self.out.push((peer, Msg::Claims { id, changes, rev })); - } - let sent = Sent { - path, - cost, - held: false, - claims: route.claims.clone(), - rev: route.rev, - }; - session.tx.insert(id, sent); - } - } - - /// Tell the reader what changed for `id`. - fn show(&mut self, id: Id) { - let route = self.routes.get(&id); - let available = route.is_some_and(|route| route.path.is_some()); - let empty = Claims::new(); - let claims = route.map_or(&empty, |route| &route.claims); - let was: Vec<(Path, (Rev, bool))> = (self.shown.range((id, Path::empty())..)) - .take_while(|((named, _), _)| *named == id) - .map(|((_, claim), shown)| (claim.clone(), *shown)) - .collect(); - for (claim, (since, shown)) in was { - if claims.get(&claim) != Some(&since) { - let kind = match route.is_some() { - true => Kind::End, - false => Kind::Offline, - }; - self.events.push((id, claim.clone(), kind)); - self.shown.remove(&(id, claim)); - } else if shown != available { - let kind = match available { - true => Kind::Available, - false => Kind::Unavailable, - }; - self.events.push((id, claim.clone(), kind)); - self.shown.insert((id, claim), (since, available)); - } - } - for (claim, since) in claims { - if !self.shown.contains_key(&(id, claim.clone())) { - self.events.push((id, claim.clone(), Kind::Start)); - self.shown.insert((id, claim.clone()), (*since, available)); - } - } - } - - /// The neighbour this relay routes `id` through: `Some(None)` at the publisher. - pub fn next(&self, id: Id) -> Option> { - let route = self.routes.get(&id)?; - route.path.as_ref().map(|_| route.next) - } - - pub fn cost(&self, id: Id) -> Option { - let route = self.routes.get(&id)?; - route.path.as_ref().map(|_| route.cost) - } - - /// Whether this relay shows `claim` from `id`, and whether it is available. - pub fn shows(&self, id: Id, claim: &Path) -> Option { - self.shown.get(&(id, claim.clone())).map(|(_, available)| *available) - } - - /// Dead hops remembered, and routes and copies heard. - pub fn retained(&self) -> (usize, usize) { - let heard = (self.sessions.values()) - .map(|session| session.rx.len() + session.copies.len()) - .sum(); - (self.dead.len(), heard) - } -} - -/// Messages sent, as protocol counts. -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] -pub(super) struct PathStats { - /// Offers, holds, ends, and causes. - pub route_msgs: usize, - /// Routes those messages name: a cause counts itself and each route it moved. - pub route_items: usize, - pub claim_msgs: usize, - pub claim_items: usize, - /// Copy offers and their ends, also counted as route messages. - pub warm_msgs: usize, - /// Subscriptions and their cancels. - pub subscribes: usize, - pub delivered: usize, -} - -/// [`Sim`] for the prototype: one [`Relay`] per node, joined by ordered queues. -pub(super) struct PathSim { - pub nodes: Vec, - links: Vec>>, - up: BTreeSet<(Node, Node)>, - queues: BTreeMap<(Node, Node), VecDeque>, - /// Subscriptions, each on a stream of its own, unordered against the routes and - /// each other: true pulls, false cancels. - subs: BTreeMap<(Node, Node, Cast, u64), VecDeque>, - held: BTreeSet<(Node, Node)>, - pub stats: PathStats, - /// Copy offers and their ends sent per broadcast. - pub warm: BTreeMap, - pub events: Vec<(Node, Id, String, Kind)>, -} - -impl PathSim { - pub fn new(nodes: usize, rules: Rules) -> Self { - Self { - nodes: (0..nodes).map(|node| Relay::new(hop(node, 0), rules)).collect(), - links: vec![vec![None; nodes]; nodes], - up: BTreeSet::new(), - queues: BTreeMap::new(), - subs: BTreeMap::new(), - held: BTreeSet::new(), - stats: PathStats::default(), - warm: BTreeMap::new(), - events: Vec::new(), - } - } - - pub fn leaf(&mut self, node: Node) { - self.nodes[node].leaf = true; - } - - /// A link costing `ab` at `a` and `ba` at `b`. - pub fn link(&mut self, a: Node, b: Node, ab: Metric, ba: Metric) { - self.links[a][b] = Some(ab); - self.links[b][a] = Some(ba); - self.connect(a, b); - } - - pub fn join(&mut self, a: Node, b: Node, cost: Metric) { - self.link(a, b, cost, cost); - } - - fn pair(a: Node, b: Node) -> (Node, Node) { - (a.min(b), a.max(b)) - } - - pub fn connect(&mut self, a: Node, b: Node) { - assert!(!self.connected(a, b), "{a}-{b} already up"); - self.up.insert(Self::pair(a, b)); - for (x, y) in [(a, b), (b, a)] { - let cost = self.links[x][y].expect("a configured link"); - let peer = hop(y, 0); - let forwards = !self.nodes[y].leaf; - let reads = forwards && !self.nodes[y].legacy; - self.nodes[x].connect(y, peer, forwards, reads, cost); - } - self.flush(); - } - - pub fn connected(&self, a: Node, b: Node) -> bool { - self.up.contains(&Self::pair(a, b)) - } - - pub fn disconnect(&mut self, a: Node, b: Node) { - self.up.remove(&Self::pair(a, b)); - for (x, y) in [(a, b), (b, a)] { - self.queues.remove(&(x, y)); - self.subs.retain(|(from, to, ..), _| (*from, *to) != (x, y)); - self.held.remove(&(x, y)); - self.nodes[y].disconnect(x); - } - self.flush(); - } - - fn peers(&self, node: Node) -> Vec { - (0..self.nodes.len()) - .filter(|&peer| self.links[node][peer].is_some() && self.connected(node, peer)) - .collect() - } - - pub fn originate(&mut self, node: Node, id: Id) { - self.nodes[node].originate(id); - self.flush(); - } - - pub fn announce(&mut self, node: Node, claim: &str) { - self.nodes[node].announce(p(claim)); - self.flush(); - } - - /// A reader for `cast` arrives at `node`. - pub fn carry(&mut self, node: Node, cast: &Cast) { - self.nodes[node].carry(cast.clone()); - self.flush(); - } - - pub fn uncarry(&mut self, node: Node, cast: &Cast) { - self.nodes[node].uncarry(cast); - self.flush(); - } - - /// `node` sends GOAWAY on every session. - pub fn goaway(&mut self, node: Node) { - for peer in self.peers(node) { - self.nodes[peer].goaway(node); - } - self.flush(); - } - - /// The linger for `cast` runs out at `node`. - pub fn expire(&mut self, node: Node, cast: &Cast) { - self.nodes[node].expire(cast); - self.flush(); - } - - pub fn hold(&mut self, from: Node, to: Node) { - self.held.insert((from, to)); - } - - pub fn release(&mut self, from: Node, to: Node) { - self.held.remove(&(from, to)); - } - - /// The subscription streams with a change to deliver, held or not. - pub fn pending_subs(&self) -> Vec<(Node, Node, Cast, u64)> { - self.subs.keys().cloned().collect() - } - - /// The subscription streams with a change to deliver that are not held. - pub fn busy_subs(&self) -> Vec<(Node, Node, Cast, u64)> { - (self.subs.keys()) - .filter(|(from, to, ..)| !self.held.contains(&(*from, *to))) - .cloned() - .collect() - } - - /// Deliver the next change on the oldest stream of `cast`'s subscriptions from - /// `from` to `to`. - pub fn deliver_sub(&mut self, from: Node, to: Node, cast: &Cast) -> bool { - let stream = (self.subs.keys()).find(|key| (key.0, key.1, &key.2) == (from, to, cast)); - match stream.map(|key| key.3) { - Some(stream) => self.deliver_stream(from, to, cast, stream), - None => false, - } - } - - /// Deliver the next change on one subscription stream from `from` to `to`. - pub fn deliver_stream(&mut self, from: Node, to: Node, cast: &Cast, stream: u64) -> bool { - let key = (from, to, cast.clone(), stream); - let Some(pull) = self.subs.get_mut(&key).and_then(VecDeque::pop_front) else { - return false; - }; - if self.subs[&key].is_empty() { - self.subs.remove(&key); - } - self.nodes[to].subscribed(from, cast.clone(), stream, pull); - self.stats.delivered += 1; - self.flush(); - true - } - - /// One hop of delay passes at every node. Whether any still waits on a timer. - pub fn tick(&mut self) -> bool { - let mut waiting = false; - for relay in &mut self.nodes { - waiting |= relay.tick(); - } - self.flush(); - waiting - } - - /// The queues with a message to deliver that are not held. - pub fn busy(&self) -> Vec<(Node, Node)> { - (self.queues.iter()) - .filter(|(key, queue)| !queue.is_empty() && !self.held.contains(key)) - .map(|(key, _)| *key) - .collect() - } - - pub fn deliver(&mut self, from: Node, to: Node) -> bool { - let Some(msg) = self.queues.get_mut(&(from, to)).and_then(VecDeque::pop_front) else { - return false; - }; - self.nodes[to].recv(from, msg); - self.stats.delivered += 1; - self.flush(); - true - } - - /// Deliver round-robin until every unheld queue, subscriptions included, is - /// empty, or give up. - pub fn run(&mut self) -> bool { - for _ in 0..10_000 { - let (ready, subs) = (self.busy(), self.busy_subs()); - if ready.is_empty() && subs.is_empty() { - return true; - } - for (from, to) in ready { - self.deliver(from, to); - } - for (from, to, cast, stream) in subs { - self.deliver_stream(from, to, &cast, stream); - } - } - false - } - - fn flush(&mut self) { - for node in 0..self.nodes.len() { - for (to, msg) in self.nodes[node].drain() { - match &msg { - Msg::Offer { .. } | Msg::Hold { .. } | Msg::End { .. } | Msg::Epoch { .. } => { - self.stats.route_msgs += 1; - self.stats.route_items += 1; - } - Msg::Warm { cast, .. } | Msg::Cold { cast } => { - self.stats.route_msgs += 1; - self.stats.route_items += 1; - self.stats.warm_msgs += 1; - *self.warm.entry(cast.clone()).or_default() += 1; - } - Msg::Dead { moved, .. } => { - self.stats.route_msgs += 1; - self.stats.route_items += 1 + moved.len(); - } - Msg::Claims { changes, .. } => { - self.stats.claim_msgs += 1; - self.stats.claim_items += changes.len(); - } - } - self.queues.entry((node, to)).or_default().push_back(msg); - } - for (to, cast, stream, pull) in self.nodes[node].drain_subs() { - self.stats.subscribes += 1; - self.subs.entry((node, to, cast, stream)).or_default().push_back(pull); - } - for (id, claim, kind) in self.nodes[node].events() { - self.events.push((node, id, claim.as_str().to_string(), kind)); - } - } - } - - pub fn next(&self, node: Node, id: Id) -> Option> { - self.nodes[node].next(id) - } - - pub fn shows(&self, node: Node, id: Id, claim: &str) -> Option { - self.nodes[node].shows(id, &p(claim)) - } -} - -/// A relay on a ring loses its link toward the publisher. The only other way runs -/// through the neighbour that routed through it, so that neighbour moves first. -/// With no holds the relay lets its claims go and gets them back; with holds they -/// turn unavailable and back, and no claim is sent again. -#[test] -fn a_ring_takes_the_other_way_round() { - for rules in [Rules::BARE, Rules::CAUSES, Rules::HOLDS, Rules::EPOCHS] { - let origin = 4; - let mut sim = PathSim::new(5, rules); - for (a, b) in [(0, 1), (1, 2), (2, 3), (3, 0)] { - sim.join(a, b, 1); - } - sim.join(origin, 0, 1); - sim.originate(origin, 7); - sim.announce(origin, "a"); - assert!(sim.run()); - assert_eq!(sim.next(1, 7), Some(Some(0))); - assert_eq!(sim.next(2, 7), Some(Some(1))); - - let (events, claims) = (sim.events.len(), sim.stats.claim_items); - sim.disconnect(0, 1); - assert!(sim.run()); - assert_eq!(sim.next(1, 7), Some(Some(2)), "{rules:?}"); - assert_eq!(sim.nodes[1].cost(7), Some(4)); - assert_eq!(sim.shows(1, 7, "a"), Some(true)); - let seen: Vec = (sim.events[events..].iter()) - .filter(|(node, ..)| *node == 1) - .map(|(.., kind)| *kind) - .collect(); - match rules.holds { - true => { - assert_eq!(seen, [Kind::Unavailable, Kind::Available], "{rules:?}"); - assert_eq!(sim.stats.claim_items, claims, "{rules:?}: no claim is sent again"); - } - false => assert_eq!(seen, [Kind::Offline, Kind::Start], "{rules:?}"), - } - - sim.disconnect(origin, 0); - assert!(sim.run()); - for node in 0..4 { - assert_eq!(sim.next(node, 7), None, "{rules:?}: node {node}"); - assert_eq!(sim.shows(node, 7, "a"), None, "{rules:?}: node {node}"); - assert_eq!(sim.nodes[node].retained(), (0, 0), "{rules:?}: node {node}"); - } - } -} diff --git a/sh/gh/test-inventory.allow b/sh/gh/test-inventory.allow index fe2c438d147..bca817e34d6 100644 --- a/sh/gh/test-inventory.allow +++ b/sh/gh/test-inventory.allow @@ -35,6 +35,3 @@ ignore moq-video::frame::tests::d3d11_resize_defaults_to_the_gpu /quest/m5/video ignore moq-video::frame::tests::d3d11_resize_matches_cpu /quest/m5/video-hardware.md ignore moq-video::render::export::dx12::tests::exports_an_nv12_texture /quest/m5/video-hardware.md ignore moq-video::render::export::dx12::tests::media_foundation_encodes_the_export_in_place /quest/m5/video-hardware.md - -# Too slow for the merge gate; the quest runs it nightly once every scenario settles. -ignore moq-net::route::tests::cache::sweep_cache_routing_long /quest/m2/cache-routing-proof.md diff --git a/test/route-sweep.sh b/test/route-sweep.sh index c9c1d72a7a2..a41d95a2d22 100755 --- a/test/route-sweep.sh +++ b/test/route-sweep.sh @@ -12,3 +12,5 @@ export CARGO_PROFILE_RELEASE_DEBUG_ASSERTIONS=true CARGO_PROFILE_RELEASE_OVERFLO just rs test -p moq-net --release --features async --test integration -E 'test(=mesh_sweep::seeded_meshes_hold_the_route_rules)' # The router's handoff sweeps replay their fixed scenarios past the gate's 150. MOQ_ROUTE_SCENARIOS=${MOQ_ROUTE_SCENARIOS:-20000} just rs test -P sweep -p moq-net --release -E 'test(/^route::tests::handoff::(distance|path)::sweep_/) | test(/^route::tests::scope::sweep_scoped_meshes_show_what_the_rule_says$/)' +# Cache-aware routing's long sweep, from the same seed, settles every churned mesh of copies: 3000 scenarios each with and without held links, about 3 minutes. +MOQ_ROUTE_CASES=3000 just rs test -P sweep -p moq-net --release --run-ignored only -E 'test(=route::tests::cache::sweep_cache_routing_long)'