diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index 57565d35fe..2f772191f5 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -1,13 +1,11 @@ --- name: verify-route -description: Model-check the lite-07 route layer (`rs/moq-net/src/route/`) with TLA+. Use when a change touches route selection, feasibility, holds, seqno requests, ROUTE_END echoes, collection, claim floors and merging, or what a node's reader is shown, or when a routing bug needs every message ordering checked rather than a random sweep. +description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteHandoff.tla`, the rule the routes epic's Plan item 27 picked, as `rs/moq-net/src/route/tests/paths.rs` prototypes it) with TLA+. Use when a change touches route paths, the DEAD message for a closed session, holds, the doubt and its timer, forgetting dead hops, routes per prefix and scoped streams, the newest-floor views, what a reader is shown, or where a request goes, or when a routing bug needs every message ordering checked rather than a random sweep. --- -# Verify the route layer +# Verify the route rule -`rs/moq-net/verify/route/RouteHandoff.tla` models one origin identity in `route::Router` as `route::tests::Sim` drives it: every router entry point is one atomic step ending in `update`, ROUTE, batches, and ROUTE_REQUEST share one FIFO per direction, and echoes travel on their own queue, free to overtake. A config picks the mesh: which nodes there are, which sessions join them, which cost 2, and which nodes are clients (`Leaves`, as `Sim::leaf`) that forward no route but echo every ROUTE_END. TLC checks every ordering of those messages under each config's bounds, or samples them at random for a mesh too large to exhaust. - -One identity is enough for meshes with several origins: the router keys every piece of state by identity and recomputes one identity from that identity's state alone, so each identity's messages, read on their own, are a run of this model with that origin. Sharing one FIFO per direction only removes orderings, and a session flap is a flap for each. The configs put the origin at the end of a line, in a ring, behind a triangle, and on a client off a ring instead. +`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and a closed session casts doubt on its far end until a timer fires. It follows the prototype `Relay` in `rs/moq-net/src/route/tests/paths.rs` under `Rules::waits`, adds a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The timer is the step `Lift`, which may fire at any moment, or with `Patient` only once no DEAD about its node is in flight. `route::Router` still runs Babel's distance; [the router quest](/quest/m2/cluster-routing/routes/scope.md) builds this rule. ## Rules @@ -15,70 +13,65 @@ Each rule is an INVARIANT; the harm says why it matters. | Rule | Holds when | Harm if broken | | --- | --- | --- | -| `Truthful` | Every view a relay keeps is the origin's claims at that view's revision. | A viewer sees a set of broadcasts the origin never had. | -| `AllCertified` | Every view and batch with claims has a floor. | The model's dropping of `loose` claims is unsound; widen the model first. | +| `Truthful` | Every view a relay keeps, and every floor it was offered, is the publisher's claims at that revision within its prefix. | A viewer sees a set of broadcasts the publisher never had. | +| `Seen` | A relay shows a claim exactly while it holds a route covering it and the newest floor a believed covering route gave it has the claim, as that floor says. | A reader sees a claim no route vouches for, or an older one than it learned. | | `NoRevival` | A reader never sees START for an item it saw END for, unless it forgot it since. | An ended broadcast comes back. | -| `NoFalseEnd` | A reader sees END only once the origin withdrew that item. | A live broadcast is reported as ended rather than offline. | +| `NoFalseEnd` | A reader sees END only once the publisher withdrew that item. | A live broadcast is reported as ended rather than offline. | | `NoRegression` | A reader never sees an UPDATE to an older item. | A stale announcement replaces a newer one. | -| `LoopFree` | Selected next hops never cycle. | Requests loop between relays. | -| `QuietIsSettled` | Once nothing is in flight, each relay reaching the origin shows its claims, available, and routes only while it has any; each cut-off relay shows nothing and keeps no route, distance, or request. | Routing never converges, or a dead origin stays shown. | -| `SimplePaths` | Every route held or in flight names a loop-free path from the origin to its sender, at that path's cost. | A node took back a route that crossed it, the first step of a count to infinity. | -| `NoCountToInfinity` | No relay selects a route longer than a loop-free path. | Stale routes bounce between relays with no bound. | -| `Budget` | From one quiet point to the next, the ROUTE, batch, ROUTE_REQUEST, and echo messages sent stay within 4 each way over each configured session, per route identity and per claim, for each external change made (a claim change, a session opening or closing, a collection). | One change sets off announce spam. | +| `Stable` | A step that only changes a route's path or lifts a doubt changes a shown claim only to a newer floor, or lets go of one it held unavailable. | A detour flaps claims, as a seqno advance did under the distance. | +| `SimplePaths` | Every route held or in flight names a loop-free path from the publisher to its sender. | A relay took back a route that crossed it. | +| `LoopsAreStale` | A request that loops crosses a relay whose route names a closed session. | Routing loops with every relay up to date. | +| `StaleIsChased` | Every stale route a relay may use has the news that ends it on its way: queued toward the relay, or toward the neighbour it came from, and so on back. | A stale route, and any loop through it, lasts for good. | +| `NoStaleReturn` | No relay may use a route through a hop it once learned had died. | A dead path comes back after a relay forgot it. | +| `QuietIsSettled` | Once nothing is in flight and every timer fired: each relay shows exactly the claims a path of open sessions carries to it, available, and a request for each reaches the publisher; no relay holds, keeps a stale route, or remembers a dead hop. | A hold never ends, a claim is shown that no path carries, or a relay behind a scoped neighbour never reaches what a farther one offers. | +| `Budget` | From one quiet point to the next, the messages sent stay within 4 each way over each configured session per prefix and per claim, for each external change. | One change sets off announce spam, or routing never goes quiet. | +| `RequestsLoopFree` | A request never loops. | Only `Loop.cfg` checks it, as an open finding. | -`SimplePaths` needs no search to believe: a node sends a route only after adding itself to hops that did not name it. It holds whatever a node forgot, which is why `NoCountToInfinity` no longer rests on the echo. +No config checks a liveness property: `Spec` has no fairness, so TLC does not show that a hold ends. That one does is an argument over what TLC does check. `Budget` bounds the messages a burst sends, each delivery takes one message off a queue, and a doubt starts only at a close or a delivery, so a burst is a finite number of steps. A delivery is enabled while a queue holds a message and a timer may fire once no news about its node is in flight, so a burst left alone reaches the state with nothing in flight and no doubt. `QuietIsSettled` says no relay holds there. -`Budget` is the one `mesh_sweep.rs` holds the implementation to, with the model's one identity. `Converges` is a PROPERTY: under fairness that something in flight is eventually delivered, routing goes quiet after the last external change and stays settled. `Budget` implies it, since the message count is part of the state: a cycle of deliveries that never goes quiet would grow the count past any budget. So every config checks `Budget`, and only `Line.cfg`, small enough for TLC's liveness checker, checks `Converges` directly. +What the timer costs, as the configs show it: fired late, it costs only the wait, and every rule holds (`RouteHandoff.cfg`). Fired early, a stale route can return, as with no doubt at all, and nothing else breaks (`Early.cfg`, which passes without `NoStaleReturn` and fails it within seconds with it). With two failures a stale route can return whatever the timer (`Twice.cfg` drops `NoStaleReturn`), and `StaleIsChased` is the bound. ## Run ```sh just rs tla moq-net/verify/route/RouteHandoff.tla -just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Behind.cfg -just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/BehindDear.cfg -just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Line.cfg just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Ring.cfg -just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Chord.cfg -simulate num=1000 -depth 600 +just test tla rs/moq-net/verify/route ``` -Times are with `-workers 4`. +`just test tla rs/moq-net/verify/route` runs every config, as Nightly does, simulating each one with a `Simulate:` line and expecting `Loop.cfg` to break. Times are with `-workers 4`. `Patient` is TRUE unless the table says otherwise. | Config | Mesh | Bounds | Distinct states | Run | | --- | --- | --- | --- | --- | -| `RouteHandoff.cfg` | The origin and two relays in a triangle. | 2 claim changes, 2 session changes, 1 collection | 24,192,399 | tens of minutes | -| `Behind.cfg` | The origin behind relay 1, which sits in a triangle with relays 2 and 3. | 1 claim change, 1 session change | 9,384,796 | tens of minutes | -| `BehindDear.cfg` | The same, with relay 3 nearer the origin than relay 2. | 1 claim change, 1 session change | 17,863,476 | an hour | -| `Line.cfg` | The origin, three relays, and a watching client in a line; also checks `Converges`. | 2 claim changes, 1 session change, 1 collection | 1,543,073 | ten minutes | -| `Ring.cfg` | The origin and three relays in a ring. | 2 claim changes, 1 session change | 18,437,833 | an hour | -| `Chord.cfg` | A publishing client on a ring of five relays with a chord, and two watching clients. Simulated: `num` counts behaviours per worker. | 2 claims, 4 claim changes, 4 session changes, 2 collections | 4,000 behaviours, 4.6M states | ten minutes | - -All pass at the sync commit, and `just test tla rs/moq-net/verify/route` runs them all, as Nightly does, simulating `Chord.cfg` as its `Simulate:` line says. Run them last, one at a time. First run what answers in seconds: - -- `just rs test -p moq-net --lib -E 'test(/route::/)'`. `sweep_no_route_outgrows_a_loop_free_path` delivers every message and echo in a random order on random meshes with unequal costs, flaps, restarts, and collections, and checks the metric bound, loop freedom, and the quiet state. Raise its case count for a wider search: 40,000 cases run in seconds in release. -- TLC's simulation mode, which samples runs the exhaustive bounds do not reach. The config's `MaxRev`, `MaxFlaps`, and `MaxCollects` still bound a simulated run, so raise them in a copy of the config first, such as three session changes, two claim changes, and a collection, then add `-simulate num=3000 -depth 150`. - -Both behind-a-triangle configs fail `SimplePaths` within seconds if `Crossed` is made false, which is the count to infinity the hops fix: a relay drops its distance once its neighbours echoed, and takes back a copy of its own route that a third relay still held. +| `RouteHandoff.cfg` | The publisher behind relay 1, in a triangle with relays 2 and 3. | 1 claim change, 1 session change | 407,240 | a minute or two | +| `Early.cfg` | The same, with a timer that fires at any moment. | 1 claim change, 1 session change | 4,178,590 | ten minutes | +| `Twice.cfg` | The same, where two sessions close, as when relay 1 dies. Simulated: `num` counts behaviours per worker. | 1 claim change, 2 session changes | 40,000 behaviours, 4.0M states | a few minutes | +| `Ring.cfg` | The publisher and three relays in a ring, with a timer that fires at any moment. | 1 claim change, 1 session change | 426,409 | a minute or two | +| `Lag.cfg` | Relay 3 hears the publisher through relays 1 and 2: the lagging neighbour. | 2 claim changes, 1 session change | 1,299,405 | a few minutes | +| `LagWide.cfg` | The same, with relay 1 asking for `"a"` alone. | 2 claim changes, 1 session change | 4,194,747 | twenty minutes | +| `Dead.cfg` | Two relays behind a relay whose only uplink asks for `"a"` alone, one with a dear session of its own to the publisher. Simulated. | 3 claim changes, 1 session change | 40,000 behaviours, 4.8M states | ten minutes | +| `Blind.cfg` | A relay whose nearest neighbour asks for `"a"` alone, and a farther one knows everything. Simulated. | 2 claim changes, 1 session change | 40,000 behaviours, 4.8M states | ten minutes | +| `Loop.cfg` | The publisher with a session to each of three relays in a triangle, all closing. Open finding: `RequestsLoopFree` breaks. | 3 session changes | 979,474 when it breaks | a few minutes | + +Run them last, one at a time. First run what answers in seconds: the prototype's tests (`just rs test -p moq-net --lib -E 'test(/route::tests::(paths|compare|copies)/)'`), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`. ## Reading a failure -Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the router with `Sim` (`hold`, `deliver`, `deliver_ack`, `disconnect`, `collect`), one call per step. +Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the prototype with `PathSim` (`deliver`, `disconnect`, `tick`), one call per step, except a timer. `Lift(n, x)` fires one timer on one relay, where `tick` moves every running relay's clock by one hop of delay: call it until that doubt lifts, and where another doubt lifted with it, the replay took a different path, which is no defect in the model. ## Coverage -Sync commit: `caa3dd8926f16fbed15760ab22298085df73b655`. Run `git log caa3dd8926.. -- rs/moq-net/src/route/` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. +Sync commit: `d3b6c2573a30541402d396f8080488646f673594`. Run `git log d3b6c2573a.. -- rs/moq-net/src/route/tests/paths.rs` to see what changed since. The model describes the prototype, extended with a route per prefix and with dead hops kept until their timer fires; `route::Router` does not run this rule yet. `Relay::collect` forgets a dead hop as soon as no route names it, so `NoStaleReturn` covers the rule the router is to run, not the prototype's. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. | Code | Model | | --- | --- | -| `rs/moq-net/src/route/mod.rs` `feasible`, `Entry::crossed`, `eligible`, `metric`, `selected`, `infeasible`, `recoverable`, `live`, `certifiers` | `Feasible`, `Crossed`, `Eligible`, `Through`, `Selected`, `InfeasibleSet`, `Recoverable`, `Live`, `Certifiers` | -| `rs/moq-net/src/route/mod.rs` `certified`, `prune`, `adopted` | `Certified`, `Prune`, `Adopted` | -| `rs/moq-net/src/route/mod.rs` `update`, `recompute`, `record`, `advertise`, `requests` | `Update` and its `prune`, `Record`, `Plan`, `Requests` | -| `rs/moq-net/src/route/mod.rs` `show`, `shown_now` | `Events`, `ShownNow` | -| `rs/moq-net/src/route/mod.rs` `route`, `batch`, `request`, `echo` | `OnRoute`, `OnBatch`, `OnRequest`, `Deliver`, `DeliverAck` | -| `rs/moq-net/src/route/mod.rs` `connect`, `peer_interest`, `disconnect`, `collect`, `change` | `Connect`, `Opened`, `Disconnect`, `Closed`, `Collect`, `ChangeClaim` | -| `rs/moq-net/src/route/view.rs` `View::merge`, `View::restrict` | `Merge`, `Certify` | -| `rs/moq-net/src/route/tests/mod.rs` `Sim` queues and `revivals`, `false_ends` | `q`, `acks`, `revivals`, `falseEnds` | -| `rs/moq-net/src/route/tests/mod.rs` `Sim::leaf`, `Stats` | `Leaves`, `burst` | +| `rs/moq-net/src/route/tests/paths.rs` `Relay::select`, `Relay::pending`, `Relay::hopes`, `Relay::doubted` | `SelectKey`, `Pending`, `Hopes`, `Doubted` | +| `rs/moq-net/src/route/tests/paths.rs` `Relay::want`, `Relay::advertise`, `Relay::report` | `Want`, `Advertise`, `Report` | +| `rs/moq-net/src/route/tests/paths.rs` `Relay::recv`, `Relay::fail`, `Relay::disconnect`, `Relay::connect`, `Relay::change` | `Recv`, `Fail`, `Disconnect`, `Connect`, `ChangeClaim` | +| `rs/moq-net/src/route/tests/paths.rs` `Relay::collect`, `Relay::tick` | `Collect`, `Lift` | +| `rs/moq-net/src/route/tests/paths.rs` `Relay::show` | `Know`, `ShownNow`, `Events` | +| `rs/moq-net/src/route/tests/paths.rs` `PathSim::deliver` | `Deliver` | ## Not covered -The model fixes every interest at the root, so it leaves out wildcards, `present`/`represent`/`retire`, interest changes, and the hidden `current` regions; prune's region bookkeeping reduces to "no certifier has a floor". It also leaves out scoped grants and qualified keys (so `advertised`, which only they read), GOAWAY, link priority, costs other than 1 and 2, the hop limit, the informational RTT, upstream links, identity rotation and restart, `node.rs`'s per-stream keys and the seqno it advances when a route link closes, and how the origin above the router ranks two identities claiming the same broadcast. A batch carries the sender's whole presentation, equal to applying its diff on a FIFO stream. The origin drops the ROUTEs and batches it hears for its own identity, which change nothing there; `Budget` still counts them. An exhaustive pass covers one claim at the bounds in the table above, and no mesh of more than four relays. `sweep_no_route_outgrows_a_loop_free_path`, `sweep_handoff`, `sweep_scoped_handoff`, and `mesh_sweep.rs` still own the rest: larger meshes, more churn, restarts, and scoped grants. +One publisher, so routes for several publishers, their copies (`Relay::warm`, `Relay::source`), and the copy rank are left out; the DEAD message's per-publisher batching reduces to one route per prefix. Two prefixes, one nested in the other, and a scoped relay asks every neighbour for the narrow one: no wildcards, interest edits, grants, qualified keys, or hidden claims. Leaves out restarts, GOAWAY, priorities, costs other than 1 and 2, RTT, and the hop limit. A claims message carries the sender's whole view for its prefix, equal to applying its changes on a FIFO stream. A route is sent on every stream whether or not the publisher has a claim there. The publisher ignores what it hears. No exhaustive pass has more than four relays or two session changes; the prototype's sweeps in `compare.rs` cover larger meshes and churn. diff --git a/quest/m1/route-inductive.md b/quest/m1/route-inductive.md index 941d0cf0a8..7c3defcf32 100644 --- a/quest/m1/route-inductive.md +++ b/quest/m1/route-inductive.md @@ -5,8 +5,9 @@ An inductive invariant of the route model implies `NoRevival`, `NoFalseEnd`, `NoRegression`, `Truthful`, and the loop rule the model proves for the path rule: no relay uses a route whose path names it, and selected next hops loop -only within the bound -[the model](/quest/m2/cluster-routing/routes/scope-model.md) found. Apalache +only within the bound the model (`rs/moq-net/verify/route/RouteHandoff.tla`) +found: through a stale route whose news is on its way (`LoopsAreStale`, +`StaleIsChased`). Apalache checks in Nightly that it holds initially and that every step preserves it from any state satisfying it, on each mesh shape of up to five relays. The rules then hold for runs of any length and any number of changes, where @@ -55,17 +56,14 @@ stay constants the config states. Say both bounds in the `verify-route` skill. the invariant also says a route is held only from a node the holder shares a link with: every route sent adds its sender to hops that did not name it ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)). Start there. -- Apalache checks safety. That every hold ends is a liveness rule the model - checks with TLC; leave it there and say so in the skill. +- Apalache checks safety. That every hold ends is no checked rule: the + model has no fairness, and the `verify-route` skill argues it from + `Budget` and `QuietIsSettled`. Leave it there. - Run it through the `tla` recipes with Apalache from the flake. If five relays does not fit Nightly, record the largest mesh that does and its time. Public API: none. Wire: none. -## Required - -- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - the rule for whose claims a relay believes, which the claim rules rest on; an invariant found before it is found for a rule about to go - ## Related - [Route sweeps are checked against the TLA+ model](/quest/m1/route-trace-validation.md) - ties the same spec to the code diff --git a/quest/m1/route-trace-validation.md b/quest/m1/route-trace-validation.md index a7652add5f..f39fb40177 100644 --- a/quest/m1/route-trace-validation.md +++ b/quest/m1/route-trace-validation.md @@ -46,5 +46,4 @@ Public API: none. Wire: none. ## Required -- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - replaces who a relay believes and adds two prefixes and a scoped relay, so traces are validated against the rule that stays - [The router runs the path rule with a route per scope](/quest/m2/cluster-routing/routes/scope.md) - the router `Sim` records, running the rule the model has diff --git a/quest/m2/cluster-routing/routes/README.md b/quest/m2/cluster-routing/routes/README.md index 75860de58c..2012786db3 100644 --- a/quest/m2/cluster-routing/routes/README.md +++ b/quest/m2/cluster-routing/routes/README.md @@ -43,9 +43,9 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout 25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **each route names the nodes it crossed, and a node never uses a route that names itself** ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487), which holds the counterexamples and the model results). Feasibility still selects routes; the list is one per origin route, not the per-claim Hop ID list item 6 removed, and the echo and the drain rule stay as they are. As built: Babel's feasibility condition is unchanged, and a node never selects or certifies through such a route. Where the condition refuses it, it still counts as infeasible, so a relay whose neighbours all route through it holds its claims while one of them finds another way. Under a qualified key it counts only while its hops extend those of the route the node last sent under that key: a copy that a scoped peer hands back under a key derived from the node's own route never moves, so nothing waits on it. The origin names itself too, so a rotated identity's route is refused where it began, and a node names itself by a Hop ID that a rotation leaves alone. The hop limit is the wire's 32 and no lower. A change of hops or RTT alone lets go of no region, unless it changes whether the node may use the route: by crossing it, or by reaching the hop limit. This supersedes the claim, in the handoff contract and in decision 6 below, that the echo alone makes forgetting safe. First decided as a relay echoing a ROUTE_END only once its own downstream had; the model ran that the same day and it fails when a session closes before answering. Rejected: that deferred echo; never dropping the distance; a timer; relays as the only origin nodes. -26. ✅ Decided 2026-11-11 (maintainer, /quest-plan after [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews), asked whose copy of a publisher's claims a relay believes: **a route is shared per publisher and per scope, and a relay believes a route by that route's own path** ([the prototype](https://github.com/moq-dev/moq/pull/8533), [the model](/quest/m2/cluster-routing/routes/scope-model.md), then [the router](/quest/m2/cluster-routing/routes/scope.md)). A scope is the prefix a relay asked a neighbour for when its grant there is narrower than its interest. Today only a feasible neighbour counts, one strictly cheaper than the distance the relay remembers, so a relay's claims move whenever a cost or seqno does: it can forget a claim for good, a mesh with two grants never settles after one seqno advance, and behind a scoped neighbour it never shows what a farther one knows. Only a broadcast some relay carries gets its own route, and that route propagates ([cache-aware routing](/quest/m3/cache-routing/README.md) Plan item 8). What keeps a route free of loops and stale copies is item 27. The contract's bullet on which neighbours certify stands until the model quest rewrites it. First decided the same day as any neighbour whose path to the publisher does not pass through the relay; tried on the router, two relays behind a scoped relay then keep a dead claim alive through each other, and the relay behind a scoped neighbour shows a claim it has no route to request. Rejected: that rule; believing a neighbour that is cheaper than the relay is now, which fixed the storm and the forgotten claim with no wire change but rests belief on distance; a path per broadcast as lite-06 has; link-state among relays; routes to relays. +26. ✅ Decided 2026-11-11 (maintainer, /quest-plan after [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews), asked whose copy of a publisher's claims a relay believes: **a route is shared per publisher and per scope, and a relay believes a route by that route's own path** ([the prototype](https://github.com/moq-dev/moq/pull/8533), the model (`rs/moq-net/verify/route/RouteHandoff.tla`), then [the router](/quest/m2/cluster-routing/routes/scope.md)). A scope is the prefix a relay asked a neighbour for when its grant there is narrower than its interest. Today only a feasible neighbour counts, one strictly cheaper than the distance the relay remembers, so a relay's claims move whenever a cost or seqno does: it can forget a claim for good, a mesh with two grants never settles after one seqno advance, and behind a scoped neighbour it never shows what a farther one knows. Only a broadcast some relay carries gets its own route, and that route propagates ([cache-aware routing](/quest/m3/cache-routing/README.md) Plan item 8). What keeps a route free of loops and stale copies is item 27. The contract's bullet on which neighbours certify now states the rule the model settled. First decided the same day as any neighbour whose path to the publisher does not pass through the relay; tried on the router, two relays behind a scoped relay then keep a dead claim alive through each other, and the relay behind a scoped neighbour shows a claim it has no route to request. Rejected: that rule; believing a neighbour that is cheaper than the relay is now, which fixed the storm and the forgotten claim with no wire change but rests belief on distance; a path per broadcast as lite-06 has; link-state among relays; routes to relays. -27. ✅ Decided 2026-11-11 (maintainer, on [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533), which measured it), asked what keeps a route free of loops and stale copies: **the route's path, a withdrawal that names the hop that died, holds, and a timed doubt. Babel's distance, the seqno, ROUTE_REQUEST, and the ROUTE_END echo go.** None of it is built: [the model](/quest/m2/cluster-routing/routes/scope-model.md) proves it, [the router](/quest/m2/cluster-routing/routes/scope.md) builds it, [the switch](/quest/m2/cluster-routing/routes/switch.md) puts it on the wire (item 28), and the contract below describes today's router until they do. The rule, as the prototype in `rs/moq-net/src/route/tests/paths.rs` runs it: +27. ✅ Decided 2026-11-11 (maintainer, on [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533), which measured it), asked what keeps a route free of loops and stale copies: **the route's path, a withdrawal that names the hop that died, holds, and a timed doubt. Babel's distance, the seqno, ROUTE_REQUEST, and the ROUTE_END echo go.** None of it is built: the model (`rs/moq-net/verify/route/RouteHandoff.tla`) checks it, with dead hops kept until their timer fires, [the router](/quest/m2/cluster-routing/routes/scope.md) builds it, [the switch](/quest/m2/cluster-routing/routes/switch.md) puts it on the wire (item 28), and the contract below describes today's router until they do. The rule, as the prototype in `rs/moq-net/src/route/tests/paths.rs` runs it: - A hop names a relay and the session it learned the route on. A relay uses the cheapest route whose path does not name it, at once. - A relay that loses a session names that hop, and every relay drops every route through it, whoever offered it. One message per session covers every publisher and lists the routes that moved to another path. - A relay with no route left holds its claims, unavailable, until each neighbour that learned the route from it has answered: with another way, or with nothing once those behind it have answered. Then it lets go. @@ -352,41 +352,31 @@ loss never reports an END: since it may certify a wildcard above both at a later revision; and withdrawing one origin's wildcard leaves another origin's copy (`wildcard_claims_end_per_origin_and_are_certified_below`). -- A node tracks the regions each neighbour's last view certifies. Only a - neighbour whose last advertisement is feasible here counts, held or - not: feasible edges strictly decrease the metric toward the origin, so two - nodes never keep a region alive through each other. While the node has a - usable route and every such neighbour has answered the current interest, a - region of its view that none of them certifies is let go: its last - certifier closed, ended its route, or no longer meets the interest. The - reader sees those claims go offline, not end. The view keeps what it knows - within the regions still certified when no lost floor is newer than every - kept one, so a lagging certifier cannot revive a withdrawal the departed one - delivered (`a_lost_scope_keeps_newer_knowledge_where_still_certified`). - Otherwise a wildcard above both was last learned through the lost floor and - is unknown at the kept floors' revision, so the view is rebuilt from the - neighbours' last views, a collection of that region. A sender sends a - route on a stream only while the route meets the stream's interest (Plan - item 15): it starts the route before its first claim in or covering the - interest, and when the last such claim goes it sends a floor certifying the - withdrawal, then ends the route; it starts the route there again only once - that end is echoed. A peer therefore holds no route, and - waits on no certifier, for an origin its interest does not meet - (`a_disjoint_interest_is_answered_with_nothing`), and a node presents each - stream only the claims that changed. - A claim is available only through a certifier that is not held; another - scope's usable route cannot serve it. A certifier that has not answered yet - counts as serving, as for a new next hop whose announcements trail its - route. Closing one of two grants - under one key takes its scope offline at the client - (`a_closed_grant_lets_its_scope_go`), and a relay downstream of two scoped - uplinks shows a lost scope unavailable while the starved relay holds, then - lets it go once a request recovers a route - (`a_relay_lets_go_of_a_scope_its_uplinks_lost`). An interest that meets no - grant scope certifies nothing, so its claims go offline - (`sweep_scoped_handoff`). Without this, the closed grant's claims stay - shown and available indefinitely, since no other grant can deliver their - withdrawal. +- Not yet built ([the router](/quest/m2/cluster-routing/routes/scope.md) + builds it and [the switch](/quest/m2/cluster-routing/routes/switch.md) puts + it on the wire; the rest of this contract describes today's router until + then), + settled by `rs/moq-net/verify/route/RouteHandoff.tla` with Plan item 27's + rule: a route is keyed by its publisher and the prefix it covers, and + carries its own path. A node that asks a neighbour for a narrower prefix + gets one route there, covering that prefix: the neighbour's live route for + that prefix if it has one, else its route for a prefix above it. A node + believes a route whose path does not name it, and keeps one view per + prefix: the newest floor any route covering that prefix gave it, kept + whole while it holds a route covering the prefix, so a lagging route + cannot revive a withdrawal another delivered, and a route that ends takes + only what no other route covers. A claim shows from the narrowest prefix's + view and is available while a live route covers it. A request starts on + the narrowest live route covering its claim, follows the route each node + offered the one it came from, and is never forwarded along a route whose + path names the node it came from. A node remembers a dead hop while a + neighbour's route names it and until the timer the hop's close started + fires, so no route through a hop it knows is dead returns. Selected routes + can loop only while the news that ends a stale route is on its way, and + only through at least one stale route. What its timer costs fired early, and the loop `Loop.cfg` pins, are in the `verify-route` skill. + A sender sends a route on a stream only while the route meets the + stream's interest (Plan item 15), and a node presents each stream only the + claims that changed. - Per identity, a node is usable (a feasible selected route: announcements shown available), held, or let go. It is held while starved, or while a neighbor's held route has a last advertisement feasible here; it diff --git a/quest/m2/cluster-routing/routes/interest-set.md b/quest/m2/cluster-routing/routes/interest-set.md index 5c1f6a10bb..002bf00c35 100644 --- a/quest/m2/cluster-routing/routes/interest-set.md +++ b/quest/m2/cluster-routing/routes/interest-set.md @@ -22,8 +22,8 @@ and every earlier routing bug was an ordering nobody had enumerated. Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for Babel): ✅ this edits the path router, after [the route per scope](/quest/m2/cluster-routing/routes/scope.md) builds it, -and the model as -[scope-model](/quest/m2/cluster-routing/routes/scope-model.md) leaves it +and the model as it has a route per prefix +(`rs/moq-net/verify/route/RouteHandoff.tla`) (rejected: landing first on Babel's router, which [the switch](/quest/m2/cluster-routing/routes/switch.md) deletes). The path rule has no echo and no route request @@ -84,7 +84,7 @@ Work: says to a neighbour shares one FIFO per direction, the hold answer included: with one stream per side nothing overtakes anything. - The model has two prefixes and a scoped relay since - [it has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md). + it has a route per scope. Give it edits in flight: add, remove, widen, narrow, and an edit that crosses a claim change, a route change, a ROUTE_END, a withdrawal that names a dead hop, and a hold with its answer. Keep it @@ -116,7 +116,6 @@ Public API: none (the router is crate-private). Wire: none. ## Required -- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - the two prefixes and the rule for whose claims a relay believes, which an edit must preserve - [The router runs the path rule with a route per scope](/quest/m2/cluster-routing/routes/scope.md) - the router this edits, with a floor per route ## Related diff --git a/quest/m2/cluster-routing/routes/scope-model.md b/quest/m2/cluster-routing/routes/scope-model.md deleted file mode 100644 index 0348f642df..0000000000 --- a/quest/m2/cluster-routing/routes/scope-model.md +++ /dev/null @@ -1,121 +0,0 @@ -# [L] The route model has a route per scope - -## Goal - -`rs/moq-net/verify/route/RouteHandoff.tla` gains two prefixes and relays that -ask a neighbour for only one of them. A route is keyed by its publisher and -the prefix it covers, carries its own path, and is believed by a relay it did -not cross. TLC shows on it: - -- A relay shows a claim exactly while it holds a route that covers the claim - and did not cross the relay, and the newest revision any covering route gave - it still has the claim. It shows the claim as it is at that revision. A - newer revision from a route that does not cover the claim says nothing - about it. -- Once nothing is in flight, no relay shows a claim that no path from its - publisher carries. -- A change of path or seqno alone starts, ends, and changes no claim a reader - sees. Whether the claim is available may change. -- A request for a shown claim follows next hops to the publisher without a - loop, also where one route's prefix lies inside another's. So a relay - behind a scoped neighbour reaches what a farther, unscoped neighbour - offers. - -The model keeps a route free of loops and stale copies by the rule the -[epic](/quest/m2/cluster-routing/routes/README.md)'s Plan item 27 picked: the -path, a withdrawal that names the hop that died, holds, and a timed doubt. -TLC also shows that every hold ends, and what bounds a loop of selected next -hops while messages are in flight. The rule this settles is written into -the epic's contract for -[the router](/quest/m2/cluster-routing/routes/scope.md) to implement. The -router's behaviour, the wire, and the draft do not change here. - -## Plan - -Decided 2026-11-11 (maintainer, /quest-plan). The decisions and what was -tried are in the [epic](/quest/m2/cluster-routing/routes/README.md)'s Plan -items 26 and 27, [moq-dev/moq#8514](https://github.com/moq-dev/moq/pull/8514), -and [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533). - -The rule for loops and stale copies has passed sweeps and no model. The -prototype in `rs/moq-net/src/route/tests/paths.rs` runs it (`Rules::waits`), -and `compare.rs` beside it holds the meshes. What the model must prove or -replace: - -- **Every hold ends.** A relay holds under the next hop it used, answers - that hop only once the neighbours that learned the route from it have - answered, and holds on its own account when that next hop's own route - names it. Two hangs the sweeps found are fixed in the prototype, and each - deserves a config: two relays that took each other's stale route each - waited on the other, and a relay answered "nothing" while a route only - doubt kept it off could still come back. -- **The doubt.** Model the timer as a step that may fire at any moment. Show - that firing late costs only the wait, and state what firing early costs: a - relay moves to a stale route, as with no doubt at all. -- **Loops.** Selected next hops form no loop once nothing is in flight. - While something is, find what bounds a loop, or the smallest change that - does: the churn sweep still loops on 32 of 120,000 steps, when two - unrelated things fail at once. -- **Forgetting.** A relay remembers a dead hop only while a neighbour's route - still names it. Show no stale route returns after that. - -Why a route per scope: a relay that asked a neighbour for prefix `a` knows a -publisher's claims only inside `a`. Today that partial knowledge lives inside -one view per publisher, as a floor per region, with claims no floor covers -carried loose, and a rule for which neighbours vouch for which region. Every -open finding is in that machinery. With the prefix on the route, every -neighbour that offers a route knows all of it, and the route ends when its -path does. - -Recommendations the model may overturn. Say in the PR what it settled on and -why: - -- **The prefix a route covers travels with the route**, as a field of - ROUTE_START. On a stream that asked for a narrower prefix the route covers - that prefix, so narrowing composes. Rejected so far: deriving a Node ID - from the publisher and the prefix, which needs no wire change but cannot - be narrowed a second time, since a relay cannot read the prefix back. -- **The newest floor wins among one publisher's routes.** A claim both a - narrow and a wide route cover is shown once, at the newer revision, and a - relay keeps what it learned while any route covers the claim - (epic Plan item 19). Nothing is pruned by an instant: a route that ends - takes only what no other route covers. -- **A request takes the narrowest route that covers its claim, and is never - forwarded along a route whose path names the neighbour it came from.** - This is the unsolved part: prove the selected next hops cannot loop when - the narrow and the wide route lead different ways, under stale messages, - or find the rule that does. -- Keep the counterexample that broke the first plan: two relays behind a - scoped relay must not keep a wider route alive through each other after - its path closed. -- Keep the lagging neighbour: a relay learns a withdrawal from one route, - that route ends, and a second route still carries the claim at the older - revision. The relay must not show the claim again - (`an_older_complete_snapshot_revives_no_known_withdrawal`). Give it a - config, with the second route covering the same prefix and a wider one. - -Work: - -- In the model, replace the fixed root interest with two prefixes and a - `Scoped` set of relays that ask for one. Key `rx`, the selection, and what - is sent by route. Replace `Certifiers`, `Certified`, `Prune`, and the - stored merged `view` with one floor per route and the newest-floor rule. - Add the Goal's four rules: `Seen`, an extended `QuietIsSettled`, `Stable`, - and `Reaches`. -- Add configs for the dead claim, the blind relay, and the meshes in the two - reproducer comments the prototype pins, at bounds that finish in minutes on - the recipe's settings. Run TLC one config at a time, simulation first. -- Rewrite the epic's bullet that begins "A node tracks the regions each - neighbour's last view certifies" to the settled rule, marked as not yet - built. -- Update the `verify-route` skill's rules table, coverage table, and - "Not covered" list. - -[The interest set](/quest/m2/cluster-routing/routes/interest-set.md) adds -edits to the two prefixes this introduces. - -Public API: none. Wire: none. - -## Related - -- [The router keeps a route per scope](/quest/m2/cluster-routing/routes/scope.md) - implements the rule this settles diff --git a/quest/m2/cluster-routing/routes/scope.md b/quest/m2/cluster-routing/routes/scope.md index 7ba6ab25ee..4fbc8ef613 100644 --- a/quest/m2/cluster-routing/routes/scope.md +++ b/quest/m2/cluster-routing/routes/scope.md @@ -2,9 +2,9 @@ ## Goal -`route::Router` runs the rule -[the model](/quest/m2/cluster-routing/routes/scope-model.md) settled, sans-IO, -and `route::tests::Sim` drives it. A route is keyed by its publisher and the +`route::Router` runs the rule the model +(`rs/moq-net/verify/route/RouteHandoff.tla`) settled, sans-IO, and +`route::tests::Sim` drives it. A route is keyed by its publisher and the prefix it covers and carries its path. A relay uses the cheapest route whose path does not name it, drops every route through a hop a withdrawal names, holds its claims while the neighbours that learned a route from it have yet @@ -45,6 +45,22 @@ the wire, the draft, and `js/net` onto it and deletes Babel's router one quest for everything, which is one PR across the router, the wire, the draft, the Rust sessions, and `js/net`). +Open, from the model +([moq-dev/moq#8541](https://github.com/moq-dev/moq/pull/8541)): how long a +relay keeps its doubt about a node. As the rule stands the doubt goes once no +route names the node, and a request can then loop while the news of three +closed sessions is still on its way: `Loop.cfg` breaks `RequestsLoopFree` at +depth 11 and is kept as that open finding. Keeping the doubt until its timer +fires showed no loop on that mesh in 7.6M states, to depth 13, and costs a +wait: no relay moves to a route through the doubted node until the timer +fires, a publisher reconnecting through another relay included, though a +relay keeps one it already has. Measure that reconnect delay on +the prototype's meshes under both rules and put the two to the maintainer +with the numbers. Recommended: keep the doubt, since the wait is bounded by +the timer and a looping request is not bounded by anything the relay +controls. If the rule changes, +`Loop.cfg` passes and `RequestsLoopFree` joins the rules every config checks. + The two rules share a tree until the switch. Pick how, and say why in the PR: the prototype (`route/tests/paths.rs`, `compare.rs`, `copies.rs`) promoted out of `tests/` into a second router, or one router generic over @@ -93,10 +109,6 @@ as built, and the deletion of today's router. Public API: none (the router is crate-private). Wire: none. -## Required - -- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - the rule, proven on two prefixes and a scoped relay - ## Related - [lite-07 sessions run the path router](/quest/m2/cluster-routing/routes/switch.md) - puts this router on the wire and deletes the one it replaces diff --git a/rs/moq-net/verify/route/Behind.cfg b/rs/moq-net/verify/route/Behind.cfg deleted file mode 100644 index 07e962b6ca..0000000000 --- a/rs/moq-net/verify/route/Behind.cfg +++ /dev/null @@ -1,27 +0,0 @@ -\* The origin (0) behind relay 1, which sits in a triangle with relays 2 and 3. -\* Relay 1 can drop its distance while 2 or 3 still holds a copy of its route, -\* which only the hops refuse once it comes back. -CONSTANTS - Nodes = {0, 1, 2, 3} - Origin = 0 - Leaves = {} - Links = {{0, 1}, {1, 2}, {1, 3}, {2, 3}} - Dear = {} - Claims = {"x"} - MaxRev = 1 - MaxFlaps = 1 - MaxCollects = 0 - -SPECIFICATION Spec - -INVARIANTS - Truthful - AllCertified - NoRevival - NoFalseEnd - NoRegression - LoopFree - SimplePaths - NoCountToInfinity - QuietIsSettled - Budget diff --git a/rs/moq-net/verify/route/BehindDear.cfg b/rs/moq-net/verify/route/BehindDear.cfg deleted file mode 100644 index 52e2ac8cbd..0000000000 --- a/rs/moq-net/verify/route/BehindDear.cfg +++ /dev/null @@ -1,26 +0,0 @@ -\* Behind.cfg with unequal costs: relay 3 is nearer the origin than relay 2, so -\* 3's copy of relay 1's route is feasible at 2 after 1 ended it. -CONSTANTS - Nodes = {0, 1, 2, 3} - Origin = 0 - Leaves = {} - Links = {{0, 1}, {1, 2}, {1, 3}, {2, 3}} - Dear = {{1, 2}, {2, 3}} - Claims = {"x"} - MaxRev = 1 - MaxFlaps = 1 - MaxCollects = 0 - -SPECIFICATION Spec - -INVARIANTS - Truthful - AllCertified - NoRevival - NoFalseEnd - NoRegression - LoopFree - SimplePaths - NoCountToInfinity - QuietIsSettled - Budget diff --git a/rs/moq-net/verify/route/Blind.cfg b/rs/moq-net/verify/route/Blind.cfg new file mode 100644 index 0000000000..750ac81f68 --- /dev/null +++ b/rs/moq-net/verify/route/Blind.cfg @@ -0,0 +1,33 @@ +\* Too large to check every ordering, so TLC samples it at random. +\* Simulate: -simulate num=10000 -depth 120 +\* The blind relay (`residual_a_relay_behind_a_scoped_neighbour_is_blind`): +\* relay 4's nearest neighbour, relay 1, asks for "a" alone, and relay 2, +\* farther from the publisher (0) through relay 3, knows everything. Relay 4, +\* and relay 5 behind it, must show "bx" and reach it through 2. +CONSTANTS + Nodes = {0, 1, 2, 3, 4, 5} + Origin = 0 + Links = {{0, 1}, {1, 4}, {0, 3}, {3, 2}, {2, 4}, {4, 5}} + Dear = {} + Scoped = {1} + Claims = {"ax", "bx"} + Narrow = {"ax"} + MaxRev = 2 + MaxFlaps = 1 + Patient = TRUE + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + NoStaleReturn + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/Chord.cfg b/rs/moq-net/verify/route/Chord.cfg deleted file mode 100644 index e2059c4fc1..0000000000 --- a/rs/moq-net/verify/route/Chord.cfg +++ /dev/null @@ -1,29 +0,0 @@ -\* Too large to check every ordering, so TLC samples it at random. -\* Simulate: -simulate num=1000 -depth 600 -\* A client (0) publishes through relay 1 of a ring of five relays (1 to 5) -\* with a chord from 2 to 4, and two clients (6 on 3, 7 on 4) watch. Enough -\* flaps cut the ring into a line. -CONSTANTS - Nodes = {0, 1, 2, 3, 4, 5, 6, 7} - Origin = 0 - Leaves = {6, 7} - Links = {{0, 1}, {1, 2}, {2, 3}, {3, 4}, {4, 5}, {5, 1}, {2, 4}, {3, 6}, {4, 7}} - Dear = {} - Claims = {"x", "y"} - MaxRev = 4 - MaxFlaps = 4 - MaxCollects = 2 - -SPECIFICATION Spec - -INVARIANTS - Truthful - AllCertified - NoRevival - NoFalseEnd - NoRegression - LoopFree - SimplePaths - NoCountToInfinity - QuietIsSettled - Budget diff --git a/rs/moq-net/verify/route/Dead.cfg b/rs/moq-net/verify/route/Dead.cfg new file mode 100644 index 0000000000..a6c0d9ac55 --- /dev/null +++ b/rs/moq-net/verify/route/Dead.cfg @@ -0,0 +1,34 @@ +\* Too large to check every ordering, so TLC samples it at random. +\* Simulate: -simulate num=10000 -depth 120 +\* The dead claim (`two_relays_behind_a_scoped_relay_let_a_dead_claim_go`): +\* relays 3 and 4 sit behind relay 2, whose only way to the publisher (0) is +\* relay 1, which asks for "a" alone. Relay 4 also has a dear session of its +\* own to the publisher. It closes, and the publisher withdraws "bx": neither +\* 3 nor 4 may keep the wider route alive through the other. +CONSTANTS + Nodes = {0, 1, 2, 3, 4} + Origin = 0 + Links = {{0, 1}, {1, 2}, {2, 3}, {2, 4}, {3, 4}, {0, 4}} + Dear = {{0, 4}} + Scoped = {1} + Claims = {"ax", "bx"} + Narrow = {"ax"} + MaxRev = 3 + MaxFlaps = 1 + Patient = TRUE + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + NoStaleReturn + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/Early.cfg b/rs/moq-net/verify/route/Early.cfg new file mode 100644 index 0000000000..c5f1c3f9f4 --- /dev/null +++ b/rs/moq-net/verify/route/Early.cfg @@ -0,0 +1,29 @@ +\* RouteHandoff.cfg with a timer that may fire at any moment, even before the +\* news of the dead hop has reached the relay: as with no doubt at all, a stale +\* route may return (NoStaleReturn breaks), and nothing else does. +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Links = {{0, 1}, {1, 2}, {1, 3}, {2, 3}} + Dear = {} + Scoped = {} + Claims = {"x"} + Narrow = {} + MaxRev = 1 + MaxFlaps = 1 + Patient = FALSE + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/Lag.cfg b/rs/moq-net/verify/route/Lag.cfg new file mode 100644 index 0000000000..f11ccb7939 --- /dev/null +++ b/rs/moq-net/verify/route/Lag.cfg @@ -0,0 +1,31 @@ +\* A lagging neighbour: relay 3 hears the publisher (0) through relays 1 and +\* 2. It may learn a withdrawal through one, lose that route, and still hold +\* the other at the older revision: it must not show the claim again. Both +\* routes cover the same prefix. +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Links = {{0, 1}, {0, 2}, {1, 3}, {2, 3}} + Dear = {} + Scoped = {} + Claims = {"ax"} + Narrow = {"ax"} + MaxRev = 2 + MaxFlaps = 1 + Patient = TRUE + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + NoStaleReturn + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/LagWide.cfg b/rs/moq-net/verify/route/LagWide.cfg new file mode 100644 index 0000000000..38cbc26358 --- /dev/null +++ b/rs/moq-net/verify/route/LagWide.cfg @@ -0,0 +1,29 @@ +\* Lag.cfg where relay 1 asks for "a" alone, so relay 3 holds a route for "a" +\* through 1 and a wider root route through 2, both covering the claim. +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Links = {{0, 1}, {0, 2}, {1, 3}, {2, 3}} + Dear = {} + Scoped = {1} + Claims = {"ax"} + Narrow = {"ax"} + MaxRev = 2 + MaxFlaps = 1 + Patient = TRUE + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + NoStaleReturn + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/Line.cfg b/rs/moq-net/verify/route/Line.cfg deleted file mode 100644 index 1c9e1999f0..0000000000 --- a/rs/moq-net/verify/route/Line.cfg +++ /dev/null @@ -1,31 +0,0 @@ -\* A line: the origin (0) publishes through relays 1, 2, and 3 to a client -\* (4) that only watches. Every route has one path, so a cut leaves the far -\* side with nothing, and the client must echo every ROUTE_END it hears. -\* Converges checks under fairness what Budget implies: routing goes quiet. -CONSTANTS - Nodes = {0, 1, 2, 3, 4} - Origin = 0 - Leaves = {4} - Links = {{0, 1}, {1, 2}, {2, 3}, {3, 4}} - Dear = {} - Claims = {"x"} - MaxRev = 2 - MaxFlaps = 1 - MaxCollects = 1 - -SPECIFICATION LiveSpec - -INVARIANTS - Truthful - AllCertified - NoRevival - NoFalseEnd - NoRegression - LoopFree - SimplePaths - NoCountToInfinity - QuietIsSettled - Budget - -PROPERTIES - Converges diff --git a/rs/moq-net/verify/route/Loop.cfg b/rs/moq-net/verify/route/Loop.cfg new file mode 100644 index 0000000000..059a32e769 --- /dev/null +++ b/rs/moq-net/verify/route/Loop.cfg @@ -0,0 +1,34 @@ +\* Open finding: RequestsLoopFree. The publisher (0) has a session to each of +\* relays 1, 2, and 3, which form a triangle, and all three close in turn. Each +\* relay drops its doubt on the publisher once no route names it, so it takes a +\* neighbour's route whose news has not reached that neighbour yet, and a +\* request goes round 1, 3, 2 until it does. LoopsAreStale and StaleIsChased +\* still hold. A relay that kept its doubt until the timer fires never takes +\* such a route: promote the rule here when the router does. +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Links = {{0, 1}, {0, 2}, {0, 3}, {1, 2}, {2, 3}, {1, 3}} + Dear = {{1, 2}} + Scoped = {} + Claims = {"x"} + Narrow = {} + MaxRev = 0 + MaxFlaps = 3 + Patient = TRUE + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + RequestsLoopFree + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/Ring.cfg b/rs/moq-net/verify/route/Ring.cfg index 3b99e392b1..6eb3e4b0ba 100644 --- a/rs/moq-net/verify/route/Ring.cfg +++ b/rs/moq-net/verify/route/Ring.cfg @@ -1,26 +1,30 @@ -\* A ring: the origin (0) and relays 1, 2, and 3, so relay 2 has two equal -\* paths and a cut anywhere leaves a longer way around. +\* The publisher (0) and relays 1, 2, and 3 in a ring, so a cut leaves a way +\* round through the far end of the closed session, which the doubt keeps a +\* relay off until its timer fires. Meanwhile the relay holds, and must not +\* let go while that route may still come back. CONSTANTS Nodes = {0, 1, 2, 3} Origin = 0 - Leaves = {} Links = {{0, 1}, {1, 2}, {2, 3}, {3, 0}} Dear = {} + Scoped = {} Claims = {"x"} - MaxRev = 2 + Narrow = {} + MaxRev = 1 MaxFlaps = 1 - MaxCollects = 0 + Patient = FALSE SPECIFICATION Spec INVARIANTS Truthful - AllCertified NoRevival NoFalseEnd NoRegression - LoopFree + Stable + Seen SimplePaths - NoCountToInfinity + LoopsAreStale + StaleIsChased QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/RouteHandoff.cfg b/rs/moq-net/verify/route/RouteHandoff.cfg index 854ef63aad..9f59549e02 100644 --- a/rs/moq-net/verify/route/RouteHandoff.cfg +++ b/rs/moq-net/verify/route/RouteHandoff.cfg @@ -1,26 +1,31 @@ -\* An origin (0) and two relays (1, 2) in a triangle, so a relay can fail -\* over through the other and Babel's feasibility decides whether it may. +\* The publisher (0) behind relay 1, which sits in a triangle with relays 2 +\* and 3. When 0-1 closes, 2 and 3 each still hold a copy of the other's route +\* through it, and an offer through the dead hop may arrive after one of them +\* dropped its own: the timer, fired late, keeps it off. CONSTANTS - Nodes = {0, 1, 2} + Nodes = {0, 1, 2, 3} Origin = 0 - Leaves = {} - Links = {{0, 1}, {0, 2}, {1, 2}} + Links = {{0, 1}, {1, 2}, {1, 3}, {2, 3}} Dear = {} + Scoped = {} Claims = {"x"} - MaxRev = 2 - MaxFlaps = 2 - MaxCollects = 1 + Narrow = {} + MaxRev = 1 + MaxFlaps = 1 + Patient = TRUE SPECIFICATION Spec INVARIANTS Truthful - AllCertified NoRevival NoFalseEnd NoRegression - LoopFree + Stable + Seen SimplePaths - NoCountToInfinity + LoopsAreStale + StaleIsChased + NoStaleReturn QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla index 726f0d2c7b..9ec5db5494 100644 --- a/rs/moq-net/verify/route/RouteHandoff.tla +++ b/rs/moq-net/verify/route/RouteHandoff.tla @@ -1,34 +1,45 @@ ---------------------------- MODULE RouteHandoff ---------------------------- (***************************************************************************) -(* The lite-07 route layer for one origin identity, as `route::Router` *) -(* (rs/moq-net/src/route/mod.rs) runs it and `route::tests::Sim` drives it. *) +(* The lite-07 route rule the routes epic's Plan item 27 picked, for one *) +(* publisher and two prefixes, as the prototype in *) +(* rs/moq-net/src/route/tests/paths.rs (`Relay` under `Rules::waits`) runs *) +(* it, extended with a route per scope. *) (* *) -(* Each node is one Router. Every router entry point (`route`, `batch`, *) -(* `request`, `echo`, `collect`, `connect`/`peer_interest`, `disconnect`, *) -(* `announce`/`unannounce`) is one atomic step that ends in `update`, as *) -(* in the code. Messages travel as the Sim queues them: ROUTE, batches, *) -(* and ROUTE_REQUEST on one FIFO per direction, echoed ROUTE_ENDs on their *) -(* own queue. A session opens and closes atomically, dropping what was in *) -(* flight. *) +(* A route is keyed by the prefix it covers: "root", or "a" inside it. A *) +(* relay in `Scoped` asks every neighbour for "a" alone, so a stream to it *) +(* carries one route, keyed "a". A route names every relay it crossed, *) +(* each with the session that relay learned it on, and a relay uses the *) +(* cheapest live route whose path does not name it. There is no distance, *) +(* seqno, ROUTE_REQUEST, or echo. Instead: *) (* *) -(* Scope: every node watches the root, links have equal priority, no *) -(* scoped grants, no GOAWAY, no hop limit, and the origin keeps its *) -(* identity. Claims are exact paths, and a claim's attributes are the *) -(* revision that last set them, standing in for a fresh Instance per *) -(* announce. *) +(* - A relay that loses a session names that hop in one DEAD message per *) +(* neighbour, listing which of its routes through it moved to another *) +(* path, and every relay drops every route through the hop. *) +(* - A relay with no route left holds its claims, unavailable, until each *) +(* neighbour that learned the route from it has answered. *) +(* - A closed session casts doubt on the node at its far end: no relay *) +(* moves to a route through it until a timer lifts the doubt. The timer *) +(* is the step `Lift`, which may fire at any moment. *) +(* *) +(* Each node's every entry point is one atomic step ending in `Update`, *) +(* as in the prototype. Messages travel on one FIFO per direction. A *) +(* session opens and closes atomically, dropping what was in flight, and *) +(* each opening is a new session with a new hop. A claim's attributes are *) +(* the revision that last set them. *) (***************************************************************************) EXTENDS Integers, Sequences, FiniteSets CONSTANTS - Nodes, \* router keys; the Sim's `Node` index, ordered as in `selected` - Origin, \* the node that declares the identity - Leaves, \* nodes that send only identities they serve (`Sim::leaf`) + Nodes, \* relays and the publisher; ordered as `select` breaks ties + Origin, \* the publisher Links, \* sessions, as 2-element sets of nodes - Dear, \* the links that add 2 to a metric; the rest add 1 (`Link::cost`) + Dear, \* the links that cost 2; the rest cost 1 + Scoped, \* relays that ask every neighbour for prefix "a" alone Claims, \* claim paths - MaxRev, \* the origin's claim changes + Narrow, \* the claims inside prefix "a" + MaxRev, \* the publisher's claim changes MaxFlaps, \* session opens and closes - MaxCollects \* `collect` calls + Patient \* TRUE: a doubt lifts only once no DEAD about its node is in flight Relays == Nodes \ {Origin} @@ -36,243 +47,335 @@ Cost == [l \in Links |-> IF l \in Dear THEN 2 ELSE 1] \* A claim's attributes: the revision that last set it, or None when absent. None == -1 -\* A view's floor: a revision, None for no floor, or Absent for no view at all. +\* A view's floor: a revision, None for no floor yet, or Absent for no view. Absent == -2 +Keys == {"root", "a"} +\* The keys in the order a node sends them in. +KeySeq == <<"root", "a">> + +\* Whether a route keyed `k` covers claim `c`. +Covers(k, c) == k = "root" \/ c \in Narrow +\* The key whose view a node shows `c` from: the narrowest that covers it. +KeyOf(c) == IF c \in Narrow THEN "a" ELSE "root" +\* What `n` asks its neighbours for. +Ask(n) == IF n \in Scoped THEN "a" ELSE "root" +\* The keys a stream to `p` carries. +Streams(p) == IF p \in Scoped THEN {"a"} ELSE Keys + NoItems == [c \in Claims |-> None] +Restrict(items, k) == [c \in Claims |-> IF Covers(k, c) THEN items[c] ELSE None] EmptyView == [floor |-> None, items |-> NoItems] NoView == [floor |-> Absent, items |-> NoItems] -\* `Reach` as sent on a stream: nothing, `Finite(Advert)`, or `Held`. `hops` is -\* the nodes the route crossed, the origin first and the sender last. -NoReach == [kind |-> "none", seq |-> 0, met |-> 0, hops |-> <<>>] -HeldReach == [kind |-> "held", seq |-> 0, met |-> 0, hops |-> <<>>] -Finite(seq, met, hops) == [kind |-> "fin", seq |-> seq, met |-> met, hops |-> hops] - -\* `rx` entry, `fd`, `heard`, and `pending` when absent. -NoEntry == [on |-> FALSE, seq |-> 0, met |-> 0, held |-> FALSE, hops |-> <<>>] -NoFd == [on |-> FALSE, seq |-> 0, met |-> 0] -NoHeard == [on |-> FALSE, raw |-> EmptyView, kept |-> NoView, ended |-> FALSE] -NoPending == [on |-> FALSE, seq |-> 0, from |-> {}, asked |-> None] +\* A hop is <>: the node, and the session to `peer` of +\* that generation it learned the route on. The publisher names itself. +OHop == <> + +\* A route a neighbour offered (`Heard`). `state` is "live", "dead" (its path +\* holds a hop this node knows is dead, and the neighbour has not said what +\* became of it), or "held". +NoHeard == [on |-> FALSE, path |-> <<>>, state |-> "live", rev |-> None, items |-> NoItems] +\* The route a node uses for a key (`Route`). Held: no path, and `next` is the +\* neighbour it holds under, or None on its own account. +NoRoute == [on |-> FALSE, held |-> FALSE, next |-> None, path |-> <<>>] +\* What a node last sent a neighbour for a key (`Sent`). `src` is the key of +\* the node's own route the offer reflects, kept after an end. +NoSent(k) == [on |-> FALSE, held |-> FALSE, path |-> <<>>, rev |-> None, items |-> NoItems, src |-> k] NotShown == [item |-> None, avail |-> FALSE] \* Messages: one record shape so TLC can compare them. -RouteMsg(reach) == [type |-> "route", reach |-> reach, view |-> NoView, seq |-> 0] -BatchMsg(view) == [type |-> "batch", reach |-> NoReach, view |-> view, seq |-> 0] -RequestMsg(seq) == [type |-> "request", reach |-> NoReach, view |-> NoView, seq |-> seq] +Blank == [t |-> "", k |-> "root", path |-> <<>>, rev |-> None, items |-> NoItems, + hop |-> OHop, moved |-> {}, gone |-> FALSE] +OfferMsg(k, path) == [Blank EXCEPT !.t = "offer", !.k = k, !.path = path] +HoldMsg(k) == [Blank EXCEPT !.t = "hold", !.k = k] +EndMsg(k) == [Blank EXCEPT !.t = "end", !.k = k] +ClaimsMsg(k, v) == [Blank EXCEPT !.t = "claims", !.k = k, !.rev = v.floor, !.items = v.items] +DeadMsg(hop, moved, gone) == + [Blank EXCEPT !.t = "dead", !.hop = hop, !.moved = moved, !.gone = gone] VARIABLES - up, \* open sessions (`Sim::up`) - q, \* q[<>]: ROUTE, batches, and ROUTE_REQUEST from m to n, in order - acks, \* acks[<>]: echoed ROUTE_ENDs from m to n - rev, \* the origin's claim revision (`Origin::view.floors[""]`) - own, \* the origin's claims (`Origin::view.items`) - seqno, \* the origin's route seqno (`Origin::seqno`) - hist, \* hist[r + 1]: the origin's claims at revision r - st, \* st[n]: node n's Router state for the identity - ended, \* ended[n][c]: the item n's reader last saw END for, until n forgets c - revivals, \* reader STARTs of an item the reader saw END for (`revivals`) - falseEnds, \* reader ENDs while the origin still has that item (`false_ends`) - regressions,\* reader UPDATEs to an older item + up, \* open sessions + gen, \* gen[l]: the generation of the session on link l + q, \* q[<>]: messages from m to n, in order + rev, \* the publisher's claim revision + own, \* the publisher's claims + hist, \* hist[r + 1]: the publisher's claims at revision r + st, \* st[n]: node n's state for the publisher + ended, \* ended[n][c]: the item n's reader last saw END for, until n forgets c + revivals, \* reader STARTs of an item the reader saw END for + falseEnds, \* reader ENDs while the publisher still has that item + regressions, \* reader UPDATEs to an older item + unstable, \* claim changes a path change or a lifted doubt alone made + learnt, \* learnt[n]: every hop n was ever told is dead flaps, - collects, - burst, \* ROUTE, batch, request, and echo messages sent since the last quiet point - churn \* external changes since the last quiet point - -vars == <> + burst, \* messages sent since the last quiet point + churn \* external changes since the last quiet point -\* The origin's view and seqno. -OV == [rev |-> rev, items |-> own, seq |-> seqno] +vars == <> Dirs == {<> \in Nodes \X Nodes : {m, n} \in Links} Peers(n, ups) == {m \in Nodes : {n, m} \in ups} -\* `Link::proxy`: a leaf sends no identity it learned. -Serves(n) == n = Origin \/ n \notin Leaves +----------------------------------------------------------------------------- +(* Paths. *) + +Names(path, n) == \E i \in DOMAIN path : path[i][1] = n +HasHop(path, h) == \E i \in DOMAIN path : path[i] = h +DeadIn(path, D) == \E i \in DOMAIN path : path[i] \in D +\* The relay that offered `path` learned it from `n`. +Child(path, n) == Len(path) >= 2 /\ path[Len(path) - 1][1] = n +\* What the route costs the last node on its path. +PathCost(path) == + LET F[i \in 0..Len(path)] == + IF i <= 1 THEN 0 ELSE F[i - 1] + Cost[{path[i - 1][1], path[i][1]}] + IN F[Len(path)] + +Live(r) == r.on /\ ~r.held ----------------------------------------------------------------------------- -(* Route selection on node `n`'s state `s` (mod.rs: `feasible`, `selected`, *) -(* `infeasible`, `recoverable`, `live`, `certifiers`). *) - -\* `Entry::crossed`: the route's hops name this node. -Crossed(n, e) == \E i \in DOMAIN e.hops : e.hops[i] = n - -Feasible(s, e) == - ~s.fd.on \/ e.seq > s.fd.seq \/ (e.seq = s.fd.seq /\ e.met < s.fd.met) - -\* `eligible`: a node selects and certifies only through a feasible route that -\* did not cross it. -Eligible(n, s, e) == ~Crossed(n, e) /\ Feasible(s, e) - -UsableSet(n, s) == {m \in Nodes : s.rx[m].on /\ ~s.rx[m].held /\ Eligible(n, s, s.rx[m])} -\* `infeasible`: a route that crossed this node counts like any other. No key -\* is qualified here, so its sender routes through this node's own route. -InfeasibleSet(s) == {m \in Nodes : s.rx[m].on /\ ~s.rx[m].held /\ ~Feasible(s, s.rx[m])} -Certifiers(n, s) == {m \in Nodes : s.rx[m].on /\ Eligible(n, s, s.rx[m])} -Recoverable(n, s) == \E m \in Certifiers(n, s) : s.rx[m].held - -\* This node's metric through the route `m` sent. -Through(n, s, m) == s.rx[m].met + Cost[{n, m}] - -\* The lowest metric, then the lowest key: `selected`'s rank with equal priority. -Best(n, s, S) == - CHOOSE m \in S : \A k \in S : - Through(n, s, m) < Through(n, s, k) \/ (Through(n, s, m) = Through(n, s, k) /\ m <= k) - -NoSel == [on |-> FALSE, next |-> None, seq |-> 0, met |-> 0, hops |-> <<>>] -Selected(n, s, ov) == - IF n = Origin THEN [on |-> TRUE, next |-> None, seq |-> ov.seq, met |-> 0, hops |-> <>] - ELSE IF UsableSet(n, s) = {} THEN NoSel - ELSE LET m == Best(n, s, UsableSet(n, s)) - IN [on |-> TRUE, next |-> m, seq |-> s.rx[m].seq, met |-> Through(n, s, m), - hops |-> Append(s.rx[m].hops, n)] - -Live(n, s) == - n = Origin \/ UsableSet(n, s) # {} \/ InfeasibleSet(s) # {} \/ Recoverable(n, s) - -\* `Heard::view`: what it certified before its route ended, else the raw view. -HeardView(h) == IF h.kept.floor # Absent THEN h.kept ELSE h.raw - -\* `certified(id, held)` at the root interest: `ok` is FALSE where the code -\* returns None (no route, or a certifier has not answered), and `root` says -\* whether any counted certifier's view has a floor over the root. -Certified(n, s, held) == - [ok |-> (\E m \in Nodes : s.rx[m].on) /\ \A m \in Certifiers(n, s) : s.heard[m].on, - root |-> \E m \in Certifiers(n, s) : - (held \/ ~s.rx[m].held) /\ HeardView(s.heard[m]).floor >= 0] - -\* The view the node shows and forwards (`adopted`). Only certified claims -\* exist at the root interest (invariant AllCertified), so `loose` adds nothing. -Adopted(n, s, ov) == - IF n = Origin THEN [floor |-> ov.rev, items |-> ov.items] - ELSE IF ~Live(n, s) \/ s.view.floor = Absent THEN EmptyView - ELSE s.view - -\* `View::floor` of what `adopted_floors` gives: the merged view even while not live. -FloorOf(n, s, ov) == IF n = Origin THEN ov.rev ELSE IF s.view.floor = Absent THEN None ELSE s.view.floor - -HasItems(v) == \E c \in Claims : v.items[c] # None +(* Selection on node n's state s (paths.rs: `select`, `pending`, `hopes`). *) + +Doubted(s, path) == \E i \in DOMAIN path : path[i][1] \in s.doubt + +\* A route a neighbour offered that this node may use: live, and its path does +\* not name this node. +Usable(n, s, m, k) == + s.rx[m][k].on /\ s.rx[m][k].state = "live" /\ ~Names(s.rx[m][k].path, n) + +\* The key of this node's own route an offer of key k to p reflects. A stream +\* to a scoped relay carries one route, "a": the node's route for "a" where it +\* has a live one, else its root route, which covers "a". +SrcKey(s, p, k) == + IF p \notin Scoped THEN k + ELSE IF Live(s.rt["a"]) THEN "a" + ELSE IF Live(s.rt["root"]) THEN "root" + ELSE IF s.rt["a"].on THEN "a" ELSE "root" + +\* A neighbour that learned this node's route for k has yet to answer its loss. +Pending(n, s, k) == + \E p \in Nodes : \E j \in Keys : + /\ s.rx[p][j].on + /\ s.rx[p][j].state # "held" + /\ Child(s.rx[p][j].path, n) + /\ s.tx[p][j].src = k + +\* A neighbour offers a route that only doubt keeps this node off. +Hopes(n, s, k) == \E m \in Nodes : Usable(n, s, m, k) /\ Doubted(s, s.rx[m][k].path) + +\* `select` for one key: the cheapest usable route, staying on the next hop +\* among equals, then the lowest key. Doubt keeps a node from moving to a route, +\* not on the one it uses. With none left, hold or let go. +SelectKey(n, s, k, gens) == + LET was == s.rt[k] + current == IF was.on THEN was.next ELSE None + using == IF Live(was) THEN was.next ELSE None + C == {m \in Nodes : Usable(n, s, m, k) /\ (m = using \/ ~Doubted(s, s.rx[m][k].path))} + Via(m) == PathCost(s.rx[m][k].path) + Cost[{n, m}] + best == CHOOSE m \in C : \A x \in C : + \/ Via(m) < Via(x) + \/ Via(m) = Via(x) /\ (m = current \/ (x # current /\ m <= x)) + under == /\ current # None + /\ s.rx[current][k].on + /\ s.rx[current][k].state # "live" + /\ ~Names(s.rx[current][k].path, n) + IN IF C # {} + THEN [on |-> TRUE, held |-> FALSE, next |-> best, + path |-> Append(s.rx[best][k].path, <>)] + ELSE IF ~was.on THEN NoRoute + ELSE IF ~under /\ ~Hopes(n, s, k) /\ ~Pending(n, s, k) THEN NoRoute + ELSE [on |-> TRUE, held |-> TRUE, next |-> IF under THEN current ELSE None, path |-> <<>>] + +\* The publisher's own route covers everything. +OwnRoutes == [k \in Keys |-> IF k = "root" + THEN [on |-> TRUE, held |-> FALSE, next |-> None, path |-> <>] + ELSE NoRoute] + +Select(n, s, gens) == + IF n = Origin THEN s + ELSE [s EXCEPT !.rt = [k \in Keys |-> SelectKey(n, s, k, gens)]] ----------------------------------------------------------------------------- -(* `update`: prune, record, advertise, drop the feasibility distance, *) -(* requests, show. Returns the node's new state, its messages per *) -(* neighbor, and its reader's events. *) - -\* `prune` at the root interest: once no certifier's view has a floor, the -\* merged view is let go and rebuilt empty. -Prune(n, s, ov) == - LET c == Certified(n, s, TRUE) IN - IF n # Origin /\ Selected(n, s, ov).on /\ c.ok /\ s.view.floor >= 0 /\ ~c.root - THEN [s EXCEPT !.view = EmptyView] - ELSE s - -\* What `advertise` decides for one peer. -Plan(n, s, m, sel, live, av, ov) == - LET was == s.sent[m] - reach0 == IF sel.on /\ Serves(n) THEN Finite(sel.seq, sel.met, sel.hops) - ELSE IF ~sel.on /\ Serves(n) /\ live /\ was.kind # "none" THEN HeldReach - ELSE NoReach - \* A route starts again only once its last end was echoed. - reach == IF was.kind # "none" \/ s.unacked[m] = 0 THEN reach0 ELSE NoReach - meets == HasItems(av) - send == IF reach.kind # "none" /\ meets THEN reach ELSE NoReach - routeOut == IF send.kind # "none" /\ was # send THEN <> ELSE <<>> - batchGo == send.kind # "none" - \/ (reach.kind # "none" /\ was.kind # "none" /\ s.sentV[m].floor # Absent) - prev == IF s.sentV[m].floor = Absent THEN EmptyView ELSE s.sentV[m] - now == [floor |-> FloorOf(n, s, ov), items |-> av.items] - batchOut == IF batchGo /\ now # prev THEN <> ELSE <<>> - endIt == send.kind = "none" /\ was.kind # "none" - IN [sent |-> send, - sentV |-> IF send.kind = "none" THEN NoView ELSE IF batchGo THEN now ELSE s.sentV[m], - unacked |-> s.unacked[m] + (IF endIt THEN 1 ELSE 0), - out |-> routeOut \o batchOut \o (IF endIt THEN <> ELSE <<>>)] - -\* `record`: the feasibility distance for a route this node sends anywhere. -Record(n, s, sel, av, ups) == - LET sends == \E m \in Peers(n, ups) : - Serves(n) /\ (s.sent[m].kind # "none" \/ s.unacked[m] = 0) /\ HasItems(av) - IN IF ~sel.on \/ ~sends THEN s - ELSE IF ~s.fd.on THEN [s EXCEPT !.fd = [on |-> TRUE, seq |-> sel.seq, met |-> sel.met]] - ELSE IF sel.seq > s.fd.seq \/ (sel.seq = s.fd.seq /\ sel.met < s.fd.met) - THEN [s EXCEPT !.fd = [on |-> TRUE, seq |-> sel.seq, met |-> sel.met]] - ELSE s - -\* `requests`: start, resend, or clear the seqno request. -Requests(n, s, sel) == - IF n = Origin THEN [pending |-> NoPending, to |-> None, seq |-> 0] - ELSE - LET starved == ~sel.on /\ InfeasibleSet(s) # {} - want == s.fd.seq + 1 - p0 == s.pending - p1 == IF ~starved THEN p0 - ELSE IF ~p0.on THEN [on |-> TRUE, seq |-> want, from |-> {}, asked |-> None] - ELSE IF p0.seq < want THEN [p0 EXCEPT !.seq = want, !.asked = None] - ELSE p0 - answered == sel.on /\ sel.seq >= p1.seq - hasTarget == sel.on \/ InfeasibleSet(s) # {} - target == IF sel.on THEN sel.next ELSE Best(n, s, InfeasibleSet(s)) - IN IF ~p1.on THEN [pending |-> p1, to |-> None, seq |-> 0] - ELSE IF answered \/ (p1.from = {} /\ ~starved) \/ ~hasTarget - THEN [pending |-> NoPending, to |-> None, seq |-> 0] - ELSE IF p1.asked # target - THEN [pending |-> [p1 EXCEPT !.asked = target], to |-> target, seq |-> p1.seq] - ELSE [pending |-> p1, to |-> None, seq |-> 0] - -\* `shown_now`: how the reader sees each claim. -ShownNow(n, s, sel, live, ov) == - LET serving == Certified(n, s, FALSE) - avail == sel.on /\ (n = Origin \/ ~serving.ok \/ serving.root) - v == Adopted(n, s, ov) - IN [c \in Claims |-> - IF v.items[c] = None THEN NotShown ELSE [item |-> v.items[c], avail |-> avail]] - -\* `show`: the reader's events, as changes to `ended` and the counters. -\* A claim leaving while live and certified is an END, else it went offline. -Events(n, s, old, now, live, ov, eb) == - LET c0 == Certified(n, s, TRUE) - uncertified == (c0.ok /\ ~c0.root) \/ (n # Origin /\ s.view.floor < 0) - isEnd(c) == old[c].item # None /\ now[c].item = None /\ live /\ ~uncertified +(* What a node knows of the claims: one view per key, the newest floor any *) +(* route covering that key gave it, kept while it holds a covering route. *) + +Presence(s, k) == s.rt[k].on \/ (k = "a" /\ s.rt["root"].on) + +\* A route this node believes: live, and it did not cross this node. +Believed(n, s, m, j) == + s.rx[m][j].on /\ s.rx[m][j].state = "live" /\ ~Names(s.rx[m][j].path, n) + +NewView(n, s, k) == + IF ~Presence(s, k) THEN NoView + ELSE LET base == IF s.view[k].floor = Absent THEN EmptyView ELSE s.view[k] + src == {x \in Nodes \X Keys : + /\ x[2] = k \/ x[2] = "root" + /\ Believed(n, s, x[1], x[2]) + /\ s.rx[x[1]][x[2]].rev > base.floor} + best == CHOOSE x \in src : \A y \in src : + s.rx[x[1]][x[2]].rev >= s.rx[y[1]][y[2]].rev + IN IF src = {} THEN base + ELSE [floor |-> s.rx[best[1]][best[2]].rev, + items |-> Restrict(s.rx[best[1]][best[2]].items, k)] + +Know(n, s, ov) == + IF n = Origin THEN [s EXCEPT !.view = [k \in Keys |-> [floor |-> ov.rev, items |-> Restrict(ov.items, k)]]] + ELSE [s EXCEPT !.view = [k \in Keys |-> NewView(n, s, k)]] + +\* How the reader sees each claim: from the narrowest key's view, available +\* while a live route covers it. +ShownNow(s) == + [c \in Claims |-> + LET v == s.view[KeyOf(c)] IN + IF v.floor = Absent \/ v.items[c] = None THEN NotShown + ELSE [item |-> v.items[c], + avail |-> Live(s.rt["root"]) \/ (c \in Narrow /\ Live(s.rt["a"]))]] + +----------------------------------------------------------------------------- +(* What a node sends (paths.rs: `want`, `advertise`, `report`). *) + +\* What this node wants a neighbour p to hold for key k: a path, a hold, an +\* end, or to keep what it was sent while the next hop this node holds under +\* waits for those behind it. +Want(n, s, k, p) == + LET src == SrcKey(s, p, k) + r == s.rt[src] + IN IF ~r.on THEN [kind |-> "end", path |-> <<>>, src |-> src] + ELSE IF ~r.held THEN [kind |-> "path", path |-> r.path, src |-> src] + ELSE IF r.next = p + THEN [kind |-> IF Pending(n, s, src) \/ Hopes(n, s, src) THEN "keep" ELSE "end", + path |-> <<>>, src |-> src] + ELSE [kind |-> "hold", path |-> <<>>, src |-> src] + +\* `advertise` for one key on the stream to p. +Advertise1(n, s, p, k) == + LET t == s.tx[p][k] + w == Want(n, s, k, p) + v == IF s.view[k].floor = Absent THEN EmptyView ELSE s.view[k] + IN CASE k \notin Streams(p) \/ w.kind = "keep" -> [tx |-> t, out |-> <<>>] + [] w.kind = "hold" -> + IF t.on /\ ~t.held THEN [tx |-> [t EXCEPT !.held = TRUE], out |-> <>] + ELSE [tx |-> t, out |-> <<>>] + [] w.kind = "end" -> + IF t.on THEN [tx |-> NoSent(t.src), out |-> <>] + ELSE [tx |-> t, out |-> <<>>] + [] OTHER -> + LET offer == ~t.on \/ t.held \/ t.path # w.path + claims == t.rev # v.floor \/ t.items # v.items + IN [tx |-> [on |-> TRUE, held |-> FALSE, path |-> w.path, + rev |-> v.floor, items |-> v.items, src |-> w.src], + out |-> (IF offer THEN <> ELSE <<>>) + \o (IF claims THEN <> ELSE <<>>)] + +Advertise(n, s, ups) == + LET a == [p \in Peers(n, ups) |-> [k \in Keys |-> Advertise1(n, s, p, k)]] + IN [s |-> [s EXCEPT !.tx = [p \in Nodes |-> + IF p \in Peers(n, ups) THEN [k \in Keys |-> a[p][k].tx] ELSE s.tx[p]]], + out |-> [p \in Nodes |-> + IF p \in Peers(n, ups) THEN a[p]["root"].out \o a[p]["a"].out ELSE <<>>]] + +\* Messages for the keys in `ks`, in key order. +Each(ks, f(_)) == + LET F[i \in 0..Len(KeySeq)] == + IF i = 0 THEN <<>> + ELSE F[i - 1] \o (IF KeySeq[i] \in ks THEN f(KeySeq[i]) ELSE <<>>) + IN F[Len(KeySeq)] + +\* `report` to one neighbour p: what became of the routes this node sent it +\* through `hop`, in one DEAD where it is the first word on that hop. +Report1(n, s, p, hop) == + LET t == s.tx[p] + through == {k \in Keys : t[k].on /\ HasHop(t[k].path, hop)} + w == [k \in Keys |-> Want(n, s, k, p)] + moved == {<> : k \in {j \in through : w[j].kind = "path"}} + holds == {k \in through : w[k].kind = "hold" /\ ~t[k].held} + \* The neighbour this node learned the route from takes silence to mean + \* it holds under it, so it is told outright. + parents == {k \in holds : Child(t[k].path, p)} + ends == {k \in through : w[k].kind = "end"} + gone == \A k \in through : w[k].kind \in {"path", "end"} + first == hop \notin s.told[p] + tx1 == [k \in Keys |-> + IF k \notin through THEN t[k] + ELSE CASE w[k].kind = "path" -> [t[k] EXCEPT !.path = w[k].path, !.held = FALSE, !.src = w[k].src] + [] w[k].kind = "hold" -> [t[k] EXCEPT !.held = TRUE] + [] w[k].kind = "end" -> NoSent(t[k].src) + [] OTHER -> t[k]] + out == IF first \/ (gone /\ ends # {}) + THEN <> + \o (IF gone THEN <<>> + ELSE Each(parents, LAMBDA k : <>) \o Each(ends, LAMBDA k : <>)) + ELSE Each({k \in through : w[k].kind = "path"}, LAMBDA k : <>) + \o Each(holds, LAMBDA k : <>) + \o Each(ends, LAMBDA k : <>) + IN IF through = {} THEN [tx |-> t, told |-> s.told[p], out |-> <<>>] + ELSE [tx |-> tx1, told |-> s.told[p] \cup {hop}, out |-> out] + +Report(n, s, ups, hop) == + LET r == [p \in Peers(n, ups) |-> Report1(n, s, p, hop)] + IN [s |-> [s EXCEPT !.tx = [p \in Nodes |-> IF p \in Peers(n, ups) THEN r[p].tx ELSE s.tx[p]], + !.told = [p \in Nodes |-> IF p \in Peers(n, ups) THEN r[p].told ELSE s.told[p]]], + out |-> [p \in Nodes |-> IF p \in Peers(n, ups) THEN r[p].out ELSE <<>>]] + +\* `collect`: forget each doubt no route names any more, each dead hop no route +\* names once the timer it started has fired, and each told hop no route sent +\* names. +Collect(s) == + LET heard == {s.rx[x[1]][x[2]].path : x \in {y \in Nodes \X Keys : s.rx[y[1]][y[2]].on}} + IN [s EXCEPT + !.dead = {h \in s.dead : h[2] \in s.wait \/ \E p \in heard : HasHop(p, h)}, + !.doubt = {x \in s.doubt : \E p \in heard : Names(p, x)}, + !.told = [m \in Nodes |-> {h \in s.told[m] : + \E k \in Keys : s.tx[m][k].on /\ HasHop(s.tx[m][k].path, h)}]] + +----------------------------------------------------------------------------- +(* `update`: select, learn, report a dead hop if any, advertise, collect, *) +(* show. Returns the node's new state, its messages per neighbour, and its *) +(* reader's events. *) + +NoReport == [on |-> FALSE, hop |-> OHop] + +\* The reader's events: changes to `ended` and the counters. A claim leaving +\* while the node still holds a covering route is an END, else it went offline. +\* `quiet` marks a step that only changed a path or lifted a doubt: in one, a +\* claim the reader sees changes only to what a newer floor says, or goes +\* offline after it was held unavailable. +Events(s0, s, old, now, ov, eb, quiet) == + LET present(c) == Presence(s, KeyOf(c)) + isEnd(c) == old[c].item # None /\ now[c].item = None /\ present(c) isStart(c) == old[c].item = None /\ now[c].item # None isUpdate(c) == old[c].item # None /\ now[c].item # None /\ old[c].item # now[c].item - IN [ended |-> [c \in Claims |-> IF isEnd(c) THEN old[c].item ELSE eb[c]], + \* Forgetting a claim clears what the reader saw end. + eb1 == [c \in Claims |-> IF present(c) THEN eb[c] ELSE None] + newer(c) == s.view[KeyOf(c)].floor > s0.view[KeyOf(c)].floor + before(c) == Presence(s0, KeyOf(c)) + \* Letting go of a claim it held unavailable ends a hold, not a path. + release(c) == ~old[c].avail /\ ~present(c) + IN [ended |-> [c \in Claims |-> IF isEnd(c) THEN old[c].item ELSE eb1[c]], revivals |-> Cardinality({c \in Claims : isStart(c) /\ eb[c] = now[c].item}), falseEnds |-> Cardinality({c \in Claims : isEnd(c) /\ ov.items[c] = old[c].item}), - regressions |-> Cardinality({c \in Claims : isUpdate(c) /\ now[c].item < old[c].item})] - -\* `update` on node n's state `s0`, with the origin's view `ov`, the open -\* sessions `ups`, and `eb`, the items n's reader saw END for. A prune forgets. -Update(n, s0, ov, ups, eb, prune) == - LET s1 == IF prune THEN Prune(n, s0, ov) ELSE s0 - eb1 == IF s1 # s0 THEN [c \in Claims |-> None] ELSE eb - sel == Selected(n, s1, ov) - s2 == Record(n, s1, sel, Adopted(n, s1, ov), ups) - live == Live(n, s2) - av == Adopted(n, s2, ov) - plans == [m \in Peers(n, ups) |-> Plan(n, s2, m, sel, live, av, ov)] - s3 == [s2 EXCEPT - !.sent = [m \in Nodes |-> IF m \in Peers(n, ups) THEN plans[m].sent ELSE s2.sent[m]], - !.sentV = [m \in Nodes |-> IF m \in Peers(n, ups) THEN plans[m].sentV ELSE s2.sentV[m]], - !.unacked = [m \in Nodes |-> IF m \in Peers(n, ups) THEN plans[m].unacked ELSE s2.unacked[m]]] - \* Drop the feasibility distance once nothing routes, advertises, or awaits an echo. - drained == n # Origin - /\ \A m \in Nodes : ~s3.rx[m].on /\ s3.sent[m].kind = "none" /\ s3.unacked[m] = 0 - s4 == IF drained THEN [s3 EXCEPT !.fd = NoFd] ELSE s3 - req == Requests(n, s4, sel) - s5 == [s4 EXCEPT !.pending = req.pending] - now == ShownNow(n, s5, sel, live, ov) - ev == Events(n, s5, s0.shown, now, live, ov, eb1) - out == [m \in Nodes |-> - (IF m \in Peers(n, ups) THEN plans[m].out ELSE <<>>) - \o (IF m = req.to THEN <> ELSE <<>>)] - IN [s |-> [s5 EXCEPT !.shown = now], out |-> out, ev |-> ev] - -\* The origin ignores every ROUTE and batch it hears for its own identity, so -\* only ROUTE_REQUEST and ROUTE_END (which it echoes) are queued toward it. -\* Dropping the rest changes no behavior and keeps the state space small. -Heeded(d, m) == - d[2] # Origin \/ m.type = "request" \/ (m.type = "route" /\ m.reach.kind = "none") + regressions |-> Cardinality({c \in Claims : isUpdate(c) /\ now[c].item < old[c].item}), + unstable |-> IF ~quiet THEN 0 + ELSE Cardinality({c \in Claims : old[c].item # now[c].item /\ before(c) /\ ~newer(c) /\ ~release(c)})] + +Update(n, s0, ov, ups, gens, eb, rep, quiet) == + LET s1 == Select(n, s0, gens) + s2 == Know(n, s1, ov) + r == IF rep.on THEN Report(n, s2, ups, rep.hop) + ELSE [s |-> s2, out |-> [p \in Nodes |-> <<>>]] + a == Advertise(n, r.s, ups) + s3 == Collect(a.s) + now == IF n = Origin THEN [c \in Claims |-> NotShown] ELSE ShownNow(s3) + ev == Events(s0, s3, s0.shown, now, ov, eb, quiet) + IN [s |-> [s3 EXCEPT !.shown = now], + out |-> [p \in Nodes |-> r.out[p] \o a.out[p]], + ev |-> ev, + dead |-> a.s.dead] + +\* The publisher ignores every message it hears for its own route, so none is +\* queued toward it. Dropping them changes no behaviour; `Budget` still counts them. +Heeded(d, m) == d[2] # Origin \* The sum of f(x) over the finite set S. SumOver(S, f(_)) == @@ -291,15 +394,18 @@ Commit(rs, q0) == /\ revivals' = revivals + Sum(LAMBDA n : rs[n].ev.revivals) /\ falseEnds' = falseEnds + Sum(LAMBDA n : rs[n].ev.falseEnds) /\ regressions' = regressions + Sum(LAMBDA n : rs[n].ev.regressions) + /\ unstable' = unstable + Sum(LAMBDA n : rs[n].ev.unstable) + /\ learnt' = [n \in Nodes |-> IF n \in ns THEN learnt[n] \cup rs[n].dead ELSE learnt[n]] -Quiet == \A d \in Dirs : q[d] = <<>> /\ acks[d] = 0 +Quiet == \A d \in Dirs : q[d] = <<>> +NoDoubt == \A n \in Nodes : st[n].wait = {} \* The messages the updates `rs` send, counted before `Heeded` drops any. Sent(rs) == SumOver(DOMAIN rs, LAMBDA n : SumOver(Nodes, LAMBDA m : Len(rs[n].out[m]))) \* A burst runs from one quiet point to the next. An external change (a claim -\* change, a session opening or closing, a collection) made while quiet starts -\* a fresh one; every message sent in it counts against the changes made in it. +\* change, a session opening or closing) made while quiet starts a fresh one; +\* every message sent in it counts against the changes made in it. External(sent) == /\ burst' = (IF Quiet THEN 0 ELSE burst) + sent /\ churn' = (IF Quiet THEN 0 ELSE churn) + 1 @@ -309,222 +415,183 @@ Internal(sent) == /\ UNCHANGED churn ----------------------------------------------------------------------------- -(* Initial state: every session open (`Sim::join`), the origin declared *) -(* with no claims (`originate`), and nothing sent. *) +(* Initial state: every session open, the publisher's route sent and no *) +(* claims yet. *) InitNode == [ - rx |-> [m \in Nodes |-> NoEntry], - fd |-> NoFd, - heard |-> [m \in Nodes |-> NoHeard], - view |-> NoView, - sent |-> [m \in Nodes |-> NoReach], - sentV |-> [m \in Nodes |-> NoView], - unacked |-> [m \in Nodes |-> 0], - pending |-> NoPending, + rx |-> [m \in Nodes |-> [k \in Keys |-> NoHeard]], + rt |-> [k \in Keys |-> NoRoute], + view |-> [k \in Keys |-> NoView], + dead |-> {}, + doubt |-> {}, + wait |-> {}, + tx |-> [m \in Nodes |-> [k \in Keys |-> NoSent(k)]], + told |-> [m \in Nodes |-> {}], shown |-> [c \in Claims |-> NotShown] ] +OV0 == [rev |-> 0, items |-> NoItems] +Gen0 == [l \in Links |-> 0] +Start == Update(Origin, [InitNode EXCEPT !.rt = OwnRoutes], OV0, Links, Gen0, + [c \in Claims |-> None], NoReport, FALSE) + Init == /\ up = Links - /\ q = [d \in Dirs |-> <<>>] - /\ acks = [d \in Dirs |-> 0] + /\ gen = Gen0 + /\ q = [d \in Dirs |-> IF d[1] = Origin THEN Start.out[d[2]] ELSE <<>>] /\ rev = 0 /\ own = NoItems - /\ seqno = 0 /\ hist = <> - /\ st = [n \in Nodes |-> InitNode] + /\ st = [n \in Nodes |-> IF n = Origin THEN Start.s ELSE InitNode] /\ ended = [n \in Nodes |-> [c \in Claims |-> None]] /\ revivals = 0 /\ falseEnds = 0 /\ regressions = 0 + /\ unstable = 0 + /\ learnt = [n \in Nodes |-> {}] /\ flaps = 0 - /\ collects = 0 /\ burst = 0 - /\ churn = 0 + /\ churn = 1 + +OV == [rev |-> rev, items |-> own] ----------------------------------------------------------------------------- -(* The origin changes a claim (`announce` / `unannounce` -> `change`). *) +(* The publisher changes a claim. *) ChangeClaim(c, item) == LET own1 == [own EXCEPT ![c] = item] - ov1 == [rev |-> rev + 1, items |-> own1, seq |-> seqno] - rs == [n \in {Origin} |-> Update(Origin, st[Origin], ov1, up, ended[Origin], TRUE)] + ov1 == [rev |-> rev + 1, items |-> own1] + rs == [n \in {Origin} |-> Update(Origin, st[Origin], ov1, up, gen, ended[Origin], NoReport, FALSE)] IN /\ rev' = rev + 1 /\ own' = own1 /\ hist' = Append(hist, own1) /\ Commit(rs, q) /\ External(Sent(rs)) - /\ UNCHANGED <> + /\ UNCHANGED <> -\* A new claim, or new attributes for one, under a fresh revision. Announce(c) == rev < MaxRev /\ ChangeClaim(c, rev + 1) Unannounce(c) == rev < MaxRev /\ own[c] # None /\ ChangeClaim(c, None) ----------------------------------------------------------------------------- -(* A message arrives (`Sim::deliver`). *) - -\* `View::restrict` at the root: a view with no floor certifies no claim. -Certify(v) == IF v.floor >= 0 THEN v ELSE [floor |-> v.floor, items |-> NoItems] - -\* `route`: ROUTE_START or ROUTE_UPDATE, ROUTE_HOLD, or ROUTE_END. After an -\* end, the next batch starts from an empty view, and the last view is kept -\* for what it certified. -OnRoute(s, from, reach) == - LET h == s.heard[from] - rx1 == CASE reach.kind = "fin" -> - [s.rx EXCEPT ![from] = [on |-> TRUE, seq |-> reach.seq, met |-> reach.met, - held |-> FALSE, hops |-> reach.hops]] - [] reach.kind = "held" -> - IF s.rx[from].on THEN [s.rx EXCEPT ![from].held = TRUE] ELSE s.rx - [] OTHER -> [s.rx EXCEPT ![from] = NoEntry] - kept == Certify(IF h.kept.floor # Absent THEN h.kept ELSE h.raw) - heard1 == IF reach.kind = "none" /\ h.on - THEN [s.heard EXCEPT ![from] = - [on |-> TRUE, raw |-> EmptyView, kept |-> kept, ended |-> TRUE]] - ELSE s.heard - IN [s EXCEPT !.rx = rx1, !.heard = heard1] - -\* `View::merge` at the root: take a view certified at a newer floor whole. -Merge(mine, theirs) == IF theirs.floor >= 0 /\ theirs.floor > mine.floor THEN theirs ELSE mine - -\* `batch`: the batch carries the sender's whole presentation, which is what -\* applying its changes to the last view gives on a FIFO stream. It replaces -\* the neighbor's view, which merges into this node's. -OnBatch(s, from, v) == - [s EXCEPT !.heard[from] = [on |-> TRUE, raw |-> v, kept |-> NoView, ended |-> FALSE], - !.view = Merge(IF s.view.floor = Absent THEN EmptyView ELSE s.view, v)] - -\* `request` at a relay that has not already advertised that seqno. -OnRequest(s, from, seq) == - LET p == IF s.pending.on THEN s.pending - ELSE [on |-> TRUE, seq |-> seq, from |-> {}, asked |-> None] - p1 == IF seq > p.seq THEN [p EXCEPT !.seq = seq, !.asked = None] ELSE p - IN [s EXCEPT !.pending = [p1 EXCEPT !.from = @ \cup {from}]] - -Max(a, b) == IF a > b THEN a ELSE b +(* A message arrives (paths.rs: `recv`, `fail`). *) + +\* `fail`: the session `hop` names closed. `from` reported it with what became +\* of its own routes through it, or this node saw it close (`from` is None). +\* It casts doubt on the node at the session's far end, `hop[2]`. +Fail(n, s, hop, from, moved, gone) == + LET Moved(k) == {mv \in moved : mv[1] = k} + rx1 == [p \in Nodes |-> [k \in Keys |-> + LET h == s.rx[p][k] IN + IF ~h.on \/ ~HasHop(h.path, hop) THEN h + ELSE IF p = from /\ Moved(k) # {} + THEN LET path == (CHOOSE mv \in Moved(k) : TRUE)[2] + IN [h EXCEPT !.path = path, + !.state = IF DeadIn(path, s.dead) THEN "dead" ELSE "live"] + ELSE IF p = from + THEN IF gone THEN NoHeard + \* A neighbour that holds under this node answers once those behind it have. + ELSE IF ~Child(h.path, n) THEN [h EXCEPT !.state = "held"] ELSE h + ELSE IF h.state = "live" THEN [h EXCEPT !.state = "dead"] ELSE h]] + IN [s EXCEPT !.rx = rx1, !.dead = @ \cup {hop}, + !.doubt = @ \cup ({hop[2]} \ {n}), !.wait = @ \cup ({hop[2]} \ {n})] + +Recv(n, s, from, msg) == + LET k == msg.k + h == s.rx[from][k] + IN CASE msg.t = "offer" -> + [s EXCEPT !.rx[from][k] = + [on |-> TRUE, path |-> msg.path, + state |-> IF DeadIn(msg.path, s.dead) THEN "dead" ELSE "live", + rev |-> IF h.on THEN h.rev ELSE None, + items |-> IF h.on THEN h.items ELSE NoItems]] + [] msg.t = "hold" -> IF h.on THEN [s EXCEPT !.rx[from][k].state = "held"] ELSE s + [] msg.t = "end" -> [s EXCEPT !.rx[from][k] = NoHeard] + [] msg.t = "claims" -> + IF h.on THEN [s EXCEPT !.rx[from][k].rev = msg.rev, !.rx[from][k].items = msg.items] ELSE s + [] OTHER -> Fail(n, s, msg.hop, from, msg.moved, msg.gone) Deliver(d) == LET m == d[1] n == d[2] msg == Head(q[d]) q0 == [q EXCEPT ![d] = Tail(@)] - isEnd == msg.type = "route" /\ msg.reach.kind = "none" - \* Only the origin advances its seqno to answer a request. - ov1 == IF n = Origin /\ msg.type = "request" - THEN [OV EXCEPT !.seq = Max(seqno, msg.seq)] ELSE OV - sel == Selected(n, st[n], OV) - \* Already advertised to the requester: `request` returns before `update`. - skip == n # Origin /\ msg.type = "request" /\ sel.on /\ sel.seq >= msg.seq - \* Only the route's hops changed: `route` then lets go of nothing. - was == st[n].rx[m] - same == /\ msg.type = "route" /\ msg.reach.kind = "fin" - /\ was.on /\ ~was.held - /\ was.seq = msg.reach.seq /\ was.met = msg.reach.met - /\ Crossed(n, was) = Crossed(n, msg.reach) - \* The origin's own identity merges nothing it hears. - s1 == IF n = Origin THEN st[n] - ELSE CASE msg.type = "route" -> OnRoute(st[n], m, msg.reach) - [] msg.type = "batch" -> OnBatch(st[n], m, msg.view) - [] OTHER -> OnRequest(st[n], m, msg.seq) - rs == [x \in {n} |-> Update(n, s1, ov1, up, ended[n], ~same)] - echoes == IF isEnd THEN 1 ELSE 0 + rep == IF msg.t = "dead" THEN [on |-> TRUE, hop |-> msg.hop] ELSE NoReport + \* Only the route's path changed. + quiet == msg.t = "offer" /\ st[n].rx[m][msg.k].on + rs == [x \in {n} |-> Update(n, Recv(n, st[n], m, msg), OV, up, gen, ended[n], rep, quiet)] IN /\ q[d] # <<>> - \* Every ROUTE_END is echoed once applied. - /\ acks' = IF isEnd THEN [acks EXCEPT ![<>] = @ + 1] ELSE acks - /\ seqno' = ov1.seq - /\ IF skip - THEN /\ q' = q0 - /\ Internal(echoes) - /\ UNCHANGED <> - ELSE /\ Commit(rs, q0) - /\ Internal(echoes + Sent(rs)) - /\ UNCHANGED <> - -\* `echo`: an echoed ROUTE_END comes back (`Sim::deliver_ack`). -DeliverAck(d) == - LET m == d[1] - n == d[2] - rs == [x \in {n} |-> Update(n, [st[n] EXCEPT !.unacked[m] = @ - 1], OV, up, ended[n], TRUE)] - IN /\ acks[d] > 0 - /\ acks' = [acks EXCEPT ![d] = @ - 1] - /\ IF st[n].unacked[m] = 0 - THEN /\ Internal(0) - /\ UNCHANGED <> - ELSE /\ Commit(rs, q) - /\ Internal(Sent(rs)) - /\ UNCHANGED <> + /\ Commit(rs, q0) + /\ Internal(Sent(rs)) + /\ UNCHANGED <> + +\* The timer node n started on x fires: the doubt on x lifts, and the dead hops +\* whose session ended at x may be forgotten. With `Patient`, only once no DEAD +\* about such a hop is in flight: a timer long enough for the news to cross +\* the mesh. +Lift(n, x) == + LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.doubt = @ \ {x}, !.wait = @ \ {x}], + OV, up, gen, ended[n], NoReport, TRUE)] + IN /\ x \in st[n].wait + /\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] : ~(q[d][i].t = "dead" /\ q[d][i].hop[2] = x) + /\ Commit(rs, q) + /\ Internal(Sent(rs)) + /\ UNCHANGED <> ----------------------------------------------------------------------------- -(* Sessions open and close (`Sim::connect`, `Sim::disconnect`). *) - -\* `peer_interest`: a new stream from `y` gets a complete snapshot. -Opened(s, y) == [s EXCEPT !.sentV[y] = NoView] +(* Sessions open and close. *) -\* `disconnect`: everything learned on the session goes, and it counts as the -\* echo of every ROUTE_END sent on it. +\* A closed session: everything learned or sent on it goes. Closed(s, y) == - [s EXCEPT !.rx[y] = NoEntry, - !.heard[y] = NoHeard, - !.unacked[y] = 0, - !.sent[y] = NoReach, - !.sentV[y] = NoView, - !.pending.from = @ \ {y}] + [s EXCEPT !.rx[y] = [k \in Keys |-> NoHeard], + !.tx[y] = [k \in Keys |-> NoSent(k)], + !.told[y] = {}] Connect(l) == LET up1 == up \cup {l} + gen1 == [gen EXCEPT ![l] = @ + 1] rs == [x \in l |-> LET y == CHOOSE y \in l : y # x - IN Update(x, Opened(st[x], y), OV, up1, ended[x], TRUE)] + IN Update(x, Closed(st[x], y), OV, up1, gen1, ended[x], NoReport, FALSE)] IN /\ l \notin up /\ flaps < MaxFlaps /\ up' = up1 + /\ gen' = gen1 /\ flaps' = flaps + 1 /\ Commit(rs, q) /\ External(Sent(rs)) - /\ UNCHANGED <> + /\ UNCHANGED <> +\* `disconnect`: each end names the hop it learned routes on there. Disconnect(l) == LET up1 == up \ {l} q0 == [d \in Dirs |-> IF {d[1], d[2]} = l THEN <<>> ELSE q[d]] rs == [x \in l |-> LET y == CHOOSE y \in l : y # x - IN Update(x, Closed(st[x], y), OV, up1, ended[x], TRUE)] + hop == <> + s == Closed(st[x], y) + IN IF x = Origin + THEN Update(x, s, OV, up1, gen, ended[x], NoReport, FALSE) + ELSE Update(x, Fail(x, s, hop, None, {}, FALSE), OV, up1, gen, ended[x], + [on |-> TRUE, hop |-> hop], FALSE)] IN /\ l \in up /\ flaps < MaxFlaps /\ up' = up1 /\ flaps' = flaps + 1 - /\ acks' = [d \in Dirs |-> IF {d[1], d[2]} = l THEN 0 ELSE acks[d]] /\ Commit(rs, q0) /\ External(Sent(rs)) - /\ UNCHANGED <> - -\* `collect`, allowed once the node holds no route (`Sim::holds`). It -\* forgets the views, so the reader's earlier ENDs no longer count. -Collect(n) == - LET rs == [x \in {n} |-> - Update(n, [st[n] EXCEPT !.view = NoView, - !.heard = [m \in Nodes |-> NoHeard], - !.pending = NoPending], - OV, up, [c \in Claims |-> None], TRUE)] - IN - /\ collects < MaxCollects - /\ \A m \in Nodes : ~st[n].rx[m].on - /\ collects' = collects + 1 - /\ Commit(rs, q) - /\ External(Sent(rs)) - /\ UNCHANGED <> - -\* Nothing is in flight and no more churn is allowed: the run is over. + /\ UNCHANGED <> + +\* Nothing is in flight and no doubt is left: the run may stop here. Done == /\ Quiet - /\ rev = MaxRev \/ flaps = MaxFlaps + /\ NoDoubt /\ UNCHANGED vars Next == \/ \E c \in Claims : Announce(c) \/ Unannounce(c) - \/ \E d \in Dirs : Deliver(d) \/ DeliverAck(d) + \/ \E d \in Dirs : Deliver(d) + \/ \E n \in Relays, x \in Nodes : Lift(n, x) \/ \E l \in Links : Connect(l) \/ Disconnect(l) - \/ \E n \in Relays : Collect(n) \/ Done Spec == Init /\ [][Next]_vars @@ -532,121 +599,168 @@ Spec == Init /\ [][Next]_vars ----------------------------------------------------------------------------- (* Rules. *) -\* Every view a relay keeps is the origin's claims at that view's revision. +\* Every floor a node keeps or was offered is the publisher's claims at that +\* revision, within its key. Truthful == \A n \in Relays : - /\ st[n].view.floor >= 0 => st[n].view.items = hist[st[n].view.floor + 1] - /\ \A m \in Nodes : - LET v == HeardView(st[n].heard[m]) - IN st[n].heard[m].on /\ v.floor >= 0 => v.items = hist[v.floor + 1] - -\* At the root interest every claim is certified, so `loose` is empty. -AllCertified == - /\ \A n \in Relays : - /\ st[n].view.floor < 0 => ~HasItems(st[n].view) - /\ \A m \in Nodes : LET v == HeardView(st[n].heard[m]) IN v.floor < 0 => ~HasItems(v) - /\ \A d \in Dirs : \A i \in DOMAIN q[d] : - q[d][i].type = "batch" => (q[d][i].view.floor >= 0 \/ ~HasItems(q[d][i].view)) + /\ \A k \in Keys : st[n].view[k].floor >= 0 => + st[n].view[k].items = Restrict(hist[st[n].view[k].floor + 1], k) + /\ \A m \in Nodes, k \in Keys : + st[n].rx[m][k].on /\ st[n].rx[m][k].rev >= 0 => + st[n].rx[m][k].items = Restrict(hist[st[n].rx[m][k].rev + 1], k) \* A reader never sees a broadcast start again after it saw it end. NoRevival == revivals = 0 -\* A reader never sees a broadcast end while the origin still has it. +\* A reader never sees a broadcast end while the publisher still has it. NoFalseEnd == falseEnds = 0 \* A reader never sees an older version replace a newer one. NoRegression == regressions = 0 -NextHop(n) == IF n = None THEN None ELSE Selected(n, st[n], OV).next -RECURSIVE Hop(_, _) -Hop(n, k) == IF k = 0 THEN n ELSE Hop(NextHop(n), k - 1) - -\* Following selected next hops from any relay never cycles. -LoopFree == \A n \in Relays : Hop(n, Cardinality(Nodes)) \notin Relays - -\* The loop-free paths from the origin over the configured links, each as the -\* nodes it crosses in order, and what each costs its last node. A path goes on -\* only from a node that forwards, so a client ends one. +\* A change of path, or a lifted doubt, alone starts, ends, and changes no claim +\* a reader sees: only a newer floor does. Whether it is available may change. +Stable == unstable = 0 + +\* Whether node n shows claim c (Seen): exactly while it holds a route that +\* covers c, as the newest floor a covering route gave it says. +Seen == + \A n \in Relays, c \in Claims : + LET v == st[n].view[KeyOf(c)] IN + /\ st[n].shown[c].item # None <=> (Presence(st[n], KeyOf(c)) /\ v.floor >= 0 /\ v.items[c] # None) + /\ st[n].shown[c].item # None => st[n].shown[c].item = v.items[c] + \* No believed covering route has a newer floor than the view. + /\ Presence(st[n], KeyOf(c)) => + \A m \in Nodes, j \in Keys : + (j = KeyOf(c) \/ j = "root") /\ Believed(n, st[n], m, j) => + st[n].rx[m][j].rev <= v.floor + +\* The loop-free paths from the publisher over the configured links, as the +\* nodes each crosses in order, and what each costs its last node. RECURSIVE Extend(_, _) Last(p) == p[Len(p)] Extend(P, k) == IF k = 0 THEN P - ELSE Extend(P \cup UNION {{Append(p, n) : n \in {m \in Relays : {Last(p), m} \in Links}} : - p \in {x \in P : Serves(Last(x))}}, k - 1) -RECURSIVE Along(_) -Along(p) == - IF Len(p) = 1 THEN 0 - ELSE LET before == SubSeq(p, 1, Len(p) - 1) - IN Along(before) + Cost[{Last(before), Last(p)}] + ELSE Extend(P \cup UNION {{Append(p, n) : n \in {m \in Relays : {Last(p), m} \in Links}} : p \in P}, k - 1) Paths == - {p \in Extend({<>}, Cardinality(Relays)) : - /\ \A i, j \in DOMAIN p : i # j => p[i] # p[j] - /\ \A i \in DOMAIN p : i > 1 => {p[i - 1], p[i]} \in Links} - -\* A route's hops are a loop-free path from the origin to its sender, and its -\* metric is that path's cost. Every route sent adds its sender to hops that did -\* not name it, so this holds of every route held or in flight, whatever any -\* node forgot. -IsPath(from, r) == r.hops \in Paths /\ Last(r.hops) = from /\ r.met = Along(r.hops) + {p \in Extend({<>}, Cardinality(Relays)) : \A i, j \in DOMAIN p : i # j => p[i] # p[j]} + +NodesOf(path) == [i \in DOMAIN path |-> path[i][1]] +IsPath(from, path) == + /\ Len(path) >= 1 + /\ path[1] = OHop + /\ NodesOf(path) \in Paths + /\ Last(NodesOf(path)) = from + /\ \A i \in DOMAIN path : i > 1 => path[i][2] = path[i - 1][1] + +\* Every route held or in flight names a loop-free path from the publisher to +\* its sender. SimplePaths == - /\ \A n, m \in Nodes : st[n].rx[m].on => IsPath(m, st[n].rx[m]) + /\ \A n, m \in Nodes, k \in Keys : st[n].rx[m][k].on => IsPath(m, st[n].rx[m][k].path) /\ \A d \in Dirs : \A i \in DOMAIN q[d] : - q[d][i].type = "route" /\ q[d][i].reach.kind = "fin" => IsPath(d[1], q[d][i].reach) - -\* No relay selects a route longer than any loop-free path: a longer one is a -\* stale route counting up as it bounces between relays. It follows from -\* SimplePaths, since no node selects a route that crossed it. -Longest == LET costs == {Along(p) : p \in Paths} - IN CHOOSE c \in costs : \A k \in costs : k <= c -NoCountToInfinity == \A n \in Relays : Selected(n, st[n], OV).met <= Longest - -\* Nodes the origin reaches over open sessions, through nodes that forward. -RECURSIVE Grow(_, _) -Grow(R, k) == IF k = 0 THEN R - ELSE Grow(R \cup {n \in Nodes : \E m \in R : Serves(m) /\ {m, n} \in up}, k - 1) -Reached == Grow({Origin}, Cardinality(Nodes)) - -\* Once nothing is in flight, each reached relay shows exactly the origin's -\* claims, available, and routes only while it has any; each cut-off relay -\* shows nothing and keeps no route or feasibility distance; and no seqno -\* request is pending. + /\ q[d][i].t = "offer" => IsPath(d[1], q[d][i].path) + /\ q[d][i].t = "dead" => \A mv \in q[d][i].moved : IsPath(d[1], mv[2]) + +\* A hop whose session has closed: the link is down or opened again since. +Gone(h) == h # OHop /\ ({h[1], h[2]} \notin up \/ gen[{h[1], h[2]}] # h[3]) +Stale(path) == \E i \in DOMAIN path : Gone(path[i]) + +\* Where a request for a claim goes. A node starts it on the narrowest live +\* route covering the claim; each node forwards it on the route it offered the +\* node it came from under that key, and never along a route whose path names +\* that node. The result is "origin", "stuck", "refused", or "loop". +RECURSIVE Fwd(_, _, _, _) +Fwd(cur, k, prev, i) == + IF cur = Origin THEN "origin" + ELSE IF i = 0 THEN "loop" + ELSE LET src == IF prev = None THEN k ELSE SrcKey(st[cur], prev, k) + r == st[cur].rt[src] + IN IF ~Live(r) THEN "stuck" + ELSE IF prev # None /\ Names(r.path, prev) THEN "refused" + ELSE Fwd(r.next, src, cur, i - 1) + +Request(n, c) == + Fwd(n, IF c \in Narrow /\ Live(st[n].rt["a"]) THEN "a" ELSE "root", None, Cardinality(Nodes)) + +\* The nodes a request from n for c crosses before it reaches the publisher, +\* stops, or has crossed every node. +RECURSIVE Walk(_, _, _, _) +Walk(cur, k, prev, i) == + IF cur = Origin \/ i = 0 THEN <<>> + ELSE LET src == IF prev = None THEN k ELSE SrcKey(st[cur], prev, k) + r == st[cur].rt[src] + IN IF ~Live(r) \/ (prev # None /\ Names(r.path, prev)) THEN <<>> + ELSE <<[node |-> cur, path |-> r.path]>> \o Walk(r.next, src, cur, i - 1) + +\* A request that loops crosses a node whose route is stale: it names a +\* session that has closed, whose news has yet to reach that node. +LoopsAreStale == + \A n \in Relays, c \in Claims : + (Live(st[n].rt["root"]) \/ Live(st[n].rt["a"])) /\ Request(n, c) = "loop" => + LET w == Walk(n, IF c \in Narrow /\ Live(st[n].rt["a"]) THEN "a" ELSE "root", None, Cardinality(Nodes)) + IN \E i \in DOMAIN w : Stale(w[i].path) + +\* A request never loops. It does while news is in flight: `Loop.cfg` pins it. +RequestsLoopFree == \A n \in Relays, c \in Claims : Request(n, c) # "loop" + +\* No node may use a route through a hop it once learned had died: after a +\* dead hop is forgotten, no stale route returns. It holds only for one failure +\* and a patient timer; `StaleIsChased` bounds the rest. +NoStaleReturn == + \A n \in Relays, m \in Nodes, k \in Keys : + Usable(n, st[n], m, k) => ~DeadIn(st[n].rx[m][k].path, learnt[n]) + +\* Whether the news that ends a stale route node n heard from m under key k is +\* on its way: queued from m to n, or m's own route is stale and its news is +\* on its way to m. +RECURSIVE Chased(_, _, _, _) +Chased(n, m, k, i) == + \/ q[<>] # <<>> + \/ /\ i > 0 + /\ m # Origin + /\ LET src == SrcKey(st[m], n, k) + r == st[m].rt[src] + IN /\ Live(r) + /\ Stale(r.path) + /\ Chased(m, r.next, src, i - 1) + +\* A stale route a node may use always has the news that ends it on its way, +\* so it lasts no longer than that news takes to arrive. +StaleIsChased == + \A n \in Relays, m \in Nodes, k \in Keys : + Usable(n, st[n], m, k) /\ Stale(st[n].rx[m][k].path) => Chased(n, m, k, Cardinality(Nodes)) + +\* The nodes a path over open sessions from the publisher reaches, through +\* nodes whose interest covers claim c. +RECURSIVE Grow(_, _, _) +Grow(R, c, i) == IF i = 0 THEN R + ELSE Grow(R \cup {n \in Relays : Covers(Ask(n), c) /\ \E m \in R : {m, n} \in up}, c, i - 1) +Carried(c) == Grow({Origin}, c, Cardinality(Nodes)) + +\* Once nothing is in flight and no doubt is left: each relay shows exactly the +\* publisher's claims that a path carries to it, available; a request for each +\* reaches the publisher; no relay holds a route or keeps a stale one; and +\* every dead hop is forgotten. Settled == - /\ \A n \in Relays \cap Reached : + /\ \A n \in Relays : /\ \A c \in Claims : - st[n].shown[c] = IF own[c] = None THEN NotShown - ELSE [item |-> own[c], avail |-> TRUE] - /\ Selected(n, st[n], OV).on = HasItems([items |-> own]) - /\ \A n \in Relays \ Reached : - /\ \A c \in Claims : st[n].shown[c] = NotShown - /\ \A m \in Nodes : ~st[n].rx[m].on - /\ ~st[n].fd.on - /\ \A n \in Nodes : ~st[n].pending.on + st[n].shown[c] = IF n \in Carried(c) /\ own[c] # None + THEN [item |-> own[c], avail |-> TRUE] ELSE NotShown + /\ \A k \in Keys : ~st[n].rt[k].held + /\ \A m \in Nodes, k \in Keys : + st[n].rx[m][k].on => st[n].rx[m][k].state = "live" /\ ~Stale(st[n].rx[m][k].path) + /\ st[n].dead = {} + /\ \A n \in Relays, c \in Claims : st[n].shown[c].avail => Request(n, c) = "origin" -QuietIsSettled == Quiet => Settled +QuietIsSettled == Quiet /\ NoDoubt => Settled \* What one external change may cost: 4 messages each way over each configured -\* session per route identity and per claim, the budget `mesh_sweep.rs` holds -\* the implementation to. Here there is one identity. -PerChange == 4 * Cardinality(Dirs) * (1 + Cardinality(Claims)) +\* session per key and per claim. +PerChange == 4 * Cardinality(Dirs) * (Cardinality(Keys) + Cardinality(Claims)) \* The messages sent since the last quiet point stay within the budget of the -\* external changes made since then, so a change never sets off announce spam. -\* `burst` is part of the state, so a cycle of deliveries that never goes quiet -\* would grow it past any budget: Budget also rules out routing that never -\* converges once the changes stop. +\* external changes made since then. `burst` is part of the state, so a cycle of +\* deliveries that never goes quiet would grow it past any budget. Budget == burst <= PerChange * churn -\* While anything is in flight, something is eventually delivered. This is -\* WF_vars(Delivery) written out, since Delivery is enabled exactly when not -\* Quiet and TLC's ENABLED on these actions does not finish. It is weaker than -\* fairness per queue, so Converges fails on any cycle of deliveries, starving -\* a queue or not. -Delivery == \E d \in Dirs : Deliver(d) \/ DeliverAck(d) -Fairness == []<>Quiet \/ []<><>_vars - -LiveSpec == Spec /\ Fairness - -\* After the last external change, routing goes quiet and stays settled. -Converges == <>[](Quiet /\ Settled) - ============================================================================= diff --git a/rs/moq-net/verify/route/RouteHandoff.view.ts b/rs/moq-net/verify/route/RouteHandoff.view.ts index fe8a6838d7..9e7c1f03af 100644 --- a/rs/moq-net/verify/route/RouteHandoff.view.ts +++ b/rs/moq-net/verify/route/RouteHandoff.view.ts @@ -1,57 +1,48 @@ -// Statechart labels for RouteHandoff.tla: each relay's route and what its +// Statechart labels for RouteHandoff.tla: each relay's routes and what its // reader shows, and each step as the message that arrives. -// A sequence such as `hops` is keyed from 1. -type Entry = { on: boolean; seq: number; met: number; held: boolean; hops: Record }; +// A sequence, such as a path, is keyed from 1; a hop is <>. +type Hop = Record; +type Path = Record; +type Route = { on: boolean; held: boolean; next: number; path: Path }; type Node = { - rx: Record; - fd: { on: boolean; seq: number; met: number }; + rt: Record; + doubt: number[]; + wait: number[]; shown: Record; }; -type Msg = { type: string; reach: { kind: string; seq: number; met: number; hops: Record } }; +type Msg = { t: string; k: string; path: Path; hop: Hop; gone: boolean }; type Vars = { st: Record; q: Record>; - acks: Record; up: number[][]; - own: Record; burst: number; churn: number; }; const ORIGIN = "0"; -const name = (n: string | number) => (String(n) === ORIGIN ? "the origin" : `relay ${n}`); +const name = (n: string | number) => (String(n) === ORIGIN ? "the publisher" : `relay ${n}`); +const hops = (path: Path) => Object.keys(path).length - 1; -const hops = (e: { hops: Record }) => Object.keys(e.hops).length; - -// Never a route that crossed node `n`, then Babel's condition. -const feasible = (n: string, s: Node, e: Entry) => - !Object.values(e.hops).some((hop) => String(hop) === n) && - (!s.fd.on || e.seq > s.fd.seq || (e.seq === s.fd.seq && e.met < s.fd.met)); - -// `selected` where every link costs the same: the usable feasible route with the -// lowest metric, then key, and the hops to the origin through it. -const selected = (n: string, s: Node) => { - let best: [string, number, number] | null = null; - for (const [m, e] of Object.entries(s.rx)) { - if (!e.on || e.held || !feasible(n, s, e)) continue; - if (best === null || e.met < best[2]) best = [m, hops(e), e.met]; - } - return best; +const route = (key: string, r: Route) => { + if (!r.on) return null; + if (r.held) return `${key} held${r.next >= 0 ? ` under ${name(r.next)}` : ""}`; + const n = hops(r.path); + return `${key} via ${name(r.next)}, ${n} hop${n === 1 ? "" : "s"}`; }; -const relay = (n: string, s: Node) => { - const route = selected(n, s); - const shows = Object.values(s.shown).some((x) => x.item >= 0); - const held = Object.values(s.rx).some((e) => e.on); - const how = route - ? `via ${route[0] === ORIGIN ? "the origin" : `relay ${route[0]}`}, ${route[1]} hop${route[1] === 1 ? "" : "s"}` - : held - ? "holding" - : s.fd.on - ? "no route, distance kept" - : "no route"; - return `${how}${shows ? ", shows it" : ""}`; +const relay = (s: Node) => { + const routes = Object.entries(s.rt) + .map(([key, r]) => route(key === "root" ? "root" : `"${key}"`, r)) + .filter((r) => r !== null); + const shows = Object.entries(s.shown) + .filter(([, x]) => x.item >= 0) + .map(([c, x]) => `${c} item ${x.item}${x.avail ? "" : " (unavailable)"}`); + const doubt = s.doubt.length ? `, doubts ${s.doubt.map(name).join(" and ")}` : ""; + // A doubt no route names any more is forgotten before its timer fires. + const timers = s.wait.filter((x) => !s.doubt.includes(x)); + const wait = timers.length ? `, timer running on ${timers.map(name).join(" and ")}` : ""; + return `${routes.join("; ") || "no route"}${shows.length ? `, shows ${shows.join(", ")}` : ""}${doubt}${wait}`; }; const sessions = (v: Vars) => @@ -60,14 +51,14 @@ const sessions = (v: Vars) => .sort() .join(", ") || "none"; -export const title = "One origin identity across a mesh of relays"; +export const title = "One publisher's routes across a mesh of relays"; export const state = (v: Vars) => ({ "Open sessions": sessions(v), ...Object.fromEntries( Object.keys(v.st) .filter((n) => n !== ORIGIN) - .map((n) => [`Relay ${n}`, relay(n, v.st[n])]), + .map((n) => [`Relay ${n}`, relay(v.st[n])]), ), "Messages this burst": `${v.burst} for ${v.churn} change${v.churn === 1 ? "" : "s"}`, }); @@ -83,9 +74,9 @@ const ends = (key: string | undefined) => { export const step = (action: string, from: Vars, to: Vars) => { switch (action) { case "Announce": - return "the origin announces the broadcast"; + return "the publisher announces a claim"; case "Unannounce": - return "the origin withdraws the broadcast"; + return "the publisher withdraws a claim"; case "Connect": case "Disconnect": { const before = new Set(from.up.map((l) => [...l].sort().join("–"))); @@ -93,28 +84,24 @@ export const step = (action: string, from: Vars, to: Vars) => { const changed = [...before, ...after].find((l) => before.has(l) !== after.has(l)); return `the ${changed} session ${action === "Connect" ? "opens" : "closes"}`; } - case "Collect": - return "a relay with no route collects its state"; - case "DeliverAck": { - const [m, n] = ends(Object.keys(from.acks).find((k) => to.acks[k] < from.acks[k])); - return `${name(m)}'s echo of a route end reaches ${name(n)}`; + case "Lift": { + const n = Object.keys(from.st).find((k) => len(to.st[k].doubt) < len(from.st[k].doubt)); + return n ? `${name(n)}'s timer fires and its doubt lifts` : "a timer fires"; } case "Deliver": { const d = Object.keys(from.q).find((k) => len(to.q[k]) === len(from.q[k]) - 1); const [m, n] = ends(d); const msg = from.q[d ?? ""]?.["1"]; if (!msg) return "a message arrives"; - if (msg.type === "batch") return `${name(m)}'s claims reach ${name(n)}`; + const key = msg.k === "root" ? "root" : `"${msg.k}"`; const what = - msg.type === "request" - ? "seqno request" - : msg.reach.kind === "none" - ? "route end" - : msg.reach.kind === "held" - ? "hold" - : String(m) === ORIGIN - ? "route" - : `route (${hops(msg.reach) - 1} hop${hops(msg.reach) === 2 ? "" : "s"})`; + msg.t === "dead" + ? `news that ${name(msg.hop["1"])}'s session to ${name(msg.hop["2"])} closed${msg.gone ? ", and its routes through it end" : ""}` + : msg.t === "offer" + ? `${key} route (${hops(msg.path)} hop${hops(msg.path) === 1 ? "" : "s"})` + : msg.t === "claims" + ? `${key} claims` + : `${key} route ${msg.t}`; return `${name(m)}'s ${what} reaches ${name(n)}`; } default: diff --git a/rs/moq-net/verify/route/Twice.cfg b/rs/moq-net/verify/route/Twice.cfg new file mode 100644 index 0000000000..e4f3a2ef76 --- /dev/null +++ b/rs/moq-net/verify/route/Twice.cfg @@ -0,0 +1,31 @@ +\* Too large to check every ordering, so TLC samples it at random. +\* Simulate: -simulate num=10000 -depth 100 +\* RouteHandoff.cfg where two sessions close, as when relay 1 dies and 2 and 3 +\* notice apart: each may take the other's stale route and hold under it, and +\* every hold must still end. +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Links = {{0, 1}, {1, 2}, {1, 3}, {2, 3}} + Dear = {} + Scoped = {} + Claims = {"x"} + Narrow = {} + MaxRev = 1 + MaxFlaps = 2 + Patient = TRUE + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + QuietIsSettled + Budget