Skip to content

fix(verify): a patient route timer waits for stale offers too - #8834

Merged
kixelated merged 2 commits into
mainfrom
quest/m2/cluster-routes/holdcycle-stale-return
Nov 15, 2026
Merged

kixelated merged 2 commits into
mainfrom
quest/m2/cluster-routes/holdcycle-stale-return

Conversation

@kixelated

@kixelated kixelated commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m2/cluster-routes/holdcycle-stale-return.md: HoldCycle.cfg now holds NoStaleReturn under any simulation seed.

Summary

HoldCycle.cfg broke NoStaleReturn under random seeds. The cause was the model's Patient guard, not the router.

The failing behaviour (48 steps, found with TLC's 2026-11-15 nightly build) runs like this:

  1. Relay 5 sends relay 2 its route through relay 3, whose session to relay 1 is still open.
  2. That session closes. Relay 5 switches to a route through relay 2 itself, so the replacement offer queues behind the stale one, and relay 5 never sends relay 2 a DEAD.
  3. Relay 2 learns of the dead hop from relay 4. Its doubt and hop timers fire, because no DEAD is in flight, and it forgets the hop.
  4. The stale offer from step 1 then arrives, and relay 2 takes it.

The router follows this path. A scratch Sim replay under Rule::PATH held the 5→2 queue while every timer fired, then delivered one message. Relay 2 routed a/ through 5 on path 0,1,3,5, which crosses the closed 1–3 session. The replay is not committed.

This is an early timer, not a router bug. The replay needs one message delayed longer than the 1 s doubt timer. That is the cost Early.cfg already documents: a timer that fires early lets a stale route return. A timer long enough for the news to cross the mesh also outlasts every offer sent before the news reached its sender. The model's Patient only waited for the DEAD.

Changes:

  • RouteHandoff.tla: with Patient, Lift(n, x) and Expire(n, hop) now wait until no message in flight names the hop they let a relay forget. That covers a DEAD's own hop, a DEAD's moved paths, and an offer's path. For Lift, a hop counts when it names a closed session ending at x. The DRAIN check is unchanged. New helpers: MsgHops, Severed, InFlight.
  • HoldCycle.cfg: drops -seed 1, so Nightly samples new orderings each night. An unseeded simulation always reruns under TLA_CACHE.
  • Twice.cfg and Loop.cfg now check NoStaleReturn. The same defect is why they left it out. On main the pinned TLC breaks it within seconds in both, and with this guard both pass.
  • verify-route skill:
    • The Patient description is updated.
    • "With two failures a stale route can return whatever the timer" is removed. Two failures were never the cause; the weak guard was.
    • The HoldCycle row says a new seed each run.
  • Deletes the quest.

No Rust changes.

Decisions

All of these were taken by the agent while the maintainer was away. Each can be revisited in review.

  1. Fix the model, not the router. The router can't tell that an offer is still in flight. Remembering dead hops until no queued message could name them would need either unbounded memory or a session-generation registry per remote link. Both cost more than a transient stale route, which the next message on the same FIFO replaces and StaleIsChased bounds.
    • Rejected: pinning the replay as a Rust fixture. It would pin the documented early-timer cost, not a bug.
  2. Lift's guard counts only hops of closed sessions (Severed). It does not count every hop ending at x. An offer through an open session to x is not stale, and blocking on it would delay many lifts and shrink coverage for no reason.
  3. The change also restores NoStaleReturn to Twice.cfg and Loop.cfg, which goes beyond the quest's HoldCycle scope. The quest said to keep Twice.cfg without it unless a behaviour showed a second session change was the cause. The behaviour shows the opposite: Twice's own counterexample (17 steps) is the same stale offer outliving a patient timer.
  4. Loop.cfg keeps its -seed 1. That is outside this quest.

Note for moq-dev/moq#8734

Its TwoHomes.cfg drops Stable because "a relay can take an offer older than the timer that names a hop it has already forgotten died". That is this mechanism. Under this guard, TwoHomes.cfg may hold Stable again, and whichever PR lands second should rerun it. Both PRs also add invariants to Twice.cfg and Loop.cfg, so expect a trivial conflict in their INVARIANTS lists.

Validation

The random-seed failure reproduces only with TLC's nightly build (2026-11-15). On the pinned TLC 1.8 and the 2026-10-02 build, 1.2M-state runs of HoldCycle.cfg passed on every model version tried, with or without the fix. So I used the nightly build for the before/after comparison:

Model Config, flags Before the fix After the fix
moq-dev/moq#8622's 12537d1c2 HoldCycle, num=3000 or 10000, -depth 100, random seed fails at 409K states (and in 3 of 3 runs in the interest-set PR) 4.0M states, no error
moq-dev/moq#8728's 98ed0f827 HoldCycle, num=3000 -depth 100, random seed fails at 883K states
main (250a13a17) HoldCycle, the failing behaviour replayed as a fixed action sequence breaks NoStaleReturn at step 48 blocked at the early Lift, no error
main HoldCycle, num=10000 -depth 100, random seed 4.0M states, no error
main + NoStaleReturn Twice, num=10000 -depth 100 fails at 297K states (17 steps) 4.0M states, no error
main + NoStaleReturn Loop, num=3000 -depth 150 fails at 30K states (28 steps) 1.8M states, no error

With the pinned TLC 1.8, as Nightly runs it:

  • HoldCycle.cfg (new Simulate: line, unseeded): 1.6M states, no error.
  • Twice.cfg and Loop.cfg with NoStaleReturn: 4.0M and 601K states, no error. On main's model the same two runs fail within 158K and 16K states.
  • Exhaustive runs:
    • RouteHandoff.cfg: no error, 434,956 distinct states, down from 476,515. The states removed are timers firing while a stale offer is queued, which Early.cfg (Patient = FALSE) still covers.
    • Prefer.cfg: 11,582 distinct states, no error.
    • Upstream.cfg: 63,335 distinct states, no error.

The stronger guard only removes behaviours, so no config that passed before can fail. A queued message keeps Deliver enabled, so the guard can't deadlock a run.

Router: the scratch Sim replay above (Rule::PATH, 1 s doubt) reproduced the stale route on the compiled router. just check passes.

Caused by

  • moq-dev/moq#8541: introduced Patient as "no DEAD about its node is in flight", which left out stale offers sent before the news (found by check: random-seed simulation).

Public API / wire impact

None. The change is limited to the model, its configs, and the skill.

Follow-ups

None.

Models

Size: M
Implementer: Claude Code, claude-opus-5-5, unknown effort (arm B)

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

Review

Rewritten by quest review after each round.

  • Head: 06a461a

  • Base: origin/main at ca88fc1

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1dfbb-b7f6-7390-8431-b3a43f8e63ce, round 1

  • Score: 9/10, No actionable defects were found; the patient-timer guards consistently cover stale offers and replacement paths, though tests and model checks were not run as instructed.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

kixelated and others added 2 commits November 15, 2026 04:50
A patient timer is one long enough for the news of a closed session to
cross the mesh, so it also outlasts every offer through that session sent
before the news reached its sender. The model let it fire once no DEAD was
in flight, while an older offer naming the dead hop could still be queued,
so a relay forgot the hop and took that offer back. HoldCycle broke
NoStaleReturn that way under random seeds, and Twice and Loop within
seconds. Lift and Expire now wait for every message naming the hop, so
all three check NoStaleReturn and HoldCycle samples a new seed each run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title HoldCycle keeps a dead path from returning off its pinned seed fix(verify): a patient route timer waits for stale offers too Nov 15, 2026
@kixelated
kixelated marked this pull request as ready for review November 15, 2026 15:04
@coderabbitai

coderabbitai Bot commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1de5b85b-a734-4385-bfe6-5f03a981aea8
Commits

Reviewing files that changed from the base of the PR and between 250a13a and 06a461a.

Files selected for processing: 6

  • .claude/skills/verify-route/SKILL.md
  • quest/m2/cluster-routes/holdcycle-stale-return.md
  • rs/moq-net/verify/route/HoldCycle.cfg
  • rs/moq-net/verify/route/Loop.cfg
  • rs/moq-net/verify/route/RouteHandoff.tla
  • rs/moq-net/verify/route/Twice.cfg
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated
kixelated enabled auto-merge November 15, 2026 15:05

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

This run reviewed the strengthened Patient guard in the route model, plus the config and skill updates that follow from it.

  • Patient guard covers stale offers: Lift and Expire now wait until no queued message names a hop they would let a relay forget. That covers an offer's path, a DEAD's hop, and a DEAD's moved paths, using the new MsgHops / Severed / InFlight helpers. I confirmed the old DEAD-only check is a subset of the new one, because a DEAD's hop always carries a closed generation. Severed excludes OHop. Disconnect empties the link's queues and Deliver stays enabled while anything is queued, so the stronger guard cannot deadlock a run.
  • Configs: HoldCycle.cfg no longer pins -seed 1, and Twice.cfg and Loop.cfg now check NoStaleReturn.
  • Skill text: the verify-route description of the timer's cost now matches the new guard. The claim that two failures let a stale route return whatever the timer is removed.

ℹ️ search.rs still uses the old guard

rs/moq-net/src/route/tests/search.rs describes itself as RouteHandoff.tla's check run on compiled code. Its patient() still waits only on DEAD and DRAIN messages. It checks no stale-return rule, so this causes no false failures today. If a stale-return rule is ever added there, it needs this PR's stronger guard.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit 0d67d88 Nov 15, 2026
10 checks passed
@kixelated
kixelated deleted the quest/m2/cluster-routes/holdcycle-stale-return branch November 15, 2026 15:13
kixelated added a commit that referenced this pull request Nov 15, 2026
Main's patient timer (#8834) waits for stale offers as well as DEAD news,
which closes the hole that let a relay take an offer naming a hop it had
forgotten died. TwoHomes.cfg with Stable passes 40,000 traces on seed 1,
where the pre-merge model failed within a minute.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant