Skip to content

fix(moq-net): a relay's finished copy keeps serving fetches - #8814

Open
kixelated wants to merge 5 commits into
mainfrom
quest/m1/failover-fetch-residuals
Open

kixelated wants to merge 5 commits into
mainfrom
quest/m1/failover-fetch-residuals

Conversation

@kixelated

@kixelated kixelated commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Works on quest/m1/failover-fetch-residuals.md.

Summary

A relay's copy of a track used to drop its serving loop once a lite upstream finished the track. That cancelled FETCHes already in flight and closed the copy, so a reader's fetch_group for an unfinished group below the end either failed or waited out the subscriber's budget. FailoverChain.tla already rules both out.

Now a finished copy keeps a fetch-only handler (FinishedServe in lite/subscriber.rs). It still serves Consumer::fetch_group fills and keeps FETCHes in flight. It is released once nobody reads it and no fetch is in flight. Readers' misses still settle through the subscription's tail.

  • track::Requests can be fetch-only. FillState counts fetchers apart from handlers, and a fill remembers whether a fetch asked or joined it.
  • kio::Tasks::retain cancels the tasks a predicate rejects. The finish uses it to keep only fetch runs.
  • The two residual tests flip to delivery: ending_the_track_keeps_a_fetch_in_flight and a_fetch_from_a_completed_copy_goes_on_the_wire. A third test checks a fetch made after the finished copies are released (a_fetch_after_the_finished_copies_linger_goes_on_the_wire).
  • EndedRelay.cfg's mesh finding is replayed on the real stack as residual_a_fetch_cannot_move_off_a_dying_route_after_the_end, which pins the hang. The quest is narrowed to that case.

Decisions

Taken while the user was away; each can be undone in review.

  • Pin the EndedRelay case instead of fixing it here. The fix is in the front: an ended logical track closes, reads as unread, and is never re-spliced when its source dies. It also needs Switch and Abort changes in FailoverChain.tla. That is a front redesign on its own, and feat(route): a dual-homed publisher's other links survive doubt about it #8734 is reworking route code in parallel. The quest file now describes only this remaining work.
  • Fetch-only handler, not a full one. A full handler on the finished copy let cursor probes and readers' misses go upstream after the end. That delivered groups outside a reader's ranges (subscribe_ranges::a_relay_with_holes) and stretched drop settlement (drop_record).
  • No linger for a finished copy. Once nobody reads it and no fetch is in flight, it is released right away. A new reader subscribes upstream again rather than reading a cache that no subscription keeps. A 30s linger broke drop_record's settle budget.
  • Readers' misses in flight are cancelled at the finish, and only fetch runs are kept. The tail already settles what the subscription owed.
  • The finished copy stays a field of TrackServeRun, not a TrackRunState phase. It runs alongside the subscription's Ending, Finish, and Done phases. Revoking the grant now ends it in any of them (a_revoked_grant_ends_a_finished_copy_awaiting_its_fin, from review round 1).
  • kio::Tasks::retain is added as a public method on kio, rather than draining and rebuilding the set.

Validation

  • just test for moq-net (with --features async) and kio: 4203 passed, clippy clean. New unit tests are a_fetch_only_handler_takes_fetches_alone and retain_cancels_only_the_rejected.
  • quest gate just check passes. After review: cargo nextest run -p moq-net --features async 3994 passed.
  • Before the fix, the two residual tests reproduced their bugs (NotFound and a hang). Afterwards both deliver.
  • TLC: FailoverChain.cfg passes (1,961,020 distinct states) and Leave.cfg passes (2,162,238). EndedRelay.cfg still breaks NoSkipWhileHeld at depth 31, as pinned. ResumeReader.tla passes on its default cfg (14,334,670) and on ThreeRoutes.cfg (7,053,230).

Public API

No breaking change. kio::Tasks::retain is a public addition. In moq-net, the fetch-only Requests and Fill::is_fetch are pub(crate).

Wire

None.

Caused by

No single PR. The finished branch dropped the serving loop from the start, and FETCH-after-end was never served.

Follow-ups

Models

Size: M
Implementer: Claude, claude-opus-5-5, unknown effort (arm A)

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: c2d0679

  • Base: origin/main at 505a86f

  • Risk: high, This change governs authorization during finished-copy FETCH serving, where a regression could expose data after permission ends and reverting cannot retract delivered data.

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1df84-139d-7382-a698-9a3b2f97c3d6, round 2

  • Score: 10/10, Code inspection confirms the revocation fix addresses the earlier finding without introducing new issues; builds and tests were not run, as instructed.

  • Raised the risk to high: This change governs authorization during finished-copy FETCH serving, where a regression could expose data after permission ends and reverting cannot retract delivered data.

  • Rounds: 2; first round 1 (medium 1); findings 1 (medium 1): fixed 1, refuted 0, open 0

  • Findings: none

  • The last round's findings, at 65e1c0c:

    • Fixed: medium, rs/moq-net/src/lite/subscriber.rs:12335: Model finished-copy serving within TrackRunState; subscriber.rs:12745 checks authorization while a finished copy exists, including Ending and Finish, and revoke_finished at line 12665 removes the copy and drops its FETCH runs while preserving FIN handling.

kixelated and others added 4 commits November 15, 2026 03:25
A relay dropped its copy the moment the upstream finished the track: a
FETCH in flight was cancelled, and a later fetch of a group below the end
found the copy closed and waited out the subscriber's budget. The copy now
keeps answering fetches while it is read, and only fetches: a reader's miss
still settles at once, as the subscription's tail already accounted for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fetch-only handler gets its own counter and a unit test. The
verify-failover skill and the quest now describe only the remaining
EndedRelay case, which the real stack replays as a hang.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title A fetch survives the track ending on a relay chain fix(moq-net): a relay's finished copy keeps serving fetches Nov 15, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 15, 2026 14:21
@coderabbitai

coderabbitai Bot commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 41 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9fcb93eb-59a4-46b1-a916-ef231381271c
Commits

Reviewing files that changed from the base of the PR and between 23b8ba4 and c2d0679.

Files selected for processing: 9

  • .claude/skills/verify-failover/SKILL.md
  • quest/m1/failover-fetch-residuals.md
  • rs/kio/src/task.rs
  • rs/moq-net/src/lite/subscriber.rs
  • rs/moq-net/src/model/track.rs
  • rs/moq-net/src/model/track/fill.rs
  • rs/moq-net/src/model/track/miss.rs
  • rs/moq-net/tests/integration/failover_chain.rs
  • rs/moq-net/verify/failover/EndedRelay.cfg
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

I reviewed the full diff at c2d0679. It keeps a relay's finished copy serving fetch_group fills, adds fetch-only Requests, adds kio::Tasks::retain, and updates the failover tests and quest docs.

  • Finished copy keeps serving fetches: on ServeEnd::Finished with FETCH support, TrackServeRun now holds a FinishedServe. It keeps only the fetch FillServeRuns and takes new fetches through Requests::fetches_only(). It is released once nobody reads the copy and no fetch is in flight, or when the session closes.
  • Fetch-only fill handlers: FillState counts fetchers separately from handlers, and a fill records whether a fetch asked for or joined it. A fetch queues when takes_fetches() is true. A reader's miss, a lookup and a flight still need a full handler, so they settle through the tail as before.
  • Revocation covers the finished copy: revoke_finished aborts the copy with Unauthorized. poll_run checks the denial during Ending and Finish while a finished copy exists, and revoke() calls revoke_finished in the other states.
  • kio::Tasks::retain: a public method that cancels rejected tasks and frees their slots the same way remove does.
  • Tests and docs: the two residual tests now assert delivery, a test checks a fetch after release, and residual_a_fetch_cannot_move_off_a_dying_route_after_the_end pins the remaining EndedRelay.cfg hang. The quest is narrowed to that case.

I checked these edge cases and none is a problem:

  • Polling Done again: once a request is admitted, Gate::poll_ready stays ready, so polling poll_run again in Done is safe.
  • No busy loop: Demand::wanted and write_unwanted read the same consumer count, so the abort_unwanted retry in FinishedServe::poll cannot spin.
  • Fetches use FETCH: fetch_group always asks for one group, so its fills become FETCH runs rather than lite-07 range_fills, which the finish drops.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant