diff --git a/doc/bin/relay/config.md b/doc/bin/relay/config.md index abbc8d9db11..346e45ca086 100644 --- a/doc/bin/relay/config.md +++ b/doc/bin/relay/config.md @@ -359,9 +359,17 @@ explicitly, it also caps how long a superseded group stays fresh, whatever max age the publisher declared; unset, each track keeps its own max age. The latest group of every track, and any group still being read, is kept past `capacity` but sheds its oldest frames instead. Whatever the capacity, a group holds at most its -newest 32 MB. A reader of a track's newest group that reaches a shed frame asks -for a new group on tracks that opt into refresh. Any other reader of a shed -frame sees `Evicted`. +newest 32 MB. A reader that reaches a shed frame, such as a late joiner of a long +group, replays it through a flight: a separate copy fetched from the upstream hop +from that frame, paced by its fastest reader and held to its own 32 MB. Readers +within 16 MB of a flight's fastest reader share it, and a reader leaves its +flight for the live copy once that copy holds its frame within 16 MB of the live +reader, which stops the flight. Each hop serves a flight from its own flights in +turn, up to the publisher, which replays only if it serves past groups. A +moq-transport hop streams a FETCH's objects after answering it, so it replays a +finished group, but it refuses a FETCH of a live group that reaches past what +the group holds, so a live group's replay stops there. When nothing replays it, a reader of a track's newest group asks for a +new group on tracks that opt into refresh, and any other reader sees `Evicted`. While a group has readers, the relay receives at most 16 MB of it past the fastest one, then stops reading its stream until that reader catches up, so diff --git a/quest/m1/unbounded-groups/README.md b/quest/m1/unbounded-groups/README.md index 526fe629144..f8bef59a954 100644 --- a/quest/m1/unbounded-groups/README.md +++ b/quest/m1/unbounded-groups/README.md @@ -23,6 +23,25 @@ holds (a tail join). ## Plan +Decided 2026-11-05 (maintainer, /quest-plan), on the open questions of +[moq-dev/moq#7944](https://github.com/moq-dev/moq/pull/7944), which landed +flights: + +- A compressed group's flight starts at frame 0, since every later frame + builds on the ones before it, and its riders skip ahead to their frame. The + prefix is transferred once per flight. Carrying the inflater's state from the + cached copy, and keeping refresh or `Evicted` for compressed groups, were + rejected. +- Paced batch fills are [their own quest](/quest/m1/unbounded-groups/paced-batch-fills.md). +- No cap on concurrent flights per group: memory is bounded by each flight's + tail and pool debt, and the cost of more flights is upstream transfer. + Revisit with relay evidence. +- A moq-transport FETCH is answered from metadata: FETCH_OK follows once the + publisher learns the group's live edge, and its objects stream after it, + riding flights, with no whole-group buffer. A FETCH of a live group is + refused unless its range ends below the live edge, and a failure after + FETCH_OK resets the stream. FETCH stays single-group. + Decided 2026-10-28 (maintainer, after an audit of the plan and [moq-dev/moq#6993](https://github.com/moq-dev/moq/pull/6993)): @@ -39,7 +58,7 @@ Decided 2026-10-28 (maintainer, after an audit of the plan and window is at most 16 MB, so a reader within the behind half of the fastest one keeps up and a slower one falls off. - A reader at a shed frame, a late joiner or one that fell off, is served by - a [flight](/quest/m1/unbounded-groups/flights.md) when the publisher has a + a flight ([moq-dev/moq#7944](https://github.com/moq-dev/moq/pull/7944)) when the publisher has a refill handler: a separate copy with its own upstream fill and 32 MB tail, paced by its fastest rider. A reader joins a flight only within 16 MB of its fastest rider, and hands over to the copy ahead at the same frame once @@ -147,15 +166,17 @@ origin can regenerate data from disk or an archive. This epic owns the end-to-end test: a relay chain carries one live group well past the old caps. A late joiner gets its head through a flight and hands -over to the live copy, and a live reader keeps the tail, over lite and -moq-transport, with memory held near 32 MB per copy. It also carries a long +over to the live copy, and a live reader keeps the tail, over lite, with +memory held near 32 MB per copy; over moq-transport, the live group's replay +is refused and a finished group's replays +([moq-dev/moq#7944](https://github.com/moq-dev/moq/pull/7944)). It also carries a long compressed group through a small cache pool with a plain reader on a legacy wire, which [moq-dev/moq#6993](https://github.com/moq-dev/moq/pull/6993) covered only at the model layer (maintainer, 2026-10-30). ## Required -- [Flights](/quest/m1/unbounded-groups/flights.md) - a reader at a shed frame replays through a separate paced copy that hands over to the copy ahead +- [Paced batch fills](/quest/m1/unbounded-groups/paced-batch-fills.md) - fills that write a batch of groups stop evicting their own earlier groups before the reader takes them - [Byte holes](/quest/m1/unbounded-groups/byte-holes.md) - segments inside a frame shed like front frames and flights start at `(frame, byte)` on the current lite wip; the frame cap goes - [JS byte holes](/quest/m1/unbounded-groups/js-byte-holes.md) - `@moq/net` mirrors the byte-offset wire and continues a frame cut off mid-way - [Compressed frames have no size cap and inflate as a bounded stream](/quest/m1/unbounded-groups/compressed-stream.md) - A compressed frame of any size publishes and is received in Rust and `@moq/net`, like a plain one diff --git a/quest/m1/unbounded-groups/byte-holes.md b/quest/m1/unbounded-groups/byte-holes.md index e1a0e5c7407..9a1c2e888e2 100644 --- a/quest/m1/unbounded-groups/byte-holes.md +++ b/quest/m1/unbounded-groups/byte-holes.md @@ -4,8 +4,8 @@ A frame of any size, such as a file delivered as one frame, passes through a relay holding only a group's 32 MB tail. Sealed segments inside the frame -are shed like front frames, and a [flight](/quest/m1/unbounded-groups/flights.md) -can start at a `(frame, byte)` position on the current lite wip version. +are shed like front frames, and a flight +([moq-dev/moq#7944](https://github.com/moq-dev/moq/pull/7944)) can start at a `(frame, byte)` position on the current lite wip version. `MAX_CACHE_BYTES` stops capping a frame; it still bounds a group's tail. ## Plan @@ -59,7 +59,3 @@ by flights, so this waits on Flights and stays in m1. Public API: `Position` gains a byte field, frames gain a chunked read, and `MAX_CACHE_BYTES` stops capping a frame. Wire: a byte offset on the current lite wip's range ends and group header. - -## Required - -- [Flights](/quest/m1/unbounded-groups/flights.md) - the paced copies a mid-frame start serves diff --git a/quest/m1/unbounded-groups/flights.md b/quest/m1/unbounded-groups/flights.md deleted file mode 100644 index a0d9d0dd7ad..00000000000 --- a/quest/m1/unbounded-groups/flights.md +++ /dev/null @@ -1,80 +0,0 @@ -# [L] Flights - -## Goal - -A reader whose frames a group copy has shed, such as a late joiner that wants -frame 0 of a long group or a reader that fell more than the 32 MB tail behind, -is served again when the publisher allows replay, without any frame being -fetched faster than its readers take it. Readers replaying the same part of a -group share one upstream fill, and a reader that catches up hands over to the -copy ahead, whose fill then stops. A track whose publisher has no -`track::Requests` handler keeps the refresh-or-`Evicted` behavior of front eviction -([moq-dev/moq#6993](https://github.com/moq-dev/moq/pull/6993)). - -## Plan - -Decided 2026-10-28 (maintainer, after an audit of the plan and -[moq-dev/moq#6993](https://github.com/moq-dev/moq/pull/6993)). This reopens -2026-10-24's rejection of paced refill "trains" that merge, and replaces -#6993's unpaced exact-hole refills written into runs below the live log. -That rejection worried that a merged reader sits at the front, the first -victim; the 16 MB rule below answers it. #6993 measured about 2.7x the -fetch cost per frame for a hole longer than the pool, because an unpaced -refill sheds its own tail before its reader gets there; a paced flight -never does. - -- A flight is a separate copy of a group: its own upstream fill from its - first rider's frame into its own group copy, with its own 32 MB tail, paced - by its fastest rider like the live edge - ([Fastest-reader pacing](https://github.com/moq-dev/moq/pull/7664)). - The live subscription is the first flight. Copies are never stitched - together: two caches sharing one group handle caused the expiry and - boundary bugs [Resume redesign](/quest/m2/resume-redesign.md) records. -- Who serves a reader at a shed frame: a flight when the track has a - `track::Requests` handler (the publisher's permission to replay), else a - new group on tracks that opt into refresh, else `Evicted`. The origin's - handler decides whether to serve old content; refresh is the fallback - (decided 2026-10-24). Byte streams register no handler, so they never - start a flight. -- Admission: a reader joins an existing flight only if that copy holds its - frame within 16 MB, half the tail, of the flight's fastest rider, less one - shedding unit (a log segment). Judge it against the worst-case front: - the ahead window is capped at 16 MB - ([Fastest-reader pacing](https://github.com/moq-dev/moq/pull/7664)), - so filling all outstanding credit still leaves an admitted reader held. - Otherwise it starts its own flight, so it never boards at a front about to - be shed. -- Handover: a rider whose frame the flight ahead holds within the same - admission bound of that flight's fastest rider switches to that copy at the same frame, the way - route failover's `Recover` continues a group from another copy. A - flight's fill is cancelled when its last rider leaves. A merge costs at - most 16 MB of duplicate transfer, and the budget never grows. Rejected: - merging as soon as the copies touch, with the merged flight's budget - growing toward 64 MB until its slow riders catch up. -- A rider that falls off its flight's tail is a reader at a shed frame: it - starts or joins another flight by the same rules. -- Upstream: a relay's flight is a frame-offset fill (lite-06+ `Frame Start`) - that the upstream serves from its own flights, recursively, until the - origin's handler serves or refuses it. Legacy lite peers widen to frame 0. - Over moq-transport a flight starts at an object boundary. -- Memory is 32 MB per flight, bounded overall by pool debt. Open, for the - implementer to measure and bring to the maintainer: whether a group needs - a cap on concurrent flights, and how a compressed flight that starts past - frame 0 gets the decoder state its plain riders need. -- Tests through a two-relay chain: a late joiner of a group many times the - tail reads it from frame 0 through a flight and hands over to the live - copy, with memory near 32 MB per flight; a rider that just handed over - survives the copy ahead filling its whole ahead window at once; joiners - within 16 MB of each other share one upstream fill; a slow rider falls off and starts its own - flight without slowing the fast one; no flight fetches a frame twice; a - track with no handler refreshes. -- Benchmarks: a case swept over readers and how far apart they join. -- Fills that write a whole batch of groups at once evict their own earlier - groups before the reader takes them (the follow-up from - [moq-dev/moq#7339](https://github.com/moq-dev/moq/pull/7339)). Moved here - from [fastest-reader pacing](https://github.com/moq-dev/moq/pull/7664) - on 2026-11-02 (maintainer): the failure crosses groups, while pacing holds - back one group copy, so a flight's fill paces the batch by its riders - instead. - -Public API: none. Wire: none, beyond the existing frame-offset fills. diff --git a/quest/m1/unbounded-groups/paced-batch-fills.md b/quest/m1/unbounded-groups/paced-batch-fills.md new file mode 100644 index 00000000000..91f77a38ca4 --- /dev/null +++ b/quest/m1/unbounded-groups/paced-batch-fills.md @@ -0,0 +1,26 @@ +# [M] Paced batch fills + +## Goal + +A `track::Requests` fill that writes a whole batch of groups at once no longer +evicts its own earlier groups before the reader takes them. A reader of a +batch fill reads every group it asked for, at its own pace, without the cache +holding the whole batch. + +## Plan + +Decided 2026-11-05 (maintainer, /quest-plan): split from +[moq-dev/moq#7944](https://github.com/moq-dev/moq/pull/7944), which landed +flights, into a quest of its own. The failure crosses groups, while a flight +paces one group copy by its riders, and pacing the batch changes what +`Fill::wanted` promises every handler. + +The follow-up from [moq-dev/moq#7339](https://github.com/moq-dev/moq/pull/7339), +moved to flights from +[fastest-reader pacing](https://github.com/moq-dev/moq/pull/7664) on +2026-11-02 (maintainer). Recommendation from #7944: `Fill::wanted` yields the +next group only once the readers took the one before, so a handler serving a +batch writes it a group at a time. + +Public API: `Fill::wanted`'s contract for handlers, with a `MIGRATE.md` note. +Wire: none. diff --git a/quest/m2/ietf-fill-refill.md b/quest/m2/ietf-fill-refill.md index dc21281f2f0..21d8ca786e2 100644 --- a/quest/m2/ietf-fill-refill.md +++ b/quest/m2/ietf-fill-refill.md @@ -3,7 +3,8 @@ ## Goal A moq-transport fill (draft 22) or joining FETCH (draft 18) serves a group's -missing head as a [flight](/quest/m1/unbounded-groups/flights.md): a separate +missing head as a flight +([moq-dev/moq#7944](https://github.com/moq-dev/moq/pull/7944)): a separate copy whose readers hand over to the subscription's tail copy at the first object the tail holds, instead of the 7-state `Fill` claim, park, and install handoff between two producers. @@ -30,7 +31,7 @@ Search by those names if the lines have moved again. Decided 2026-10-28 (maintainer): nothing is written into a group copy below its live log, so the head no longer fills the tail's group in place. Once -[Flights](/quest/m1/unbounded-groups/flights.md) lands, the head is a flight +[Flights](https://github.com/moq-dev/moq/pull/7944) lands, the head is a flight from object 0 that hands over to the tail copy, which also keeps a tail whose head never arrives, as the headless group quest decided. Joining at the next group instead was considered and rejected: it costs a GOP of startup and drops objects already received. @@ -39,5 +40,4 @@ Public API: none. Wire: none. ## Required -- [Flights](/quest/m1/unbounded-groups/flights.md) - separate paced copies and the handover between them - [moq-transport requests end through one lease](/quest/m2/ietf-request-lease.md) - restructures the same files first diff --git a/quest/m2/resume-redesign.md b/quest/m2/resume-redesign.md index 0966f8fa422..bccb84bb3ef 100644 --- a/quest/m2/resume-redesign.md +++ b/quest/m2/resume-redesign.md @@ -64,7 +64,7 @@ paths diverged. `Retain`, `Candidate`, and per-reader `learn_end`. 4. `Recover` attaches to the slot, not a handed-out group: continuing a group whose copy failed is a - [flight](/quest/m1/unbounded-groups/flights.md) from the reader's frame on + [flight](https://github.com/moq-dev/moq/pull/7944) from the reader's frame on the new route, which hands over like any other flight. Hard constraint from the splice era: two caches sharing one group handle @@ -90,7 +90,6 @@ serving copy's upstream closed, followed by a read. ## Required -- [Flights](/quest/m1/unbounded-groups/flights.md) - the paced copy and handover `Recover` folds into - [Release cut](/quest/m2/release/cut.md) - wait for the release in flight to ship ## Related diff --git a/rs/kio/src/log.rs b/rs/kio/src/log.rs index 28788ab7358..d8a6a448511 100644 --- a/rs/kio/src/log.rs +++ b/rs/kio/src/log.rs @@ -268,6 +268,9 @@ impl Writer { /// first segment. pub const HEAP: usize = Self::SHARED + Segment::::heap(FIRST_SEGMENT); + /// Bytes the largest segment occupies: the most one eviction frees at once. + pub const SEGMENT_HEAP: usize = Segment::::heap(MAX_SEGMENT); + /// An empty log whose first entry gets index `start`. pub fn new(start: u64) -> Self { Self { diff --git a/rs/moq-net/MIGRATE.md b/rs/moq-net/MIGRATE.md index 0ea701763f4..44ed73d14c5 100644 --- a/rs/moq-net/MIGRATE.md +++ b/rs/moq-net/MIGRATE.md @@ -4,6 +4,12 @@ How to update your code for each breaking change, newest first. ## Unreleased +### Readers at a shed frame ask a `track::Requests` handler for a flight ([#7944](https://github.com/moq-dev/moq/pull/7944)) + +A reader that reaches a frame its group copy shed no longer gets `Error::Evicted` when the track has a live `track::Requests` handle: the handler receives a `Fill` of that one group from that frame (`Fill::start`), or from frame 0 on a compressed track, even though the group is still cached and live. +Accept it as usual and write from that frame, waiting on `group::Producer::ready` so its readers pace it, and stop once `Fill::is_wanted` is false and the group's `demand()` has no reader: they left it for the live copy. +A handler that cannot serve old frames refuses the fill (`Fill::refuse_groups` or dropping it), and the reader falls back to refresh or `Evicted` as before. A track without a handler is unchanged. + ### The `async` feature is opt-in ([#7607](https://github.com/moq-dev/moq/pull/7607)) `async` is no longer a default feature. Enable it to keep the sessions (`Client`, `Server`, `Session`), `time::run`, the announcement `Stream` adapter, and every async wrapper such as `next_group`, `read_frame`, or `closed`: diff --git a/rs/moq-net/benches/group.rs b/rs/moq-net/benches/group.rs index 847accd20c0..6ee7b357554 100644 --- a/rs/moq-net/benches/group.rs +++ b/rs/moq-net/benches/group.rs @@ -30,6 +30,11 @@ //! reader, as a paced ingest runs, swept over readers: each frame read reports progress, //! and the fastest reader's read wakes the parked writer. //! +//! `group_flights` replays a live group half again past its 32 MiB tail to late readers +//! through flights, swept over readers and how far apart they join: within the 16 MiB a +//! flight admits they share one, further apart each asks its own. Each reader hands over +//! to the live copy once it catches up, and its flight stops. +//! //! `group_compressed_updates` republishes single-frame compressed groups, swept over //! tracks and overlapping groups, to expose encoder allocation on the update path. //! @@ -996,6 +1001,145 @@ fn bench_paced(c: &mut Criterion) { g.finish(); } +/// 48 MiB of 16 KiB frames: half again the live tail. +const FLIGHT_FRAMES: usize = 3_072; + +/// A live group of [`FLIGHT_FRAMES`] on a track with a handler, its live reader at the edge. +struct Flights { + _broadcast: broadcast::Producer, + pool: moq_net::cache::Pool, + requests: track::Requests, + live: group::Producer, + _reader: group::Consumer, +} + +fn shed_live_group(payloads: &[Bytes]) -> Flights { + let info = broadcast::Info::default(); + let pool = info.pool.clone(); + let broadcast = broadcast::Producer::new(info); + let track = broadcast.create_track("bench", track::Info::default()).unwrap(); + let requests = track.requests(); + let mut live = track.append_group().unwrap(); + let mut reader = live.consume(); + let waiter = kio::Waiter::noop(); + for payload in payloads { + live.write_frame(None, payload.clone()).unwrap(); + while let Poll::Ready(Ok(Some(_))) = reader.poll_read_frame(&waiter) {} + } + Flights { + _broadcast: broadcast, + pool, + requests, + live, + _reader: reader, + } +} + +/// A flight the handler writes, from the frame it was asked at. +struct Flight { + group: group::Producer, + demand: group::Demand, + next: usize, + ridden: bool, +} + +/// Replay the live group from frame 0 to `readers` late readers, each joining once the one +/// before it has read `apart` frames, and return how many flights the handler served, the +/// frames it wrote, and the most the pool held. +fn replay_flights(ctx: &Flights, payloads: &[Bytes], readers: usize, apart: usize) -> (usize, usize, u64) { + let waiter = kio::Waiter::noop(); + let mut riders: Vec<(group::Consumer, usize)> = Vec::new(); + let mut flights: Vec = Vec::new(); + let (mut served, mut written) = (0, 0); + let mut peak = 0; + while riders.len() < readers || riders.iter().any(|(_, read)| *read < payloads.len()) { + while riders.len() < readers && riders.last().is_none_or(|(_, read)| *read >= apart) { + riders.push((ctx.live.consume(), 0)); + } + // A megabyte a round each, so a reader joins within a megabyte of when it is due. + for (rider, read) in &mut riders { + for _ in 0..64 { + let Poll::Ready(frame) = rider.poll_read_frame(&waiter) else { + break; + }; + let frame = frame.expect("a late reader replays every frame"); + black_box(frame.expect("the live group is open")); + *read += 1; + } + } + while let Poll::Ready(Ok(fill)) = ctx.requests.poll_requested_fill(&waiter) { + let start = fill.start(); + let group = fill.accept(start.group, None).unwrap(); + let demand = group.demand(); + served += 1; + flights.push(Flight { + group, + demand, + next: start.frame as usize, + ridden: false, + }); + } + // Riders that left their flight end it. + flights.retain_mut(|flight| { + let wanted = flight.demand.wanted(); + let keep = wanted || !flight.ridden; + flight.ridden |= wanted; + keep + }); + for flight in &mut flights { + while let Some(payload) = payloads.get(flight.next) + && flight.group.poll_ready(&waiter, &frame_info(payload)).is_ready() + { + flight.group.write_frame(None, payload.clone()).unwrap(); + flight.next += 1; + written += 1; + } + } + peak = peak.max(ctx.pool.used()); + } + (served, written, peak) +} + +fn frame_info(payload: &Bytes) -> frame::Info { + frame::Info { + size: payload.len() as u64, + timestamp: None, + } +} + +/// See `group_flights` in the module docs. Throughput counts frames the late readers read, +/// so the flight machinery (boarding, admission, and the handover checks a rider makes) shows +/// up as a slope across readers and how far apart they join. +fn bench_flights(c: &mut Criterion) { + let payloads: Vec = (0..FLIGHT_FRAMES as u32) + .map(|index| Bytes::from(index.to_le_bytes().repeat(4096))) + .collect(); + let mut g = c.benchmark_group("group_flights"); + g.sample_size(10); + for readers in [1, 4, 16] { + // Joining together, 8 MiB apart (within reach), and 24 MiB apart (past it). + for apart_mib in [0, 8, 24] { + let apart = apart_mib * 64; + g.throughput(Throughput::Elements((readers * FLIGHT_FRAMES) as u64)); + g.bench_function(BenchmarkId::new(format!("apart_{apart_mib}MiB"), readers), |b| { + b.iter_batched( + || shed_live_group(&payloads), + |ctx| { + let (served, written, peak) = replay_flights(&ctx, &payloads, readers, apart); + // Joiners within reach share a flight; flights share the group's + // content, so the pool holds about one live tail and one flight. + assert!(served >= 1 && written < readers * FLIGHT_FRAMES); + assert!(peak < 64 << 20, "{} MiB held", peak >> 20); + ctx + }, + BatchSize::PerIteration, + ); + }); + } + } + g.finish(); +} + criterion_group!( benches, bench_write, @@ -1009,6 +1153,7 @@ criterion_group!( bench_contended, bench_streaming, bench_shed, - bench_paced + bench_paced, + bench_flights ); criterion_main!(benches); diff --git a/rs/moq-net/src/ietf/publisher.rs b/rs/moq-net/src/ietf/publisher.rs index f49f04b8416..3f73ab77ae1 100644 --- a/rs/moq-net/src/ietf/publisher.rs +++ b/rs/moq-net/src/ietf/publisher.rs @@ -1,7 +1,6 @@ use crate::driver::Transport; use crate::transport::poll::Session as _; use crate::{frame, group, origin, track}; -use futures::FutureExt as _; use std::{ collections::{HashMap, VecDeque}, ops::Bound, @@ -139,15 +138,20 @@ impl FetchRange { } } -/// A FETCH resolved against its track, with its group read ahead. +/// A FETCH resolved against its track, with the group it answers from, none of it read. struct PreparedFetch { range: FetchRange, - /// Read ahead so FETCH_OK can say where the response ends. - last: FetchedGroup, + /// The track's info, which FETCH_OK describes and whose units stamp the objects. + info: track::Info, + group: group::Consumer, + /// One past the finished group's last object, or `None` while the group is live and the + /// range ends at or below its edge. + end: Option, } -/// Resolves a one-group FETCH and reads its objects before promising them in FETCH_OK. -/// A whole-group request waits for an open group to finish. +/// Resolves a one-group FETCH and learns where its group ends before FETCH_OK promises it, +/// without reading the objects: they stream after FETCH_OK. A live group is answered only +/// for a range that ends at or below its live edge. struct PrepareFetch { range: FetchRange, largest_filter: Option, @@ -202,17 +206,29 @@ impl PrepareFetch { }); resolved.map_err(with_reason)?; let range: FetchRange = self.judge(track)?; - let mut read: ReadFetch = ReadFetch::new( + let mut find: FindFetch = FindFetch::new( track, range.start.group, range.start.object, - range.until(range.start.group), self.priority, self.compression, self.ahead.take(), ); - let last = wait!(Read => read.poll(waiter, track)).map_err(with_reason)?; - Poll::Ready(Ok(PreparedFetch { range, last })) + let found: (track::Info, group::Consumer) = wait!(Find => find.poll(waiter, track)).map_err(with_reason)?; + // Parked past every frame, so the group's paced ingest runs on while this waits for + // its end rather than stalling on a reader that takes nothing. + let mut past: group::Consumer = found.1.clone().with_range(u64::MAX..); + let end = wait!(Settle => { + let _ = past.poll_next_frame(waiter); + poll_settle(waiter, track, range, &found.1, self.joined) + })?; + let (info, group) = found; + Poll::Ready(Ok(PreparedFetch { + range, + info, + group, + end, + })) } /// Judge the range against the track, now that its info resolved. @@ -298,19 +314,47 @@ impl Ahead { } } -/// The group a FETCH answers with, read out of the cache so a later eviction cannot -/// truncate what FETCH_OK already promised. -struct FetchedGroup { - sequence: u64, - /// The Object ID of the first frame. - first: u64, - frames: Vec, - /// The group's END_OF_GROUP status object, when it falls in the range. - end_object: Option>, - /// The fetched range reached the finished group's final object. - complete: bool, - /// The track's info, which FETCH_OK describes and whose units stamp the objects. - info: track::Info, +/// Learn where a FETCH's group ends before FETCH_OK promises it: one past its last object +/// once it finished, or `None` once a range bounded inside it ends at or below its live +/// edge. A range reaching past the edge of the track's newest group, which is live, is +/// refused; an older group still arriving, such as a relay's fill, is waited on to finish. +/// Each change to the group wakes it, so a bounded range is answered once the edge passes it. +fn poll_settle( + waiter: &kio::Waiter, + track: &track::Consumer, + range: &FetchRange, + group: &group::Consumer, + joined: bool, +) -> Poll, (Error, String)>> { + let sequence = range.start.group; + match group.poll_closed(waiter) { + // FETCH objects carry no status, so a status object only bounds the frames before it. + Poll::Ready(Ok(())) => { + let end = match group.end_object() { + Some(end) => end.id, + None => group.last_object().map_or(0, |last| last.saturating_add(1)), + }; + return Poll::Ready(Ok(Some(end))); + } + Poll::Ready(Err(err)) => return Poll::Ready(Err(with_reason(err))), + Poll::Pending => {} + } + let edge = group.last_object().map_or(0, |last| last.saturating_add(1)); + if range.until(sequence).is_some_and(|until| until <= edge) { + return Poll::Ready(Ok(None)); + } + // A joining FETCH ends where its subscription starts, at the saved Largest Object, so + // its prefix is still arriving rather than live. + if joined { + return Poll::Pending; + } + if track.is_live() && track.latest_sequence().is_none_or(|latest| sequence >= latest) { + return Poll::Ready(Err(( + Error::Unsupported, + "FETCH of a live group must end below its live edge".into(), + ))); + } + Poll::Pending } struct FetchObjectInfo<'a> { @@ -339,48 +383,23 @@ fn fetch_prior(prior: Option, object: u64) -> Result { Ok(FetchPrior::Same((step != 1).then_some(step))) } -impl FetchedGroup { - /// Drop the END_OF_GROUP object: FETCH objects carry no status, so each group ends at its - /// last frame, which is all a receiver can account for. - fn without_status(mut self) -> Self { - self.end_object = None; - self - } - - /// One past the last Object ID read, including the END_OF_GROUP object. - fn end(&self) -> u64 { - if let Some(end) = &self.end_object { - return end.id.saturating_add(1); - } - self.frames.last().map_or(self.first, |frame| { - frame - .fields - .as_ref() - .map_or(self.first + self.frames.len() as u64, |fields| { - fields.id.saturating_add(1) - }) - }) - } -} - -/// Reads group `sequence` of a track for a FETCH, from object `skip` up to the exclusive -/// `until`, or through the end of the group without one. +/// Finds group `sequence` of a track for a FETCH from object `skip`, without reading it. /// /// This is a [`track::Consumer::fetch_group`], so a relay fetches a miss upstream. The /// track's info resolves first, as it would for a SUBSCRIBE; `ahead` is the whole group /// already asked for, used for a miss. -struct ReadFetch { +struct FindFetch { sequence: u64, skip: u64, - until: Option, priority: i8, compression: bool, - /// Held until the read ends, even when the read asks for its own part of the group. + /// Held until the group is found, even when the find asks for its own part of the group. ahead: Option, - state: ReadFetchState, + state: FindFetchState, } -enum ReadFetchState { +#[expect(clippy::large_enum_variant, reason = "a machine's enum is its storage")] +enum FindFetchState { /// Resolving the track's info. Info(track::Resolving), /// Waiting for the group. @@ -388,25 +407,14 @@ enum ReadFetchState { info: track::Info, fetching: kio::Pending, }, - /// Reading the group's objects in the range. - Read { - info: track::Info, - group: group::Consumer, - /// The Object ID of the first frame read. - first: u64, - frames: Vec, - /// A long cached group is read a slice per poll, so the session's other tasks still run. - budget: kio::coop::Budget, - }, Done, } -impl ReadFetch { +impl FindFetch { fn new( track: &track::Consumer, sequence: u64, skip: u64, - until: Option, priority: i8, compression: bool, ahead: Option, @@ -414,24 +422,34 @@ impl ReadFetch { Self { sequence, skip, - until, priority, compression, ahead, - state: ReadFetchState::Info(track.info().into_inner()), + state: FindFetchState::Info(track.info().into_inner()), } } - fn poll(&mut self, waiter: &kio::Waiter, track: &track::Consumer) -> Poll> { - let res = ready!(self.poll_read(waiter, track)); - self.state = ReadFetchState::Done; - Poll::Ready(res) + fn poll( + &mut self, + waiter: &kio::Waiter, + track: &track::Consumer, + ) -> Poll> { + let res = ready!(self.poll_find(waiter, track)); + self.state = FindFetchState::Done; + Poll::Ready(res.map(|(info, group)| match self.compression { + true => (info, group.compressed()), + false => (info, group), + })) } - fn poll_read(&mut self, waiter: &kio::Waiter, track: &track::Consumer) -> Poll> { + fn poll_find( + &mut self, + waiter: &kio::Waiter, + track: &track::Consumer, + ) -> Poll> { loop { match &mut self.state { - ReadFetchState::Info(resolving) => { + FindFetchState::Info(resolving) => { let info = ready!(resolving.poll_ok(waiter))?; let cached = track.peek_group(self.sequence); let wire = cached.as_ref().and_then(|group| group.wire()); @@ -449,10 +467,7 @@ impl ReadFetch { (_, None) => 0, }; let fetching = match self.ahead.take_if(|_| frame_start == 0) { - Some(Ahead::Fetched(result)) => { - self.state = self.reading(info, result?); - continue; - } + Some(Ahead::Fetched(result)) => return Poll::Ready(result.map(|group| (info, group))), Some(Ahead::Fetching(fetching)) => fetching, None => track.fetch_group( self.sequence, @@ -463,123 +478,16 @@ impl ReadFetch { }, ), }; - self.state = ReadFetchState::Fetch { info, fetching }; + self.state = FindFetchState::Fetch { info, fetching }; } - ReadFetchState::Fetch { info, fetching } => { + FindFetchState::Fetch { info, fetching } => { let group = ready!(kio::Task::poll(&mut **fetching, waiter))?; - let info = info.clone(); - self.state = self.reading(info, group); + return Poll::Ready(Ok((info.clone(), group))); } - ReadFetchState::Read { - group, - first, - frames, - budget, - .. - } => { - // Reading an object past the range proves the range missed the group's end. - let mut overshot = false; - loop { - ready!(budget.poll_yield(waiter)); - if group.wire().is_some_and(|wire| !wire.fetchable) { - return Poll::Ready(Err(Error::Unsupported)); - } - let ordinal = group.index(); - let Some(frame) = ready!(group.poll_read_object(waiter))? else { - break; - }; - let id = frame.fields.as_ref().map_or(ordinal, |fields| fields.id); - if self.until.is_some_and(|until| id >= until) { - overshot = true; - break; - } - if id < self.skip { - continue; - } - if frames.is_empty() { - *first = id; - } - frames.push(frame); - if self.until.is_some_and(|until| id.saturating_add(1) >= until) { - break; - } - } - let ReadFetchState::Read { - info, - group, - first, - frames, - .. - } = std::mem::replace(&mut self.state, ReadFetchState::Done) - else { - unreachable!("matched above"); - }; - return Poll::Ready(self.read(info, group, first, frames, overshot)); - } - ReadFetchState::Done => panic!("polled after it resolved"), + FindFetchState::Done => panic!("polled after it resolved"), } } } - - /// Start reading `group`, the one the FETCH answers with. - fn reading(&self, info: track::Info, mut group: group::Consumer) -> ReadFetchState { - if self.compression { - group = group.compressed(); - } - ReadFetchState::Read { - info, - group, - first: self.skip, - frames: Vec::new(), - budget: kio::coop::Budget::new(32), - } - } - - /// What the read found, once it reached the end of its range or of the group. - fn read( - &self, - info: track::Info, - mut group: group::Consumer, - first: u64, - frames: Vec, - overshot: bool, - ) -> Result { - // A bounded range can reach the group's end without reading its terminal None, and a - // start past the end is already there. - let complete = !overshot - && matches!(group.finished().now_or_never(), Some(Ok(_))) - && group.index() >= group.frame_count() as u64; - let end_object = group - .end_object() - .filter(|end| complete && end.id >= self.skip && self.until.is_none_or(|until| end.id < until)); - - if group.wire().is_some_and(|wire| !wire.fetchable) { - return Err(Error::Unsupported); - } - Ok(FetchedGroup { - sequence: self.sequence, - first, - frames, - end_object, - complete, - info, - }) - } -} - -/// Await [`ReadFetch`]. -#[cfg(test)] -async fn read_fetch( - track: &track::Consumer, - sequence: u64, - skip: u64, - until: Option, - priority: i8, - compression: bool, - ahead: Option, -) -> Result { - let mut read = ReadFetch::new(track, sequence, skip, until, priority, compression, ahead); - kio::wait(|waiter| read.poll(waiter, track)).await } /// A broadcast whose route table is watched for changes in what we advertise: the @@ -4664,7 +4572,6 @@ struct FetchServe { state: FetchState, } -#[expect(clippy::large_enum_variant, reason = "a machine's enum is its storage")] enum FetchState { Start { msg: ietf::Fetch<'static>, @@ -4712,7 +4619,7 @@ enum FetchState { ok: Option, /// Where the response starts, which ranks its stream. start: Location, - write: FetchedGroupWrite, + write: FetchGroupWrite, }, /// The response on its own stream. Cancellation releases the buffered answer even while /// its stream is flow controlled, and a joining one no longer ends with its subscription. @@ -4927,7 +4834,8 @@ impl FetchServe { Ok(()) } - /// Judge the group read for the FETCH, then promise it in FETCH_OK or refuse it. + /// Judge the group found for the FETCH, then promise it in FETCH_OK or refuse it. Its + /// objects stream after FETCH_OK, so a failure from here on resets the response. fn answer( &mut self, admitted: Admitted, @@ -4936,57 +4844,72 @@ impl FetchServe { joined: bool, ) -> Result<(), Error> { let version = self.publisher.version; - let PreparedFetch { range, last: group } = prepared; - let end = range.end; - let group = group.without_status(); + let PreparedFetch { + range, + info, + group, + end, + } = prepared; + let sequence = range.start.group; + let (skip, until) = (range.skip(sequence), range.until(sequence)); // Only the TIMESCALE in FETCH_OK opts the answer into timestamps, so a requester that // opted out of the properties gets an untimed track. - let timescale = group.info.timescale.filter(|_| self.properties_wanted); - let frames = group - .frames - .iter() - .filter_map(|frame| Some((frame.fields.as_deref()?, frame.timestamp))); - for (fields, frame) in frames.chain(group.end_object.as_deref().map(|end| (end, None))) { - if let Err(err) = super::object::write(&fields.properties, group.info.ietf(), frame, version, timescale) { - return self.reject(&err, &err.to_string()); - } - } - let properties = match track_properties(&group.info, self.properties_wanted, version, compression) { + let timescale = info.timescale.filter(|_| self.properties_wanted); + let properties = match track_properties(&info, self.properties_wanted, version, compression) { Ok(properties) => properties, Err(err) => return self.reject(&err, &err.to_string()), }; let draft20 = Filter::is_draft20(version); // Only a finished track's final object ends it, never a live track's Largest Object. - let track = self.track.as_ref().expect("read before answering"); - let end_of_track = !joined && group.complete && track.final_sequence() == group.sequence.checked_add(1); + let track = self.track.as_ref().expect("found before answering"); + let complete = end.is_some_and(|end| until.is_none_or(|until| until >= end)); + let end_of_track = !joined && complete && track.final_sequence() == sequence.checked_add(1); + // One past the last object the answer carries; settling bounds a live group's range. + let stop = match (end, until) { + (Some(end), until) => until.map_or(end, |until| until.min(end)), + (None, until) => until.expect("a live group's range is bounded"), + } + .max(skip); let end_location = if joined { // The subscription starts at the saved Largest Object, so the prefix of that // group has to be here in full or the two leave a gap. - if group.sequence != end.group || group.end() != end.object { + if stop != range.end.object { return self.reject(&Error::NotFound, "joining prefix unavailable"); } - end + range.end } else if draft20 { // Draft-22's End Location is inclusive. An empty answer still covers its start. - let object = match range.until(group.sequence) { + let object = match until { // The track's last object as this draft writes it, without a status object. - Some(until) if end_of_track => (until - 1).min(group.end().saturating_sub(1)), + Some(until) if end_of_track => (until - 1).min(stop.saturating_sub(1)), Some(until) => until - 1, - None => group.end().saturating_sub(1).max(range.skip(group.sequence)), + None => stop.saturating_sub(1).max(skip), }; Location { - group: group.sequence, + group: sequence, object, } - } else if end_of_track && end.object > 0 { + } else if end_of_track && range.end.object > 0 { // A bounded response cannot promise objects past the track's end. Location { - group: group.sequence, - object: group.end(), + group: sequence, + object: stop, } } else { - end + range.end + }; + + // Capped only where the range ends inside the group: a finished group read to its end + // has nothing missing. + let slice = GroupSlice { + skip, + until: until.filter(|until| end.is_none_or(|end| *until < end)), + }; + let priority = super::priority::to_wire(info.priority); + let write = match FetchGroupWrite::new(group, sequence, slice, timescale, version, priority) { + Ok(write) => write, + Err(err) => return self.reject(&err, &err.to_string()), }; // FETCH_OK on every draft, never REQUEST_OK: section 5.2 allows exactly one FETCH_OK or @@ -5000,7 +4923,7 @@ impl FetchServe { properties, }), start: range.start, - write: FetchedGroupWrite::new(group, version, timescale), + write, }; Ok(()) } @@ -5303,7 +5226,7 @@ struct FetchRespond { /// Taken by the stream once it opens. rank: Option>>, open: crate::transport::open::Open, - write: FetchedGroupWrite, + write: FetchGroupWrite, state: FetchRespondState, } @@ -5314,7 +5237,7 @@ impl FetchRespond { admitted: &Admitted, start: Location, request_id: RequestId, - write: FetchedGroupWrite, + write: FetchGroupWrite, ) -> Self { Self { session: publisher.session.clone(), @@ -6981,9 +6904,10 @@ impl FetchGroupWrite { return Err(Error::Unsupported); } Ok(Self { + // A FETCH's group may be found from its first object in range. + index: group.index(), group, sequence, - index: 0, slice, timescale, version, @@ -7033,6 +6957,9 @@ impl FetchGroupWrite { match &mut self.state { FetchGroupState::Yield => { ready!(self.budget.poll_yield(waiter)); + if self.group.wire().is_some_and(|wire| !wire.fetchable) { + return Poll::Ready(Err(Error::Unsupported)); + } if Self::capped(self.slice, self.index) { return Poll::Ready(self.finish()); } @@ -7145,77 +7072,6 @@ impl FetchGroupWrite { } } -/// Write one group's objects to a fetch stream, which carries no other group. -struct FetchedGroupWrite { - frames: std::vec::IntoIter, - /// The Object ID of the next frame without its own. - ordinal: u64, - sequence: u64, - source: Option, - priority: u8, - version: Version, - timescale: Option, - prior: Option, - budget: kio::coop::Budget, - payload: Option, -} - -impl FetchedGroupWrite { - fn new(group: FetchedGroup, version: Version, timescale: Option) -> Self { - Self { - source: group.info.ietf(), - priority: super::priority::to_wire(group.info.priority), - frames: group.frames.into_iter(), - ordinal: group.first, - sequence: group.sequence, - version, - timescale, - prior: None, - budget: kio::coop::Budget::new(32), - payload: None, - } - } - - fn poll( - &mut self, - waiter: &kio::Waiter, - writer: &mut Writer, Version>, - ) -> Poll> { - loop { - if let Some(write) = &mut self.payload { - ready!(write.poll(waiter, writer))?; - self.payload = None; - } - if self.frames.as_slice().is_empty() { - return Poll::Ready(Ok(())); - } - ready!(self.budget.poll_yield(waiter)); - let frame = self.frames.next().expect("checked non-empty"); - let ordinal = self.ordinal; - self.ordinal += 1; - let object = frame.fields.as_ref().map_or(ordinal, |fields| fields.id); - let position = fetch_prior(self.prior, object)?; - self.prior = Some(object); - Publisher::buffer_fetch_object( - writer, - FetchObjectInfo { - sequence: self.sequence, - object, - prior: position, - priority: self.priority, - timestamp: frame.timestamp, - inflated: frame.inflated, - timescale: self.timescale, - version: self.version, - fields: frame.fields.as_deref(), - source: self.source, - }, - )?; - self.payload = Some(FetchPayloadWrite::new(frame.frame.payload)); - } - } -} - /// Write a whole fetch object's length and payload, after its header. struct FetchPayloadWrite { payload: crate::frame::Payload, @@ -8897,33 +8753,6 @@ mod subscribe_cursor_test { use super::*; use crate::lite::test_transport::{Log, SinkSession}; - /// A group that goes stale while a FETCH reads it is refused as too old, rather - /// than answered with the prefix read so far. - #[kio::test] - async fn a_fetch_past_max_age_is_refused_not_truncated() { - let track = track::Producer::new( - std::sync::Arc::new(crate::broadcast::Info::default()), - "video", - crate::track::Info::timed().with_max_age(std::time::Duration::from_millis(500)), - ); - let mut open = track.append_group().unwrap(); - open.write_frame(crate::Timestamp::ZERO, b"a".to_vec()).unwrap(); - - let consumer = track.consume(); - let read = - kio::clock::sim::spawn(async move { read_fetch(&consumer, 0, 0, None, 0, false, None).await.map(|_| ()) }); - kio::clock::sim::yield_now().await; - // The track runs a full second past the group's successor. - for ms in [1000, 2000] { - let mut group = track.append_group().unwrap(); - group - .write_frame(crate::Timestamp::from_millis(ms).unwrap(), b"x".to_vec()) - .unwrap(); - group.finish().unwrap(); - } - assert!(matches!(read.await.unwrap(), Err(Error::Old))); - } - /// A subscription's cursor starts at the oldest cached group, so serving it verbatim /// replays every retained group at once, each on its own stream. Relays reject the burst /// and players skip straight back to the live edge, so the catch-up is pure waste. @@ -11565,7 +11394,7 @@ mod serve_tests { )); let done = futures::poll!(serving.as_mut()); assert!( - claim.requested_broadcast().now_or_never().is_none(), + futures::FutureExt::now_or_never(claim.requested_broadcast()).is_none(), "{version}: {path} reached the application" ); assert!(done.is_ready(), "{version}: {path} was left waiting"); @@ -14448,26 +14277,63 @@ mod serve_tests { } } - /// A whole live group waits for its end and never claims the track ends. + /// A FETCH of the live group is refused unless it ends at or below the group's live + /// edge, which is answered at once and never claims the track ends. #[kio::test] - async fn a_standalone_fetch_waits_for_the_largest_object_group() { + async fn a_standalone_fetch_of_the_live_group_ends_below_its_edge() { for version in FETCH_DRAFTS { let mut h = serve(version); publish_pairs(&mut h, 4, None); let mut live = h.track.create_group(group::Info { sequence: 4 }).unwrap(); - live.write_frame(timestamp(), b"4-0".to_vec()).unwrap(); + for object in 0..2 { + live.write_frame(timestamp(), format!("4-{object}").into_bytes()) + .unwrap(); + } settle().await; - let fetch = standalone_fetch( - &h, - Location { group: 4, object: 0 }, - Location { group: 4, object: 0 }, - GroupOrder::Ascending, + let whole = Location { group: 4, object: 0 }; + let buf = standalone_fetch(&h, whole, whole, GroupOrder::Ascending).await; + assert_eq!(fetch_refusal(buf, version), request::NOT_SUPPORTED, "{version}"); + let past = Location { group: 4, object: 3 }; + let buf = standalone_fetch(&h, whole, past, GroupOrder::Ascending).await; + assert_eq!(fetch_refusal(buf, version), request::NOT_SUPPORTED, "{version}"); + + let edge = Location { group: 4, object: 2 }; + let buf = standalone_fetch(&h, whole, edge, GroupOrder::Ascending).await; + let (ok, objects) = fetch_answer(buf, version); + assert_eq!( + ok.end_location, + if Filter::is_draft20(version) { + Location { group: 4, object: 1 } + } else { + edge + }, + "{version}" ); + assert!(!ok.end_of_track, "{version}"); + assert_eq!(objects, pairs([4]), "{version}"); + } + } + + /// An older group still arriving is not the live edge: a FETCH of it whole waits for + /// its end, then answers. + #[kio::test] + async fn a_standalone_fetch_waits_for_an_older_group_to_finish() { + for version in FETCH_DRAFTS { + let mut h = serve(version); + publish_pairs(&mut h, 4, None); + let mut older = h.track.create_group(group::Info { sequence: 4 }).unwrap(); + older.write_frame(timestamp(), b"4-0".to_vec()).unwrap(); + let mut newer = h.track.create_group(group::Info { sequence: 5 }).unwrap(); + newer.write_frame(timestamp(), b"5-0".to_vec()).unwrap(); + settle().await; + + let whole = Location { group: 4, object: 0 }; + let fetch = standalone_fetch(&h, whole, whole, GroupOrder::Ascending); let finish = async { settle().await; - live.write_frame(timestamp(), b"4-1".to_vec()).unwrap(); - live.finish().unwrap(); + older.write_frame(timestamp(), b"4-1".to_vec()).unwrap(); + older.finish().unwrap(); }; let (buf, ()) = futures::join!(fetch, finish); let (ok, objects) = fetch_answer(buf, version); @@ -14476,7 +14342,7 @@ mod serve_tests { if Filter::is_draft20(version) { Location { group: 4, object: 1 } } else { - Location { group: 4, object: 0 } + whole }, "{version}" ); diff --git a/rs/moq-net/src/model/group.rs b/rs/moq-net/src/model/group.rs index 7cba1e9aa1f..cd0443a8a77 100644 --- a/rs/moq-net/src/model/group.rs +++ b/rs/moq-net/src/model/group.rs @@ -15,7 +15,10 @@ //! //! A group has no size cap. Each copy holds at most its newest [`MAX_CACHE_BYTES`], less //! under the cache pool's memory pressure, and sheds whole frames from its front to stay -//! there; reading a shed frame is [`Error::Evicted`]. +//! there. A reader that reaches a shed frame rides a flight when the track has a +//! [`track::Requests`] handler: a separate copy from that frame, paced by its fastest +//! reader, which it leaves for the copy ahead once that holds its frame within reach. +//! Otherwise reading a shed frame is [`Error::Evicted`]. //! //! The stream is closed with [Error] when all writers or readers are dropped. use super::flate; @@ -43,10 +46,15 @@ use crate::{Error, IntoBytes, Result, StreamError, Timestamp}; /// /// A group has no size cap: a write that takes the frames held past this sheds whole /// frames from the copy's front until it fits, and a compressed group's inflated copy -/// sheds the same way. A reader that reaches a shed frame gets [`Error::Evicted`]. A -/// larger declared frame size is [`Error::FrameTooLarge`] before allocating. +/// sheds the same way. A reader that reaches a shed frame rides a flight, or gets +/// [`Error::Evicted`] when nothing serves one (see the module docs). A larger declared +/// frame size is [`Error::FrameTooLarge`] before allocating. pub const MAX_CACHE_BYTES: u64 = 32 * 1024 * 1024; // 32 MB +/// How far behind a copy's fastest reader a reader may board it: half the live tail, less +/// the most one shed frees at once, since the other half may be all ahead of that reader. +pub(super) const ADMIT: u64 = super::pace::AHEAD - kio::log::Writer::::SEGMENT_HEAP as u64; + /// Heap one cached group costs beyond its frame payloads, excluding the track-side /// bookkeeping in [`track::CACHE_OVERHEAD`]. /// @@ -165,6 +173,13 @@ impl Clone for Entry { } impl Entry { + /// Where the frame sits on its copy's [`Pace`] scale: its payload, metadata, and slot, + /// as the live tail charges it, so a paced writer of tiny frames is bounded too. + fn cost(&self) -> u64 { + let metadata = self.fields.as_ref().map_or(0, |fields| fields.heap()); + self.frame.payload.len() as u64 + metadata + size_of::() as u64 + } + /// A frame without object fields. fn plain(frame: Cached, inflated: u64) -> Self { Self { @@ -965,15 +980,9 @@ impl GroupState { } /// Append a completed frame, returning the slot growth it charged. - /// - /// The frame's place on the pace scale costs its payload, metadata, and slot, as the - /// live tail does, so a paced writer of tiny frames is bounded too. fn push(&mut self, mut entry: Entry) -> u64 { let slots = self.reserve_frame(); - let metadata = entry.fields.as_ref().map_or(0, |fields| fields.heap()); - entry.end = self - .pace - .commit(entry.frame.payload.len() as u64 + metadata + size_of::() as u64); + entry.end = self.pace.commit(entry.cost()); self.log.push(entry); slots } @@ -1417,6 +1426,18 @@ impl Detached { }; (!handle.is_settled()).then_some(handle) } + + /// The group, while anything still holds it and it is not aborted. + pub(super) fn held(&self) -> Option { + let handle = Handle { + state: SendCell::new(self.state.upgrade()?), + info: self.info, + track: self.track.clone(), + cache: self.cache.clone(), + alive: self.alive.clone(), + }; + (!handle.is_aborted()).then_some(handle) + } } /// Whether a group failed, without holding it open or pinning its frames. @@ -2291,6 +2312,43 @@ impl Producer { } impl Handle { + /// Whether `reader` may ride this copy at its frame: it holds the frame, no further + /// than [`ADMIT`] behind the end of the last frame its fastest reader took, so the + /// front shedding as the writer runs a whole window ahead still leaves it in place. + pub(super) fn admits(&self, reader: &Consumer) -> bool { + let index = reader.cursor.index; + let state = self.state.read(); + // Frames adopted below the log sit off this copy's pace scale. + if state.abort.is_some() || (index as u64) < state.log.base() { + return false; + } + // A plain read of a compressed copy inflates its frame from every slice since the + // last one inflated, which must all still be held, from frame 0. + if let Some(plain) = state.plain(reader.cursor.raw) + && (state.offset != 0 + || plain.len.load(Ordering::Acquire).min(index) < state.held() + || plain.shed_past(index)) + { + return false; + } + let start = match state.log.get(index as u64) { + Some(entry) => entry.end - entry.cost(), + None if index == state.committed() => state.pace.written(), + None => return false, + }; + state.pace.read().saturating_sub(start) <= ADMIT + } + + /// One past the last frame this copy committed: how far its writer got. + pub(super) fn committed(&self) -> usize { + self.state.read().committed() + } + + /// Whether `reader` reads this copy. + pub(super) fn is_read_by(&self, reader: &Consumer) -> bool { + Arc::ptr_eq(&self.alive, &reader.cursor.alive) + } + /// Abort the group without taking ownership of its writer. // Sessions cancel a group while its writer is in a frame. pub(crate) fn abort(&self, err: Error) -> Result<()> { @@ -2881,7 +2939,7 @@ impl Handle { expiry: None, expired: false, adopted: false, - recover: None, + carry: None, idle: None, parked: false, } @@ -3130,6 +3188,11 @@ impl Prefetch { self.batch.as_ref().map_or(0, |batch| batch.len) } + /// Whether every buffered frame was taken. + fn is_drained(&self) -> bool { + self.batch.as_ref().is_none_or(|batch| batch.pos == batch.len) + } + /// Buffer `entry` at the end of the batch, with its inflated size if `inflated`, /// returning where it ends on the group's pace scale. fn push(&mut self, entry: &Entry, inflated: bool) -> u64 { @@ -3327,9 +3390,8 @@ pub struct Consumer { // A replacement copy was adopted and has not yet returned a frame that matches. Its // ordinals match this group's only while its Object IDs do. adopted: bool, - // Handed out from a front's logical track: carries the read across route changes. - // Boxed: it is the rare case. - recover: Option>, + // Carries the read across route changes and onto flights. Boxed: it is the rare case. + carry: Option>, // The frame read that last parked, answered again without the read path until something // it parked on fires, once a read parks twice in a row; see [`Self::poll_read_as`]. idle: Option>>, @@ -3341,6 +3403,51 @@ pub struct Consumer { #[cfg(target_pointer_width = "64")] const _: () = assert!(size_of::() <= 184); +/// What carries a read past the copy it reads; see [`Consumer::with_recover`] and +/// [`track::board`]. +#[derive(Default)] +struct Carry { + // Handed out from a front's logical track: carries the read across route changes. + recover: Option, + // Boarding or riding a flight, once the copy read shed the frame read next. + ride: Option, +} + +impl Clone for Carry { + fn clone(&self) -> Self { + Self { + recover: self.recover.clone(), + // A clone reads in parallel, so it boards for itself. + ride: self.ride.as_ref().map(|ride| Ride { + boarding: None, + ..*ride + }), + } + } +} + +/// A reader's flight: a separate copy of its group, from a frame the copy it read shed. +struct Ride { + // The flight asked for, until its copy is accepted. + boarding: Option, + // Whether the copy read is a flight, which the reader hands over from. + aboard: bool, + // Frames still to read and drop before the reader's own: a compressed flight starts at + // frame 0, and its rider inflates its way to the frame it boarded at. + skip: usize, +} + +impl Ride { + /// Reading the copy boarded, a flight when `flight`, with `skip` frames to drop first. + fn aboard(flight: bool, skip: usize) -> Self { + Self { + boarding: None, + aboard: flight, + skip, + } + } +} + /// Per-delivery state shared by a consumer's clones and its frame readers. /// Kept across replacement copies, so provenance and arrival remain the original's. pub(super) struct Delivery { @@ -3433,7 +3540,7 @@ impl Clone for Consumer { expiry: self.expiry.clone(), expired: self.expired, adopted: self.adopted, - recover: self.recover.clone(), + carry: self.carry.clone(), // A clone reads in parallel, so it parks for itself. idle: None, parked: false, @@ -3475,6 +3582,13 @@ impl Consumer { self } + /// Resolves once the group's outcome is final, wherever this cursor is: `Ok` once + /// finished, or the abort error. + #[cfg(feature = "async")] + pub(crate) fn poll_closed(&self, waiter: &kio::Waiter) -> Poll> { + self.cursor.poll(waiter, |state| state.poll_outcome()) + } + /// The index of the first frame the group holds (see [`Producer::start_at`]). pub(crate) fn first_frame(&self) -> u64 { self.cursor.state.read().offset as u64 @@ -3545,7 +3659,7 @@ impl Consumer { /// Carry this group across a front's route changes; see [`super::resume`]. pub(super) fn with_recover(mut self, recover: super::resume::Recover) -> Self { Idle::disarm(&mut self.idle); - self.recover = Some(Box::new(recover)); + self.carry.get_or_insert_default().recover = Some(recover); self } @@ -3562,7 +3676,7 @@ impl Consumer { if self.expired { return res; } - let Some(recover) = self.recover.as_mut() else { + let Some(recover) = self.carry.as_mut().and_then(|carry| carry.recover.as_mut()) else { return res; }; let failed = match &res { @@ -3581,7 +3695,9 @@ impl Consumer { } // Read to its end: no route is needed for it any more. Poll::Ready(Ok(None)) => { - self.recover = None; + if let Some(carry) = &mut self.carry { + carry.recover = None; + } return res; } // Content that failed to inflate is the group's under its name, so no copy fixes it. @@ -3679,7 +3795,7 @@ impl Consumer { if !self.expired && (held || self.expiry_pending()) && self.expiry.as_ref().is_some_and(|expiry| { - let budget = self.recover.as_ref().and_then(|recover| recover.poll_budget(waiter)); + let budget = self.recover().and_then(|recover| recover.poll_budget(waiter)); expiry.is_expired(budget, waiter) }) { self.expired = true; @@ -3787,15 +3903,15 @@ impl Consumer { /// passed the [`Self::with_range`] cap. #[cfg_attr(feature = "async", kio::wrapper)] pub fn poll_next_frame(&mut self, waiter: &kio::Waiter) -> Poll>> { - let res = match self.recover.is_none() { + let res = match self.recover().is_none() { true => { let res = self.poll_next_frame_once(waiter); self.ended_on(res) } - false => match (self.poll_resumed(waiter, Self::poll_next_frame_once), &self.recover) { + false => match (self.poll_resumed(waiter, Self::poll_next_frame_once), self.recover()) { // A frame read from a copy that may fail too carries on the same way. (Poll::Ready(Ok(Some(frame))), Some(recover)) => Poll::Ready(Ok(Some( - frame.with_recover((**recover).clone(), self.cursor.index as u64 - 1), + frame.with_recover(recover.clone(), self.cursor.index as u64 - 1), ))), (res, _) => self.ended_on(res), }, @@ -3834,6 +3950,10 @@ impl Consumer { } fn poll_next_frame_once(&mut self, waiter: &kio::Waiter) -> Poll>> { + self.poll_flown(waiter, Self::poll_next_frame_copy) + } + + fn poll_next_frame_copy(&mut self, waiter: &kio::Waiter) -> Poll>> { if self.expired { return Poll::Ready(Err(Error::Old)); } @@ -3902,7 +4022,7 @@ impl Consumer { waiter: &kio::Waiter, take: impl Fn(&mut Prefetch) -> Option + Copy, ) -> Poll>> { - match self.recover.is_none() { + match self.recover().is_none() { true => self.poll_read_frame_once(waiter, take), false => self.poll_resumed(waiter, |this, waiter| this.poll_read_frame_once(waiter, take)), } @@ -3924,6 +4044,142 @@ impl Consumer { &mut self, waiter: &kio::Waiter, take: impl Fn(&mut Prefetch) -> Option + Copy, + ) -> Poll>> { + self.poll_flown(waiter, |this, waiter| this.poll_read_frame_copy(waiter, take)) + } + + /// Read through `read` from the copy the cursor reads, carried onto a flight once that + /// copy shed the frame read next, and from a flight onto a copy further along once one + /// admits the reader (see [`track::board`]). + fn poll_flown( + &mut self, + waiter: &kio::Waiter, + mut read: impl FnMut(&mut Self, &kio::Waiter) -> Poll>>, + ) -> Poll>> { + let mut boarded = false; + loop { + if let Some(ride) = self.ride_mut() { + // Frames still to drop when the flight boarded, before the ones it starts below + // the reader. + let pending = ride.skip; + if let Some(boarding) = &mut ride.boarding { + let copy = match boarding.poll(waiter) { + Poll::Ready(Ok(copy)) => copy, + // Nothing serves the frame: the shed read stands. + Poll::Ready(Err(_)) => { + self.set_ride(None); + return Poll::Ready(Err(Error::Evicted)); + } + // A read waiting on its flight is blocked like one at the edge, so the + // reader's budget still gives up on it. + Poll::Pending => return self.poll_expired_if_blocked(waiter, Poll::Pending), + }; + let start = usize::try_from(boarding.frame()).unwrap_or(usize::MAX); + let start = start.min(self.cursor.index); + let skip = pending + (self.cursor.index - start); + self.adopt_copy(copy); + self.cursor.index = start; + self.set_ride(Some(Ride::aboard(true, skip))); + } else if ride.aboard + && self.cursor.prefetch.is_drained() + && let Some(track) = self.cursor.cache.track() + && let Some((copy, flight)) = track::hand_over(&track, self) + { + let skip = self.ride_mut().map_or(0, |ride| ride.skip); + self.adopt_copy(copy); + self.set_ride((flight || skip > 0).then(|| Ride::aboard(flight, skip))); + } + } + let res = read(self, waiter); + if let Poll::Ready(Ok(Some(_))) = &res + && let Some(ride) = self.ride_mut() + && ride.skip > 0 + { + ride.skip -= 1; + continue; + } + if !matches!(res, Poll::Ready(Err(_))) || !self.shed_next() { + return res; + } + // The copy just boarded shed the frame before the reader took it: pool debt + // outruns the reader, and boarding again now would fare no better. + if std::mem::replace(&mut boarded, true) { + return res; + } + let Some(track) = self.cursor.cache.track() else { + return res; + }; + // A rider still skipping to its frame keeps what it has left to skip. + let skip = self.ride_mut().map_or(0, |ride| ride.skip); + match track::board(&track, self) { + None => return res, + Some(track::Board::Ride(copy, flight)) => { + self.adopt_copy(copy); + self.set_ride(Some(Ride::aboard(flight, skip))); + } + Some(track::Board::Wait(boarding)) => self.set_ride(Some(Ride { + boarding: Some(boarding), + aboard: true, + skip, + })), + } + } + } + + fn recover(&self) -> Option<&super::resume::Recover> { + self.carry.as_ref()?.recover.as_ref() + } + + fn ride_mut(&mut self) -> Option<&mut Ride> { + self.carry.as_mut()?.ride.as_mut() + } + + fn set_ride(&mut self, ride: Option) { + match (&mut self.carry, ride) { + (Some(carry), ride) => carry.ride = ride, + (None, Some(ride)) => { + self.carry = Some(Box::new(Carry { + recover: None, + ride: Some(ride), + })) + } + (None, None) => {} + } + } + + /// Whether the copy read shed the frame read next, rather than failing or never holding + /// it. Ordinals stand for frames only on a dense group, so only one boards a flight. A + /// plain read of a compressed copy also loses a frame its inflated copy shed, which the + /// decoder, already past it, cannot inflate again. + fn shed_next(&self) -> bool { + let state = self.cursor.state.read(); + let index = self.cursor.index; + if state.abort.is_some() || state.sparse() || index < state.offset { + return false; + } + index < state.held() || state.plain(self.cursor.raw).is_some_and(|plain| plain.shed_past(index)) + } + + /// Read the same frames from `copy` from here on, keeping this reader's place and cap. + fn adopt_copy(&mut self, copy: Consumer) { + let (index, end, raw) = (self.cursor.index, self.cursor.end, self.cursor.raw); + // Another read may move the cursor while a frame read is parked on the old copy. + Idle::disarm(&mut self.idle); + self.cursor = copy.cursor; + self.cursor.index = index; + self.cursor.end = end; + self.cursor.raw = raw; + } + + /// One past the last frame the copy read committed. + pub(super) fn committed(&self) -> usize { + self.cursor.state.read().committed() + } + + fn poll_read_frame_copy( + &mut self, + waiter: &kio::Waiter, + take: impl Fn(&mut Prefetch) -> Option + Copy, ) -> Poll>> { if self.expired { return Poll::Ready(Err(Error::Old)); @@ -4005,7 +4261,7 @@ impl Consumer { /// failed upstream) and no route will continue them. #[cfg_attr(feature = "async", kio::wrapper)] pub fn poll_finished(&mut self, waiter: &kio::Waiter) -> Poll> { - let res = match self.recover.is_none() { + let res = match self.recover().is_none() { true => self.poll_finished_once(waiter), false => self .poll_resumed(waiter, |this, waiter| { diff --git a/rs/moq-net/src/model/pace.rs b/rs/moq-net/src/model/pace.rs index db667bb7682..5db5ab20e29 100644 --- a/rs/moq-net/src/model/pace.rs +++ b/rs/moq-net/src/model/pace.rs @@ -99,10 +99,16 @@ impl Pace { self.wake(); } - fn written(&self) -> u64 { + /// Where the last committed frame ends. + pub fn written(&self) -> u64 { self.written.load(Ordering::Acquire) } + /// Where the last frame the fastest reader took ends. + pub fn read(&self) -> u64 { + self.read.load(Ordering::Acquire) + } + /// Whether a writer `partial` bytes into an uncommitted frame is more than `window` /// bytes past the fastest reader, given it read up to `read`. /// diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 782f4d194d4..fcc5eb091f5 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -386,6 +386,10 @@ pub(crate) struct TrackState { // past `detached_kept`, the count its last prune left. detached: Vec, detached_kept: usize, + // The flights of each group: copies of their own, from a frame a cached copy shed, for + // the readers riding them; see [`Self::board`]. Held weakly, so the last rider leaving + // frees one, and pruned as they are looked up. + flights: BTreeMap>, // Publisher-produced groups in arrival order as (sequence, stamp), walked by // subscriptions; an entry only resolves while its stamp matches the slot's. @@ -2454,6 +2458,17 @@ impl TrackState { paid = paid.saturating_add(freed); } } + // Then the flights, which readers ride for the same reason, cached group or not. + for flight in self.flights.values().flatten() { + if self.debt == 0 || paid >= cap { + break; + } + if let Some(group) = flight.held() { + let freed = group.shed(self.debt, cap - paid.min(cap)); + self.debt -= freed.min(self.debt); + paid = paid.saturating_add(freed); + } + } paid } @@ -3095,6 +3110,161 @@ impl TrackState { } } +impl TrackState { + /// Store a flight a [`Fill`] answered: a copy of group `sequence` of its own, from the + /// frame its riders start at, which the cache never serves in place of its cached copy. + pub(super) fn insert_flight( + &mut self, + sequence: u64, + frames: group::Bounds, + info: Option, + ) -> Result { + if let Some(err) = &self.abort { + return Err(err.clone()); + } + if self.info.is_none() { + self.install(info.unwrap_or_else(|| Info::default().with_max_age(None))); + } + let info = self.info.clone().unwrap(); + let mut group = group::Producer::new(group::Info { sequence }, info.clone(), self.cache.clone()); + // A compressed flight inflates from frame 0 with a decoder of its own, so it keeps + // its own content rather than the cached copy's, whose inflated copy is further on. + if !info.compress { + if let Some(content) = self.content_for_copy(sequence)? { + group.handle().share_content(content); + } + group.handle().register_content(&self.contents); + } + group.bound(frames)?; + self.cap(&group.handle())?; + self.charge_debt(); + self.detach(&group.handle()); + // Forget every flight gone since, so only flights still held are kept. + self.flights.retain(|_, flights| { + flights.retain(|flight| flight.held().is_some()); + !flights.is_empty() + }); + self.flights.entry(sequence).or_default().push(group.handle().detach()); + Ok(group) + } + + /// The copy of its group `reader` rides at its frame, furthest along of those that admit + /// it (see [`group::Handle::admits`]), and whether it is a flight rather than a cached + /// copy. When `ahead`, only one further along than the copy it reads, which it hands + /// over from. + pub(super) fn ride(&self, reader: &group::Consumer, ahead: bool) -> Option<(group::Handle, bool)> { + let sequence = reader.sequence; + let cached = self + .lookup + .get(&sequence) + .into_iter() + .flat_map(Slot::copies) + .map(|copy| (copy.clone(), false)); + let flights = self + .flights + .get(&sequence) + .into_iter() + .flatten() + .filter_map(|flight| Some((flight.held()?, true))); + let past = ahead.then(|| reader.committed()); + cached + .chain(flights) + .filter(|(copy, _)| !ahead || !copy.is_read_by(reader)) + .map(|(copy, flight)| (copy.committed(), copy, flight)) + .filter(|(committed, copy, _)| past.is_none_or(|past| *committed > past) && copy.admits(reader)) + .max_by_key(|(committed, ..)| *committed) + .map(|(_, copy, flight)| (copy, flight)) + } +} + +/// Where a reader at a frame its copy shed goes: see [`board`]. +pub(super) enum Board { + /// Ride this copy, a flight or a cached copy, which holds the frame. + Ride(group::Consumer, bool), + /// Wait for a flight the track's handlers were asked for. + Wait(Boarding), +} + +/// Board `reader`'s group at its frame, which the copy it read shed: ride a copy that +/// admits it, else ask the track's [`Requests`] handlers for a flight from that frame. +/// `None` when no handler can serve one. +/// +/// A compressed group's flight starts at frame 0, since a later frame builds on the ones +/// before it, and its riders skip to their frame. +pub(super) fn board(track: &kio::Producer, reader: &group::Consumer) -> Option { + let state = track.read(); + let compress = state.info.as_ref()?.compress; + if let Some((copy, flight)) = state.ride(reader, false) + && let Some(copy) = copy.try_consume() + { + return Some(Board::Ride(copy, flight)); + } + let fills = state.fills.clone(); + drop(state); + let asking = fill::Asking { + priority: 0, + max_delay: None, + order: Order::Ascending, + }; + let (sequence, frame) = (reader.sequence, if compress { 0 } else { reader.index() }); + let join = fills.lock().fly(&fills, sequence, frame, asking)?; + Some(Board::Wait(Boarding { + join: Box::new(join), + sequence, + frame, + })) +} + +/// The copy `reader`, riding a flight, hands over to: one further along that admits it, +/// and whether that is a flight. See [`TrackState::ride`]. +pub(super) fn hand_over( + track: &kio::Producer, + reader: &group::Consumer, +) -> Option<(group::Consumer, bool)> { + let (copy, flight) = track.read().ride(reader, true)?; + Some((copy.try_consume()?, flight)) +} + +/// A reader waiting on the flight it asked for. +pub(super) struct Boarding { + // Boxed: a reader holds one only while boarding. + join: Box, + sequence: u64, + frame: u64, +} + +impl Boarding { + /// The frame the flight starts at. + pub(super) fn frame(&self) -> u64 { + self.frame + } + + /// Poll for a reader of the flight, or why none came: the handlers refused it, or went + /// away without answering. + pub(super) fn poll(&mut self, waiter: &kio::Waiter) -> Poll> { + let frames = group::Bounds { + start: self.frame, + end: None, + }; + let sequence = self.sequence; + let answer = self + .join + .outcome() + .poll(waiter, |outcome| match outcome.refused(sequence) { + Some((err, _)) => Poll::Ready(Err(err.clone())), + None if outcome.is_accepted(sequence) => { + Poll::Ready(outcome.partial(sequence, frames).ok_or(Error::Cancel)) + } + None => Poll::Pending, + }); + Poll::Ready(match ready!(answer) { + Ok(Ok(copy)) => copy.try_consume().ok_or(Error::Cancel), + Ok(Err(err)) => Err(err), + Err(_closed) => Err(Error::NotFound), + }) + } +} + /// Demand wakes taken out of the track state, fired once its guard is gone. #[must_use = "fire the wakes once the guard is released"] pub(super) struct Deferred(Vec); @@ -3975,6 +4145,11 @@ impl Producer { /// from upstream. While at least one is alive a reader's miss queues a [`Fill`] for it /// to serve; with none, the miss is named dropped, and a cache-miss /// [`Consumer::fetch_group`] on an accepted track fails fast with [`Error::NotFound`]. +/// +/// A live handle also lets a reader at a frame a group copy shed replay it: a [`Fill`] of +/// that one group from that frame asks for a flight, a copy of its own paced by the readers +/// riding it, which they leave for the copy ahead once it holds their frame within reach. +/// Serve it like any group, and stop once nobody reads it: see [`Fill::is_wanted`]. pub struct Requests { name: Arc, // Kept to insert served groups into the cache and observe track abort. @@ -23097,6 +23272,324 @@ mod test { assert_eq!(producer.demand().wanted(), Some(1), "a join asks for the same group"); } + /// The frame a read takes right now, as its number in [`numbered`]. + fn read_numbered(reader: &mut group::Consumer) -> Option { + let frame = reader.read_frame().now_or_never()?; + let frame = frame.expect("read failed").expect("the group ended"); + Some(u32::from_le_bytes(frame.payload[..4].try_into().unwrap())) + } + + /// A late joiner of a live group that shed its front boards a flight from frame 0, + /// served by the track's handler, reads it as the handler writes it, and hands over to + /// the live copy once that copy holds its frame within reach of the live reader. The + /// flight is then unwanted, so its handler stops. + #[test] + #[cfg(feature = "async")] + fn a_late_joiner_rides_a_flight_and_hands_over() { + let producer = track_producer("test", Info::default()); + let requests = producer.requests(); + let mut live = producer.append_group().unwrap(); + let mut reader = live.consume(); + let mut late = live.consume(); + // 48 MiB, half again the live tail. + const FRAMES: u32 = 3_000; + for index in 0..FRAMES { + live.write_frame(None, numbered(index)).unwrap(); + assert_eq!(read_numbered(&mut reader), Some(index)); + } + + // Frame 0 was shed: the late reader asks the handler for a flight from it. + assert_eq!(read_numbered(&mut late), None); + let fill = match requests.poll_requested_fill(&kio::Waiter::noop()) { + Poll::Ready(fill) => fill.unwrap(), + Poll::Pending => panic!("a flight is asked"), + }; + assert_eq!(fill.start(), Position { group: 0, frame: 0 }); + let mut flight = fill.accept(0, None).unwrap(); + assert_eq!(read_numbered(&mut late), None, "aboard, waiting on the handler"); + let unwanted = |flight: &group::Producer| flight.handle().poll_unwanted(&kio::Waiter::noop()).is_ready(); + assert!(!unwanted(&flight), "ridden"); + + // The handler writes as the rider reads, until the rider leaves for the live copy. + let mut index = 0; + while !unwanted(&flight) { + flight.write_frame(None, numbered(index)).unwrap(); + assert_eq!(read_numbered(&mut late), Some(index)); + index += 1; + assert!(index < FRAMES, "handed over before the flight caught up"); + } + // The live writer runs its whole window ahead of the live reader at once: the + // rider, as far behind that reader as it was admitted, keeps its frame. + let mut written = FRAMES; + while live + .poll_ready(&kio::Waiter::noop(), &frame_info(&numbered(0))) + .is_ready() + { + live.write_frame(None, numbered(written)).unwrap(); + written += 1; + } + assert!(written > FRAMES + 900, "a whole window: {written}"); + while index < written { + assert_eq!( + read_numbered(&mut late), + Some(index), + "frame {index} from the live copy" + ); + index += 1; + } + assert_eq!(read_numbered(&mut late), None, "at the live edge"); + } + + /// A plain reader of a compressed group that fell off boards a flight from frame 0, + /// since every later frame builds on the ones before it, and skips the frames before its + /// own as the handler writes them. It then hands over to the live copy. + #[test] + #[cfg(feature = "async")] + fn a_compressed_flight_starts_at_frame_0_and_its_rider_skips_ahead() { + let producer = track_producer("test", Info::default().with_compress(true)); + let requests = producer.requests(); + let mut live = producer.append_group().unwrap(); + let mut reader = live.consume(); + let mut behind = live.consume(); + const FRAMES: u32 = 3_000; + for index in 0..FRAMES { + live.write_frame(None, numbered(index)).unwrap(); + assert_eq!(read_numbered(&mut reader), Some(index)); + if index < 10 { + assert_eq!(read_numbered(&mut behind), Some(index)); + } + } + + // The live reader's inflated copy shed frame 10, which the decoder is past. + assert_eq!(read_numbered(&mut behind), None); + let fill = asked(&requests).expect("a flight"); + assert_eq!(fill.start(), Position { group: 0, frame: 0 }, "from frame 0"); + let mut flight = fill.accept(0, None).unwrap(); + let unwanted = |flight: &group::Producer| flight.handle().poll_unwanted(&kio::Waiter::noop()).is_ready(); + for index in 0..10 { + flight.write_frame(None, numbered(index)).unwrap(); + assert_eq!(read_numbered(&mut behind), None, "frame {index} is skipped"); + } + let mut index = 10; + while !unwanted(&flight) { + flight.write_frame(None, numbered(index)).unwrap(); + assert_eq!(read_numbered(&mut behind), Some(index)); + index += 1; + assert!(index < FRAMES, "handed over before the flight caught up"); + } + while index < FRAMES { + assert_eq!( + read_numbered(&mut behind), + Some(index), + "frame {index} from the live copy" + ); + index += 1; + } + assert_eq!(read_numbered(&mut behind), None, "at the live edge"); + } + + /// The size of `payload` as a frame, for [`group::Producer::poll_ready`]. + fn frame_info(payload: &bytes::Bytes) -> frame::Info { + frame::Info { + size: payload.len() as u64, + timestamp: None, + } + } + + /// A live group of `frames` numbered frames past its tail, read along by a live reader, + /// on a track with a handler. + fn shed_live_group(frames: u32) -> (Producer, Requests, group::Producer, group::Consumer) { + let producer = track_producer("test", Info::default()); + let requests = producer.requests(); + let mut live = producer.append_group().unwrap(); + let mut reader = live.consume(); + for index in 0..frames { + live.write_frame(None, numbered(index)).unwrap(); + assert_eq!(read_numbered(&mut reader), Some(index)); + } + (producer, requests, live, reader) + } + + /// The next fill asked of `requests`, if any. + fn asked(requests: &Requests) -> Option { + match requests.poll_requested_fill(&kio::Waiter::noop()) { + Poll::Ready(fill) => Some(fill.unwrap()), + Poll::Pending => None, + } + } + + /// Joiners at the same shed frame share one flight; one joining later boards it while + /// its frame is within reach of the flight's fastest rider, and asks a flight of its + /// own once it is not. + #[test] + #[cfg(feature = "async")] + fn joiners_within_reach_share_one_flight() { + let (_producer, requests, live, _reader) = shed_live_group(3_000); + let (mut first, mut second) = (live.consume(), live.consume()); + assert_eq!(read_numbered(&mut first), None); + assert_eq!(read_numbered(&mut second), None); + let fill = asked(&requests).expect("a flight"); + assert!(asked(&requests).is_none(), "the second joiner shares it"); + let mut flight = fill.accept(0, None).unwrap(); + // 8 MiB in, half the reach. + for index in 0..500 { + flight.write_frame(None, numbered(index)).unwrap(); + assert_eq!(read_numbered(&mut first), Some(index)); + } + let mut near = live.consume(); + assert_eq!(read_numbered(&mut near), Some(0), "boards the flight at once"); + assert_eq!(read_numbered(&mut second), Some(0)); + assert!(asked(&requests).is_none(), "one upstream fill"); + // 17 MiB in, past the reach. + for index in 500..1_100 { + flight.write_frame(None, numbered(index)).unwrap(); + assert_eq!(read_numbered(&mut first), Some(index)); + } + let mut far = live.consume(); + assert_eq!(read_numbered(&mut far), None, "too far behind the flight's rider"); + let own = asked(&requests).expect("a flight of its own"); + assert_eq!(own.start(), Position { group: 0, frame: 0 }); + } + + /// A flight sheds its front under pool debt like the copy it replays, so a pool far + /// smaller than one tail holds a live group and its flight near the pool. + #[test] + #[cfg(feature = "async")] + fn pool_debt_sheds_a_flight() { + const POOL: u64 = 4 * 1024 * 1024; + let pool = cache::Pool::new(cache::Config::default().with_capacity(POOL)); + let producer = Producer::new( + Arc::new(broadcast::Info { + pool: pool.clone(), + ..Default::default() + }), + "test", + Info::default(), + ); + let requests = producer.requests(); + let mut live = producer.append_group().unwrap(); + let mut late = live.consume(); + for index in 0..1_000 { + live.write_frame(None, numbered(index)).unwrap(); + } + assert_eq!(read_numbered(&mut late), None); + let mut flight = asked(&requests).expect("a flight").accept(0, None).unwrap(); + assert_eq!(read_numbered(&mut late), None); + for index in 0..1_000 { + flight.write_frame(None, numbered(index)).unwrap(); + assert_eq!(read_numbered(&mut late), Some(index)); + assert!(pool.used() < 2 * POOL, "frame {index}: {} bytes held", pool.used()); + } + } + + /// A reader waiting on the flight it asked for still goes stale: its budget gives up on + /// the group once a newer one supersedes it, though the handler never answered. + #[test] + #[cfg(feature = "async")] + fn a_reader_boarding_a_flight_goes_stale() { + let producer = track_producer("test", Info::timed()); + let requests = producer.requests(); + let mut sub = producer.subscribe(None); + let mut live = producer.append_group().unwrap(); + for index in 0..3_000u32 { + let at = Timestamp::from_millis(index.into()).unwrap(); + live.write_frame(at, numbered(index)).unwrap(); + } + let mut late = sub.recv_group().now_or_never().unwrap().unwrap().expect("group 0"); + assert!(late.read_frame().now_or_never().is_none(), "boarding"); + let _flight = asked(&requests).expect("a flight, never answered"); + let mut newer = producer.append_group().unwrap(); + newer + .write_frame(Timestamp::from_millis(100_000).unwrap(), numbered(0)) + .unwrap(); + assert!(matches!(late.read_frame().now_or_never(), Some(Err(Error::Old)))); + } + + /// Pool debt sheds a flight even once the cached copy of its group was evicted whole. + #[test] + #[cfg(feature = "async")] + fn pool_debt_sheds_a_flight_without_its_cached_group() { + const POOL: u64 = 4 * 1024 * 1024; + let pool = cache::Pool::new(cache::Config::default().with_capacity(POOL)); + let producer = Producer::new( + Arc::new(broadcast::Info { + pool: pool.clone(), + ..Default::default() + }), + "test", + Info::default(), + ); + let requests = producer.requests(); + let mut old = producer.append_group().unwrap(); + let mut late = old.consume(); + for index in 0..1_000 { + old.write_frame(None, numbered(index)).unwrap(); + } + assert_eq!(read_numbered(&mut late), None); + let mut flight = asked(&requests).expect("a flight").accept(0, None).unwrap(); + assert_eq!(read_numbered(&mut late), None); + // A newer group leaves group 0 cold, and its cached copy goes whole. + let mut newer = producer.append_group().unwrap(); + for index in 0..1_000 { + newer.write_frame(None, numbered(index)).unwrap(); + } + assert!(!producer.state.read().lookup.contains_key(&0), "group 0 evicted"); + for index in 0..1_000 { + flight.write_frame(None, numbered(index)).unwrap(); + assert!(pool.used() < 2 * POOL, "frame {index}: {} bytes held", pool.used()); + } + } + + /// A handler that refuses a flight leaves the reader where a track without one does: a + /// track that opts into refresh asks for a new group, and the read is `Evicted`. + #[test] + #[cfg(feature = "async")] + fn a_refused_flight_falls_back_to_refresh() { + let producer = track_producer("test", Info::default().with_refresh(1_000_000)); + let requests = producer.requests(); + // Joined before the group grew, so the join asks for nothing. + let _subscriber = producer.subscribe(None); + let mut live = producer.append_group().unwrap(); + let mut late = live.consume(); + for index in 0..3_000 { + live.write_frame(None, numbered(index)).unwrap(); + } + assert_eq!(read_numbered(&mut late), None, "asks a flight"); + assert_eq!(producer.demand().wanted(), Some(0), "nothing refreshed yet"); + asked(&requests).expect("a flight").refuse_groups(0..1, Error::NotFound); + assert!(matches!(late.read_frame().now_or_never().unwrap(), Err(Error::Evicted))); + assert_eq!(producer.demand().wanted(), Some(1), "the reader asked for a new group"); + } + + /// A rider that falls off its flight's tail never holds the flight's writer back, and + /// boards a flight of its own from the frame it fell off at. + #[test] + #[cfg(feature = "async")] + fn a_slow_rider_falls_off_and_flies_on_its_own() { + let (_producer, requests, live, _reader) = shed_live_group(3_000); + let (mut fast, mut slow) = (live.consume(), live.consume()); + assert_eq!(read_numbered(&mut fast), None); + assert_eq!(read_numbered(&mut slow), None); + let mut flight = asked(&requests).unwrap().accept(0, None).unwrap(); + assert_eq!(read_numbered(&mut fast), None); + assert_eq!(read_numbered(&mut slow), None); + assert_eq!(read_numbered(&mut slow), None); + for index in 0..2_500 { + let payload = numbered(index); + assert!( + flight + .poll_ready(&kio::Waiter::noop(), &frame_info(&payload)) + .is_ready(), + "frame {index}: paced by the fast rider only" + ); + flight.write_frame(None, payload).unwrap(); + assert_eq!(read_numbered(&mut fast), Some(index)); + } + assert_eq!(read_numbered(&mut slow), None, "fell off"); + let own = asked(&requests).expect("a flight of its own"); + assert_eq!(own.start(), Position { group: 0, frame: 0 }); + } + /// What a read returns right now: `Some(Some(sequence))` for a group, `Some(None)` for /// its end, `None` while it parks. fn read_now(subscriber: &mut Subscriber, ordered: bool) -> Option> { diff --git a/rs/moq-net/src/model/track/fill.rs b/rs/moq-net/src/model/track/fill.rs index c238259ecd3..24bef42df54 100644 --- a/rs/moq-net/src/model/track/fill.rs +++ b/rs/moq-net/src/model/track/fill.rs @@ -49,6 +49,9 @@ struct Pending { // What its joins ask with now, shared with them and its handler. asks: kio::Shared, queued: bool, + // A flight: a separate copy of one group from its start frame, which no other reader + // joins by sequence; see [`FillState::fly`]. + flight: bool, // How many joins still read each sequence: one a reader narrowed away is no longer owed. wants: RangeMap, // Readers hold consumers, so it counts the fill's demand; the handler's copy says what @@ -519,6 +522,7 @@ impl FillState { time: None, asks, queued: true, + flight: false, wants: RangeMap::from_iter([(groups.clone(), 1)]), result, }, @@ -528,6 +532,68 @@ impl FillState { (joins, unfillable) } + /// Join or queue a flight of group `sequence` from `frame`: a fill answered with a copy + /// of its own, which no reader joins by sequence, since a copy shed past its readers + /// holds nothing for one asking earlier. Readers asking from the same frame before it is + /// answered share it. `None` while no handler can take it. + pub(super) fn fly( + &mut self, + shared: &kio::Shared, + sequence: u64, + frame: u64, + asking: Asking, + ) -> Option { + let groups = sequence..sequence.checked_add(1)?; + if self.handlers == 0 { + return None; + } + let boarding = self.pending.iter_mut().find(|(_, pending)| { + pending.flight + && pending.start == Position { group: sequence, frame } + && !pending.result.read().resolved.contains(&sequence) + }); + let id = match boarding { + Some((id, pending)) => { + pending.want(groups.clone()); + pending.asks.lock().add(&asking); + *id + } + None => { + let id = self.next; + self.next += 1; + let mut asks = Asks::default(); + asks.add(&asking); + self.pending.insert( + id, + Pending { + start: Position { group: sequence, frame }, + end: groups.end, + frame_end: None, + time: None, + asks: kio::Shared::new(asks), + queued: true, + flight: true, + wants: RangeMap::from_iter([(groups.clone(), 1)]), + result: Default::default(), + }, + ); + self.queue.push_back(id); + id + } + }; + let pending = &self.pending[&id]; + Some(Join { + fills: shared.clone(), + id, + held: RangeSet::from_iter([groups.clone()]), + groups, + outcome: Some(pending.result.consume()), + attempt: pending.result.weak(), + asking, + asks: pending.asks.clone(), + }) + } + /// Queue a fill looking up `time` within `ask`'s run, with the join waiting on its answer. fn queue_lookup(&mut self, shared: &kio::Shared, ask: &Ask, time: Timestamp) -> Join { let id = self.next; @@ -555,6 +621,7 @@ impl FillState { time: Some(time), asks, queued: true, + flight: false, wants: RangeMap::new(), result, }, @@ -678,7 +745,13 @@ pub(super) fn poll_requested( }) { let id = guard.pop().expect("predicate guaranteed a fill"); let pending = guard.pending.get(&id).expect("popped fill is pending"); - let (start, end, frame_end, time) = (pending.start, pending.end, pending.frame_end, pending.time); + let (start, end, frame_end, time, flight) = ( + pending.start, + pending.end, + pending.frame_end, + pending.time, + pending.flight, + ); let (asks, result) = (pending.asks.clone(), pending.result.clone()); drop(guard); // Built with the fills unlocked, so no other lock nests under theirs. @@ -690,6 +763,7 @@ pub(super) fn poll_requested( end, frame_end, time, + flight, asks, result, }; @@ -753,6 +827,8 @@ struct Inner { // Exclusive frame cap in the last group; None asks through the original end. frame_end: Option, time: Option, + // Answered with a copy of its own rather than the cache's; see [`FillState::fly`]. + flight: bool, asks: kio::Shared, result: kio::Producer, } @@ -892,12 +968,14 @@ impl Fill { } }?; // The live feed may have delivered it since: answered by the cache, not the source. - let cached = self - .inner - .state - .read() - .covering_range(sequence, self.inner.bounds(sequence)) - .is_some(); + // Never a flight's, which the cached copy shed. + let cached = !self.inner.flight + && self + .inner + .state + .read() + .covering_range(sequence, self.inner.bounds(sequence)) + .is_some(); if !cached { return Some(sequence); } @@ -1133,7 +1211,10 @@ impl Inner { } // Cache the group before answering: readers resolve an accept through the cache. let res = TrackState::modify(&self.state).and_then(|mut state| { - let res = state.insert_group_request(sequence, frames, info); + let res = match self.flight { + true => state.insert_flight(sequence, frames, info), + false => state.insert_group_request(sequence, frames, info), + }; let deferred = state.deferred(); drop(state); deferred.wake(); @@ -1142,7 +1223,7 @@ impl Inner { if let Ok(mut outcome) = self.result.write() { outcome.accept(sequence..end); if let Ok(group) = &res - && (frames.start > 0 || frames.end.is_some()) + && (self.flight || frames.start > 0 || frames.end.is_some()) { outcome.partial.insert(sequence, group.handle()); } diff --git a/rs/moq-net/tests/integration/flights.rs b/rs/moq-net/tests/integration/flights.rs new file mode 100644 index 00000000000..b51236a7ce0 --- /dev/null +++ b/rs/moq-net/tests/integration/flights.rs @@ -0,0 +1,315 @@ +#![cfg(feature = "async")] + +//! A late joiner of a live group far past its 32 MiB tail replays it from frame 0 through +//! a flight: each relay in the chain asks the hop above for the group from the frame it +//! shed, and the origin's handler serves it. The joiner hands over to the live copy once it +//! catches up, and every flight in the chain then stops. + +use std::sync::{ + Arc, + atomic::{AtomicU64, Ordering}, +}; +use std::task::Poll; +use std::time::Duration; + +use bytes::Bytes; +use moq_net::{Hop, Version, cache, frame, group, origin, track}; + +use crate::support::harness::{MockConnectOptions, connect_mock, spawn}; + +const TIMEOUT: Duration = Duration::from_secs(10); + +/// 48 MiB of 16 KiB frames: half again the live tail. +const FRAMES: u32 = 3_000; + +const VERSIONS: &[&str] = &["moq-lite-05", "moq-lite-06", "moq-lite-07-wip"]; + +/// Frame `index`: 16 KiB naming its index. +fn numbered(index: u32) -> Bytes { + Bytes::from(index.to_le_bytes().repeat(4096)) +} + +fn number(frame: &frame::Frame) -> u32 { + u32::from_le_bytes(frame.payload[..4].try_into().unwrap()) +} + +fn info(payload: &Bytes) -> frame::Info { + frame::Info { + size: payload.len() as u64, + timestamp: None, + } +} + +fn origin(hop: u64, pool: &cache::Pool) -> origin::Producer { + let mut config = origin::Config::new(Hop::new(hop).unwrap()); + config.pool = pool.clone(); + let (origin, driver) = origin::Producer::new(config); + spawn(driver); + origin +} + +/// Write `payload` once `group`'s fastest reader leaves room, or return `false` once no +/// reader waits on it: its fill gave it up and nobody reads it. Without a payload, the +/// archive has nothing newer yet, so only wait for that. +async fn write_paced( + group: &mut group::Producer, + fill: &track::Fill, + demand: &mut group::Demand, + payload: Option, +) -> bool { + let sequence = group.sequence; + let room = kio::wait(|waiter| { + while let Poll::Ready(Ok(_)) = demand.poll_wanted_changed(waiter) {} + if fill.poll_unwanted(sequence, waiter).is_ready() && !demand.wanted() { + return Poll::Ready(false); + } + match &payload { + Some(payload) => group.poll_ready(waiter, &info(payload)).map(|()| true), + None => Poll::Pending, + } + }) + .await; + room && payload.is_some_and(|payload| group.write_frame(None, payload).is_ok()) +} + +/// The origin's handler: serves each fill from the frame it starts at, paced by its riders, +/// counting the frames it writes and the fills it ends. With `finish`, the archive holds the +/// whole group, so each fill finishes after its last frame. +fn serve(requests: track::Requests, written: Arc, ended: Arc, finish: bool) { + drop(kio::clock::sim::spawn(async move { + while let Ok(fill) = requests.requested_fill().await { + let start = fill.start(); + let mut group = fill.accept(start.group, None).expect("accept"); + let mut demand = group.demand(); + let (written, ended) = (written.clone(), ended.clone()); + drop(kio::clock::sim::spawn(async move { + for index in start.frame as u32.. { + let payload = (index < FRAMES).then(|| numbered(index)); + if payload.is_none() && finish { + group.finish().expect("finish"); + break; + } + if !write_paced(&mut group, &fill, &mut demand, payload).await { + break; + } + written.fetch_add(1, Ordering::Relaxed); + } + ended.fetch_add(1, Ordering::Relaxed); + })); + } + })); +} + +/// Read the next frame of `group`, failing on anything else. +async fn next(group: &mut group::Consumer, version: &str, index: u32) -> u32 { + let frame = kio::clock::timeout(TIMEOUT, group.read_frame()) + .await + .unwrap_or_else(|_| panic!("{version}: frame {index} never arrived")) + .unwrap_or_else(|err| panic!("{version}: frame {index}: {err}")) + .unwrap_or_else(|| panic!("{version}: the group ended before frame {index}")); + number(&frame) +} + +/// Whether `group` ends cleanly at its next read. +async fn next_end(group: &mut group::Consumer, version: &str) -> bool { + kio::clock::timeout(TIMEOUT, group.read_frame()) + .await + .unwrap_or_else(|_| panic!("{version}: the group never ended")) + .unwrap_or_else(|err| panic!("{version}: the end: {err}")) + .is_none() +} + +#[kio::test] +async fn a_late_joiner_replays_through_a_relay_chain() { + for version in VERSIONS { + chain(version, Replay::Live).await; + } +} + +/// A moq-transport hop refuses a FETCH reaching past the live edge of a live group, as a +/// flight's whole-group FETCH does, before reading any of it: the late joiner is refused +/// rather than replayed, and no flight reaches the origin. +#[kio::test] +async fn moq_transport_refuses_a_replay_of_a_live_group() { + chain("moq-transport-22", Replay::Refused).await; +} + +/// A moq-transport hop answers a FETCH of a finished group before reading it, then streams +/// its objects, so a group far past one tail replays through each hop's flight. +#[kio::test] +async fn moq_transport_replays_a_finished_group() { + chain("moq-transport-22", Replay::Finished).await; +} + +/// A moq-transport relay holding none of a finished group, far past its ingest window, +/// answers a FETCH of it once its own copy finishes arriving, which nothing reads until +/// FETCH_OK, then streams it, riding a flight for the front its copy shed. +#[kio::test] +async fn moq_transport_relays_a_cold_finished_group() { + let version = "moq-transport-22"; + let version_parsed: Version = version.parse().unwrap(); + let pools: [cache::Pool; 3] = Default::default(); + let publisher = origin(1, &pools[0]); + let upper = origin(2, &pools[1]); + let lower = origin(3, &pools[2]); + + let broadcast = publisher.create_broadcast("live").unwrap(); + let track = broadcast.create_track("data", track::Info::default()).unwrap(); + broadcast.announce(Default::default()).unwrap(); + let (written, ended) = (Arc::new(AtomicU64::new(0)), Arc::new(AtomicU64::new(0))); + serve(track.requests(), written, ended, true); + let mut group = track.append_group().unwrap(); + for index in 0..FRAMES { + group.write_frame(None, numbered(index)).unwrap(); + } + group.finish().unwrap(); + + let mut options = MockConnectOptions::new(version_parsed); + options.server_publish = Some(publisher.consume()); + options.client_subscribe = Some(upper.clone()); + let _upper_link = connect_mock(options).await; + let mut options = MockConnectOptions::new(version_parsed); + options.server_publish = Some(upper.consume()); + options.client_subscribe = Some(lower.clone()); + let _lower_link = connect_mock(options).await; + + let remote = kio::clock::timeout(TIMEOUT, lower.consume().follow("live", None).unwrap().available()) + .await + .expect("resolve timeout") + .expect("broadcast resolves") + .expect("an instance"); + let mut fetched = kio::clock::timeout(TIMEOUT, remote.track("data").unwrap().fetch_group(0, None)) + .await + .expect("fetch timeout") + .expect("fetch"); + for index in 0..FRAMES { + assert_eq!(next(&mut fetched, version, index).await, index, "{version}: fetched"); + } + assert!( + next_end(&mut fetched, version).await, + "{version}: the fetch ends with the group" + ); +} + +/// What a late joiner of group 0 finds at the bottom of the chain. +#[derive(Clone, Copy, PartialEq)] +enum Replay { + /// The live group, replayed from frame 0 until the joiner hands over to the live copy. + Live, + /// The finished group, replayed from frame 0 to its end. + Finished, + /// A refusal. + Refused, +} + +/// Join group 0 late at the bottom of a two-relay chain, expecting `replay`. +async fn chain(version: &str, replay: Replay) { + let version_parsed: Version = version.parse().unwrap(); + let pools: [cache::Pool; 3] = Default::default(); + let publisher = origin(1, &pools[0]); + let upper = origin(2, &pools[1]); + let lower = origin(3, &pools[2]); + + let broadcast = publisher.create_broadcast("live").unwrap(); + let track = broadcast.create_track("data", track::Info::default()).unwrap(); + broadcast.announce(Default::default()).unwrap(); + let (written, ended) = (Arc::new(AtomicU64::new(0)), Arc::new(AtomicU64::new(0))); + serve( + track.requests(), + written.clone(), + ended.clone(), + replay == Replay::Finished, + ); + + let mut options = MockConnectOptions::new(version_parsed); + options.server_publish = Some(publisher.consume()); + options.client_subscribe = Some(upper.clone()); + let _upper_link = connect_mock(options).await; + let mut options = MockConnectOptions::new(version_parsed); + options.server_publish = Some(upper.consume()); + options.client_subscribe = Some(lower.clone()); + let _lower_link = connect_mock(options).await; + + let remote = kio::clock::timeout(TIMEOUT, lower.consume().follow("live", None).unwrap().available()) + .await + .expect("resolve timeout") + .expect("broadcast resolves") + .expect("an instance"); + let subscription = || { + track::Subscription::default() + .with_range(0..) + .with_delay(track::Delay::default().with_max(Duration::from_secs(3600))) + }; + let mut live_sub = kio::clock::timeout(TIMEOUT, remote.track("data").unwrap().subscribe(subscription())) + .await + .expect("subscribe timeout") + .expect("subscribe"); + + // The live reader at the bottom of the chain keeps up with the publisher. + let mut live = track.append_group().unwrap(); + live.write_frame(None, numbered(0)).unwrap(); + let mut reader = kio::clock::timeout(TIMEOUT, live_sub.recv_group()) + .await + .expect("group timeout") + .unwrap() + .expect("group 0"); + assert_eq!(next(&mut reader, version, 0).await, 0); + for index in 1..FRAMES { + kio::wait(|waiter| live.poll_ready(waiter, &info(&numbered(index)))).await; + live.write_frame(None, numbered(index)).unwrap(); + assert_eq!(next(&mut reader, version, index).await, index, "{version}: live"); + } + + if replay == Replay::Finished { + live.finish().unwrap(); + assert!( + next_end(&mut reader, version).await, + "{version}: the live reader sees the end" + ); + } + + // A late joiner at the bottom: every copy in the chain shed frame 0. + let mut late_sub = kio::clock::timeout(TIMEOUT, remote.track("data").unwrap().subscribe(subscription())) + .await + .expect("subscribe timeout") + .expect("subscribe"); + let mut late = kio::clock::timeout(TIMEOUT, late_sub.recv_group()) + .await + .expect("group timeout") + .unwrap() + .expect("group 0"); + if replay == Replay::Refused { + let refused = kio::clock::timeout(TIMEOUT, late.read_frame()) + .await + .unwrap_or_else(|_| panic!("{version}: never answered")); + assert!(refused.is_err(), "{version}: {refused:?}"); + kio::clock::sleep(Duration::from_secs(1)).await; + assert_eq!(written.load(Ordering::Relaxed), 0, "{version}: no flight at the origin"); + return; + } + let mut held = 0; + for index in 0..FRAMES { + assert_eq!(next(&mut late, version, index).await, index, "{version}: replayed"); + held = pools.iter().map(cache::Pool::used).max().unwrap().max(held); + } + + if replay == Replay::Finished { + assert!( + next_end(&mut late, version).await, + "{version}: the replay ends with the group" + ); + } + + // The joiner left its flight for the live copy before the flights fetched the whole + // group, and every flight in the chain stopped. A finished group is short enough that + // its flight, a window ahead of the handover, reaches its end. + let fetched = written.load(Ordering::Relaxed); + assert!( + replay == Replay::Finished || fetched < u64::from(FRAMES), + "{version}: the origin wrote {fetched} frames" + ); + kio::clock::sleep(Duration::from_secs(1)).await; + assert_eq!(ended.load(Ordering::Relaxed), 1, "{version}: one flight, stopped"); + // A live copy and one flight per hop, each held to its 32 MiB tail. + assert!(held < 80 << 20, "{version}: a hop held {} MiB", held >> 20); +} diff --git a/rs/moq-net/tests/integration/main.rs b/rs/moq-net/tests/integration/main.rs index fa361127848..78b2f3a2b22 100644 --- a/rs/moq-net/tests/integration/main.rs +++ b/rs/moq-net/tests/integration/main.rs @@ -31,6 +31,7 @@ mod fetch_only; mod fetch_pipeline; mod finished_broadcast_mock; mod finite_relay; +mod flights; mod goaway; mod group_order; mod history_groups; diff --git a/rs/moq-net/tests/integration/session_idle.rs b/rs/moq-net/tests/integration/session_idle.rs index 0d200234a21..7e798485a75 100644 --- a/rs/moq-net/tests/integration/session_idle.rs +++ b/rs/moq-net/tests/integration/session_idle.rs @@ -113,9 +113,10 @@ async fn a_subscription_keeps_both_ends_busy() { } /// A fetch keeps both ends busy until it ends: here, until the group it reads finishes. +/// moq-transport refuses a FETCH of the whole live group rather than waiting on it. #[kio::test] async fn a_fetch_keeps_both_ends_busy() { - for version in ["moq-lite-06", "moq-lite-07-wip", "moq-transport-22"] { + for version in ["moq-lite-06", "moq-lite-07-wip"] { let Setup { pair, track,