diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 3f475ef672b..761e64a9f00 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -19,6 +19,7 @@ author: normative: moqt: I-D.ietf-moq-transport qmux: I-D.ietf-quic-qmux + reset-stream-at: I-D.ietf-quic-reliable-stream-reset qmuxws: title: "QMux over WebSocket" target: https://datatracker.ietf.org/doc/draft-lcurley-qmux-websocket/ @@ -714,6 +715,7 @@ A subscriber assembling a Group from more than one publisher does so across sepa Both the publisher and subscriber MAY reset the stream at any time. This is not a fatal error and the session remains active. +A publisher that declared [Reliable Reset](#reliable-reset-parameter) resets a Group Stream with RESET_STREAM_AT, so the GROUP message still arrives and the subscriber can account for the stream. The subscriber MAY cache the error and potentially retry later. ## Datagrams @@ -840,6 +842,8 @@ The following Setup Parameters are defined: |------|-----------|-------------| | 0x5 | Hop | Hop ID (i) | |------|-----------|-------------| +| 0x6 | Reliable Reset | Flag (i) | +|------|-----------|-------------| ### Probe Parameter {#probe-parameter} The Probe Parameter advertises the sender's capability level when acting as a publisher on a [Probe Stream](#probe). @@ -901,6 +905,21 @@ Declaring it at setup gives the receiver the peer's identity before any other st Either endpoint MAY send it; a subscriber-only endpoint with no identity MAY omit it, but a publisher SHOULD have a Hop ID regardless (see [ANNOUNCE_OK](#announce-ok)). A relay MUST NOT forward it. +### Reliable Reset Parameter {#reliable-reset-parameter} +The Reliable Reset Parameter declares that every Group Stream the sender counts in a [SUBSCRIBE_END](#subscribe-end) `Stream Count` delivers its GROUP message, even when the sender resets it. +The Parameter Value is a variable-length integer: `1` declares it, and `0` is equivalent to omitting the parameter. +A receiver MUST close the session with a PROTOCOL_VIOLATION on any other value. + +A sender that declares it MUST count a Group Stream only once its GROUP message is written, and MUST reset a counted Group Stream with RESET_STREAM_AT [reset-stream-at], giving a Reliable Size that covers the GROUP message. +A Group Stream reset before its GROUP message is written is not counted. +It MUST NOT declare it unless the peer negotiated the reset_stream_at transport parameter, so it applies only to bindings 1 and 2, and only on a version whose SUBSCRIBE_END carries a `Stream Count`. +An endpoint whose transport API cannot reset a stream that way does not declare it. + +A subscriber whose publisher declared it waits for every counted Group Stream without a bound, since each is sure to arrive; see [SUBSCRIBE_END](#subscribe-end). +A lost datagram is not counted and does not change that. + +Either endpoint MAY send it, and the two directions are independent. A relay MUST NOT forward it: each hop declares what its own transport does. + ## ANNOUNCE_REQUEST {#announce-request} A subscriber sends an ANNOUNCE_REQUEST message to indicate it wants to receive announcements for any broadcasts with a path that starts with the requested prefix. @@ -1331,6 +1350,7 @@ A relay counts the Group Streams it opened itself, never the count it received u The publisher MUST NOT send SUBSCRIBE_END until every Group Stream it will open for the subscription has been opened, so the count is final; it does not wait for them to finish. The subscriber has received every Group Stream once it has read the header of `Stream Count` of them, which MAY happen after SUBSCRIBE_END or the FIN since streams are not ordered. A Group Stream reset before its header arrived is never seen, so a subscriber SHOULD bound how long it waits for the rest, for example by `Subscriber Max Age`. +When the publisher declared [Reliable Reset](#reliable-reset-parameter), every counted Group Stream delivers its header, so the subscriber SHOULD NOT bound the wait. ## FETCH FETCH is sent by a subscriber to request a single group from a track. @@ -1600,6 +1620,7 @@ The `Message Length` describes the payload size on the wire. - Hid routes with a `.`-prefixed segment below the requested prefix from announce discovery, and added the ANNOUNCE_REQUEST `Hidden` field to opt in. - Added `Stream Count` to SUBSCRIBE_END: the number of Group Streams opened for the subscription. SUBSCRIBE_END is now sent once every counted Group Stream has opened, rather than as soon as the final group is known. - Removed SUBSCRIBE_DROP and its type 0x2; a group without a Group Stream is not counted. +- Added the SETUP `Reliable Reset` parameter (0x6): the sender counts a Group Stream only once its GROUP message is written and resets it with RESET_STREAM_AT keeping that message, so a subscriber waits for every counted stream without a bound. - The Subscribe Stream FIN now follows once every counted Group Stream has finished or been reset. - Added the Spread Hash tie-break after the shortest path: a hash of the requested path and the route's Hop IDs, so equal-cost advertisers of one prefix share its paths. - Added a local link preference to route selection, after the newest Epoch and before the 0 Hop ID check: the higher preference a subscriber assigns to the arriving link wins, and an unassigned link is neutral. Not carried on the wire. diff --git a/js/net/src/lite/setup.test.ts b/js/net/src/lite/setup.test.ts index 3b26a8b76ca..a543bd9dcc3 100644 --- a/js/net/src/lite/setup.test.ts +++ b/js/net/src/lite/setup.test.ts @@ -11,6 +11,7 @@ const PARAM_PROBE = 0x1n; const PARAM_PATH = 0x2n; const PARAM_ROLE = 0x3n; const PARAM_HOP = 0x5n; +const PARAM_RELIABLE_RESET = 0x6n; function concat(chunks: Uint8Array[]): Uint8Array { const total = chunks.reduce((sum, c) => sum + c.byteLength, 0); @@ -147,6 +148,14 @@ test("hop 0 decodes as absent", async () => { expect(got.hop).toBeUndefined(); }); +test("reliable reset round-trips as a 0/1 flag", async () => { + expect((await roundTrip(new Setup({ reliableReset: true }))).reliableReset).toBe(true); + expect((await roundTrip(new Setup())).reliableReset).toBe(false); + expect((await decodeParam(PARAM_RELIABLE_RESET, Varint.encode(0))).reliableReset).toBe(false); + expect((await decodeParam(PARAM_RELIABLE_RESET, Varint.encode(1))).reliableReset).toBe(true); + await expect(decodeParam(PARAM_RELIABLE_RESET, Varint.encode(2))).rejects.toThrow(); +}); + test("SETUP is rejected before draft-05", async () => { await expect(bytes((w) => new Setup().encode(w, Version.DRAFT_04), Version.DRAFT_04)).rejects.toThrow(); }); diff --git a/js/net/src/lite/setup.ts b/js/net/src/lite/setup.ts index 1cea1263d67..b57f40998f0 100644 --- a/js/net/src/lite/setup.ts +++ b/js/net/src/lite/setup.ts @@ -26,6 +26,8 @@ const PARAM_PATH = 0x2n; const PARAM_ROLE = 0x3n; /** Setup Parameter id for the endpoint's Hop ID. */ const PARAM_HOP = 0x5n; +/** Setup Parameter id declaring that the endpoint's group resets keep their header. */ +const PARAM_RELIABLE_RESET = 0x6n; /** Cap on the number of parameters in a bag, matching the Rust decoder. */ const MAX_PARAMS = 64; @@ -182,6 +184,8 @@ export interface SetupProps { role?: Role; /** See {@link Setup.hop}. Omitted by default. */ hop?: Hop; + /** See {@link Setup.reliableReset}. Defaults to `false`. */ + reliableReset?: boolean; } /** @@ -219,11 +223,20 @@ export class Setup { */ hop?: Hop; - constructor({ probe, path, role, hop }: SetupProps = {}) { + /** + * Whether this endpoint counts a group stream only once its GROUP message is written, + * and delivers that message even when it resets the stream (lite-07+). The peer then + * waits for each counted stream rather than giving up after a grace. A browser cannot + * reset that way, so `@moq/net` never declares it. + */ + reliableReset: boolean; + + constructor({ probe, path, role, hop, reliableReset }: SetupProps = {}) { this.probe = probe ?? ProbeLevel.None; this.path = path; this.role = role ?? Role.Both; this.hop = hop; + this.reliableReset = reliableReset ?? false; } static #guard(version: Version) { @@ -248,6 +261,9 @@ export class Setup { if (this.hop !== undefined && this.hop !== 0n) { params.setVarint(PARAM_HOP, this.hop, version); } + if (this.reliableReset) { + params.setVarint(PARAM_RELIABLE_RESET, 1n, version); + } await params.encode(w); } @@ -269,7 +285,13 @@ export class Setup { const hopRaw = await params.getVarint(PARAM_HOP, version); const hop = hopRaw === undefined || hopRaw === 0n ? undefined : HopSchema.parse(hopRaw); - const setup = new Setup({ probe, path, role, hop }); + const reliableRaw = await params.getVarint(PARAM_RELIABLE_RESET, version); + if (reliableRaw !== undefined && reliableRaw > 1n) { + throw new Error(`invalid reliable reset value: ${reliableRaw}`); + } + const reliableReset = reliableRaw === 1n; + + const setup = new Setup({ probe, path, role, hop, reliableReset }); if (pathBytes !== undefined || roleCode !== undefined) clientParameters.add(setup); return setup; } diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 09dc62b08c8..53d88b243f0 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -1192,7 +1192,8 @@ export class Subscriber { // // lite-07 counts streams, so skipped sequences owe nothing. Older drafts account for // the range using received headers and SUBSCRIBE_DROP. A counted stream reset before - // its header leaves no trace, so the grace still bounds that wait. Streams whose + // its header leaves no trace, so the grace still bounds that wait, unless the publisher + // declared reliable reset: then every counted stream delivers its header. Streams whose // headers arrived keep reading until their own FIN or reset. #settleTail(entry: SubscribeEntry): Promise { const { tail, track } = entry; @@ -1213,7 +1214,8 @@ export class Subscriber { return tail.covers(start, end); }; - return tail.settle(complete, track.closed); + const exact = entry.streams !== undefined && this.#peerSetup?.peek()?.reliableReset === true; + return tail.settle(complete, track.closed, { exact }); } /** diff --git a/js/net/src/lite/tail.test.ts b/js/net/src/lite/tail.test.ts index 7e717210342..0c478078c2e 100644 --- a/js/net/src/lite/tail.test.ts +++ b/js/net/src/lite/tail.test.ts @@ -1,5 +1,6 @@ import { describe, expect, test } from "bun:test"; import { Milli } from "@moq/clock"; +import { Signal } from "@moq/signals"; import { ProtocolViolation, StreamCode, StreamError } from "../error.ts"; import type { Consumer as GroupConsumer } from "../group.ts"; import { randomHop } from "../hop.ts"; @@ -9,6 +10,7 @@ import { Reader, Stream } from "../stream.ts"; import { Timescale } from "../time.ts"; import type { Groups } from "../track.ts"; import { Group as GroupMessage } from "./group.ts"; +import { Setup } from "./setup.ts"; import { StreamId } from "./stream.ts"; import { encodeSubscribeResponse, @@ -53,9 +55,15 @@ function groupStream(subscriber: Subscriber, sequence: number) { * it answers TRACK_INFO, then writes whatever responses the test asks for on the subscribe * stream and FINs it when told. */ -async function subscribed(version: Version, maxDelay = GRACE, groups?: Groups) { +async function subscribed(version: Version, maxDelay = GRACE, groups?: Groups, peerSetup?: Setup) { const pair = createMockTransportPair(ALPN_05); - const subscriber = new Subscriber(pair.client, version, randomHop()); + const subscriber = new Subscriber( + pair.client, + version, + randomHop(), + undefined, + new Signal(peerSetup), + ); const reader = subscriber.consume(Path.from("room")).track("video").subscribe({ maxDelay, groups }); const info = await Stream.accept(pair.server, version); @@ -92,7 +100,7 @@ async function readAll(group: GroupConsumer | undefined): Promise { } /** Whether `promise` settles within `ms`. */ -async function settlesWithin(promise: Promise, ms: number): Promise { +async function settlesWithin(promise: PromiseLike, ms: number): Promise { let timer: ReturnType | undefined; const pending = new Promise((resolve) => { timer = setTimeout(() => resolve(false), ms); @@ -363,6 +371,41 @@ test("lite-07 zero count ends without grace even when the announced range is non expect(reader.end()).toEqual({ group: 3, frame: 0 }); }); +// A publisher that declared reliable reset delivers the header of every stream it counts, so +// a missing one is still on its way: the subscription waits past the grace for it. +for (const reliableReset of [false, true]) { + test(`lite-07 ${reliableReset ? "waits past" : "gives up after"} the grace for a counted stream with reliable reset ${reliableReset ? "declared" : "undeclared"}`, async () => { + const peer = new Setup({ reliableReset }); + const { subscriber, reader, respond, fin } = await subscribed(Version.DRAFT_07, GRACE, undefined, peer); + await respond({ start: new SubscribeStart(0) }); + await respond({ end: new SubscribeEnd(1, 1) }); + await fin(); + + const ended = settlesWithin(reader.closed, GRACE * 3); + expect(await ended).toBe(!reliableReset); + if (!reliableReset) return; + + const late = groupStream(subscriber, 0); + late.write("late"); + late.finish(); + await late.handled; + expect(await readAll(await reader.recvGroup())).toEqual(["late"]); + expect(await settlesWithin(reader.recvGroup(), 1000)).toBe(true); + expect(await reader.closed).toBeNull(); + }); +} + +// The declaration only drops the grace where SUBSCRIBE_END counts streams; older drafts +// cannot tell a datagram hole from a stream in flight. +test("lite-06 keeps the grace even when the publisher declared reliable reset", async () => { + const peer = new Setup({ reliableReset: true }); + const { reader, respond, fin } = await subscribed(Version.DRAFT_06, GRACE, undefined, peer); + await respond({ start: new SubscribeStart(0) }); + await respond({ end: new SubscribeEnd(1) }); + await fin(); + expect(await settlesWithin(reader.closed, GRACE * 3)).toBe(true); +}); + for (const started of [false, true]) { test(`a bare FIN ${started ? "after SUBSCRIBE_START" : "without responses"} aborts the track`, async () => { const { reader, respond, fin } = await subscribed(Version.DRAFT_05); diff --git a/js/net/src/tail.ts b/js/net/src/tail.ts index 61a9c2d852c..2cf4a3b6b08 100644 --- a/js/net/src/tail.ts +++ b/js/net/src/tail.ts @@ -30,7 +30,8 @@ export interface TailOptions { * waits on them. * * A lost datagram is not owed: it leaves a hole that waits out the grace like a stream reset - * before its header. + * before its header. A publisher whose resets keep the header (reliable reset) needs no + * grace: on lite-07 every stream it counts arrives. * * @internal */ @@ -103,21 +104,23 @@ export class Tail { * * A stream still being read is always waited for: a group ends on its own stream's FIN * or reset, never because its track ended. The grace only gives up on streams that never - * arrived. + * arrived, and `exact` skips it for a publisher that promised every owed stream arrives. */ - async settle(complete: () => boolean, closed: GetPromise): Promise { + async settle(complete: () => boolean, closed: GetPromise, { exact = false } = {}): Promise { // A gap that aged past the grace since the last insert is no longer waited for either. this.#expire(now()); let expired = false; const timer = new AbortController(); - void sleep(Nano.fromMilli(this.#grace()), { signal: timer.signal }).then( - () => { - expired = true; - this.#bump(); - }, - () => {}, - ); + if (!exact) { + void sleep(Nano.fromMilli(this.#grace()), { signal: timer.signal }).then( + () => { + expired = true; + this.#bump(); + }, + () => {}, + ); + } try { while (closed.peek() === undefined) { if (this.#active === 0 && (expired || complete())) return; diff --git a/rs/moq-net/src/client.rs b/rs/moq-net/src/client.rs index 395fde91f38..6e2c93b7044 100644 --- a/rs/moq-net/src/client.rs +++ b/rs/moq-net/src/client.rs @@ -200,6 +200,7 @@ impl Client { cost: self.cost, // Filled by `lite::start` from the attached origin handles. hop: None, + reliable_reset: lite::Setup::detect_reliable_reset(&session, version), } } else { lite::Setup::default() diff --git a/rs/moq-net/src/coding/writer.rs b/rs/moq-net/src/coding/writer.rs index 037a095676c..406ee898f03 100644 --- a/rs/moq-net/src/coding/writer.rs +++ b/rs/moq-net/src/coding/writer.rs @@ -211,6 +211,13 @@ impl Writer { self.stream.as_mut().unwrap().set_priority(send_order.into()); } + /// Mark what has flushed so far for a later reset to still deliver, where the session + /// supports it. See [`crate::transport::poll::SendStream::checkpoint`]. + pub fn checkpoint(&mut self) { + debug_assert_eq!(self.flushed, self.buffer.len(), "checkpoint with unflushed bytes"); + self.stream.as_mut().unwrap().checkpoint(); + } + /// Cast the writer to a different version, used during version negotiation. pub fn with_version(mut self, version: O) -> Writer { Writer { diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 75a6cb6fa70..b8910b82504 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -3156,8 +3156,10 @@ struct Subscription { /// Counts a subscription's group streams for lite-07's SUBSCRIBE_END. /// -/// A group is pending from the moment it is queued until its stream opens, or it gives -/// up first (expired, or the open failed) and is never counted. +/// A group is pending from the moment it is queued until its GROUP message is written to +/// its stream, or it gives up first (expired, cancelled, or the open failed) and is never +/// counted. Only a written header is checkpointed, so on a session that declared reliable +/// resets every counted stream reaches the subscriber. /// /// `unencodable` rides along because every group machine and the run loop already share /// this handle. A header that cannot be encoded writes nothing, so the group-stream @@ -3169,6 +3171,16 @@ struct Opens { unencodable: Mutex>, } +impl Opens { + /// Stop waiting on a pending group, counting its stream if its header was written. + fn settle(&self, opened: bool) { + self.pending.fetch_sub(1, Ordering::Relaxed); + if opened { + self.opened.fetch_add(1, Ordering::Relaxed); + } + } +} + impl Subscription { /// Send one datagram best-effort over a QUIC datagram (lite-05 ยง6.4). /// @@ -3649,6 +3661,8 @@ enum GroupState { /// then the pending frame, then the next frame. Serve { writer: Writer, + /// The GROUP message is still in the write buffer, so the stream is not counted. + header: bool, frame: Option, chunk: Option, batch: Box, @@ -3683,14 +3697,6 @@ impl GroupServe { } } - /// Leave [`GroupState::Open`], counting the stream if it opened. - fn settle_open(&mut self, opened: bool) { - self.ctx.opens.pending.fetch_sub(1, Ordering::Relaxed); - if opened { - self.ctx.opens.opened.fetch_add(1, Ordering::Relaxed); - } - } - /// Serve the group, aborting the stream with the real reason (Old, Lagged, /// Evicted, ...) on failure so the subscriber can tell a truncated group from /// a routine cancel. Without this the Writer's Drop fallback would report @@ -3723,7 +3729,7 @@ impl GroupServe { match &mut self.state { GroupState::Open => { if self.group.poll_expired(waiter) { - self.settle_open(false); + self.ctx.opens.settle(false); self.state = GroupState::Done; return Poll::Ready(Err(Error::Old)); } @@ -3731,12 +3737,11 @@ impl GroupServe { let stream = match ready!(self.ctx.session.poll_open_uni(&mut cx)) { Ok(stream) => stream, Err(err) => { - self.settle_open(false); + self.ctx.opens.settle(false); self.state = GroupState::Done; return Poll::Ready(Err(crate::lite::error::from_transport(err))); } }; - self.settle_open(true); let mut writer = Writer::new(stream, self.ctx.version); writer.set_priority(self.priority.send_order()); @@ -3750,12 +3755,14 @@ impl GroupServe { // it to this subscription. The run loop reads the slot and resets the // subscribe stream instead. self.ctx.note_unencodable(&err); + self.ctx.opens.settle(false); self.state = GroupState::Done; writer.abort(&err); return Poll::Ready(Err(err)); } self.state = GroupState::Serve { writer, + header: true, frame: None, chunk: None, batch: Box::new(frame::Buffer::new()), @@ -3764,6 +3771,7 @@ impl GroupServe { } GroupState::Serve { writer, + header, frame, chunk, batch, @@ -3779,6 +3787,13 @@ impl GroupServe { loop { ready!(self.budget.poll_yield(waiter)); match writer.poll_flush(&mut cx) { + // The header is the first flush, so a reset from here on still + // tells the subscriber which group it was. + Poll::Ready(Ok(())) if *header => { + *header = false; + writer.checkpoint(); + self.ctx.opens.settle(true); + } Poll::Ready(Ok(())) => {} Poll::Ready(Err(err)) => break 'serve Err(err), // Parking on the transport is the one stall the group cursor cannot @@ -3871,9 +3886,13 @@ impl GroupServe { } }; - let GroupState::Serve { writer, .. } = std::mem::replace(&mut self.state, GroupState::Done) else { + let GroupState::Serve { writer, header, .. } = std::mem::replace(&mut self.state, GroupState::Done) + else { unreachable!() }; + if header { + self.ctx.opens.settle(false); + } match outcome { Ok(()) => { let mut writer = writer; @@ -4099,6 +4118,63 @@ mod serve_group_test { assert!(matches!(serve.await, Err(Error::Old))); assert_eq!(log.resets(), vec![crate::StreamError::Old.to_code()]); + + // The checkpoint sits right after the header, so the reset keeps it and drops the + // frame behind it. + let trail: Vec<_> = log.trail().into_iter().map(|(_, sent)| sent).collect(); + let [Sent::Write(_), Sent::Checkpoint, .., Sent::Reset(_)] = trail.as_slice() else { + panic!("{trail:?}"); + }; + } + + /// A group stream counts once its header is written, which is also what its checkpoint + /// keeps. One that gives up before then is not counted, since no reset could deliver it. + #[moq_net_sim::test] + async fn a_stream_counts_once_its_header_is_written() { + for open in [true, false] { + let gate = kio::Producer::new(open); + let session = SinkSession::gated_uni(gate.consume()); + let log = session.log.clone(); + let steer = kio::Producer::new(Steer::default()); + let opens = Arc::new(Opens::default()); + let subscription = Subscription { + session, + id: 0, + track_name: "test".into(), + priority: PriorityQueue::default(), + steer: steer.consume(), + steer_seen: Steer::default(), + version: Version::Lite07, + timescale: Some(crate::Timescale::MILLI), + runtime: crate::time::Clock::sim(), + opens: opens.clone(), + }; + + let track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + let mut subscriber = track.subscribe(None); + let mut old = track.append_group().unwrap(); + old.write_frame(Timestamp::ZERO, b"old".as_slice()).unwrap(); + let group = subscriber.recv_group().await.unwrap().expect("old group"); + let handle = subscription.priority.insert(Priority::new(0, 0, 0)); + let mut serve = std::pin::pin!(subscription.serve_group(0, 0, handle, group)); + assert!(futures::poll!(serve.as_mut()).is_pending()); + assert_eq!(opens.pending.load(Ordering::Relaxed), u64::from(!open)); + + // A successor a second on gives the group up, header written or not. + moq_net_sim::advance(Duration::from_secs(1)).await; + let mut edge = track.append_group().unwrap(); + edge.write_frame(Timestamp::from_millis(1000).unwrap(), b"edge".as_slice()) + .unwrap(); + let _ = serve.await; + assert_eq!(opens.pending.load(Ordering::Relaxed), 0); + assert_eq!(opens.opened.load(Ordering::Relaxed), u64::from(open), "{open}"); + let checkpoints = log.trail().iter().filter(|(_, sent)| *sent == Sent::Checkpoint).count(); + assert_eq!(checkpoints, usize::from(open)); + } } /// A subscription group keeps checking the max delay while a transport write is diff --git a/rs/moq-net/src/lite/setup.rs b/rs/moq-net/src/lite/setup.rs index df7fd821a4d..5f0a865ab78 100644 --- a/rs/moq-net/src/lite/setup.rs +++ b/rs/moq-net/src/lite/setup.rs @@ -15,6 +15,8 @@ const PARAM_ROLE: u64 = 0x3; const PARAM_COST: u64 = 0x4; /// Setup Parameter id for the endpoint's Hop ID. const PARAM_HOP: u64 = 0x5; +/// Setup Parameter id declaring that the endpoint's group resets keep their header. +const PARAM_RELIABLE_RESET: u64 = 0x6; /// The cost of crossing a link that neither end priced. /// @@ -182,6 +184,18 @@ pub struct Setup { /// applies). `None` when the endpoint has no meaningful identity (a /// leaf that never forwards); a wire value of 0 decodes as `None`. pub hop: Option, + /// Whether every Group Stream this endpoint counts in SUBSCRIBE_END delivers its + /// GROUP message, even when reset (lite-07+). The peer then waits for each counted + /// stream rather than giving up after a grace. + pub reliable_reset: bool, +} + +impl Setup { + /// Whether `session` can keep that promise on `version`: it needs the stream count to + /// make, and a transport whose resets keep a checkpoint. + pub fn detect_reliable_reset(session: &S, version: Version) -> bool { + version.has_stream_count() && session.reliable_reset() + } } impl Message for Setup { @@ -215,6 +229,9 @@ impl Message for Setup { if let Some(hop) = self.hop { params.set_varint(PARAM_HOP, hop.id(), version)?; } + if self.reliable_reset { + params.set_varint(PARAM_RELIABLE_RESET, 1, version)?; + } params.encode(w, version) } @@ -256,6 +273,11 @@ impl Message for ReceivedSetup { let hop = params .get_varint(PARAM_HOP, version)? .and_then(|id| crate::Hop::new(id).ok()); + let reliable_reset = match params.get_varint(PARAM_RELIABLE_RESET, version)? { + None | Some(0) => false, + Some(1) => true, + Some(_) => return Err(DecodeError::InvalidValue), + }; Ok(Self { role_present: role_code.is_some(), @@ -265,6 +287,7 @@ impl Message for ReceivedSetup { role, cost, hop, + reliable_reset, }, }) } @@ -345,6 +368,12 @@ impl PeerSetup { self.poll_get(waiter, |setup| setup.hop) } + /// Whether the peer declared that its group resets keep their header. False until its + /// SETUP arrives, which only costs a grace. + pub fn reliable_reset(&self) -> bool { + self.0.read().setup.as_ref().is_some_and(|setup| setup.reliable_reset) + } + /// Poll for a field of the peer's SETUP. /// /// The peer MUST send exactly one SETUP, so this resolves once that stream is read. @@ -497,6 +526,42 @@ mod tests { assert_eq!(round_trip(&msg), msg); } + /// The declaration is a 0/1 flag; any other value is malformed. + #[test] + fn reliable_reset_round_trip() { + let msg = Setup { + reliable_reset: true, + ..Default::default() + }; + assert_eq!(round_trip(&msg), msg); + + let version = Version::Lite07; + for (value, want) in [(0, Some(false)), (1, Some(true)), (2, None)] { + let mut params = Parameters::default(); + params.set_varint(super::PARAM_RELIABLE_RESET, value, version).unwrap(); + let mut buf = Vec::new(); + params + .encode(&mut Encoder::new(&mut buf, version.into()), version) + .unwrap(); + let got = Setup::decode_msg(&mut Decoder::new(&buf, version.into()), version); + assert_eq!(got.ok().map(|setup| setup.reliable_reset), want, "{value}"); + } + } + + /// Only lite-07 counts the streams the promise is about. + #[test] + fn reliable_reset_needs_the_stream_count_and_the_transport() { + use crate::transport::memory; + for reliable_reset in [true, false] { + let (session, _peer) = memory::pair(memory::Config { + reliable_reset, + ..Default::default() + }); + assert_eq!(Setup::detect_reliable_reset(&session, Version::Lite07), reliable_reset); + assert!(!Setup::detect_reliable_reset(&session, Version::Lite06)); + } + } + // A declared id of 0 carries no identity (it cannot be excluded), so it // decodes as absent rather than erroring. #[test] diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 1ad386c4fc5..9da5ea5045c 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -5294,7 +5294,7 @@ enum ServeMode { /// The upstream FIN'd, so the track is over, but QUIC does not order streams: keep /// the subscription routable until the counted headers arrive (lite-07), or every /// owed group is accounted for (older drafts). The grace bounds a stream reset - /// before its header arrived. + /// before its header arrived, unless the publisher declared its resets keep it. Tail { settle: Settle, owed: Option>, @@ -5726,14 +5726,21 @@ impl ServeLoop { )); tail.expire(serve.subscriber.runtime.now()); } + let streams = active + .end + .as_ref() + .filter(|_| serve.subscriber.version.has_stream_count()) + .map(|end| end.streams); + // Every counted stream delivers its header when the publisher + // promised its resets keep it, so nothing is given up on. + let settle = match streams.is_some() && serve.subscriber.peer_setup.reliable_reset() { + true => Settle::exact(active.tail.consume()), + false => Settle::new(&serve.subscriber.runtime, active.tail.consume()), + }; self.mode = ServeMode::Tail { - settle: Settle::new(&serve.subscriber.runtime, active.tail.consume()), + settle, owed: active.owed(requested_end), - streams: active - .end - .as_ref() - .filter(|_| serve.subscriber.version.has_stream_count()) - .map(|end| end.streams), + streams, }; continue; } diff --git a/rs/moq-net/src/lite/test_transport.rs b/rs/moq-net/src/lite/test_transport.rs index f47528eec5d..2535ea99052 100644 --- a/rs/moq-net/src/lite/test_transport.rs +++ b/rs/moq-net/src/lite/test_transport.rs @@ -58,6 +58,8 @@ pub enum Sent { Write(Vec), Finish, Reset(u32), + /// A later reset keeps what was written so far. + Checkpoint, } impl Log { @@ -197,6 +199,10 @@ impl poll::SendStream for SinkSend { self.log.record(self.id, Sent::Reset(code)); } + fn checkpoint(&mut self) { + self.log.record(self.id, Sent::Checkpoint); + } + fn poll_closed(&mut self, _cx: &mut Context<'_>) -> Poll> { match self.finished && !self.unacked_fin { true => Poll::Ready(Ok(())), diff --git a/rs/moq-net/src/server.rs b/rs/moq-net/src/server.rs index 371e347a161..a0b44ba3e67 100644 --- a/rs/moq-net/src/server.rs +++ b/rs/moq-net/src/server.rs @@ -167,6 +167,7 @@ impl Server { cost: None, // Filled by `lite::start` from the attached origin handles. hop: None, + reliable_reset: lite::Setup::detect_reliable_reset(&session, version), } } else { lite::Setup::default() @@ -1160,6 +1161,7 @@ mod tests { role, cost: None, hop, + reliable_reset: false, } .encode(&mut crate::coding::Encoder::new(&mut buf, v.into()), v) .unwrap(); diff --git a/rs/moq-net/src/tail.rs b/rs/moq-net/src/tail.rs index f6ac0c58eb5..24a954635fa 100644 --- a/rs/moq-net/src/tail.rs +++ b/rs/moq-net/src/tail.rs @@ -4,7 +4,8 @@ //! but QUIC does not order streams, so one opened before the end can still reach us after //! it. The subscriber keeps the subscription routable until each owed group is accounted //! for, or a grace expires for a group whose stream was reset before its header arrived. -//! A reset that keeps the header (reliable reset) would make the grace unnecessary. +//! A publisher whose resets keep the header (reliable reset) needs no grace: on lite-07 +//! every stream it counts arrives. //! //! A stream whose header arrived is read to its end whatever the grace says: a group ends //! on its own stream's FIN or reset, never because its track ended. A lost datagram is not @@ -291,7 +292,8 @@ impl Drop for Reading { /// Waits out a subscription's tail: until the owed streams are accounted for, or the grace. pub(crate) struct Settle { tail: kio::Consumer, - grace: Deadline, + /// `None` when every owed stream is sure to arrive. + grace: Option, } impl Settle { @@ -300,14 +302,20 @@ impl Settle { let grace = tail.read().grace; Self { tail, - grace: Deadline::after(runtime, grace), + grace: Some(Deadline::after(runtime, grace)), } } + /// Wait on `tail` without a grace, for a publisher that promised every owed stream + /// delivers its header. + pub fn exact(tail: kio::Consumer) -> Self { + Self { tail, grace: None } + } + /// Ready once no stream is being read and either `complete` holds or the grace /// expired, or once the subscription is gone. pub fn poll(&mut self, waiter: &kio::Waiter, mut complete: impl FnMut(&Tail) -> bool) -> Poll<()> { - let expired = self.grace.poll(waiter).is_ready(); + let expired = self.grace.as_mut().is_some_and(|grace| grace.poll(waiter).is_ready()); self.tail .poll(waiter, |tail| match tail.active == 0 && (expired || complete(tail)) { true => Poll::Ready(()), diff --git a/rs/moq-net/src/transport.rs b/rs/moq-net/src/transport.rs index 409ed432e7b..d42fcf7fc42 100644 --- a/rs/moq-net/src/transport.rs +++ b/rs/moq-net/src/transport.rs @@ -207,6 +207,15 @@ pub mod poll { /// rather than defaulted for the same reason as [`protocol`](Self::protocol). fn stats(&self) -> impl Stats; + /// Whether a reset after [`SendStream::checkpoint`] still delivers the data written + /// before it to the peer's application, as QUIC's RESET_STREAM_AT does. + /// + /// Only true once the peer has agreed to it. A transport that reports this must + /// implement [`SendStream::checkpoint`], since a peer is told to rely on it. + fn reliable_reset(&self) -> bool { + false + } + /// Accept the next unidirectional stream opened by the peer. fn accept_uni(&mut self) -> AcceptUni<'_, Self> { AcceptUni(self) @@ -371,6 +380,12 @@ pub mod poll { /// for exactly that reason. fn reset(&mut self, code: u32); + /// Mark the data written so far, so a later [`reset`](Self::reset) still delivers it. + /// + /// Only does so on a session whose [`Session::reliable_reset`] holds; elsewhere a + /// reset discards it as before. A peer that stopped the stream gets nothing. + fn checkpoint(&mut self) {} + /// Poll until the stream is closed by either side. /// /// This includes: diff --git a/rs/moq-net/src/transport/memory.rs b/rs/moq-net/src/transport/memory.rs index 1bccc9948bb..575a224cc15 100644 --- a/rs/moq-net/src/transport/memory.rs +++ b/rs/moq-net/src/transport/memory.rs @@ -10,7 +10,8 @@ //! //! [`finish`](poll::SendStream::finish) closes the channel cleanly once its messages are //! read. A [`reset`](poll::SendStream::reset) or [`stop`](poll::RecvStream::stop) closes it -//! with a code and drops what it queued. Closing the session keeps queued messages +//! with a code and drops what it queued, except that a reset with +//! [`Config::reliable_reset`] still delivers what was written before its checkpoint. Closing the session keeps queued messages //! readable, and leaves streams as they were, so the peer reads them, then the close. //! //! Datagrams are copied, since [`poll::Session::poll_send_datagram`] borrows them, and @@ -42,6 +43,9 @@ pub struct Config { pub stream_buffer: usize, /// The application protocol both sides report, standing in for a negotiated ALPN. pub protocol: Option, + /// Whether a reset still delivers the data before a stream's checkpoint, like a QUIC + /// peer that accepts RESET_STREAM_AT. + pub reliable_reset: bool, } impl Default for Config { @@ -51,6 +55,7 @@ impl Default for Config { Self { stream_buffer: 1024 * 1024, protocol: None, + reliable_reset: false, } } } @@ -160,6 +165,8 @@ impl Conn { channel: channel.clone(), close: self.close.clone(), buffer: self.config.stream_buffer, + reliable: self.config.reliable_reset, + checkpoint: None, done: false, park: Default::default(), }; @@ -334,6 +341,10 @@ impl poll::Session for Session { fn stats(&self) -> impl super::Stats { StatsUnavailable } + + fn reliable_reset(&self) -> bool { + self.conn().config.reliable_reset + } } /// One direction of a stream: the messages its writer queued for its reader. @@ -342,6 +353,9 @@ struct Channel { messages: VecDeque, /// The bytes in `messages`. queued: usize, + /// The bytes ever queued and read, which place a checkpoint among the messages. + written: u64, + read: u64, /// How the channel closed, once it has. end: Option, } @@ -357,14 +371,19 @@ enum End { } impl Channel { - /// Close with a reset or stop code and drop what is queued. An earlier end stands, - /// unless it was a finish the reader has not fully read. - fn abort(&mut self, end: End) { + /// Close with a reset or stop code and drop what is queued past its first `keep` + /// bytes. An earlier end stands, unless it was a finish the reader has not fully read. + fn abort(&mut self, end: End, keep: usize) { if self.end.is_none() || !self.messages.is_empty() { self.end = Some(end); } - self.messages.clear(); - self.queued = 0; + let mut kept = 0; + self.messages.retain_mut(|message| { + message.truncate(keep - kept); + kept += message.len(); + !message.is_empty() + }); + self.queued = kept; } /// How the writer sees the channel close: reset, stopped, or finished and drained. @@ -383,6 +402,10 @@ pub struct SendStream { channel: kio::Shared, close: Close, buffer: usize, + /// Whether a reset keeps the bytes before the checkpoint. + reliable: bool, + /// The bytes written when the stream was last checkpointed. + checkpoint: Option, /// This side finished or reset the stream. done: bool, park: kio::Park, @@ -425,6 +448,7 @@ impl poll::SendStream for SendStream { let size = buf.remaining(); channel.messages.push_back(buf.copy_to_bytes(size)); channel.queued += size; + channel.written += size as u64; Poll::Ready(Ok(size)) } @@ -456,7 +480,15 @@ impl poll::SendStream for SendStream { // close is checked under the channel lock so a concurrent close can't land between. let mut channel = self.channel.lock(); if closed(&self.close).is_none() { - channel.abort(End::Reset(code)); + // What the reader has yet to take of the checkpointed prefix. + let keep = self.checkpoint.map_or(0, |at| at.saturating_sub(channel.read)); + channel.abort(End::Reset(code), keep as usize); + } + } + + fn checkpoint(&mut self) { + if self.reliable { + self.checkpoint = Some(self.channel.lock().written); } } @@ -546,6 +578,7 @@ impl poll::RecvStream for RecvStream { false => channel.messages.pop_front().expect("front exists"), }; channel.queued -= chunk.len(); + channel.read += chunk.len() as u64; Poll::Ready(Ok(Some(chunk))) } @@ -558,7 +591,7 @@ impl poll::RecvStream for RecvStream { // as in `reset`. let mut channel = self.channel.lock(); if closed(&self.close).is_none() { - channel.abort(End::Stop(code)); + channel.abort(End::Stop(code), 0); } } @@ -734,6 +767,49 @@ mod tests { )); } + /// A reliable reset delivers the unread part of what came before the checkpoint, then + /// the code; a checkpoint without it changes nothing. + #[test] + fn reset_keeps_the_checkpoint() { + for reliable_reset in [true, false] { + let (mut a, mut b) = pair(Config { + reliable_reset, + ..Default::default() + }); + assert_eq!(a.reliable_reset(), reliable_reset); + let (mut send, mut recv) = uni(&mut a, &mut b); + + ready(write(&mut send, &Bytes::from_static(b"0123"))).unwrap(); + assert_eq!(text(read(&mut recv, 1)), "0"); + ready(write(&mut send, &Bytes::from_static(b"45"))).unwrap(); + send.checkpoint(); + ready(write(&mut send, &Bytes::from_static(b"6789"))).unwrap(); + send.reset(7); + + if reliable_reset { + assert_eq!(text(read(&mut recv, usize::MAX)), "123"); + assert_eq!(text(read(&mut recv, usize::MAX)), "45"); + } + assert!(matches!(ready(read(&mut recv, usize::MAX)), Err(Error::Reset(7)))); + } + } + + /// A peer that stopped reading gets nothing, checkpoint or not. + #[test] + fn stop_drops_the_checkpoint() { + let (mut a, mut b) = pair(Config { + reliable_reset: true, + ..Default::default() + }); + let (mut send, mut recv) = uni(&mut a, &mut b); + ready(write(&mut send, &Bytes::from_static(b"01"))).unwrap(); + send.checkpoint(); + recv.stop(3); + send.reset(7); + assert!(matches!(ready(send.poll_closed(&mut noop())), Err(Error::Stopped(3)))); + assert!(matches!(ready(read(&mut recv, usize::MAX)), Err(Error::Closed))); + } + #[test] fn finish_closes_once_drained() { let (mut a, mut b) = pair(Config::default()); diff --git a/rs/moq-net/tests/integration/support/mock.rs b/rs/moq-net/tests/integration/support/mock.rs index f9889c45227..cb51c25e829 100644 --- a/rs/moq-net/tests/integration/support/mock.rs +++ b/rs/moq-net/tests/integration/support/mock.rs @@ -255,6 +255,10 @@ impl poll::SendStream for MockSendStream { } } + fn checkpoint(&mut self) { + self.inner.checkpoint(); + } + fn poll_closed(&mut self, cx: &mut Context<'_>) -> Poll> { if self.conn.hold_fins.load(Ordering::Relaxed) { return Poll::Ready(Err(ready!(poll::Session::poll_closed(&mut self.session, cx)))); @@ -622,6 +626,10 @@ impl poll::Session for MockSession { self.inner.poll_closed(cx) } + fn reliable_reset(&self) -> bool { + self.inner.reliable_reset() + } + fn stats(&self) -> impl moq_net::transport::Stats { assert!( !self.side.panic_on_stats.load(Ordering::Relaxed), diff --git a/rs/moq-net/tests/integration/track_tail.rs b/rs/moq-net/tests/integration/track_tail.rs index 3cce8c6dcf6..2bd6f225b82 100644 --- a/rs/moq-net/tests/integration/track_tail.rs +++ b/rs/moq-net/tests/integration/track_tail.rs @@ -75,7 +75,7 @@ struct Pair { ), } -async fn connect(version: &str) -> Pair { +async fn connect(version: &str, transport: moq_net::transport::memory::Config) -> Pair { let publisher = produce_origin(1); let broadcast = publisher.create_broadcast("bcast").unwrap(); let track = broadcast.create_track("video", timed()).unwrap(); @@ -83,6 +83,7 @@ async fn connect(version: &str) -> Pair { let subscriber = produce_origin(2); let mut options = MockConnectOptions::new(version.parse::().unwrap()); + options.transport = transport; options.server_publish = Some(publisher.consume()); options.client_subscribe = Some(subscriber.clone()); let pair = connect_mock(options).await; @@ -113,7 +114,7 @@ async fn round(version: &str, late: Late, final_sequence: u64) -> Outcome { mut track, remote, _keep, - } = connect(version).await; + } = connect(version, Default::default()).await; let reader = moq_net_sim::spawn(async move { let subscription = moq_net::track::Subscription::default().with_start(moq_net::track::Position::group(0)); @@ -273,7 +274,7 @@ async fn ietf_leaving_cancels_a_blocked_end_of_track() { track, remote, _keep, - } = connect(version).await; + } = connect(version, Default::default()).await; let subscription = moq_net::track::Subscription::default().with_start(moq_net::track::Position::group(0)); let mut sub = remote @@ -331,7 +332,7 @@ async fn a_lost_datagram_delays_the_end_only_without_a_stream_count() { mut track, remote, _keep, - } = connect(version).await; + } = connect(version, Default::default()).await; let reader = moq_net_sim::spawn(async move { let subscription = moq_net::track::Subscription::default().with_start(moq_net::track::Position::group(0)); @@ -393,7 +394,7 @@ async fn a_group_whose_first_frame_trails_its_header_is_delivered() { mut track, remote, _keep, - } = connect(version).await; + } = connect(version, Default::default()).await; let reader = moq_net_sim::spawn(async move { let subscription = moq_net::track::Subscription::default() @@ -441,3 +442,211 @@ async fn a_group_whose_first_frame_trails_its_header_is_delivered() { drop((track, pair, _keep)); } } + +/// A transport whose resets keep a stream's checkpoint, or not. +fn reliable(reliable_reset: bool) -> moq_net::transport::memory::Config { + let mut config = moq_net::transport::memory::Config::default(); + config.reliable_reset = reliable_reset; + config +} + +/// What a reader saw of the track: its frames, the groups that failed, and when it ended. +type Read = (Vec>, Vec, moq_net::time::Instant); + +/// Read every group of the track from its first, noting the ones that failed, until it ends. +fn read_track(remote: moq_net::broadcast::Consumer, max_delay: Duration) -> moq_net_sim::JoinHandle { + moq_net_sim::spawn(async move { + let subscription = moq_net::track::Subscription::default() + .with_start(moq_net::track::Position::group(0)) + .with_max_delay(max_delay); + let mut sub = remote + .track("video") + .unwrap() + .subscribe(subscription) + .await + .expect("subscribe"); + let mut frames = Vec::new(); + let mut failed = Vec::new(); + while let Some(mut group) = sub.recv_group().await.expect("track aborted") { + loop { + match group.read_frame().await { + Ok(Some(frame)) => frames.push(frame.payload.to_vec()), + Ok(None) => break, + Err(_) => { + failed.push(group.sequence); + break; + } + } + } + } + (frames, failed, moq_net_sim::now()) + }) +} + +/// A group reset before any payload still delivers its header when the transport's resets +/// keep a checkpoint, so the lite-07 subscriber counts its stream and ends at once. A plain +/// reset loses the header, and the subscriber waits out the grace for it. +#[moq_net_sim::test] +async fn a_reset_group_that_keeps_its_header_ends_without_the_grace() { + // The grace is the subscription's max delay, which also expires the stalled group. + let max_delay = GRACE / 2; + for reliable_reset in [true, false] { + // Each stream takes one write until it is read, so a held group stream takes its + // header and stalls on the frame behind it. + let mut transport = reliable(reliable_reset); + transport.stream_buffer = 0; + let Pair { + pair, + mut track, + remote, + _keep, + } = connect("moq-lite-07-wip", transport).await; + let reader = read_track(remote, max_delay); + moq_net_sim::timeout(TIMEOUT, support::harness::subscribed(&track)) + .await + .expect("no subscriber appeared"); + + pair.server_transport.hold_unis(); + // The publisher resets this group once its successor has been out for the max + // delay, with its frame still stalled behind the header. + let mut stalled = track.append_group().unwrap(); + stalled.write_frame(Timestamp::ZERO, PAYLOAD).unwrap(); + stalled.finish().unwrap(); + moq_net_sim::sleep(max_delay / 10).await; + let mut group = track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, PAYLOAD).unwrap(); + group.finish().unwrap(); + moq_net_sim::sleep(max_delay * 2).await; + track.finish_at(moq_net::track::Position::group(2)).unwrap(); + drop(track); + moq_net_sim::sleep(max_delay / 10).await; + assert!( + !reader.is_finished(), + "{reliable_reset}: ended before the streams arrived" + ); + + let released = moq_net_sim::now(); + pair.server_transport.release_unis(); + let (frames, failed, ended) = moq_net_sim::timeout(TIMEOUT, reader) + .await + .expect("the subscription never ended") + .expect("reader panicked"); + // A group never shows before its first frame, so the reset one is not read either way. + assert_eq!(frames, [PAYLOAD], "{reliable_reset}"); + assert!(failed.is_empty(), "{reliable_reset}"); + let elapsed = ended - released; + match reliable_reset { + true => assert!(elapsed < max_delay / 10, "ended after {elapsed:?}"), + false => assert!(elapsed >= max_delay / 2, "ended after {elapsed:?}"), + } + drop((pair, _keep, stalled)); + } +} + +/// A publisher that declared its resets keep the header is waited on for every stream it +/// counted, however late: one arriving long after the grace is still delivered. Without the +/// declaration the subscriber gives up on it after the grace. +#[moq_net_sim::test] +async fn a_declared_publisher_is_waited_on_past_the_grace() { + for reliable_reset in [true, false] { + let Pair { + pair, + mut track, + remote, + _keep, + } = connect("moq-lite-07-wip", reliable(reliable_reset)).await; + let reader = read_track(remote, Duration::ZERO); + moq_net_sim::timeout(TIMEOUT, support::harness::subscribed(&track)) + .await + .expect("no subscriber appeared"); + + pair.server_transport.hold_unis(); + let mut group = track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, PAYLOAD).unwrap(); + group.finish().unwrap(); + track.finish_at(moq_net::track::Position::group(1)).unwrap(); + drop(track); + + moq_net_sim::sleep(GRACE * 3).await; + assert_eq!(reader.is_finished(), !reliable_reset, "gave up on the stream"); + let released = moq_net_sim::now(); + pair.server_transport.release_unis(); + let (frames, failed, ended) = moq_net_sim::timeout(TIMEOUT, reader) + .await + .expect("the subscription never ended") + .expect("reader panicked"); + assert!(failed.is_empty(), "{reliable_reset}"); + match reliable_reset { + true => { + assert_eq!(frames, [PAYLOAD]); + assert!(ended - released < GRACE / 10); + } + false => assert!(frames.is_empty()), + } + drop((pair, _keep)); + } +} + +/// The declaration holds per hop: a relay whose upstream publisher declared waits for its +/// late stream past the grace, and its own downstream subscriber, also declared to, waits +/// on the relay in turn. Without it the relay gives up after the grace and ends the track. +#[moq_net_sim::test] +async fn a_declared_publisher_is_waited_on_through_a_relay() { + for reliable_reset in [true, false] { + let version: Version = "moq-lite-07-wip".parse().unwrap(); + let publisher = produce_origin(1); + let broadcast = publisher.create_broadcast("bcast").unwrap(); + let mut track = broadcast.create_track("video", timed()).unwrap(); + broadcast.announce(Default::default()).unwrap(); + + let relay = produce_origin(2); + let mut options = MockConnectOptions::new(version); + options.transport = reliable(reliable_reset); + options.server_publish = Some(publisher.consume()); + options.client_subscribe = Some(relay.clone()); + let upstream = connect_mock(options).await; + + let subscriber = produce_origin(3); + let mut options = MockConnectOptions::new(version); + options.transport = reliable(reliable_reset); + options.server_publish = Some(relay.consume()); + options.client_subscribe = Some(subscriber.clone()); + let downstream = connect_mock(options).await; + + let consumer = subscriber.consume(); + moq_net_sim::timeout(TIMEOUT, consumer.routed("bcast")) + .await + .expect("announce timeout") + .expect("routed"); + let remote = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast", None)) + .await + .expect("resolve timeout") + .expect("broadcast resolves"); + + let reader = read_track(remote, Duration::ZERO); + moq_net_sim::timeout(TIMEOUT, support::harness::subscribed(&track)) + .await + .expect("no subscriber appeared"); + + upstream.server_transport.hold_unis(); + let mut group = track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, PAYLOAD).unwrap(); + group.finish().unwrap(); + track.finish_at(moq_net::track::Position::group(1)).unwrap(); + drop(track); + + moq_net_sim::sleep(GRACE * 3).await; + assert_eq!(reader.is_finished(), !reliable_reset, "gave up on the stream"); + upstream.server_transport.release_unis(); + let (frames, failed, _) = moq_net_sim::timeout(TIMEOUT, reader) + .await + .expect("the subscription never ended") + .expect("reader panicked"); + assert!(failed.is_empty(), "{reliable_reset}"); + match reliable_reset { + true => assert_eq!(frames, [PAYLOAD]), + false => assert!(frames.is_empty()), + } + drop((upstream, downstream, broadcast, publisher, relay, subscriber)); + } +} diff --git a/rs/moq-quic/src/connection/mod.rs b/rs/moq-quic/src/connection/mod.rs index 3425f00ba2b..e87afe96431 100644 --- a/rs/moq-quic/src/connection/mod.rs +++ b/rs/moq-quic/src/connection/mod.rs @@ -1469,6 +1469,14 @@ impl Connection { !self.spaces[SpaceId::Data].pending.is_empty(&self.streams) } + /// Whether a [`Checkpoint::reset`] reaches the peer's application, which needs the peer to + /// accept RESET_STREAM_AT + /// + /// Without it, the data before the checkpoint only reaches the peer's QUIC stack. + pub fn reliable_reset(&self) -> bool { + self.peer_params.reset_stream_at + } + /// Look up whether we're the client or server of this Connection pub fn side(&self) -> Side { self.side.side() diff --git a/rs/moq-quic/src/tests/reset_stream_at.rs b/rs/moq-quic/src/tests/reset_stream_at.rs index 8e1ef9a27bb..be7533d6f17 100644 --- a/rs/moq-quic/src/tests/reset_stream_at.rs +++ b/rs/moq-quic/src/tests/reset_stream_at.rs @@ -142,6 +142,17 @@ fn stop_sending_abandons_prefix_unsupported() { stop_sending_abandons_prefix(false); } +/// The client learns from the server's transport parameters whether its checkpoints survive +#[test] +fn reports_reliable_reset() { + let _guard = subscribe(); + for supported in [true, false] { + let mut pair = pair(supported); + let (client_ch, _) = pair.connect(); + assert_eq!(pair.client_conn_mut(client_ch).reliable_reset(), supported); + } +} + #[test] fn delivers_prefix_before_error() { let _guard = subscribe(); diff --git a/rs/moq-tokio/src/quic/driver/connection.rs b/rs/moq-tokio/src/quic/driver/connection.rs index 17ab2f70ea3..a5ede4c1220 100644 --- a/rs/moq-tokio/src/quic/driver/connection.rs +++ b/rs/moq-tokio/src/quic/driver/connection.rs @@ -596,6 +596,12 @@ impl Connection { self.0.state.lock("side").inner.side() } + /// Whether a [`Checkpoint::reset()`](crate::quic::driver::Checkpoint::reset) reaches the + /// peer's application: the peer accepts RESET_STREAM_AT + pub fn reliable_reset(&self) -> bool { + self.0.state.lock("reliable_reset").inner.reliable_reset() + } + /// The peer's UDP address /// /// If `ServerConfig::migration` is `true`, clients may change addresses at will, e.g. when diff --git a/rs/moq-tokio/src/quic/web/send.rs b/rs/moq-tokio/src/quic/web/send.rs index a6c040a045f..4f3bbd7437c 100644 --- a/rs/moq-tokio/src/quic/web/send.rs +++ b/rs/moq-tokio/src/quic/web/send.rs @@ -44,6 +44,12 @@ impl SendStream { } } + /// Mark the data written so far, so a later [`reset`](Self::reset) still delivers it. It + /// replaces the checkpoint after the WebTransport header, which it covers. + pub(crate) fn checkpoint(&mut self) { + self.header = Some(self.stream.checkpoint()); + } + /// Abruptly reset the stream with the provided error code. See [`crate::quic::driver::SendStream::reset`]. /// This is a u32 with WebTransport because we share the error space with HTTP/3. /// A raw QUIC session sends the code as is. diff --git a/rs/moq-tokio/src/quic/web/session.rs b/rs/moq-tokio/src/quic/web/session.rs index c84ecfc0305..5583f6a001c 100644 --- a/rs/moq-tokio/src/quic/web/session.rs +++ b/rs/moq-tokio/src/quic/web/session.rs @@ -557,6 +557,11 @@ impl Session { } } + /// Whether a stream's checkpoint survives a reset: the peer accepts RESET_STREAM_AT. + pub(crate) fn reliable_reset(&self) -> bool { + self.conn.reliable_reset() + } + /// Returns the [`ConnectRequest`] if this session was established over HTTP/3, /// or `None` for a raw QUIC session. pub fn request(&self) -> Option<&ConnectRequest> { @@ -1326,6 +1331,44 @@ mod tests { assert!(matches!(bi, crate::quic::web::ReadError::Reset(8)), "{bi:?}"); } + /// Through moq-net's transport, a checkpoint keeps what was written before it across a + /// reset, past the WebTransport header. + #[tokio::test] + async fn reset_delivers_the_checkpoint() { + use moq_net::transport::poll::{SendStream as _, Session as _}; + + let (client_conn, server_conn, _endpoints) = pair().await; + let (client, server) = tokio::join!( + Session::connect(client_conn, url::Url::parse("https://localhost/").unwrap()), + async { crate::quic::web::Request::accept(server_conn).await.unwrap().ok().await }, + ); + let (client, server) = (client.unwrap(), server.unwrap()); + let mut client = crate::transport::Session::new(client, moq_net::transport::Binding::Uri); + assert!(client.reliable_reset()); + + let mut send = std::future::poll_fn(|cx| client.poll_open_uni(cx)).await.unwrap(); + let n = std::future::poll_fn(|cx| send.poll_write(cx, b"kept")).await.unwrap(); + assert_eq!(n, 4); + send.checkpoint(); + std::future::poll_fn(|cx| send.poll_write(cx, b"maybe")).await.unwrap(); + send.reset(7); + + let read = async { + let mut uni = server.accept_uni().await.unwrap(); + let mut data = Vec::new(); + loop { + match uni.read_chunk(usize::MAX).await { + Ok(Some(chunk)) => data.extend_from_slice(&chunk), + Ok(None) => panic!("finished instead of reset"), + Err(err) => return (data, err), + } + } + }; + let (data, err) = tokio::time::timeout(Duration::from_secs(10), read).await.unwrap(); + assert!(data.starts_with(b"kept"), "{data:?}"); + assert!(matches!(err, crate::quic::web::ReadError::Reset(7)), "{err:?}"); + } + /// A stream that names another session is still reported as `UnknownSession`. #[tokio::test] async fn uni_other_session() { diff --git a/rs/moq-tokio/src/transport.rs b/rs/moq-tokio/src/transport.rs index 5ab97f52a9e..e3702e40047 100644 --- a/rs/moq-tokio/src/transport.rs +++ b/rs/moq-tokio/src/transport.rs @@ -132,6 +132,16 @@ fn poll_op(slot: &mut OpSlot, cx: &mut Context<'_>, start: impl FnOnce() - } impl Session { + /// Whether a send stream's checkpoint survives a reset. Only moq-quic's sessions can + /// say, and the async interface has no word for it, so they are recognized by type. + pub(crate) fn reliable_reset(&self) -> bool { + #[cfg(feature = "quic")] + if let Some(session) = (&self.session as &dyn std::any::Any).downcast_ref::() { + return session.reliable_reset(); + } + false + } + /// Wrap an async-interface transport session. pub fn new(session: S, binding: moq_net::transport::Binding) -> Self { Self { @@ -249,6 +259,8 @@ pub struct SendStream { state: std::sync::Mutex>>, // Deferred actions, applied when the in-flight operation settles. priority: Option, + /// Applied before a deferred reset, so the reset still delivers what it marks. + checkpoint: bool, finish: bool, reset: Option, /// Bytes already transmitted but not yet reported to the caller: a write the @@ -288,6 +300,7 @@ impl SendStream { Self { state: std::sync::Mutex::new(Some(SendState::Idle(stream))), priority: None, + checkpoint: false, finish: false, reset: None, completed: Bytes::new(), @@ -295,11 +308,33 @@ impl SendStream { } } + /// Mark the data written so far for a later reset to deliver, deferred like a reset + /// while an operation holds the stream. Only moq-quic's streams can, recognized by + /// type as in [`Session::reliable_reset`]. + pub(crate) fn checkpoint(&mut self) { + match self.state.get_mut().unwrap().as_mut() { + Some(SendState::Idle(stream)) => Self::checkpoint_stream(stream), + _ => self.checkpoint = true, + } + } + + fn checkpoint_stream(stream: &mut S) { + #[cfg(feature = "quic")] + if let Some(stream) = (stream as &mut dyn std::any::Any).downcast_mut::() { + stream.checkpoint(); + } + #[cfg(not(feature = "quic"))] + let _ = stream; + } + /// Apply actions deferred while an operation was in flight. fn settle(&mut self, stream: &mut S) { if let Some(order) = self.priority.take() { stream.set_priority(order); } + if std::mem::take(&mut self.checkpoint) { + Self::checkpoint_stream(stream); + } if let Some(code) = self.reset.take() { self.finish = false; stream.reset(code); diff --git a/rs/moq-tokio/src/transport/owned.rs b/rs/moq-tokio/src/transport/owned.rs index 44d98ef0d7e..c5e15ca23c6 100644 --- a/rs/moq-tokio/src/transport/owned.rs +++ b/rs/moq-tokio/src/transport/owned.rs @@ -84,6 +84,9 @@ impl> poll::Session for fn stats(&self) -> impl moq::Stats { Stats(backend::Session::stats(self)) } + fn reliable_reset(&self) -> bool { + super::Session::reliable_reset(self) + } } impl poll::SendStream for super::SendStream { type Error = Error; @@ -102,6 +105,9 @@ impl poll::SendStream for super::SendStream { fn reset(&mut self, code: u32) { backend::SendStream::reset(self, code) } + fn checkpoint(&mut self) { + super::SendStream::checkpoint(self) + } fn poll_closed(&mut self, cx: &mut Context<'_>) -> Poll> { backend::SendStream::poll_closed(self, cx).map_err(Error) } diff --git a/rs/moq-uring/src/quic/driver/connection.rs b/rs/moq-uring/src/quic/driver/connection.rs index 2b7c834b89e..19e97a9f23d 100644 --- a/rs/moq-uring/src/quic/driver/connection.rs +++ b/rs/moq-uring/src/quic/driver/connection.rs @@ -354,6 +354,11 @@ impl Connection { pub fn stats(&self) -> moq_quic::ConnectionStats { self.shared.conn.borrow().stats() } + + /// Whether a stream's checkpoint survives a reset: the peer accepts RESET_STREAM_AT. + pub fn reliable_reset(&self) -> bool { + self.shared.conn.borrow().reliable_reset() + } } impl Clone for Connection { diff --git a/rs/moq-uring/src/quic/web.rs b/rs/moq-uring/src/quic/web.rs index b6854e5b715..95e483a72c4 100644 --- a/rs/moq-uring/src/quic/web.rs +++ b/rs/moq-uring/src/quic/web.rs @@ -443,6 +443,11 @@ pub struct Session { } impl Session { + /// Whether a stream's checkpoint survives a reset: the peer accepts RESET_STREAM_AT. + pub(crate) fn reliable_reset(&self) -> bool { + self.conn.reliable_reset() + } + /// Wrap a raw-QUIC connection in a session, with no /// WebTransport layering: streams and datagrams pass through untouched. pub fn raw(conn: Connection) -> Self { @@ -842,10 +847,16 @@ impl SendStream { } /// Reset a WebTransport stream, still delivering the header of one we - /// opened. + /// opened, or what was written before [`checkpoint`](Self::checkpoint). fn reset_web(&mut self, code: u64) { self.inner.reset_code(code, self.header.take()); } + + /// Mark the data written so far, so a later reset still delivers it. It + /// replaces the checkpoint after the WebTransport header, which it covers. + pub(crate) fn checkpoint(&mut self) { + self.header = Some(self.inner.checkpoint()); + } } impl web_transport_trait::poll::SendStream for SendStream { @@ -876,7 +887,7 @@ impl web_transport_trait::poll::SendStream for SendStream { fn reset(&mut self, code: u32) { match self.web { true => self.reset_web(proto::error_to_http3(code)), - false => web_transport_trait::poll::SendStream::reset(&mut self.inner, code), + false => self.inner.reset_code(u64::from(code), self.header.take()), } } diff --git a/rs/moq-uring/src/transport.rs b/rs/moq-uring/src/transport.rs index 6a26f1d66b7..bdbb22b899f 100644 --- a/rs/moq-uring/src/transport.rs +++ b/rs/moq-uring/src/transport.rs @@ -1,3 +1,22 @@ //! Adapt backend poll sessions to moq-net without changing thread affinity. mod adapter; pub use adapter::{Error, RecvStream, SendStream, Session}; + +// The backend traits have no word for a reliable reset, so only moq-quic's +// sessions and streams can offer one, recognized by type. +fn reliable_reset(session: &dyn std::any::Any) -> bool { + #[cfg(feature = "quic")] + if let Some(session) = session.downcast_ref::() { + return session.reliable_reset(); + } + let _ = session; + false +} + +fn checkpoint(stream: &mut dyn std::any::Any) { + #[cfg(feature = "quic")] + if let Some(stream) = stream.downcast_mut::() { + stream.checkpoint(); + } + let _ = stream; +} diff --git a/rs/moq-uring/src/transport/adapter.rs b/rs/moq-uring/src/transport/adapter.rs index 0a751d76718..a81f230a7b2 100644 --- a/rs/moq-uring/src/transport/adapter.rs +++ b/rs/moq-uring/src/transport/adapter.rs @@ -101,6 +101,9 @@ impl + Clone + 'st fn stats(&self) -> impl moq::Stats { Stats(backend::Session::stats(&self.0)) } + fn reliable_reset(&self) -> bool { + super::reliable_reset(&self.0) + } } /// An outgoing WebTransport stream adapted to moq-net. pub struct SendStream(S); @@ -121,6 +124,9 @@ impl poll::SendStream for SendStream { fn reset(&mut self, code: u32) { backend::SendStream::reset(&mut self.0, code) } + fn checkpoint(&mut self) { + super::checkpoint(&mut self.0) + } fn poll_closed(&mut self, cx: &mut Context<'_>) -> Poll> { backend::SendStream::poll_closed(&mut self.0, cx).map_err(Error) } diff --git a/rs/moq-uring/tests/integration/transport.rs b/rs/moq-uring/tests/integration/transport.rs index eb3353e6f25..c9e5459a71e 100644 --- a/rs/moq-uring/tests/integration/transport.rs +++ b/rs/moq-uring/tests/integration/transport.rs @@ -1,3 +1,10 @@ // Exercise the transport adapter on native hosts without a browser or ring. #[path = "../../src/transport/adapter.rs"] mod adapter; + +// The adapter's hooks for moq-quic's reliable reset, which these backends lack. +fn reliable_reset(_: &dyn std::any::Any) -> bool { + false +} + +fn checkpoint(_: &mut dyn std::any::Any) {} diff --git a/rs/moq-uring/tests/integration/web.rs b/rs/moq-uring/tests/integration/web.rs index 33783c3c846..ea8b052cb3e 100644 --- a/rs/moq-uring/tests/integration/web.rs +++ b/rs/moq-uring/tests/integration/web.rs @@ -644,6 +644,72 @@ fn a_reset_stream_keeps_its_header() { client.join().expect("client thread"); } +/// Through moq-net's transport, a checkpoint keeps what was written before it +/// across a reset, past the WebTransport header. +#[test] +fn a_reset_stream_keeps_its_checkpoint() { + let Some(mut worker) = worker() else { return }; + let handle = worker.handle(); + let certs = support::certs().expect("certificates"); + let endpoint = h3_endpoint(&handle, &certs); + let addr = endpoint.local_addr(); + + let client = tokio_client(format!("https://{addr}/"), |session| { + Box::pin(async move { + let mut recv = session.accept_uni().await.expect("accept_uni"); + let mut data = Vec::new(); + let err = loop { + match recv.read_chunk(usize::MAX).await { + Ok(Some(chunk)) => data.extend_from_slice(&chunk), + Ok(None) => panic!("finished instead of reset"), + Err(err) => break err, + } + }; + assert!(data.starts_with(b"kept"), "got {data:?}"); + assert!(matches!(err, moq_tokio::quic::web::ReadError::Reset(7)), "got {err:?}"); + session.close(CLOSE_CODE, CLOSE_REASON.as_bytes()); + session.closed().await; + }) + }); + + worker + .block_on(async move { + use moq_net::transport::poll::{SendStream as _, Session as _}; + + let conn = endpoint.accept().await.expect("accept"); + let request = quic::web::Request::accept(conn).await.expect("handshake"); + let session = request + .respond(quic::web::Response::default().with_protocol(PROTO)) + .await + .expect("respond"); + let mut session = moq_uring::transport::Session::new(session, moq_net::transport::Binding::Uri); + assert!(session.reliable_reset()); + + let mut send = std::future::poll_fn(|cx| session.poll_open_uni(cx)) + .await + .expect("open_uni"); + let n = std::future::poll_fn(|cx| send.poll_write(cx, b"kept")) + .await + .expect("write"); + assert_eq!(n, 4); + send.checkpoint(); + std::future::poll_fn(|cx| send.poll_write(cx, b"maybe")) + .await + .expect("write"); + send.reset(7); + + within( + &handle, + "the client's close", + std::future::poll_fn(|cx| session.poll_closed(cx)), + ) + .await; + }) + .expect("worker"); + + client.join().expect("client thread"); +} + /// A stream opened with no connection credit left still delivers its header /// when reset, rather than a plain reset of a stream the peer cannot place. #[test]