diff --git a/drafts/draft-lcurley-moq-ietf.md b/drafts/draft-lcurley-moq-ietf.md index f1d6514818c..5dde8cbb77a 100644 --- a/drafts/draft-lcurley-moq-ietf.md +++ b/drafts/draft-lcurley-moq-ietf.md @@ -70,6 +70,7 @@ A Group Stream, FETCH response, or datagram is read with the Version of its trac # Group Streams {#group} When the extension is negotiated, a Group Stream carries a Group State after the GROUP message, and an Object Record before each FRAME. A FETCH response carries the same, with the Group State first. +A FETCH for a track with a Version MUST NOT set a non-zero `Start Byte`, since an Object Record names a whole object; a publisher MUST reset the response with a PROTOCOL_VIOLATION otherwise. ~~~ Group Stream { @@ -265,6 +266,7 @@ A high, distinctive value is used to avoid collisions with the low values {{moql ## Since draft-lcurley-moq-ietf-00 (in progress) +- A FETCH for a track with a Version never starts partway into a frame. - A relay never restates the Max Age as MAX_CACHE_DURATION, and a sealed MAX_CACHE_DURATION is advisory. - Preserve relayed track timestamps and their unchecked MoQ Transport provenance across negotiated hops. Without the extension a relay serves them as native moq-lite timestamps rather than untimed. - Initial version. diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index c3a068e5e2e..758ce1c7cce 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -213,6 +213,7 @@ A subscriber that has received frames 0 through `N-1` of group `G` and wants the This is what lets a subscriber move a Track to a different publisher partway through a Group instead of waiting for the next Group to start, which matters for Tracks whose current Group may stay open indefinitely. A Group can therefore be assembled from more than one publisher: each contributes a disjoint run of frames, and the subscriber concatenates them in index order. +A subscriber that has received the first `B` bytes of frame `N` FETCHes `G` from frame `N` with a `Start Byte` of `B`, so a frame read partway on one publisher continues on another without sending its first bytes again. A partial Group is only ever delivered to a subscriber that asked for one with FETCH, or as the [continuation](#yield) of a paused Group. A publisher that holds only part of a Group a subscription covers names it in a [SUBSCRIBE_DROP](#subscribe-drop) rather than deliver the part it holds. @@ -2132,6 +2133,7 @@ FETCH Message { Subscriber Max Age (i) Group Sequence (i) Frame Start (i) + Start Byte (i) Frame End (i) } ~~~ @@ -2160,6 +2162,11 @@ The sequence number of the group to fetch. The index of the first frame to return within the group (see [Positions](#positions)). A plain absolute index; 0 means the start of the group (default). +**Start Byte**: +The offset of the first payload byte to return within the frame at `Frame Start`; 0 means the whole frame (default). +The first FRAME of the response still states that frame's full size, and carries only its payload from `Start Byte` on; later frames are whole. +A non-zero `Start Byte` at or past the frame's size is a protocol violation, since the rest of a frame is never empty, and so is a non-zero `Start Byte` on a Track whose `Compression` is not none, since a compressed frame inflates only from its first byte, or on a Track carrying a MoqTransport Version under the `ietf` extension, whose object records name whole objects. + **Frame End**: The last frame to return (inclusive), encoded as the absolute frame index + 1. A value of 0 means through the end of the group (default). @@ -2457,6 +2464,7 @@ An application-specific payload. - Added `Group Order` to SUBSCRIBE and SUBSCRIBE_UPDATE: descending (0x0, the previous fixed order) or ascending (0x1). - Added an optional `Subscriber Min Age` to SUBSCRIBE and SUBSCRIBE_UPDATE, the subscriber's target latency, encoded as milliseconds plus one with zero meaning not stated. A relay forwards the smallest stated value. - FETCH reads one Group or a slice of one, and SUBSCRIBE reads whole Groups: a read starting partway through a Group FETCHes it. A publisher holding only part of a Group a subscription covers drops it. +- Added `Start Byte` to FETCH, after `Frame Start`: the response's first frame states its full size and carries only its payload from that byte, so a frame read partway on one publisher continues on another from where it stopped. - A SUBSCRIBE_UPDATE that lowers a `Start` covers the groups below the old start that the publisher still holds, not only those that arrive later. A group already sent is not sent again. - Replaced SUBSCRIBE_OK `Group` with a `Range Start` per Range: the Group where that Range's delivery starts, including at a Relative or Time Anchor, from which its Groups are owed. - Each Range takes an implicit Range ID, and SUBSCRIBE_UPDATE edits the live Ranges instead of restating them: `Remove Count` Range IDs to retire, then the Ranges to add. The live Ranges must stay non-empty and fit in one SUBSCRIBE. A Range resolves its Anchors once, however many updates leave it live. diff --git a/js/net/src/lite/fetch.test.ts b/js/net/src/lite/fetch.test.ts index ef18e73946c..9a6dbc8516d 100644 --- a/js/net/src/lite/fetch.test.ts +++ b/js/net/src/lite/fetch.test.ts @@ -87,8 +87,9 @@ test("Fetch decodes the optional delay offset before converting to milliseconds" await w.u8(0); await w.u62(delay + 1n); await w.u62(42n); - await w.u53(0); - await w.u53(0); + await w.u53(0); // start frame + await w.u53(0); // start byte + await w.u53(0); // through the end }), ); const decoded = Fetch.decode(new Reader(undefined, encoded, version), version); @@ -99,3 +100,20 @@ test("Fetch decodes the optional delay offset before converting to milliseconds" unsafe.maxDelay = Number.MAX_SAFE_INTEGER + 1; await expect(encode(version, unsafe)).rejects.toThrow(RangeError); }); + +test("Fetch carries a start byte only on draft-07, beside its start frame", async () => { + const partway = () => + new Fetch({ + broadcast: Path.from("room/1"), + track: "video", + priority: 3, + group: 42n, + startFrame: 2, + startByte: 1 << 20, + endFrame: 5, + }); + const got = await roundtrip(Version.DRAFT_07, partway()); + expect([got.startFrame, got.startByte, got.endFrame]).toEqual([2, 1 << 20, 5]); + await expect(encode(Version.DRAFT_06, partway())).rejects.toThrow("start byte"); + expect((await roundtrip(Version.DRAFT_06, sample())).startByte).toBe(0); +}); diff --git a/js/net/src/lite/fetch.ts b/js/net/src/lite/fetch.ts index 8c54edc57c0..92f71dfa8c1 100644 --- a/js/net/src/lite/fetch.ts +++ b/js/net/src/lite/fetch.ts @@ -5,7 +5,7 @@ import * as Duration from "./duration.ts"; import { decodeNode, decodePin, encodeNode, encodePin } from "./instance.ts"; import * as Message from "./message.ts"; import { fromWire, toWire } from "./priority.ts"; -import { hasFetchMaxDelay, hasFrameBounds, Version } from "./version.ts"; +import { hasByteStart, hasFetchMaxDelay, hasFrameBounds, Version } from "./version.ts"; function guardFetch(version: Version) { switch (version) { @@ -32,6 +32,12 @@ export class Fetch { /** Index of the first frame to return; 0 is the start of the group. Lite-06+. */ startFrame: number; + /** + * Offset of the first byte to return within frame {@link startFrame}; 0 is the whole + * frame. The response still states that frame's full size. Lite-07+. + */ + startByte: number; + /** * Index of the last frame to return (inclusive), or undefined for through the end * of the group. Lite-06+. @@ -47,6 +53,7 @@ export class Fetch { maxDelay, group, startFrame = 0, + startByte = 0, endFrame, }: { broadcast: Path.Valid; @@ -57,6 +64,7 @@ export class Fetch { maxDelay?: number; group: bigint; startFrame?: number; + startByte?: number; endFrame?: number; }) { this.broadcast = broadcast; @@ -67,6 +75,7 @@ export class Fetch { this.maxDelay = maxDelay; this.group = group; this.startFrame = startFrame; + this.startByte = startByte; this.endFrame = endFrame; } @@ -80,13 +89,17 @@ export class Fetch { else if (this.maxDelay !== undefined) throw new Error("fetch maxDelay not supported for this version"); await w.u62(this.group); - if (hasFrameBounds(version)) { - await w.u53(this.startFrame); - await w.u53(this.endFrame !== undefined ? this.endFrame + 1 : 0); - } else if (this.startFrame !== 0 || this.endFrame !== undefined) { + if (!hasFrameBounds(version) && (this.startFrame !== 0 || this.endFrame !== undefined)) { // The peer would serve the whole group, including frames we excluded. throw new Error("frame bounds not supported for this version"); } + if (!hasByteStart(version) && this.startByte !== 0) { + // The peer would resend the frame from its start, misaligning the reader. + throw new Error("fetch start byte not supported for this version"); + } + if (hasFrameBounds(version)) await w.u53(this.startFrame); + if (hasByteStart(version)) await w.u53(this.startByte); + if (hasFrameBounds(version)) await w.u53(this.endFrame !== undefined ? this.endFrame + 1 : 0); } static async #decode(r: Reader, version: Version): Promise { @@ -103,6 +116,7 @@ export class Fetch { } const startFrame = await r.u53(); + const startByte = hasByteStart(version) ? await r.u53() : 0; const endFrame = await r.u53(); if (endFrame > 0 && endFrame - 1 < startFrame) { throw new Error("fetch frame range ends before it starts"); @@ -117,6 +131,7 @@ export class Fetch { maxDelay, group, startFrame, + startByte, endFrame: endFrame > 0 ? endFrame - 1 : undefined, }); } diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index 452e557c944..8b1b057bf25 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -36,7 +36,7 @@ import { } from "../internal.ts"; import type { Consumer as OriginConsumer } from "../origin.ts"; import * as Path from "../path.ts"; -import { readSize, sendPayload } from "../payload.ts"; +import { readSize, sendPayload, sendPayloadFrom } from "../payload.ts"; import { Priority, sendOrder } from "../priority.ts"; import { rescale } from "../rescale.ts"; import { concat, encodeVarint, type Reader, type Stream, type StreamVersion, Writer } from "../stream.ts"; @@ -279,7 +279,10 @@ type ServeGroup = { }; /** What serving fetched frames needs beyond the group and destination stream. */ -type ServeFetch = Omit; +type ServeFetch = Omit & { + /** The byte frame `start` is served from: only its rest follows its full-size header. */ + byte: number; +}; /** What the serving loop takes from the subscribe stream instead of serving a group. */ type Control = @@ -1415,6 +1418,7 @@ export class Publisher { stamp: stamping(this.version, info), compress: info.compress, start: msg.startFrame, + byte: msg.startByte, end: msg.endFrame, }); console.debug(`fetch done: broadcast=${msg.broadcast} track=${msg.track} group=${msg.group}`); @@ -2476,8 +2480,10 @@ export class Publisher { async #runFetchGroup( group: group.Consumer, stream: Writer, - { stamp, compress, start: startFrame, end: endFrame }: ServeFetch, + { stamp, compress, start: startFrame, byte, end: endFrame }: ServeFetch, ) { + // A compressed frame inflates only from its first byte; no subscriber asks for less. + if (byte > 0 && compress) throw new Error("a compressed frame is served whole"); // The response carries no header, so the receiver numbers the first frame it gets // as `startFrame`. Skipping the head here is the only thing keeping those numbers // honest; a group that ends before we reach it can't be served at all. @@ -2502,11 +2508,14 @@ export class Publisher { prevTs = ts; } - await stream.write( - frameHeader(stream.version, delta, read.inflated, readSize(read), hasLengthFirst(this.version)), - ); - if (read.chunks) await sendPayload(read.chunks, (chunk) => stream.write(chunk), stream.closed); - else if (frame.payload.length > 0) await stream.write(frame.payload); + // The first frame states its full size, then carries only the bytes from `byte`. + const skip = index === startFrame ? byte : 0; + const size = readSize(read); + // A non-zero start byte names a byte of the frame, so the rest is never empty. + if (skip > 0 && skip >= size) throw new Error(`fetch start byte ${skip} not within the frame's ${size}`); + await stream.write(frameHeader(stream.version, delta, read.inflated, size, hasLengthFirst(this.version))); + if (read.chunks) await sendPayloadFrom(read.chunks, skip, (chunk) => stream.write(chunk), stream.closed); + else if (frame.payload.length > skip) await stream.write(frame.payload.subarray(skip)); } } diff --git a/js/net/src/lite/version.ts b/js/net/src/lite/version.ts index 56dc6455bdc..84f337694af 100644 --- a/js/net/src/lite/version.ts +++ b/js/net/src/lite/version.ts @@ -311,6 +311,12 @@ export function hasRanges(version: Version): boolean { } } +/** Whether FETCH carries a Start Byte within its first frame, so a reader cut off partway + * through a frame asks for only the rest of it. Added in lite-07. */ +export function hasByteStart(version: Version): boolean { + return hasFetchMaxDelay(version); +} + /** Whether FETCH carries the subscriber's optional max delay. Added in lite-07; an earlier * fetch is bounded by the publisher's max age alone. */ export function hasFetchMaxDelay(version: Version): boolean { diff --git a/js/net/src/payload.ts b/js/net/src/payload.ts index 9c1da311429..7a94dc0c3cb 100644 --- a/js/net/src/payload.ts +++ b/js/net/src/payload.ts @@ -73,13 +73,24 @@ export async function sendPayload( write: (chunk: Uint8Array) => Promise, closed: Promise, ): Promise { - for (let sent = 0; ; ) { + return sendPayloadFrom(chunks, 0, write, closed); +} + +/** {@link sendPayload}, writing only the bytes from offset `skip` on. */ +export async function sendPayloadFrom( + chunks: frame.Consumer, + skip: number, + write: (chunk: Uint8Array) => Promise, + closed: Promise, +): Promise { + for (let read = 0; ; ) { const chunk = await hooks.readFrameChunk(chunks, closed); if (!(chunk instanceof Uint8Array)) { - if (sent < chunks.size) throw new Error("stream closed mid-frame"); + if (read < chunks.size) throw new Error("stream closed mid-frame"); return; } - sent += chunk.byteLength; - await write(chunk); + const from = Math.max(0, skip - read); + read += chunk.byteLength; + if (from < chunk.byteLength) await write(chunk.subarray(from)); } } diff --git a/quest/m3/unbounded-groups/byte-holes.md b/quest/m3/unbounded-groups/byte-holes.md index b9e3e82daaa..9955c243b28 100644 --- a/quest/m3/unbounded-groups/byte-holes.md +++ b/quest/m3/unbounded-groups/byte-holes.md @@ -25,12 +25,40 @@ Moved to m3 2026-11-12 (maintainer, /quest-audit), with its epic: the cut does n Decided 2026-11-05 (maintainer, /quest-audit): ✅ the byte offset rides FETCH's frame start plus the group header, not a SUBSCRIBE range end. -SUBSCRIBE ranges are whole groups since 2026-11-03 -(Lite-07 FETCH is single-group only, -[moq-dev/moq#8616](https://github.com/moq-dev/moq/pull/8616) and -[moq-dev/moq#7675](https://github.com/moq-dev/moq/pull/7675)), and flights -ride lite FETCH since [moq-dev/moq#7944](https://github.com/moq-dev/moq/pull/7944). -This waits for that quest. + +Decided 2026-11-13 (maintainer, /quest-complete): + +- ✅ The byte stays crate-private until a public API needs it (rejected: add + `Position::byte` now, about 90 literals across moq-net, moq-archive, and + moq-ffi). `Position` also names subscribe ranges and largest locations, + where a byte means nothing. +- ✅ [moq-dev/moq#8699](https://github.com/moq-dev/moq/pull/8699) merges its + slice alone, and this quest keeps the rest (rejected: grow that PR to the + whole quest). + +Landed in [moq-dev/moq#8699](https://github.com/moq-dev/moq/pull/8699), the +first slice: + +- Lite-07 FETCH carries a `Start Byte` after `Frame Start`, in Rust, JS, and the + lite draft. The response's first frame states its full size and carries only + its bytes from there. A FETCH response has no header, so this needed no + group-header field. +- A group copy can start partway into a frame: `group::Bounds::byte` keeps the + rest of that frame aside as the copy's `Lead`, outside the frame log. Every + whole-frame read finds that frame missing (`Lagged`). Only + `group::Consumer::at` with `poll_next_frame_at` reads it, from a byte at or + past the copy's start. +- Route failover mid-frame (`frame::Consumer`'s `Recover`) asks for only the + missing suffix by byte. Its integration regression is + `rs/moq-net/tests/integration/byte_holes.rs`, with a 5 MiB frame. A frame + larger than the 32 MB tail waits on the frame cap below. +- Rust readers already had a chunked payload read + (`frame::Consumer::read_chunk`), so nothing was added. +- The byte stays crate-private (`group::Fetch::byte_start`, the fill's start). + Public `Position` did not gain a field, since nothing public reads a byte + position yet. + +What remains: - Segments are `frame::SEGMENT` (1 MiB) long, so segment `i` always holds bytes from `i * SEGMENT`, and a completed chain is `frame::Payload::Chain`. @@ -39,36 +67,29 @@ This waits for that quest. stays one `Bytes` at any size, so evicting part of it needs it split into segment slices, whose memory frees only once every slice drops, or copied into segments. - -- `Position` gains a byte offset within the frame. A lite FETCH carries it - beside its `frame_start`, and so does a group stream header. A - stream that starts mid-frame still states the frame's full size. Older - lite versions and moq-transport start flights only at frame or object - boundaries. A reader inside a shed object refreshes or is `Evicted` there. - Every sealed segment is evictable under the same policy as front frames, including one a reader is on. A reader keeps its current segment alive through its own reference, as prefetched frames are today. That memory - is bounded by readers times one segment, not by the cache. Only the - writer's open segment stays resident. A reader at a shed segment boards - a flight that starts at its `(frame, byte)`. -- Route failover mid-frame (`frame::Consumer`'s `Recover`) requests only - the missing suffix by byte, instead of refetching the frame from its - start. Cover it with a regression: a route fails partway through a frame - larger than the 32 MB tail. -- Update `drafts/draft-lcurley-moq-lite.md` and run `just drafts check`. - Also drop the 32 MiB interop cap in `drafts/draft-lcurley-moq-e2ee.md` if - it no longer follows from a model limit. -- Rust readers get a chunked payload read, so a frame larger than memory - never has to be collected whole. [moq-dev/moq#6628](https://github.com/moq-dev/moq/pull/6628) - kept its segments internal. JS already has its chunked read - (`nextFrame` and `readChunk`, from - [moq-dev/moq#7411](https://github.com/moq-dev/moq/pull/7411)); its wire - half is in [JS byte holes](/quest/m3/unbounded-groups/js-byte-holes.md). + is bounded by readers times one segment, not by the cache. A reader holding + a `Payload::Chain` holds every segment through its `Arc<[Bytes]>` today, so + the chain has to hand a reader one segment at a time. Only the writer's open + segment stays resident. A reader at a shed segment boards a flight that + starts at its `(frame, byte)`: `FillState::fly` and `group::Bounds` already + carry the byte, and the flight's copy starts with a `Lead`. +- A group stream that starts mid-frame (a relay holding only a group's tail, + once segments shed) carries the byte offset in the lite wip GROUP header + beside `Frame Start`, and still states the frame's full size. Older lite + versions and moq-transport start only at frame or object boundaries. A + reader inside a shed object refreshes or is `Evicted` there. +- `MAX_CACHE_BYTES` stops capping a frame (`group::Producer::open_frame` and + `open_lead` refuse a larger one today). Then extend the failover regression + to a frame larger than the 32 MB tail. +- Drop the 32 MiB interop cap in `drafts/draft-lcurley-moq-e2ee.md` if it no + longer follows from a model limit, and run `just drafts check`. - Test a frame several times the 32 MB tail through a relay chain: a late joiner reads it whole through a flight, and memory stays near 32 MB per copy. -Public API: `Position` gains a byte field, frames gain a chunked read, and -`MAX_CACHE_BYTES` stops capping a frame. -Wire: a byte offset on the current lite wip's FETCH frame start and group -header. +Public API: `Position` gains a byte field only once a public reader needs a +byte position, and `MAX_CACHE_BYTES` stops capping a frame. Wire: a byte +offset on the current lite wip's group header. diff --git a/rs/moq-net/src/lite/fetch.rs b/rs/moq-net/src/lite/fetch.rs index 288655be66a..81024882e4e 100644 --- a/rs/moq-net/src/lite/fetch.rs +++ b/rs/moq-net/src/lite/fetch.rs @@ -26,6 +26,9 @@ pub struct Fetch<'a> { pub group: u64, /// Index of the first frame to return; 0 is the start of the group. Lite06+ only. pub start_frame: u64, + /// Offset of the first byte to return within frame `start_frame`; 0 is the whole + /// frame. The response still states that frame's full size. Lite07+ only. + pub start_byte: u64, /// Index of the last frame to return (inclusive), or `None` for through the end of /// the group. Lite06+ only. pub end_frame: Option, @@ -43,6 +46,7 @@ impl Fetch<'_> { max_delay: self.max_delay, group: self.group, start_frame: self.start_frame, + start_byte: self.start_byte, end_frame: self.end_frame, } } @@ -69,9 +73,17 @@ impl<'a> Message<'a> for Fetch<'a> { }; let group = r.varint()?; - let (start_frame, end_frame) = match version.has_frame_bounds() { - true => (r.varint()?, r.varint_opt()?), - false => (0, None), + let start_frame = match version.has_frame_bounds() { + true => r.varint()?, + false => 0, + }; + let start_byte = match version.has_byte_start() { + true => r.varint()?, + false => 0, + }; + let end_frame = match version.has_frame_bounds() { + true => r.varint_opt()?, + false => None, }; // A range that ends before it starts can never be served. if end_frame.is_some_and(|end| end < start_frame) { @@ -87,6 +99,7 @@ impl<'a> Message<'a> for Fetch<'a> { max_delay, group, start_frame, + start_byte, end_frame, }) } @@ -116,12 +129,22 @@ impl<'a> Message<'a> for Fetch<'a> { } w.varint(self.group)?; + if !version.has_frame_bounds() && (self.start_frame != 0 || self.end_frame.is_some()) { + // The peer would serve the whole group, including frames the caller excluded. + return Err(EncodeError::Version); + } + if !version.has_byte_start() && self.start_byte != 0 { + // The peer would resend the frame from its start, misaligning the reader. + return Err(EncodeError::Version); + } if version.has_frame_bounds() { w.varint(self.start_frame)?; + } + if version.has_byte_start() { + w.varint(self.start_byte)?; + } + if version.has_frame_bounds() { w.varint_opt(self.end_frame)?; - } else if self.start_frame != 0 || self.end_frame.is_some() { - // The peer would serve the whole group, including frames the caller excluded. - return Err(EncodeError::Version); } Ok(()) @@ -142,10 +165,30 @@ mod test { max_delay: None, group: 7, start_frame: 0, + start_byte: 0, end_frame: None, } } + /// Lite-07 asks for the rest of a frame; an older peer cannot, so encoding refuses. + #[test] + fn fetch_start_byte_roundtrips_only_on_lite07() { + let mut msg = fetch_sample(); + msg.start_frame = 2; + msg.start_byte = 1 << 20; + let got = fetch_roundtrip(Version::Lite07, &msg); + assert_eq!((got.start_frame, got.start_byte), (2, 1 << 20)); + assert!( + msg.encode_msg( + &mut Encoder::new(&mut Vec::new(), Version::Lite06.into()), + Version::Lite06 + ) + .is_err() + ); + msg.start_byte = 0; + assert_eq!(fetch_roundtrip(Version::Lite06, &msg).start_byte, 0); + } + #[test] fn fetch_max_delay_roundtrips_on_lite07() { let mut msg = fetch_sample(); diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 9b8f28b0000..482e88c8f48 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -3176,6 +3176,9 @@ enum FetchServe { batch_pos: usize, /// The `ietf` extension's object records, when negotiated. records: Option, + /// The byte the first frame is served from, until it is: the response states that + /// frame's full size and carries only its bytes from here. + lead: Option, }, } @@ -3226,6 +3229,7 @@ impl Request for FetchServe { priority: msg.priority, max_delay: msg.max_delay, frame_start: msg.start_frame, + byte_start: msg.start_byte, }, end: msg.end_frame.and_then(|end| end.checked_add(1)), }, @@ -3249,15 +3253,24 @@ impl Request for FetchServe { Self::Fetch { msg, fetching } => { // Before the group: a fetched group polled while this waited would be lost. let ietf = ready!(shared.peer_setup.poll_ietf(waiter)); - let group = ready!(kio::Task::poll(fetching, waiter))?.with_range(msg.start_frame..); - // The response carries no header, so a short run is indistinguishable // from one that started elsewhere: only serve a range we can cover // exactly. `fetch_group` already positions the consumer, so this is a // belt-and-braces check on a promise the wire can't restate. - if group.index() != msg.start_frame { + let Some(group) = ready!(kio::Task::poll(fetching, waiter))?.at(msg.start_frame, msg.start_byte) + else { return Poll::Ready(Err(Error::Lagged)); - } + }; + // A compressed frame inflates only from its first byte, and an object record + // names a whole object, so neither is served from a later byte; this side + // never asks for it. + let lead = match msg.start_byte { + 0 => None, + _ if group.is_compressed() || (ietf && group.ietf().is_some()) => { + return Poll::Ready(Err(Error::ProtocolViolation)); + } + byte => Some(usize::try_from(byte).map_err(|_| Error::WrongSize)?), + }; // The end is a serving cap only: the cached group runs to the end of // the group so it stays usable for anyone else (see // `group::Fetch::frame_start`). Only the end, so the start stays where @@ -3284,6 +3297,7 @@ impl Request for FetchServe { batch: Box::default(), batch_pos: 0, records, + lead, }; } Self::Serve { @@ -3295,11 +3309,36 @@ impl Request for FetchServe { batch, batch_pos, records, + lead, } => loop { ready!(writer.poll_flush(waiter))?; if let Some(records) = records { ready!(records.poll_state(writer, group, waiter))?; } + if let Some(byte) = *lead { + let Some(next) = ready!(group.poll_next_frame_at(byte, waiter))? else { + // The group ends before the frame the request starts in. + return Poll::Ready(Err(Error::WrongSize)); + }; + *lead = None; + if let Some(records) = records { + records.frame(writer, group.index() - 1, next.fields.as_deref())?; + } + // The frame's full size: the subscriber knows it asked for the rest. + buffer_frame_info( + writer, + shared.version, + next.timestamp, + next.inflated, + next.size, + *stamp, + prev_ts, + &shared.runtime, + next.size - byte as u64, + )?; + *frame = Some(next); + continue; + } if let Some(pending) = chunk { ready!(writer.poll_write(waiter, pending))?; if !bytes::Buf::has_remaining(pending) { @@ -3325,6 +3364,7 @@ impl Request for FetchServe { *stamp, prev_ts, &shared.runtime, + batched.frame.payload.len() as u64, )?; let payload = std::mem::take(&mut batched.frame.payload); if !payload.is_empty() { @@ -3361,6 +3401,7 @@ impl Request for FetchServe { *stamp, prev_ts, &shared.runtime, + next.size, )?; *frame = Some(next); return Poll::Ready(Ok(ControlFlow::Continue(()))); @@ -4623,6 +4664,8 @@ fn buffer_frame_info( stamp: Option, prev_ts: &mut u64, runtime: &crate::time::Deadlines, + // Payload bytes that follow: fewer than `size` only for a frame served from a byte. + sent: u64, ) -> Result<(), Error> { // Lite-07 leads with the size plus one, leaving 0 for GROUP_END. let length_first = version.has_length_first(); @@ -4642,7 +4685,7 @@ fn buffer_frame_info( if !length_first { writer.buffer_varint(size)?; } - writer.frame(size); + writer.frame(sent); Ok(()) } @@ -6739,6 +6782,7 @@ impl GroupServe { self.ctx.stamp, &mut self.prev_ts, &self.ctx.runtime, + batched.frame.payload.len() as u64, ) }); if let Err(err) = buffered { @@ -6794,6 +6838,7 @@ impl GroupServe { self.ctx.stamp, &mut self.prev_ts, &self.ctx.runtime, + next.size, ) }); if let Err(err) = buffered { @@ -11078,6 +11123,7 @@ mod tests { max_delay: None, group: 0, start_frame: 0, + start_byte: 0, end_frame: None, }; leave_request::(version, &fetch, wait, close).await; @@ -12136,6 +12182,7 @@ mod tests { max_delay: None, group: 0, start_frame: 2, + start_byte: 0, end_frame: Some(3), }, "video", @@ -12206,6 +12253,7 @@ mod tests { max_delay: None, group: 0, start_frame: 0, + start_byte: 0, end_frame: None, } .encode(&mut crate::coding::Encoder::new(&mut script, VERSION.into()), VERSION) @@ -12282,6 +12330,7 @@ mod tests { max_delay: Some(Duration::from_millis(500)), group: 0, start_frame: 0, + start_byte: 0, end_frame: None, } .encode(&mut crate::coding::Encoder::new(&mut script, VERSION.into()), VERSION) @@ -12311,6 +12360,89 @@ mod tests { assert_eq!(session.log.resets(), vec![crate::StreamError::Old.to_code()]); } + /// A FETCH from partway into a frame of a moq-transport track, whose object records name + /// whole objects, is refused once the `ietf` extension carries those records; a moq-lite + /// track on the same session is served from the byte. + #[kio::test] + async fn a_start_byte_is_refused_on_a_track_with_object_records() { + const VERSION: Version = Version::Lite07; + for transport in [false, true] { + let origin = crate::origin::Config::new(std::num::NonZeroU64::new(1).unwrap()).produce(); + let broadcast = origin.route("room", crate::origin::RouteOptions::hidden()).unwrap(); + let mut info = track::Info::timed(); + if transport { + info.properties = Some(Arc::new(crate::model::object::TrackProperties { + version: crate::ietf::Version::Draft18, + values: Vec::new(), + })); + } + let track = broadcast.broadcast().create_track("video", info).unwrap(); + broadcast.announce().unwrap(); + let mut group = track.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"abcd".to_vec()).unwrap(); + group.finish().unwrap(); + + let peer_setup = crate::lite::PeerSetup::new(false, crate::transport::Address::Setup).offer_ietf(true); + peer_setup + .set(crate::lite::Setup { + ietf: true, + ..Default::default() + }) + .unwrap(); + let (_, goaway) = crate::goaway::Handle::new(true); + let publisher = Publisher::new(PublisherConfig { + runtime: crate::time::sim(), + session: crate::driver::Transport::new(ScriptedSession::new(Vec::new())), + origin: origin.consume(), + version: VERSION, + peer_setup, + goaway, + peer_hop: None, + auth: crate::auth::Link::new(false), + peer_grant: Default::default(), + client: false, + subscriptions: Default::default(), + estimate: kio::Producer::default().weak(), + }); + let mut script = Vec::new(); + lite::Fetch { + instance: None, + node: 0, + broadcast: crate::Path::from_static("room"), + track: "video".into(), + priority: 0, + max_delay: None, + group: 0, + start_frame: 0, + start_byte: 2, + end_frame: None, + } + .encode(&mut crate::coding::Encoder::new(&mut script, VERSION.into()), VERSION) + .unwrap(); + let session = ScriptedSession::new(script); + let (send, recv) = kio::wait(|waiter| { + ::poll_open_bi(&session, waiter, Default::default()) + }) + .await + .unwrap(); + let stream = Stream:: { + writer: Writer::new(crate::transport::boxed::SendStream::new(send), VERSION), + reader: crate::coding::Reader::new(crate::transport::boxed::RecvStream::new(recv), VERSION), + }; + let mut serve = RequestServe::::new(publisher.shared.clone(), stream); + assert!(drive(&mut serve).await, "the fetch ended"); + let refused = crate::StreamError::Session(SessionError::ProtocolViolation).to_code(); + match transport { + true => assert_eq!(session.log.resets(), vec![refused]), + false => { + assert!(session.log.resets().is_empty(), "a moq-lite track serves from the byte"); + let written = session.log.writes.lock().unwrap(); + assert!(written.ends_with(b"cd") && !written.ends_with(b"abcd")); + } + } + } + } + #[kio::test] async fn report_consumes_probe_targets_until_fin() { for version in [ diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index b5cd3f3916c..faf9436b70d 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -2029,10 +2029,17 @@ impl FrameIngest { /// Where the frame `header` opens goes: a new frame of the group, or on a continuation, /// one the group holds already, skipped, or the one a YIELDED reset cut short, completed. - fn dispatch(&mut self, group: &crate::recv::Group, header: frame::Header) -> Result { + fn dispatch(&mut self, group: &mut crate::recv::Group, header: frame::Header) -> Result { let index = self.index; // The peer's Frame Start can number a frame past any count. self.index = index.checked_add(1).ok_or(Error::ProtocolViolation)?; + // A response from a byte: the header states the frame's full size, and only the rest + // of it follows. + if group.lead_at(index).is_some() { + return Ok(IngestPhase::Payload { + frame: group.create_lead(header)?, + }); + } let held = group.frame_count() as u64; if index >= held { return Ok(IngestPhase::Admit { header }); @@ -2083,7 +2090,9 @@ impl FrameIngest { let lite::ietf::Decoded { properties, gap } = lite::ietf::decode_properties(&block, self.framing, true)?; group.mark_gap(id, gap)?; - let plain = id == group.frame_count() as u64 && properties.is_empty(); + // The rest of a frame the copy starts partway into is numbered below its count. + let next = id == group.frame_count() as u64 || group.lead_at(id).is_some(); + let plain = next && properties.is_empty(); if !plain && !records.relayed { return Err(Error::ProtocolViolation); } @@ -13969,6 +13978,7 @@ enum FetchRunState { request: Option, stream: Stream, frame_start: u64, + byte_start: u64, frame_end: Option, }, /// Flushed; waiting for the publisher to answer before accepting. @@ -13976,6 +13986,7 @@ enum FetchRunState { request: Option, stream: Stream, frame_start: u64, + byte_start: u64, frame_end: Option, }, Ingest { @@ -14138,6 +14149,15 @@ impl kio::Task for FetchServeRun { false => 0, }; + // Only the rest of a frame a reader got partway through, which a compressed + // frame cannot start at: it inflates only from its first byte. Nor can a + // moq-transport track's, whose object records name whole objects. + let whole_objects = matches!(self.info.peek(), Some(Ok(info)) if info.ietf().is_some()); + let byte_start = match self.serve.subscriber.version.has_byte_start() { + true if plain && !whole_objects && frame_start == request.frame_start() => request.byte_start(), + _ => 0, + }; + let frame_end = if self.serve.subscriber.version.has_frame_bounds() && plain { request.frame_end() } else { @@ -14156,6 +14176,7 @@ impl kio::Task for FetchServeRun { .filter(|_| self.serve.subscriber.version.has_fetch_max_delay()), group: self.group, start_frame: frame_start, + start_byte: byte_start, // A bounded plain copy keeps this provenance in the cache; older wires // and compressed groups widen to a complete representation. end_frame: frame_end.and_then(|end| end.checked_sub(1)), @@ -14175,6 +14196,7 @@ impl kio::Task for FetchServeRun { request: Some(request), stream, frame_start, + byte_start, frame_end, }; } @@ -14193,6 +14215,7 @@ impl kio::Task for FetchServeRun { request, mut stream, frame_start, + byte_start, frame_end, } = std::mem::replace(&mut self.state, FetchRunState::Done) else { @@ -14205,6 +14228,7 @@ impl kio::Task for FetchServeRun { request, stream, frame_start, + byte_start, frame_end, }; } @@ -14241,6 +14265,7 @@ impl kio::Task for FetchServeRun { request, mut stream, frame_start, + byte_start, frame_end, } = std::mem::replace(&mut self.state, FetchRunState::Done) else { @@ -14272,6 +14297,7 @@ impl kio::Task for FetchServeRun { group_info, group::Bounds { start: frame_start, + byte: byte_start, end: frame_end, }, ) { diff --git a/rs/moq-net/src/lite/version.rs b/rs/moq-net/src/lite/version.rs index 03e35385345..7626719c2e4 100644 --- a/rs/moq-net/src/lite/version.rs +++ b/rs/moq-net/src/lite/version.rs @@ -210,6 +210,15 @@ impl Version { ) } + /// Whether FETCH carries a `Start Byte` within its first frame, so a reader cut off + /// partway through a frame asks for only the rest of it. Added in lite-07. + pub(crate) fn has_byte_start(self) -> bool { + !matches!( + self, + Self::Lite01 | Self::Lite02 | Self::Lite03 | Self::Lite04 | Self::Lite05 | Self::Lite06 + ) + } + /// Whether this version's SUBSCRIBE, SUBSCRIBE_UPDATE, SUBSCRIBE_OK, and TRACK_INFO /// carry the retired `Ordered` byte. /// diff --git a/rs/moq-net/src/model/frame.rs b/rs/moq-net/src/model/frame.rs index 103731bb08c..3f0c7f5399b 100644 --- a/rs/moq-net/src/model/frame.rs +++ b/rs/moq-net/src/model/frame.rs @@ -1001,7 +1001,6 @@ impl FrameBuf { } /// The canonical allocation's charge, counted once by its cache owner. - #[cfg(test)] pub(super) fn content_size(&self) -> u64 { self.content.bill.get().map_or(0, cache::Extra::size) } @@ -1299,6 +1298,9 @@ pub struct Producer { stats: stats::Meter, // The frame's share of the origin's [`Budget`], if it got one, held until it ends. admission: Option, + // Writes the rest of the frame its copy starts partway into (see + // [`group::Producer::create_lead`]), which stays out of the frame log. + lead: bool, } impl std::ops::Deref for Producer { @@ -1319,9 +1321,17 @@ impl Producer { cancel_on_drop: false, stats: stats::Meter::default(), admission, + lead: false, } } + /// Write the rest of the frame a copy starts partway into. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(super) fn lead(mut self) -> Self { + self.lead = true; + self + } + /// Attach the parent group's ingress meter, so written chunks bump `bytes`. pub(super) fn with_meter(mut self, meter: stats::Meter) -> Self { self.stats = meter; @@ -1405,8 +1415,14 @@ impl Producer { if self.buf.written(Ordering::Acquire) != self.buf.size() { return Err(Error::WrongSize); } - let payload = self.buf.freeze(); let mut group = self.group.take().expect("frame owns its group"); + if self.lead { + self.done = true; + // Its readers already hold its buffer, so only the wake is left. + group.frame_notify(); + return Ok(group); + } + let payload = self.buf.freeze(); let committed = group.frame_commit(Cached { timestamp: self.info.timestamp, payload, @@ -1516,7 +1532,10 @@ pub struct Consumer { #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] pub(crate) inflated: Option, source: Source, - // Byte offset consumed so far. + // The frame offset `source` starts at: non-zero only for the rest of a frame a copy + // started partway into (see [`group::Consumer::at`]). + base: usize, + // Byte offset consumed so far, from the frame's start. read_idx: usize, // Egress payload meter, so chunks bump `bytes` exactly once as they're read out. // Empty (no-op) for an untagged group. @@ -1546,6 +1565,7 @@ impl Consumer { fields: header.fields, inflated: header.inflated, source, + base: 0, read_idx: 0, stats: stats::Meter::default(), expiry: None, @@ -1554,6 +1574,15 @@ impl Consumer { } } + /// Read from frame offset `from`, with `source` holding the frame from `base`, which + /// is at most `from`. + pub(super) fn starting_at(mut self, base: usize, from: usize) -> Self { + debug_assert!(base <= from); + self.base = base; + self.read_idx = from; + self + } + /// Carry this frame across a front's route changes; see [`super::resume`]. pub(super) fn with_recover(mut self, recover: super::resume::Recover, index: u64) -> Self { self.recover = Some(Box::new((recover, index))); @@ -1592,7 +1621,9 @@ impl Consumer { if self.state.read().sparse() { return Poll::Ready(Err(Error::Unsupported)); } - let mut replacement = match recover.poll(*index, failed.as_ref(), waiter) { + // Only the rest of the frame is asked for: the bytes already read are not sent again. + let byte = self.read_idx as u64; + let mut replacement = match recover.poll(*index, byte, failed.as_ref(), waiter) { Poll::Ready(Ok(group)) => group, Poll::Ready(Err(err)) => return Poll::Ready(Err(err)), Poll::Pending => return Poll::Pending, @@ -1607,7 +1638,7 @@ impl Consumer { if self.inflated.is_some() { replacement.group = replacement.group.compressed(); } - let frame = match replacement.group.poll_next_frame(waiter) { + let frame = match replacement.group.poll_next_frame_at(self.read_idx, waiter) { Poll::Ready(Ok(Some(frame))) => frame, Poll::Ready(Ok(None)) => return Poll::Ready(Err(Error::WrongSize)), Poll::Ready(Err(err)) => return Poll::Ready(Err(err)), @@ -1627,9 +1658,14 @@ impl Consumer { if frame.info.size != self.info.size || frame.inflated != self.inflated { return Poll::Ready(Err(Error::ProtocolViolation)); } + // Positioned at this read's byte, so the replacement holds every byte from it. + if frame.base > self.read_idx { + return Poll::Ready(Err(Error::Lagged)); + } recover.adopt(&replacement); self.state = frame.state; self.source = frame.source; + self.base = frame.base; } } @@ -1647,7 +1683,7 @@ impl Consumer { fn size(&self) -> usize { match &self.source { - Source::Complete(payload) => payload.len(), + Source::Complete(payload) => self.base + payload.len(), Source::Partial(_) => self.info.size as usize, } } @@ -1695,15 +1731,17 @@ impl Consumer { if self.expired { return Poll::Ready(Err(Error::Old)); } - let size = self.info.size as usize; + // The source's own offsets: it holds the frame from `base`. + let size = self.info.size as usize - self.base; loop { + let at = self.read_idx - self.base; // Borrowed rather than cloned: every reader of the frame shares its count. let buf = match &self.source { Source::Complete(payload) => { - if self.read_idx >= payload.len() { + if at >= payload.len() { return Poll::Ready(Ok(None)); } - let out = payload.chunk_at(self.read_idx); + let out = payload.chunk_at(at); self.read_idx += out.len(); self.stats.bytes(out.len() as u64); return Poll::Ready(Ok(Some(out))); @@ -1711,8 +1749,8 @@ impl Consumer { Source::Partial(buf) => buf, }; let written = buf.written(Ordering::Acquire); - if written > self.read_idx { - let out = buf.slice(self.read_idx, written); + if written > at { + let out = buf.slice(at, written); self.read_idx += out.len(); self.stats.bytes(out.len() as u64); return Poll::Ready(Ok(Some(out))); @@ -1728,7 +1766,7 @@ impl Consumer { let Source::Partial(buf) = &self.source else { unreachable!("only a partial frame waits for its bytes"); }; - let read_idx = self.read_idx; + let read_idx = at; // Park on the group's channel; the producer notifies it on each write and // on abort. Re-check the atomic on wake. ready!(poll_state(&self.state, waiter, |state| { @@ -1758,18 +1796,18 @@ impl Consumer { if self.expired { return Poll::Ready(Err(Error::Old)); } + let read_idx = self.read_idx - self.base; let buf = match &self.source { Source::Complete(payload) => { - let out = payload.contiguous(self.read_idx); - self.read_idx = payload.len(); + let out = payload.contiguous(read_idx); + self.read_idx = self.base + payload.len(); self.stats.bytes(out.len() as u64); return Poll::Ready(Ok(out)); } Source::Partial(buf) => buf.clone(), }; - let size = self.info.size as usize; - let read_idx = self.read_idx; + let size = self.info.size as usize - self.base; // Waiting on the rest of the payload is a stall; see `poll_read_chunk`. if buf.written(Ordering::Acquire) < size && self.poll_expired(waiter) { return Poll::Ready(Err(Error::Old)); @@ -1785,7 +1823,7 @@ impl Consumer { } })?); let out = buf.freeze().contiguous(read_idx); - self.read_idx = size; + self.read_idx = self.base + size; self.stats.bytes(out.len() as u64); Poll::Ready(Ok(out)) } diff --git a/rs/moq-net/src/model/group.rs b/rs/moq-net/src/model/group.rs index d6dce13fb05..46198acacc8 100644 --- a/rs/moq-net/src/model/group.rs +++ b/rs/moq-net/src/model/group.rs @@ -211,9 +211,26 @@ impl Entry { #[derive(Clone, Copy, Debug, Default)] pub(crate) struct Bounds { pub start: u64, + /// The first byte held of frame `start`: non-zero when the copy starts partway into + /// it, which only a read continuing that frame from a later byte can use (see [`Lead`]). + pub byte: u64, pub end: Option, } +/// The frame a copy holds only the rest of, from byte `byte` on: a transport started the copy +/// partway into frame `index`, the one before [`GroupState::offset`]. +/// +/// Kept beside the frame log rather than in it, so every whole-frame read of the copy finds +/// frame `index` missing ([`Error::Lagged`]), as it would on a copy starting at the next +/// frame. Only a read continuing the frame from `byte` or later reaches it, through +/// [`Consumer::at`]. +pub(super) struct Lead { + index: usize, + byte: usize, + // Opened by the first frame the copy's ingest writes, with the frame's full size. + open: Option<(frame::Info, FrameBuf)>, +} + /// Shared group state. Model-visible so [`frame`] handles can observe its outcome /// while streaming a partial frame. #[derive(Default)] @@ -234,6 +251,10 @@ pub(super) struct GroupState { // The single in-flight frame, if one is open. pub(super) partial: Option, + // The frame before `offset` this copy holds only a suffix of, if it started partway into + // it. Boxed: it is the rare case. + lead: Option>, + // Index of the first frame this handle holds: any the group deliberately started // past (see [`Producer::start_at`]). Reading below it is [`Error::Lagged`]; the // frames are not here. Frames shed from the front since sit between it and @@ -903,10 +924,19 @@ impl GroupState { payload + fields } - /// The bytes the live tail holds: every committed frame, their fields, and the - /// in-flight frame at its declared size. + /// The bytes the live tail holds: every committed frame, their fields, the rest of a + /// frame the copy started partway into, and the in-flight frame at its declared size. fn tail(&self) -> u64 { - self.cache + self.relay.as_ref().map_or(0, |relay| relay.bytes) + self.unwritten() + self.cache + self.relay.as_ref().map_or(0, |relay| relay.bytes) + self.lead_bytes() + self.unwritten() + } + + /// The rest of the frame the copy started partway into, at its declared size like the + /// in-flight frame, since its later writes trim nothing. + fn lead_bytes(&self) -> u64 { + self.lead + .as_deref() + .and_then(|lead| lead.open.as_ref()) + .map_or(0, |(_, buf)| buf.size() as u64) } /// The frame slots the tail holds beyond the log's first segment, so a group of empty @@ -999,6 +1029,12 @@ impl GroupState { if tail.saturating_add(slots) <= MAX_CACHE_BYTES { return true; } + // The lead is the oldest frame, so it goes first. Only a later frame trims, once the + // lead is written; its readers keep the bytes they hold. + if self.lead.as_deref().is_some_and(|lead| lead.open.is_some()) { + tail -= self.lead_bytes(); + self.lead = None; + } let mut end = self.held(); while tail.saturating_add(slots) > MAX_CACHE_BYTES && let Some(entry) = self.entry(end) @@ -1447,6 +1483,7 @@ impl GroupState { self.front = VecDeque::new(); self.evict_log(self.log.end()); self.partial = None; + self.lead = None; if let Some(content) = self.content.take() { content.detach(self.outcome.error().cloned().unwrap_or(Error::Dropped)); } @@ -1820,8 +1857,26 @@ impl Producer { } /// Apply transport bounds before this copy is published to the cache. + /// + /// A copy starting partway into frame `start` starts at the next frame, holding the + /// rest of `start` aside as its [`Lead`], which the first frame written opens. pub(crate) fn bound(&mut self, frames: Bounds) -> Result<()> { - self.start_at(frames.start)?; + match frames.byte { + 0 => self.start_at(frames.start)?, + byte => { + let next = frames + .start + .checked_add(1) + .ok_or(Error::BoundsExceeded(crate::coding::BoundsExceeded))?; + self.start_at(next)?; + let byte = usize::try_from(byte).map_err(|_| Error::BoundsExceeded(crate::coding::BoundsExceeded))?; + modify(&self.handle.state)?.lead = Some(Box::new(Lead { + index: frames.start as usize, + byte, + open: None, + })); + } + } if let Some(end) = frames.end { let end = usize::try_from(end).map_err(|_| Error::BoundsExceeded(crate::coding::BoundsExceeded))?; let mut state = modify(&self.handle.state)?; @@ -2081,6 +2136,65 @@ impl Producer { Ok(frame::Producer::new(self, buf, info, admission).with_meter(meter)) } + /// Open the rest of the frame this copy starts partway into (see [`Bounds::byte`]): + /// `frame` declares the frame's full size, and the producer takes only the bytes past + /// the copy's start. + /// + /// [`Error::ProtocolViolation`] unless the copy starts partway into a frame it has not + /// opened yet, or when the frame is no larger than that start: a transport answering + /// from a byte always has more of the frame to send. A compressed frame never starts + /// partway, since its bytes inflate only from its first. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn create_lead(mut self, frame: frame::Header) -> Result { + let meter = self.chunk_meter(None); + let res = self.open_lead(frame); + match res { + Ok((info, buf, admission)) => { + // Ingress: one frame opened, its bytes counted per chunk, as `create_object` does. + self.stats.frames(1); + Ok(frame::Producer::new(self, buf, info, admission) + .lead() + .with_meter(meter)) + } + Err(err) => { + let _ = self.handle.abort(err.clone()); + Err(err) + } + } + } + + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + fn open_lead(&mut self, header: frame::Header) -> Result<(frame::Info, FrameBuf, Option)> { + let frame = header.info; + let timestamp = self.handle.track.rescale(frame.timestamp)?; + if frame.size > MAX_CACHE_BYTES { + return Err(Error::FrameTooLarge); + } + if header.inflated.is_some() || header.fields.is_some() { + return Err(Error::ProtocolViolation); + } + let mut state = modify(&self.handle.state)?; + let Some(lead) = state.lead.as_deref().filter(|lead| lead.open.is_none()) else { + return Err(Error::ProtocolViolation); + }; + let rest = usize::try_from(frame.size) + .ok() + .and_then(|size| size.checked_sub(lead.byte)) + .filter(|rest| *rest > 0) + .ok_or(Error::ProtocolViolation)?; + let (buf, admission) = FrameBuf::admitted(rest, self.handle.cache.frames()); + // Charged as its bytes arrive, and freed with the last reader holding them. + buf.shared().bill_content(state.charge.extra()); + let info = frame::Info { + size: frame.size, + timestamp, + }; + state.lead.as_deref_mut().expect("checked above").open = Some((info, buf.clone())); + // The ingest settled the relay state before writing the lead, as before any frame. + state.wrote = true; + Ok((info, buf, admission)) + } + /// The meter a streamed frame's chunks bump: none for a plain frame a compressed group /// compresses on commit, which counts its stored slice then, as `bytes` counts payload /// as stored. @@ -2677,7 +2791,15 @@ impl Handle { let same_form = head.compressed == state.flate.is_some(); // A copy that shed its front holds no run from `offset` for the head to precede. let shed = state.held() != offset; - if state.outcome.error().is_some() || state.sparse() || !same_form || shed || head.start >= offset || !reaches { + // Nor does one holding only the rest of the frame below `offset` (see [`Lead`]). + let lead = state.lead.is_some(); + if state.outcome.error().is_some() + || state.sparse() + || !same_form + || shed || lead + || head.start >= offset + || !reaches + { return; } let count = offset - head.start; @@ -2912,7 +3034,11 @@ impl Handle { pub(super) fn covers(&self, frames: Bounds) -> bool { let state = self.state.read(); state.outcome.error().is_none() - && state.offset as u64 <= frames.start + && (state.offset as u64 <= frames.start + || state + .lead + .as_ref() + .is_some_and(|lead| lead.index as u64 == frames.start && lead.byte as u64 <= frames.byte)) && match (state.range_end, frames.end) { (None, _) => true, (Some(cap), Some(end)) => end <= cap as u64, @@ -2920,6 +3046,15 @@ impl Handle { } } + /// The byte this copy starts frame `index` at, if it holds only the rest of it and + /// has not opened it yet (see [`Producer::create_lead`]). + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn lead_at(&self, index: u64) -> Option { + let state = self.state.read(); + let lead = state.lead.as_deref()?; + (lead.index as u64 == index && lead.open.is_none()).then_some(lead.byte as u64) + } + /// Whether this copy was cut short at the back by its transport request. pub(super) fn back_cut(&self) -> bool { self.state.read().range_end.is_some() @@ -3078,6 +3213,12 @@ impl Handle { || state.flate.as_ref().map_or(0, |flate| flate.plain.bill.size()), |content| content.size(), ) + state.partial.as_ref().map_or(0, |partial| partial.buf.view_size()) + // The rest of a frame the copy started partway into, billed apart from its charge. + + state + .lead + .as_deref() + .and_then(|lead| lead.open.as_ref()) + .map_or(0, |(_, buf)| buf.content_size()) } /// Whether the payload cached after the first frame exceeds `ratio` times the first @@ -4033,7 +4174,7 @@ impl Consumer { } let index = self.cursor.index as u64; let recover = self.recover_mut().expect("only a recovering group needs a replacement"); - let replacement = ready!(recover.poll(index, failed.as_ref(), waiter))?; + let replacement = ready!(recover.poll(index, 0, failed.as_ref(), waiter))?; // A plain read inflates from frame 0, which a replacement starting later lacks // when the failed route's head was not lent to it. if !self.cursor.raw @@ -4177,6 +4318,99 @@ impl Consumer { self } + /// This reader positioned at frame `index`, to continue that frame from byte `byte` + /// with [`Self::poll_next_frame_at`], or `None` when this copy does not hold it from + /// there: it starts past the frame, or partway into it past `byte`. + pub(crate) fn at(self, index: u64, byte: u64) -> Option { + let mut group = self.with_range(index..); + if group.index() == index { + return Some(group); + } + let starts = { + let state = group.cursor.state.read(); + state + .lead + .as_deref() + .is_some_and(|lead| lead.index as u64 == index && lead.byte as u64 <= byte) + }; + if !starts || group.index() != index + 1 { + return None; + } + // Below the copy's start: only `poll_next_frame_at` reads the frame there. + group.cursor.index = index as usize; + group.cursor.prefetch = Prefetch::default(); + group.cursor.partial = false; + Some(group) + } + + /// Whether the group's track compresses its frames. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn is_compressed(&self) -> bool { + self.cursor.compressed + } + + /// The next frame, to stream from byte `from` on, which a copy starting partway into + /// it (see [`Bounds::byte`]) serves if it starts at or before `from`. + /// + /// [`Error::WrongSize`] when a non-zero `from` is not within the frame. + pub(crate) fn poll_next_frame_at( + &mut self, + from: usize, + waiter: &kio::Waiter, + ) -> Poll>> { + if self.expired { + return Poll::Ready(Err(Error::Old)); + } + let index = self.cursor.index; + let res = self.cursor.poll(waiter, |state| { + let Some(lead) = state.lead.as_deref().filter(|lead| lead.index == index) else { + return Poll::Ready(Ok(None)); + }; + if from < lead.byte { + return Poll::Ready(Err(Error::Lagged)); + } + match &lead.open { + Some((info, buf)) => Poll::Ready(Ok(Some((*info, buf.clone(), lead.byte)))), + None if state.outcome.error().is_some() || state.outcome.frames().is_some() => { + Poll::Ready(Err(state.outcome.error().cloned().unwrap_or(Error::Lagged))) + } + None => Poll::Pending, + } + }); + // A lead that stops arriving is judged by the read's budget, as a whole frame is. + let lead = ready!(self.poll_expired_if_blocked(waiter, res))?; + let frame = match lead { + Some((info, _, _)) if from as u64 >= info.size => return Poll::Ready(Err(Error::WrongSize)), + Some((info, buf, base)) => { + self.cursor.index += 1; + self.stats.frames(1); + let header = frame::Header { + info, + fields: None, + inflated: None, + }; + let frame = frame::Consumer::new(self.cursor.state.clone(), header, frame::Source::Partial(buf)) + .with_meter(self.stats.clone()) + .starting_at(base, from); + let end = self.cursor.end.unwrap_or(usize::MAX); + match &self.expiry { + Some(policy) => frame.with_expiry( + frame::Expiry::new(policy.clone(), self.stats.clone(), self.delivery.clone()) + .for_frame(self.cursor.index..end, true), + ), + None => frame, + } + } + None => match ready!(self.poll_next_frame(waiter))? { + // A non-zero start names a byte of the frame, so the rest is never empty. + Some(frame) if from > 0 && from as u64 >= frame.size => return Poll::Ready(Err(Error::WrongSize)), + Some(frame) => frame.starting_at(0, from), + None => return Poll::Ready(Ok(None)), + }, + }; + Poll::Ready(Ok(Some(frame))) + } + /// Cap reads before frame `end`, or remove the cap with `None`, without moving the /// start, which [`Self::with_range`] would clamp up to the first available frame. pub(crate) fn with_end(mut self, end: impl Into>) -> Self { @@ -4555,6 +4789,7 @@ impl Consumer { pub(super) fn unread(&self, index: usize) -> Bounds { Bounds { start: index as u64, + byte: 0, end: self.cursor.end.map(|end| end as u64), } } @@ -5041,6 +5276,10 @@ pub struct Fetch { /// There is no public matching end. Private transport caps retain incomplete cache /// provenance, so a bounded reply never answers a later whole-group fetch. pub(crate) frame_start: u64, + + /// The first byte to fetch of frame `frame_start`. Like that start, only a route change + /// sets it: a frame read partway when its route failed asks for the rest of it. + pub(crate) byte_start: u64, } impl Fetch { @@ -5720,6 +5959,30 @@ mod test { ); } + /// A copy holding only the rest of a frame adopts no head over it: the head would hold + /// that frame whole beside its rest, outside the live tail's admission. + #[test] + fn a_copy_with_a_lead_adopts_no_head() { + let mut producer = Info { sequence: 0 }.produce(); + producer + .bound(Bounds { + start: 1, + byte: 1, + end: None, + }) + .unwrap(); + let adopter = producer.handle(); + let head = Head { + content: None, + start: 0, + first_object: true, + entries: vec![Entry::plain(Frame::new(None, Bytes::from_static(b"ab")).into(), 0); 2], + compressed: false, + }; + adopter.adopt_head(&head); + assert_eq!(adopter.state.read().offset, 2, "the head was adopted over the lead"); + } + /// A head's slots count against the live tail, so a head of empty frames that would /// take the group's slots past it is not adopted. #[test] @@ -6635,6 +6898,140 @@ mod test { } } + /// A copy started partway into frame 1 holds its rest aside: whole reads find the frame + /// missing, a read from at or past the copy's start byte streams the rest, and the frames + /// after it read as usual. + #[test] + #[cfg(feature = "async")] + fn a_copy_started_partway_into_a_frame_serves_only_its_rest() { + let bounded = || { + let mut producer = Info { sequence: 0 }.produce(); + producer + .bound(Bounds { + start: 1, + byte: 3, + end: None, + }) + .unwrap(); + producer + }; + let next_at = |group: &mut Consumer, from| match group.poll_next_frame_at(from, &kio::Waiter::noop()) { + Poll::Ready(res) => Some(res), + Poll::Pending => None, + }; + + // The rest of a frame no larger than the start is no frame at all. + assert!(matches!( + bounded().create_lead(sized(3).into()), + Err(Error::ProtocolViolation) + )); + + let producer = bounded(); + let handle = producer.handle(); + assert_eq!(handle.lead_at(1), Some(3)); + let covers = |start, byte| handle.covers(Bounds { start, byte, end: None }); + assert!(covers(1, 3) && covers(1, 4) && covers(2, 0)); + assert!(!covers(1, 2) && !covers(1, 0) && !covers(0, 0)); + + let mut lagged = producer.consume(); + lagged.skip_to(1); + assert!(matches!(lagged.read_frame().now_or_never(), Some(Err(Error::Lagged)))); + let mut plain = producer.consume().with_range(0..); + assert_eq!(plain.index(), 2, "a whole read starts past the frame"); + assert!(plain.clone().at(1, 2).is_none(), "the copy starts past byte 2"); + let mut rest = plain.clone().at(1, 4).unwrap(); + let mut early = plain.clone().at(1, 3).unwrap(); + let mut past = plain.clone().at(1, 10).unwrap(); + let mut edge = plain.clone().at(1, 6).unwrap(); + assert!(next_at(&mut rest, 4).is_none(), "waits for the frame to open"); + + let before = handle.cache_size(); + let mut lead = producer.create_lead(sized(6).into()).unwrap(); + assert_eq!(handle.lead_at(1), None, "opened"); + assert!( + plain.clone().poll_wire(&kio::Waiter::noop()).is_ready(), + "the lead settles the relay state" + ); + assert_eq!(lead.remaining(), 3, "only the rest is written"); + lead.write(b"de".as_ref()).unwrap(); + assert!( + matches!(next_at(&mut past, 10), Some(Err(Error::WrongSize))), + "a byte past the frame's end" + ); + assert!( + matches!(next_at(&mut edge, 6), Some(Err(Error::WrongSize))), + "the frame's end leaves no rest" + ); + let mut frame = next_at(&mut rest, 4).unwrap().unwrap().unwrap(); + assert_eq!(frame.size, 6, "it states the frame's full size"); + assert_eq!(&frame.read_chunk().now_or_never().unwrap().unwrap().unwrap()[..], b"e"); + lead.write(b"f".as_ref()).unwrap(); + assert!( + handle.cache_size() >= before + 3, + "the rest counts toward the group's eviction size" + ); + let mut producer = lead.finish().unwrap(); + producer.write_frame(Timestamp::ZERO, b"g".as_ref()).unwrap(); + producer.finish().unwrap(); + assert_eq!(&frame.read_chunk().now_or_never().unwrap().unwrap().unwrap()[..], b"f"); + assert!(frame.read_chunk().now_or_never().unwrap().unwrap().is_none()); + assert_eq!( + rest.read_frame().now_or_never().unwrap().unwrap().unwrap().payload, + b"g"[..] + ); + + let mut frame = next_at(&mut early, 3).unwrap().unwrap().unwrap(); + assert_eq!(frame.read_all().now_or_never().unwrap().unwrap(), b"def"[..]); + + let read = plain.read_frame().now_or_never().unwrap().unwrap().unwrap(); + assert_eq!(read.payload, b"g"[..], "whole reads skip the frame held in part"); + } + + /// The rest of a frame the copy started partway into counts toward the live tail, and + /// is the first frame a later one sheds past it; a reader already holding it keeps it. + #[test] + #[cfg(feature = "async")] + fn a_lead_is_shed_first_past_the_live_tail() { + let half = MAX_CACHE_BYTES / 2; + let mut producer = Info { sequence: 0 }.produce(); + producer + .bound(Bounds { + start: 0, + byte: 1, + end: None, + }) + .unwrap(); + let handle = producer.handle(); + let mut holding = producer.consume().at(0, 1).unwrap(); + let mut late = producer.consume().at(0, 1).unwrap(); + + let mut lead = producer.create_lead(sized(half + 1).into()).unwrap(); + lead.write(Bytes::from(vec![1u8; half as usize])).unwrap(); + let mut producer = lead.finish().unwrap(); + let mut frame = match holding.poll_next_frame_at(1, &kio::Waiter::noop()) { + Poll::Ready(Ok(Some(frame))) => frame, + _ => panic!("the lead is readable"), + }; + assert!(handle.cache_size() >= half); + + producer + .write_frame(Timestamp::ZERO, Bytes::from(vec![2u8; (half + half / 2) as usize])) + .unwrap(); + assert!(handle.cache_size() < MAX_CACHE_BYTES, "the lead was not shed"); + assert!( + matches!( + late.poll_next_frame_at(1, &kio::Waiter::noop()), + Poll::Ready(Err(Error::Lagged)) + ), + "a reader that had not taken the lead finds it gone" + ); + assert_eq!( + frame.read_all().now_or_never().unwrap().unwrap().len() as u64, + half, + "a reader holding the lead keeps it" + ); + } + /// A declared size costs the peer nothing, so the pool is charged as the bytes /// arrive. Charging the declaration would let a peer evict the cache without /// sending anything. @@ -7252,7 +7649,13 @@ mod test { let cache = cache::Track::new(pool.clone(), Default::default(), kio::Weak::new()); let info = track::Info::default().with_compression(crate::track::Compression::Deflate); let mut released = Producer::new(Info::from(0u64), info.clone(), cache.clone()); - released.bound(Bounds { start: 0, end: Some(2) }).unwrap(); + released + .bound(Bounds { + byte: 0, + start: 0, + end: Some(2), + }) + .unwrap(); for index in 0..2 { released.write_frame(None, verbose(index)).unwrap(); } @@ -7509,6 +7912,57 @@ mod test { assert!(matches!(finished.poll_finished(&waiter), Poll::Ready(Err(Error::Old)))); } + /// A read continuing a frame from a byte gives up with the read's budget, whether the + /// rest of the frame never opens or stops arriving partway. + #[test] + #[cfg(feature = "async")] + fn a_lead_read_expires_as_old() { + let bounded = || { + let mut producer = Info { sequence: 0 }.produce(); + producer + .bound(Bounds { + start: 0, + byte: 1, + end: None, + }) + .unwrap(); + producer + }; + let waiter = kio::Waiter::noop(); + + let producer = bounded(); + let expired = Switch::default(); + let mut group = producer + .consume() + .with_expiry(Arc::new(expired.clone())) + .at(0, 1) + .unwrap(); + assert!(group.poll_next_frame_at(1, &waiter).is_pending(), "waits for the lead"); + expired.set(true); + assert!(matches!( + group.poll_next_frame_at(1, &waiter), + Poll::Ready(Err(Error::Old)) + )); + + let producer = bounded(); + let expired = Switch::default(); + let mut group = producer + .consume() + .with_expiry(Arc::new(expired.clone())) + .at(0, 1) + .unwrap(); + let mut lead = producer.create_lead(sized(4).into()).unwrap(); + lead.write(b"b".as_ref()).unwrap(); + let mut frame = match group.poll_next_frame_at(1, &waiter) { + Poll::Ready(Ok(Some(frame))) => frame, + _ => panic!("the lead is readable"), + }; + assert_eq!(&frame.read_chunk().now_or_never().unwrap().unwrap().unwrap()[..], b"b"); + assert!(frame.read_chunk().now_or_never().is_none(), "waits for the rest"); + expired.set(true); + assert!(matches!(frame.read_chunk().now_or_never(), Some(Err(Error::Old)))); + } + /// Records a wake. #[derive(Default)] struct Woken(AtomicBool); @@ -7623,7 +8077,13 @@ mod test { #[test] fn a_bounded_copy_finished_short_of_its_bound_ends_cleanly() { let mut producer = Info { sequence: 0 }.produce(); - producer.bound(Bounds { start: 0, end: Some(4) }).unwrap(); + producer + .bound(Bounds { + byte: 0, + start: 0, + end: Some(4), + }) + .unwrap(); let mut reader = producer.consume(); for payload in [b"a", b"b"] { producer diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index aee3619e1f3..7516fe21eed 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -27,7 +27,8 @@ //! //! A group carries on across the switch through its [`Recover`]: once its copy stalls while //! a newer route serves, it continues from the serving route's copy of the same group at the -//! same frame (and byte), found in that copy's cache or fetched from it. Once its copy fails, +//! same frame (and byte), found in that copy's cache or fetched from it, which over lite-07 +//! asks only for the bytes of a frame past those already read. Once its copy fails, //! any other copy still holding the group continues it first, since a group's name fixes its //! content. //! @@ -2150,13 +2151,15 @@ impl Recover { || (self.reader.is_none() && matches!(StreamError::from(err), StreamError::Old | StreamError::Evicted)) } - /// Poll for the serving route's copy of the group, positioned at frame `index`. + /// Poll for the serving route's copy of the group, positioned at frame `index` to read + /// it from byte `byte` on: a frame read partway asks only for the rest of it. /// /// `failed` is the error the current copy ended with, if it did; a stall is `None`. /// Returns the copy's error when nothing can continue the group. pub(super) fn poll( &mut self, index: u64, + byte: u64, failed: Option<&Error>, waiter: &kio::Waiter, ) -> Poll> { @@ -2196,7 +2199,7 @@ impl Recover { // fixes its content. A stall only moves to the serving copy, so two stalled copies // never trade the group back and forth. if failed.is_some() - && let Some(replacement) = self.held(index) + && let Some(replacement) = self.held(index, byte) { return Poll::Ready(Ok(replacement)); } @@ -2226,7 +2229,7 @@ impl Recover { if generation == self.generation && failed.is_none() { return Poll::Pending; } - let res = self.poll_serving(generation, &serving, index, failed, withdrawn.is_none(), waiter); + let res = self.poll_serving(generation, &serving, (index, byte), failed, withdrawn.is_none(), waiter); // The serving route failed too, and no front is left to replace it. if let (Poll::Pending, Some(err)) = (&res, failed) && terminal && copy_failed(&serving) @@ -2247,8 +2250,8 @@ impl Recover { } /// The group from the newest of the reader's other copies that still holds it unfailed, - /// positioned at frame `index`. - fn held(&self, index: u64) -> Option { + /// positioned at frame `index` to read from byte `byte`. + fn held(&self, index: u64, byte: u64) -> Option { let reader = self.reader.as_ref()?.upgrade()?; let reader = reader.lock().unwrap_or_else(PoisonError::into_inner); reader.copies.iter().rev().find_map(|copy| { @@ -2257,8 +2260,7 @@ impl Recover { } let group = copy.track.peek_group(self.sequence)?; lend_head(&self.handoff, self.sequence, ©.track, &group); - let group = group.with_range(index..); - (group.index() == index).then(|| Replacement { + group.at(index, byte).map(|group| Replacement { group, generation: copy.generation, copy: copy.track.clone(), @@ -2272,7 +2274,7 @@ impl Recover { &mut self, generation: u64, serving: &track::Consumer, - index: u64, + (index, byte): (u64, u64), failed: Option<&Error>, ask: bool, waiter: &kio::Waiter, @@ -2283,8 +2285,7 @@ impl Recover { match serving.poll_group(self.sequence, waiter) { Poll::Ready(Ok(Some(group))) if !same => { lend_head(&self.handoff, self.sequence, serving, &group); - let group = group.with_range(index..); - if group.index() == index { + if let Some(group) = group.at(index, byte) { return Poll::Ready(Ok(Replacement { group, generation, @@ -2314,19 +2315,27 @@ impl Recover { { verdict = Some(err); } - match self.poll_fetch(generation, serving, index, failed, ask && verdict.is_none(), waiter) { + match self.poll_fetch( + generation, + serving, + (index, byte), + failed, + ask && verdict.is_none(), + waiter, + ) { Poll::Ready(Ok(replacement)) => Poll::Ready(Ok(replacement)), res => verdict.map_or(res, |err| Poll::Ready(Err(err))), } } - /// Ask `serving` for the rest of the group from frame `index`, unless told not to `ask`, and - /// poll the answer: the group, a refusal that ends a copy that failed, or nothing yet. + /// Ask `serving` for the rest of the group from byte `byte` of frame `index`, unless told + /// not to `ask`, and poll the answer: the group, a refusal that ends a copy that failed, + /// or nothing yet. fn poll_fetch( &mut self, generation: u64, serving: &track::Consumer, - index: u64, + (index, byte): (u64, u64), failed: Option<&Error>, ask: bool, waiter: &kio::Waiter, @@ -2339,6 +2348,7 @@ impl Recover { let options = group::Fetch { max_delay: self.max_delay, frame_start: index, + byte_start: byte, ..Default::default() }; self.fetch = Some(( @@ -2356,8 +2366,7 @@ impl Recover { Fetch::Pending(pending) => match ready!(kio::Task::poll(&mut **pending, waiter)) { Ok(group) => { lend_head(&self.handoff, self.sequence, serving, &group); - let group = group.with_range(index..); - if group.index() == index { + if let Some(group) = group.at(index, byte) { return Poll::Ready(Ok(Replacement { group, generation, @@ -3452,6 +3461,122 @@ mod test { assert!(chunks.read_chunk().now_or_never().unwrap().unwrap().is_none()); } + /// A frame read partway when its route dies asks the next route for only the rest of it, + /// which a copy starting partway into the frame serves. + #[test] + #[cfg(feature = "async")] + fn a_half_read_frame_asks_only_for_the_rest() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + + let open = a.create_group(group::Info { sequence: 0 }).unwrap(); + let mut frame = open + .create_frame(crate::frame::Info { + size: 4, + timestamp: Some(ts(0)), + }) + .unwrap(); + frame.write(b"ab".as_ref()).unwrap(); + let mut reading = recv(&mut sub); + let mut chunks = reading.next_frame().now_or_never().unwrap().unwrap().unwrap(); + assert_eq!( + &chunks.read_chunk().now_or_never().unwrap().unwrap().unwrap()[..], + b"ab" + ); + + drop(frame); + a.clone().abort(Error::Dropped).unwrap(); + let b = copy(); + let dynamic = b.requests(); + routes.serve(b.consume()); + assert!(chunks.read_chunk().now_or_never().is_none()); + let request = dynamic.requested_fill_group().now_or_never().unwrap().unwrap(); + assert_eq!((request.frame_start(), request.byte_start()), (0, 2)); + + // The answer holds the frame from byte 2 only, so a whole read of it finds nothing. + let rest = request + .accept_from( + crate::track::Info::timed(), + group::Bounds { + start: 0, + byte: 2, + end: None, + }, + ) + .unwrap(); + let mut lead = rest + .create_lead( + crate::frame::Info { + size: 4, + timestamp: Some(ts(0)), + } + .into(), + ) + .unwrap(); + lead.write(b"cd".as_ref()).unwrap(); + let mut rest = lead.finish().unwrap(); + rest.write_frame(ts(1), b"e".as_ref()).unwrap(); + rest.finish().unwrap(); + + assert_eq!( + &chunks.read_chunk().now_or_never().unwrap().unwrap().unwrap()[..], + b"cd" + ); + assert!(chunks.read_chunk().now_or_never().unwrap().unwrap().is_none()); + assert_eq!(read(&mut reading), Some(b"e".to_vec())); + } + + /// A route that answers a byte ask with whole frames, as moq-transport and app + /// handlers do, keeps each frame at its index, and the read skips the bytes it had. + #[test] + #[cfg(feature = "async")] + fn a_half_read_frame_continues_from_a_whole_frame_answer() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + + let open = a.create_group(group::Info { sequence: 0 }).unwrap(); + let mut frame = open + .create_frame(crate::frame::Info { + size: 4, + timestamp: Some(ts(0)), + }) + .unwrap(); + frame.write(b"ab".as_ref()).unwrap(); + let mut reading = recv(&mut sub); + let mut chunks = reading.next_frame().now_or_never().unwrap().unwrap().unwrap(); + assert_eq!( + &chunks.read_chunk().now_or_never().unwrap().unwrap().unwrap()[..], + b"ab" + ); + + drop(frame); + a.clone().abort(Error::Dropped).unwrap(); + let b = copy(); + let dynamic = b.requests(); + routes.serve(b.consume()); + assert!(chunks.read_chunk().now_or_never().is_none()); + let request = dynamic.requested_fill_group().now_or_never().unwrap().unwrap(); + assert_eq!((request.frame_start(), request.byte_start()), (0, 2)); + + let mut rest = request.accept(None).unwrap(); + rest.write_frame(ts(0), b"abcd".as_ref()).unwrap(); + rest.write_frame(ts(1), b"e".as_ref()).unwrap(); + rest.finish().unwrap(); + + assert_eq!( + &chunks.read_chunk().now_or_never().unwrap().unwrap().unwrap()[..], + b"cd" + ); + assert!(chunks.read_chunk().now_or_never().unwrap().unwrap().is_none()); + assert_eq!(read(&mut reading), Some(b"e".to_vec())); + } + /// A resumed group that the serving route gives up or aborts ends with that route's /// error, even with no successor or later writes and its fetch still unanswered. #[test] diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index faf05f2f2f9..34831abef56 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -1833,6 +1833,7 @@ impl TrackState { sequence, group::Bounds { start: frame_start, + byte: 0, end: None, }, ) @@ -3929,6 +3930,7 @@ impl Boarding { pub(super) fn poll(&mut self, waiter: &kio::Waiter) -> Poll> { let frames = group::Bounds { start: self.frame, + byte: 0, end: None, }; let sequence = self.sequence; @@ -4286,7 +4288,11 @@ impl Producer { state.claim_sequence(info.sequence, start)?; let mut group = group::Producer::new(info, track, state.cache.clone()).with_meter(state.ingress.meter()); if start != 0 { - group.bound(group::Bounds { start, end: None })?; + group.bound(group::Bounds { + start, + byte: 0, + end: None, + })?; } if let Some(content) = content { group.handle().share_content(content); @@ -6341,6 +6347,7 @@ impl Consumer { || state.info.as_ref().is_none_or(|info| info.compressed()) { request.options.frame_start = 0; + request.options.byte_start = 0; } // A compressed stream needs the entire dictionary; an unknown track is conservative // until TRACK_INFO has told the relay which representation it will receive. @@ -6371,6 +6378,7 @@ impl Consumer { fills, sequence, frame_start: options.frame_start, + byte_start: options.byte_start, frame_end, priority: options.priority, max_delay: options.max_delay, @@ -6392,6 +6400,7 @@ impl Consumer { fills: state.read().fills.clone(), sequence, frame_start: options.frame_start, + byte_start: options.byte_start, frame_end, priority: options.priority, max_delay: options.max_delay, @@ -6713,6 +6722,7 @@ fn queue(fills: &kio::Shared, sequence: u64, request: &FetchOptions) group: sequence, frame: options.frame_start, }, + byte: options.byte_start, end, frame_end: request.end, asking: fill::Asking { @@ -6748,6 +6758,7 @@ pub struct Fetching { // This caller's own start, so a cached group that begins above it is a miss // rather than a short answer. frame_start: u64, + byte_start: u64, frame_end: Option, // This caller's priority, for a fetch that moves to a front's routes. priority: i8, @@ -6781,6 +6792,7 @@ impl Fetching { ) -> Self { let frames = group::Bounds { start: request.options.frame_start, + byte: request.options.byte_start, end: request.end, }; let (fills, cached) = { @@ -6802,6 +6814,7 @@ impl Fetching { fills, sequence, frame_start: request.options.frame_start, + byte_start: request.options.byte_start, frame_end: request.end, priority: request.options.priority, max_delay: request.options.max_delay, @@ -6849,6 +6862,7 @@ impl Fetching { priority: self.priority, max_delay: self.max_delay, frame_start: self.frame_start, + byte_start: self.byte_start, }; self.resume = Some(Box::new(routes.fetch_group( self.sequence, @@ -6905,6 +6919,7 @@ impl Fetching { // error is the channel closing without any of those. let frames = group::Bounds { start: frame_start, + byte: self.byte_start, end: self.frame_end, }; let cached = |waiter: &kio::Waiter| match state.poll(waiter, |state| state.poll_fetch_cached(sequence, frames)) @@ -6971,6 +6986,7 @@ impl Fetching { priority: self.priority, max_delay: self.max_delay, frame_start: self.frame_start, + byte_start: self.byte_start, }; self.result = queue( &self.fills, @@ -8866,6 +8882,7 @@ impl Cursor { } let ask = fill::Ask { start: Position::group(0), + byte: 0, end: reach, frame_end: None, asking: *asking, @@ -9193,6 +9210,7 @@ impl Cursor { priority: asking.priority, max_delay: asking.delay.max, frame_start: 0, + byte_start: 0, }, end: None, }; @@ -17807,7 +17825,15 @@ mod test { .write() .ok() .unwrap() - .insert_group_request(3, group::Bounds { start: 0, end: Some(3) }, None) + .insert_group_request( + 3, + group::Bounds { + byte: 0, + start: 0, + end: Some(3), + }, + None, + ) .unwrap(); for (millis, payload) in [(300, b"a"), (301, b"b"), (302, b"c")] { head.write_frame(at(millis), &payload[..]).unwrap(); @@ -17833,7 +17859,15 @@ mod test { .write() .ok() .unwrap() - .insert_group_request(3, group::Bounds { start: 0, end: Some(3) }, None) + .insert_group_request( + 3, + group::Bounds { + byte: 0, + start: 0, + end: Some(3), + }, + None, + ) .unwrap(); let mut reader = head.consume().with_range(..3); for (millis, payload) in [(300, b"a"), (301, b"b"), (302, b"c")] { @@ -17879,7 +17913,15 @@ mod test { .write() .ok() .unwrap() - .insert_group_request(3, group::Bounds { start: 2, end: Some(3) }, None) + .insert_group_request( + 3, + group::Bounds { + byte: 0, + start: 2, + end: Some(3), + }, + None, + ) .unwrap(); drop(hold); cut.write_frame(at(302), &b"c"[..]).unwrap(); @@ -17968,7 +18010,15 @@ mod test { .write() .ok() .unwrap() - .insert_group_request(3, group::Bounds { start: 0, end: Some(3) }, None) + .insert_group_request( + 3, + group::Bounds { + byte: 0, + start: 0, + end: Some(3), + }, + None, + ) .unwrap(); for (millis, payload) in [(300, b"a"), (301, b"b"), (302, b"c")] { head.write_frame(at(millis), &payload[..]).unwrap(); @@ -26667,6 +26717,7 @@ mod test { 0, group::Bounds { start, + byte: 0, end: Some(start + 1), }, None, @@ -26695,7 +26746,15 @@ mod test { .write() .ok() .unwrap() - .insert_group_request(0, group::Bounds { start: 0, end: Some(1) }, None) + .insert_group_request( + 0, + group::Bounds { + byte: 0, + start: 0, + end: Some(1), + }, + None, + ) .unwrap(); assert!(Arc::ptr_eq(&owner, &refill.handle().content().unwrap())); refill.write_frame(Timestamp::ZERO, b"zero".as_slice()).unwrap(); @@ -26713,7 +26772,15 @@ mod test { .write() .ok() .unwrap() - .insert_group_request(0, group::Bounds { start: 0, end: Some(1) }, None) + .insert_group_request( + 0, + group::Bounds { + byte: 0, + start: 0, + end: Some(1), + }, + None, + ) .unwrap(); new.write_frame(Timestamp::ZERO, b"new zero".as_slice()).unwrap(); assert!(!Arc::ptr_eq(&owner, &new.handle().content().unwrap())); @@ -26750,7 +26817,15 @@ mod test { .write() .ok() .unwrap() - .insert_group_request(0, group::Bounds { start: 3, end: Some(5) }, None) + .insert_group_request( + 0, + group::Bounds { + byte: 0, + start: 3, + end: Some(5), + }, + None, + ) .unwrap(); second.write_frame(Timestamp::ZERO, b"three".as_slice()).unwrap(); let mut second_read = second.consume().with_range(4..); @@ -26859,7 +26934,15 @@ mod test { .write() .ok() .unwrap() - .insert_group_request(0, group::Bounds { start: 1, end: Some(3) }, None) + .insert_group_request( + 0, + group::Bounds { + byte: 0, + start: 1, + end: Some(3), + }, + None, + ) .unwrap(); head.set_wire(wire(90)).unwrap(); let head = write(head, 90, b"b"); @@ -26875,7 +26958,15 @@ mod test { .write() .ok() .unwrap() - .insert_group_request(0, group::Bounds { start: 0, end: Some(2) }, None) + .insert_group_request( + 0, + group::Bounds { + byte: 0, + start: 0, + end: Some(2), + }, + None, + ) .unwrap(); earlier.set_wire(wire(90)).unwrap(); let earlier = write(earlier, 90, b"b"); @@ -27091,7 +27182,15 @@ mod test { .write() .ok() .unwrap() - .insert_group_request(0, group::Bounds { start: 3, end: Some(4) }, None) + .insert_group_request( + 0, + group::Bounds { + byte: 0, + start: 3, + end: Some(4), + }, + None, + ) .unwrap(); head.write_frame(Timestamp::ZERO, b"three".as_slice()).unwrap(); head.finish().unwrap(); @@ -27114,7 +27213,15 @@ mod test { .write() .ok() .unwrap() - .insert_group_request(0, group::Bounds { start: 4, end: Some(5) }, None) + .insert_group_request( + 0, + group::Bounds { + byte: 0, + start: 4, + end: Some(5), + }, + None, + ) .unwrap(); assert!(Arc::ptr_eq( &identity.upgrade().unwrap(), @@ -27154,7 +27261,15 @@ mod test { .write() .ok() .unwrap() - .insert_group_request(0, group::Bounds { start: 0, end: Some(1) }, None) + .insert_group_request( + 0, + group::Bounds { + byte: 0, + start: 0, + end: Some(1), + }, + None, + ) .unwrap(); head.write_frame(Timestamp::ZERO, bytes::Bytes::new()).unwrap(); let head = head.finish().unwrap(); diff --git a/rs/moq-net/src/model/track/fill.rs b/rs/moq-net/src/model/track/fill.rs index d7ffc1e83a9..8ab3a7c32d2 100644 --- a/rs/moq-net/src/model/track/fill.rs +++ b/rs/moq-net/src/model/track/fill.rs @@ -41,6 +41,8 @@ pub(in crate::model) struct FillState { /// One fill until it ends. struct Pending { start: Position, + // The first byte asked of frame `start`, for a reader continuing it partway. + byte: u64, end: u64, // Exclusive frame cap in the last group; None asks through the original end. frame_end: Option, @@ -278,6 +280,8 @@ impl Eq for Refusal {} /// A reader's request: the run `start.group..end`, from `start.frame` in its first group. pub(super) struct Ask { pub start: Position, + /// The first byte of frame `start` the reader wants: it already holds those before. + pub byte: u64, pub end: u64, pub frame_end: Option, pub asking: Asking, @@ -458,13 +462,13 @@ impl FillState { .expect("a covered sequence names a pending fill"); if groups.start == pending.start.group { // A reader starting in an earlier group wants this one whole. - let frame = match ask.start.group == groups.start { - true => ask.start.frame, - false => 0, + let (frame, byte) = match ask.start.group == groups.start { + true => (ask.start.frame, ask.byte), + false => (0, 0), }; - if frame < pending.start.frame { + if (frame, byte) < (pending.start.frame, pending.byte) { match pending.queued { - true => pending.start.frame = frame, + true => (pending.start.frame, pending.byte) = (frame, byte), // Taken: its handler may have asked its source from the later frame // already, so the earlier frames are a fill of their own. false => { @@ -514,9 +518,9 @@ impl FillState { } let id = self.next; self.next += 1; - let start = match groups.start == ask.start.group { - true => ask.start, - false => Position::group(groups.start), + let (start, byte) = match groups.start == ask.start.group { + true => (ask.start, ask.byte), + false => (Position::group(groups.start), 0), }; let result = kio::Producer::::default(); let mut asks = Asks::default(); @@ -538,6 +542,7 @@ impl FillState { id, Pending { start, + byte, end: groups.end, frame_end: if groups.end == ask.end { ask.frame_end } else { None }, time: None, @@ -589,6 +594,7 @@ impl FillState { id, Pending { start: Position { group: sequence, frame }, + byte: 0, end: groups.end, frame_end: None, time: None, @@ -639,6 +645,7 @@ impl FillState { id, Pending { start: ask.start, + byte: 0, end: ask.end, frame_end: None, time: Some(time), @@ -769,8 +776,9 @@ pub(super) fn poll_requested( }) { let id = guard.pop().expect("predicate guaranteed a fill"); let pending = guard.pending.get(&id).expect("popped fill is pending"); - let (start, end, frame_end, time, flight) = ( + let (start, byte, end, frame_end, time, flight) = ( pending.start, + pending.byte, pending.end, pending.frame_end, pending.time, @@ -784,6 +792,7 @@ pub(super) fn poll_requested( fills: fills.clone(), id, start, + byte, end, frame_end, time, @@ -850,6 +859,8 @@ struct Inner { fills: kio::Shared, id: u64, start: Position, + // The first byte of frame `start` asked for. + byte: u64, end: u64, // Exclusive frame cap in the last group; None asks through the original end. frame_end: Option, @@ -1191,6 +1202,7 @@ impl Fill { inner: self.inner.clone(), sequence, frame_start: self.inner.frame_start(sequence), + byte_start: self.inner.byte_start(sequence), frame_end: self.inner.bounds(sequence).end, priority: self.priority(), max_delay: self.max_delay(), @@ -1249,9 +1261,18 @@ impl Inner { } } + /// The first byte group `sequence` starts from within its first frame. + fn byte_start(&self, sequence: u64) -> u64 { + match sequence == self.start.group { + true => self.byte, + false => 0, + } + } + fn bounds(&self, sequence: u64) -> group::Bounds { group::Bounds { start: self.frame_start(sequence), + byte: self.byte_start(sequence), end: if sequence.checked_add(1) == Some(self.end) { self.frame_end } else { @@ -1260,8 +1281,15 @@ impl Inner { } } + /// A writer that never says it answers from a byte writes whole frames, so its copy + /// starts at a frame boundary even when the fill asked from a byte: only + /// [`GroupFill::accept_from`] starts one partway into a frame. fn accept(&self, sequence: u64, info: Option) -> Result { - self.accept_from(sequence, info, self.bounds(sequence)) + let frames = group::Bounds { + byte: 0, + ..self.bounds(sequence) + }; + self.accept_from(sequence, info, frames) } fn accept_from(&self, sequence: u64, info: Option, frames: group::Bounds) -> Result { @@ -1307,7 +1335,7 @@ impl Inner { if let Ok(mut outcome) = self.result.write() { outcome.accept(sequence..end); if let Ok(group) = &res - && (self.flight || frames.start > 0 || frames.end.is_some()) + && (self.flight || frames.start > 0 || frames.byte > 0 || frames.end.is_some()) { outcome.partial.insert(sequence, group.handle()); } @@ -1519,6 +1547,7 @@ pub(crate) struct GroupFill { inner: Arc>, sequence: u64, frame_start: u64, + byte_start: u64, frame_end: Option, priority: i8, max_delay: Option, @@ -1547,6 +1576,11 @@ impl GroupFill { self.frame_start } + /// The first byte of frame [`Self::frame_start`] the readers want; 0 is the whole frame. + pub(crate) fn byte_start(&self) -> u64 { + self.byte_start + } + /// The requested exclusive last frame, if this copy is bounded at the back. pub(crate) fn frame_end(&self) -> Option { self.frame_end diff --git a/rs/moq-net/src/model/track/miss.rs b/rs/moq-net/src/model/track/miss.rs index a86acf0cbb3..b5ea111fecc 100644 --- a/rs/moq-net/src/model/track/miss.rs +++ b/rs/moq-net/src/model/track/miss.rs @@ -1329,6 +1329,7 @@ impl Misses { for run in runs { let ask = Ask { start: Position::group(run.start), + byte: 0, end: run.end, frame_end: None, asking: *asking, diff --git a/rs/moq-net/src/recv.rs b/rs/moq-net/src/recv.rs index 70aab3393c5..a85ebc2d01e 100644 --- a/rs/moq-net/src/recv.rs +++ b/rs/moq-net/src/recv.rs @@ -40,6 +40,22 @@ impl Group { .map(frame::Producer::cancel_on_drop) } + /// The byte the copy starts frame `index` at, if it holds only the rest of that frame + /// (see [`group::Producer::create_lead`]). + pub fn lead_at(&self, index: u64) -> Option { + self.handle.lead_at(index) + } + + /// Move the writer and cancellation cleanup into the rest of the frame the copy starts + /// partway into. + pub fn create_lead(&mut self, frame: frame::Header) -> Result { + self.writer + .take() + .expect("group writer is available between frames") + .create_lead(frame) + .map(frame::Producer::cancel_on_drop) + } + /// Poll until the writer may admit the frame `header` opens, its payload and metadata, /// within `window` bytes past its fastest reader, so admitting it sheds no byte that /// reader has yet to take; see [`group::Producer::poll_paced`]. diff --git a/rs/moq-net/tests/integration/byte_holes.rs b/rs/moq-net/tests/integration/byte_holes.rs new file mode 100644 index 00000000000..a200dc706fe --- /dev/null +++ b/rs/moq-net/tests/integration/byte_holes.rs @@ -0,0 +1,120 @@ +#![cfg(feature = "async")] + +//! A frame read partway when its route dies continues on the next route from the byte it +//! reached: over lite-07 the subscriber asks for only the rest of the frame, so the new route +//! sends none of the bytes already read. +//! +//! The publisher `P` is pulled by relays `A` and `B`. The subscribing relay `E` reads a large +//! frame of group 0 halfway through `A`, then a cheaper route through `B` wins and `A`'s +//! session to `E` dies. `B`'s subscription starts at group 1, so `E` FETCHes the rest of the +//! frame from `B`. + +use crate::route_change::{link, priced_link, produce_origin, settle, timed}; +use crate::support::harness::{MockConnectOptions, connect_mock}; + +use moq_net::{Error, Timestamp, Version, frame, origin, stats, track}; + +/// Several storage segments, so the rest spans more than one. +const SIZE: usize = 5 * 1024 * 1024 + 7; +/// Where the first route dies, partway into a segment. +const HALF: usize = 2 * 1024 * 1024 + 3; + +fn pattern(len: usize) -> Vec { + (0..len).map(|i| (i % 251) as u8).collect() +} + +/// The payload bytes `registry`'s sessions sent as publisher. +fn sent(registry: &stats::Registry) -> u64 { + registry + .snapshot() + .traffic() + .into_iter() + .filter(|(_, role, _)| *role == stats::Role::Publisher) + .map(|(_, _, traffic)| traffic.bytes) + .sum() +} + +#[kio::test] +async fn a_half_read_frame_continues_from_its_byte() { + let version: Version = "moq-lite-07-wip".parse().unwrap(); + let publisher = produce_origin(1); + let relay_a = produce_origin(2); + let relay_b = produce_origin(3); + let relay_e = produce_origin(4); + + let broadcast = publisher.route("live", origin::RouteOptions::hidden()).unwrap(); + let track = broadcast.broadcast().create_track("file", timed()).unwrap(); + broadcast.announce().unwrap(); + + let _p_to_a = link(version, &publisher, &relay_a).await; + let _p_to_b = link(version, &publisher, &relay_b).await; + let a_to_e = priced_link(version, &relay_a, &relay_e, Some(1_000)).await; + + let remote = relay_e + .consume() + .follow("live", None) + .unwrap() + .available() + .await + .unwrap() + .unwrap(); + let prefs = + track::Subscription::default().with_delay(track::Delay::default().with_max(std::time::Duration::from_secs(60))); + let mut sub = remote.track("file").unwrap().subscribe(prefs).await.unwrap(); + + let payload = pattern(SIZE); + let group0 = track.append_group().unwrap(); + let mut writing = group0 + .create_frame(frame::Info { + size: SIZE as u64, + timestamp: Some(Timestamp::ZERO), + }) + .unwrap(); + writing.write(payload[..HALF].to_vec()).unwrap(); + + let mut read0 = sub.recv_group().await.unwrap().unwrap(); + assert_eq!(read0.sequence, 0); + let mut reading = read0.next_frame().await.unwrap().unwrap(); + let mut got = Vec::new(); + while got.len() < HALF { + got.extend_from_slice(&reading.read_chunk().await.unwrap().unwrap()); + } + assert_eq!(got, payload[..HALF]); + + let mut group1 = track.append_group().unwrap(); + group1.write_frame(Timestamp::ZERO, b"next".to_vec()).unwrap(); + group1.finish().unwrap(); + let mut read1 = sub.recv_group().await.unwrap().unwrap(); + assert_eq!(read1.sequence, 1); + assert_eq!(&read1.read_frame().await.unwrap().unwrap().payload[..], b"next"); + + // The cheaper route through `B` wins, metered so its egress shows what it sent `E`. + let registry = stats::Registry::new(Default::default()); + let mut options = MockConnectOptions::new(version); + options.server_publish = Some(relay_b.consume()); + options.client_subscribe = Some(relay_e.clone()); + options.server_stats = registry + .tier(stats::Tier::default()) + .session(stats::SessionInfo::new(moq_net::Path::from_static(""))); + let _b_to_e = connect_mock(options).await; + settle().await; + a_to_e.server.abort(Error::Cancel); + a_to_e.client.abort(Error::Cancel); + settle().await; + + writing.write(payload[HALF..].to_vec()).unwrap(); + writing.finish().unwrap().finish().unwrap(); + + while let Some(chunk) = reading.read_chunk().await.unwrap() { + got.extend_from_slice(&chunk); + } + assert!(got == payload, "the frame continues at its byte"); + + let rest = (SIZE - HALF) as u64; + let sent = sent(®istry); + assert!(sent >= rest, "{sent} bytes cannot carry the {rest} left"); + assert!( + sent < rest + 1024, + "{sent} bytes resent what `E` read: only {rest} were left" + ); +} diff --git a/rs/moq-net/tests/integration/main.rs b/rs/moq-net/tests/integration/main.rs index bc236d2d23e..4ddefce3700 100644 --- a/rs/moq-net/tests/integration/main.rs +++ b/rs/moq-net/tests/integration/main.rs @@ -12,6 +12,7 @@ mod announce_to_serve; mod answer_behind_cache; mod auth; mod broadcast_close; +mod byte_holes; mod catalog_acquisition; mod catalog_churn; mod catalog_handover;