diff --git a/doc/concept/moq-lite.md b/doc/concept/moq-lite.md index 898cae6ea35..c1a70d92985 100644 --- a/doc/concept/moq-lite.md +++ b/doc/concept/moq-lite.md @@ -184,7 +184,12 @@ to a prefix. Right after setup both sides present the credential the connection already carried (the URL token, a client certificate, or nothing), so a publisher learns before anyone subscribes whether its broadcasts can reach the peer. More tokens can be added without reconnecting; the session's scope is the -union of every open token's grant. A token the peer has no verifier for, or +union of every open token's grant. A request outside the known union waits +while a non-empty token already presented is unanswered, then proceeds or +fails against the settled union; a covered request does not wait. Each wait +logs its path and duration. To renew without interrupting live requests the +new grant still covers, await the new token's acceptance before dropping the +old one. A token the peer has no verifier for, or whose grant is too large to send, fails with `Unsupported` and ends only that token; the session and its other grants stay up. A relay with an [auth server](/bin/relay/auth#in-band-renewal) verifies these tokens through it, diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 941072fbbe6..a18db08d670 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -563,6 +563,7 @@ Either endpoint can open an Auth Stream (0x7) to present a token and learn what The opener sends a single AUTH message and keeps its side open for as long as the token applies. Each endpoint SHOULD open one Auth Stream with an empty token right after the session starts, so both learn what the credential the connection already presented (the request URI, a client certificate, or nothing) allows. The acceptor replies with an AUTH_OK carrying the grant, or an AUTH_ERROR refusing the token. +The grant MUST join the acceptor's scope before it sends AUTH_OK. It MAY send further AUTH_OK messages to replace the grant, such as with a lowered expiry, and MAY send an AUTH_ERROR after an AUTH_OK to revoke it; the latest AUTH_OK is the token's grant. After an AUTH_ERROR the acceptor closes its side of the stream. The opener withdraws a token by closing or resetting its side of the stream, and the acceptor then closes its own. @@ -571,7 +572,13 @@ A grant names the paths the opener may publish to the acceptor and subscribe to An endpoint's scope is the union of the grants of its open Auth Streams, and a stream that ends removes its grant from the union. When the union shrinks, an endpoint SHOULD withdraw its announcements and cancel its subscriptions that the union no longer covers, keeping the session and everything still covered. An endpoint that cancels a subscription, or refuses or stops serving a request, because the union does not cover it resets the stream with UNAUTHORIZED. -An endpoint that announces a broadcast outside the union once its own setup tokens are answered SHOULD close the session with UNAUTHORIZED, rather than wait for a subscription that will never come. +The acceptor judges each request against the union when it reads the request. +A presenter MUST NOT send a request outside its known union while any non-empty token it presented remains unanswered; an unknown union covers no paths for this rule. +Once those tokens are answered, it sends the request if the union covers it, or refuses it locally. +An empty token does not hold requests: the connection credential is judged before the session serves requests. +The same wait applies before closing the session for announcing a broadcast outside the union. +An endpoint that announces a broadcast outside the settled union, and still announces it once the wait ends, SHOULD close the session with UNAUTHORIZED, rather than wait for a subscription that will never come. +To renew a token without interrupting requests the new grant still covers, the presenter waits for the new token's AUTH_OK before withdrawing the old token. An acceptor that does not verify a token in band, or cannot express its grant in AUTH_OK, MUST send AUTH_ERROR with NOT_SUPPORTED and close its side of the stream. This applies before or after an AUTH_OK, including when a grant exceeds the message size limit. @@ -1635,6 +1642,7 @@ The `Message Length` describes the payload size on the wire. - A Report-level publisher reads and ignores PROBE targets instead of resetting the Probe Stream. - Added NOT_SUPPORTED (0x30) to session errors; unsupported Auth Stream tokens and grants use AUTH_ERROR before or after a grant. - Added the Auth Stream (0x7) with AUTH, AUTH_OK, and AUTH_ERROR: either endpoint presents a token on its own stream and learns the [path patterns](#path-pattern) it may publish and subscribe to. The union of a session's open grants is its scope. A shrink withdraws what it no longer covers, and an announcement outside the scope closes the session with UNAUTHORIZED. A peer without the stream resets it. +- Required scope widening before AUTH_OK and presenter-side waiting for unanswered non-empty tokens before sending an uncovered request or rejecting a publication; renewal waits for AUTH_OK before withdrawing the old token. ## moq-lite-06 diff --git a/js/net/src/auth.test.ts b/js/net/src/auth.test.ts index 1b9f44e325d..27a50d510b2 100644 --- a/js/net/src/auth.test.ts +++ b/js/net/src/auth.test.ts @@ -1,5 +1,5 @@ import { describe, expect, spyOn, test } from "bun:test"; -import { Milli, Nano } from "@moq/clock"; +import { Manual, Milli, Nano } from "@moq/clock"; import type { Getter } from "@moq/signals"; import { type Grant, type Issued, Unsupported } from "./auth.ts"; import type { AuthSession } from "./auth_session.ts"; @@ -17,6 +17,7 @@ import { createMockTransportPair, type MockTransport } from "./mock.ts"; import { Producer as OriginProducer } from "./origin.ts"; import * as Path from "./path.ts"; import { type Stream, Writer } from "./stream.ts"; +import { abandoned } from "./util/demand.ts"; import { withTimeout } from "./util/timeout.ts"; import { nextTask } from "./util/yield.ts"; import { wireOf } from "./wire.ts"; @@ -47,6 +48,19 @@ async function waitFor(getter: Getter, ready: (value: T) => boolean): Prom return value; } +/** Whether a request on `path` logged the end of its authorization wait. */ +function waited(logged: { mock: { calls: unknown[][] } }, path: string): boolean { + return logged.mock.calls.some(([line]) => + String(line).startsWith(`request waited for authorization: path=${path} `), + ); +} + +/** Yield until `done` holds, failing if it never does. */ +async function settles(done: () => boolean): Promise { + for (let i = 0; i < 50 && !done(); i++) await nextTask(); + expect(done()).toBe(true); +} + interface Pair { client: Established; server: Established; @@ -141,6 +155,39 @@ describe.each([Lite.ALPN_07_WIP, Ietf.ALPN.DRAFT_17, Ietf.ALPN.DRAFT_22])("%s", server.abort(); }); + test("a broadcast withdrawn while its token waits is not refused", async () => { + const origin = new OriginProducer(); + const { client, server, transport } = await connect({ publish: origin, protocol }); + const requests = server.auth.requests(); + const setup = await requests.next(); + if (!setup) throw new Error("missing setup token"); + const initial = setup.accept(grant(["baz"], [])); + await waitFor(client.auth.grant, (g) => g !== undefined); + const adding = client.auth.add("new"); + try { + const unanswered = await requests.next(); + if (!unanswered) throw new Error("missing token"); + const withdrawn = origin.createBroadcast(Path.from("foo/bar")); + withdrawn.announce(); + await nextTask(); + withdrawn.unannounce(); + await nextTask(); + const issued = unanswered.accept(grant(["other"], [])); + await adding; + // Still enforcing: the next uncovered broadcast is the one named. + origin.createBroadcast(Path.from("still-bad")).announce(); + const info = await transport.closed; + expect(info.closeCode).toBe(SessionCode.Unauthorized); + expect(info.reason).toBe("unauthorized: still-bad"); + issued.close(); + } finally { + initial.close(); + client.abort(); + server.abort(); + await adding.catch(() => void 0); + } + }); + test("a revoked grant withdraws its broadcasts without closing the session", async () => { const origin = new OriginProducer(); const { client, server, transport } = await connect({ publish: origin, protocol }); @@ -240,6 +287,175 @@ describe.each([Lite.ALPN_07_WIP, Ietf.ALPN.DRAFT_17, Ietf.ALPN.DRAFT_22])("%s", server.abort(); }); + test("a request waits for its token and survives union renewal", async () => { + const clock = new Manual(); + const restore = clock.install(); + const logged = spyOn(console, "debug"); + const origin = new OriginProducer(); + const broadcast = origin.createBroadcast(Path.from("room/x")); + broadcast.announce(); + const published = broadcast.createTrack("video"); + published.appendGroup().writeFrame({ payload: new Uint8Array([1]) }); + const { client, server } = await connect({ serverPublish: origin, protocol }); + const requests = server.auth.requests(); + await (client.auth as AuthSession).ready(Path.from("room/x"), "subscribe"); + const setup = await requests.next(); + if (!setup) throw new Error("missing setup token"); + const initial = setup.accept(grant([], [])); + await waitFor(client.auth.grant, (g) => g !== undefined); + const adding = client.auth.add("first"); + try { + const unanswered = await requests.next(); + if (!unanswered) throw new Error("missing token"); + const track = wireOf(client).consume(Path.from("room/x")).track("video").subscribe(); + await nextTask(); + expect(track.closed.peek()).toBeUndefined(); + clock.advance(Nano.fromMilli(Milli(7))); + const firstGrant = unanswered.accept(grant([], ["room"])); + const first = await adding; + expect(await track.recvGroup()).toBeDefined(); + expect( + logged.mock.calls.filter(([line]) => String(line).startsWith("request waited for authorization:")), + ).toEqual([["request waited for authorization: path=room/x waited=7ms"]]); + const renewing = client.auth.add("renewal"); + const renewal = await requests.next(); + if (!renewal) throw new Error("missing renewal"); + broadcast + .createTrack("audio") + .appendGroup() + .writeFrame({ payload: new Uint8Array([3]) }); + const covered = wireOf(client).consume(Path.from("room/x")).track("audio").subscribe(); + expect(await covered.recvGroup()).toBeDefined(); + covered.close(); + const renewedGrant = renewal.accept(grant([], ["room"])); + const renewed = await renewing; + first.close(); + await nextTask(); + expect(track.closed.peek()).toBeUndefined(); + published.appendGroup().writeFrame({ payload: new Uint8Array([2]) }); + expect(await track.recvGroup()).toBeDefined(); + track.close(); + renewed.close(); + firstGrant.close(); + renewedGrant.close(); + } finally { + initial.close(); + client.abort(); + server.abort(); + await adding.catch(() => void 0); + logged.mockRestore(); + restore(); + } + }); + + test("an abandoned request stops waiting for its token", async () => { + const { client, server } = await connect({ protocol }); + const requests = server.auth.requests(); + const setup = await requests.next(); + if (!setup) throw new Error("missing setup token"); + const initial = setup.accept(grant([], [])); + await waitFor(client.auth.grant, (g) => g !== undefined); + const adding = client.auth.add("first"); + try { + const unanswered = await requests.next(); + if (!unanswered) throw new Error("missing token"); + const track = new OriginProducer().createBroadcast(Path.from("room/x")).createTrack("video"); + const reader = track.subscribe(); + const ready = (client.auth as AuthSession).ready(Path.from("room/x"), "subscribe", { + until: () => abandoned(track.demand()), + }); + let settled = false; + void ready.then(() => { + settled = true; + }); + await nextTask(); + expect(settled).toBe(false); + reader.close(); + expect(await ready).toBe(false); + unanswered.accept(grant([], [])).close(); + } finally { + initial.close(); + client.abort(); + server.abort(); + await adding.catch(() => void 0); + } + }); + + test("a subscribe abandoned while its token waits never reaches the wire", async () => { + const logged = spyOn(console, "debug"); + const origin = new OriginProducer(); + const broadcast = origin.createBroadcast(Path.from("room/x")); + broadcast.announce(); + broadcast.createTrack("video"); + const { client, server, transport } = await connect({ serverPublish: origin, protocol }); + const requests = server.auth.requests(); + const setup = await requests.next(); + if (!setup) throw new Error("missing setup token"); + const initial = setup.accept(grant([], [])); + await waitFor(client.auth.grant, (g) => g !== undefined); + const adding = client.auth.add("first"); + try { + const unanswered = await requests.next(); + if (!unanswered) throw new Error("missing token"); + const track = wireOf(client).consume(Path.from("room/x")).track("video").subscribe(); + await nextTask(); + track.close(); + await settles(() => waited(logged, "room/x")); + const opened = transport.sendStreams.bidi.length; + const issued = unanswered.accept(grant([], ["room"])); + await adding; + for (let i = 0; i < 10; i++) await nextTask(); + expect(transport.sendStreams.bidi.length).toBe(opened); + issued.close(); + } finally { + initial.close(); + client.abort(); + server.abort(); + await adding.catch(() => void 0); + logged.mockRestore(); + } + }); + + // The window between the wait seeing the reader leave and acting on it spans a few + // microtasks, so sweep where the new reader lands. + test.each(Array.from({ length: 17 }, (_, gap) => gap))( + "a subscribe that regains a reader while its token waits keeps waiting (%d)", + async (gap) => { + const origin = new OriginProducer(); + const broadcast = origin.createBroadcast(Path.from("room/x")); + broadcast.announce(); + const published = broadcast.createTrack("video"); + published.appendGroup().writeFrame({ payload: new Uint8Array([7]) }); + const { client, server } = await connect({ serverPublish: origin, protocol }); + const requests = server.auth.requests(); + const setup = await requests.next(); + if (!setup) throw new Error("missing setup token"); + const initial = setup.accept(grant([], [])); + await waitFor(client.auth.grant, (g) => g !== undefined); + const adding = client.auth.add("first"); + try { + const unanswered = await requests.next(); + if (!unanswered) throw new Error("missing token"); + const track = wireOf(client).consume(Path.from("room/x")).track("video"); + const first = track.subscribe(); + await nextTask(); + first.close(); + for (let i = 0; i < gap; i++) await Promise.resolve(); + const second = track.subscribe(); + const issued = unanswered.accept(grant([], ["room"])); + await adding; + expect(await second.recvGroup()).toBeDefined(); + second.close(); + issued.close(); + } finally { + initial.close(); + client.abort(); + server.abort(); + await adding.catch(() => void 0); + } + }, + ); + test("a session's close is not a revocation of its subscriptions", async () => { const origin = new OriginProducer(); const broadcast = origin.createBroadcast(Path.from("room/x")); @@ -364,6 +580,94 @@ describe.each([Ietf.ALPN.DRAFT_17, Ietf.ALPN.DRAFT_22])("%s", (protocol) => { }); // moq-lite carries patterns, so every grant arrives exactly as issued. +test("a fetch abandoned while its token waits never reaches the wire", async () => { + const logged = spyOn(console, "debug"); + const origin = new OriginProducer(); + const broadcast = origin.createBroadcast(Path.from("room/x")); + broadcast.announce(); + broadcast + .createTrack("video") + .appendGroup() + .writeFrame({ payload: new Uint8Array([1]) }); + const { client, server, transport } = await connect({ serverPublish: origin, protocol: Lite.ALPN_07_WIP }); + const requests = server.auth.requests(); + const setup = await requests.next(); + if (!setup) throw new Error("missing setup token"); + const initial = setup.accept(grant([], [])); + await waitFor(client.auth.grant, (g) => g !== undefined); + const adding = client.auth.add("first"); + try { + const unanswered = await requests.next(); + if (!unanswered) throw new Error("missing token"); + const abort = new AbortController(); + const fetched = wireOf(client) + .consume(Path.from("room/x")) + .track("video") + .fetchGroup(0, { signal: abort.signal }); + await nextTask(); + abort.abort(new Error("left")); + await expect(fetched).rejects.toThrow(); + await settles(() => waited(logged, "room/x")); + const opened = transport.sendStreams.bidi.length; + const issued = unanswered.accept(grant([], ["room"])); + await adding; + for (let i = 0; i < 10; i++) await nextTask(); + expect(transport.sendStreams.bidi.length).toBe(opened); + issued.close(); + } finally { + initial.close(); + client.abort(); + server.abort(); + await adding.catch(() => void 0); + logged.mockRestore(); + } +}); + +// The window between the wait seeing the caller leave and acting on it spans a few +// microtasks, so sweep where the new caller lands. +test.each(Array.from({ length: 17 }, (_, gap) => gap))( + "a fetch that regains a caller while its token waits keeps waiting (%d)", + async (gap) => { + const origin = new OriginProducer(); + const broadcast = origin.createBroadcast(Path.from("room/x")); + broadcast.announce(); + broadcast + .createTrack("video") + .appendGroup() + .writeFrame({ payload: new Uint8Array([7]) }); + const { client, server } = await connect({ serverPublish: origin, protocol: Lite.ALPN_07_WIP }); + const requests = server.auth.requests(); + const setup = await requests.next(); + if (!setup) throw new Error("missing setup token"); + const initial = setup.accept(grant([], [])); + await waitFor(client.auth.grant, (g) => g !== undefined); + const adding = client.auth.add("first"); + try { + const unanswered = await requests.next(); + if (!unanswered) throw new Error("missing token"); + const track = wireOf(client).consume(Path.from("room/x")).track("video"); + const abort = new AbortController(); + const first = track.fetchGroup(0, { signal: abort.signal }); + await nextTask(); + // The last caller leaves and another coalesces before the wait observes it. + abort.abort(new Error("left")); + for (let i = 0; i < gap; i++) await Promise.resolve(); + const second = track.fetchGroup(0); + await expect(first).rejects.toThrow(); + const issued = unanswered.accept(grant([], ["room"])); + await adding; + const group = await second; + expect((await group.readFrame())?.payload).toEqual(new Uint8Array([7])); + issued.close(); + } finally { + initial.close(); + client.abort(); + server.abort(); + await adding.catch(() => void 0); + } + }, +); + test("lite-07 carries literal and wildcard grants exactly", async () => { const { client, server } = await connect({ publish: new OriginProducer(), protocol: Lite.ALPN_07_WIP }); const requests = server.auth.requests(); diff --git a/js/net/src/auth_session.ts b/js/net/src/auth_session.ts index daeb0afcccd..33532e0068b 100644 --- a/js/net/src/auth_session.ts +++ b/js/net/src/auth_session.ts @@ -1,4 +1,4 @@ -import { wall } from "@moq/clock"; +import { Instant, Nano, now, wall } from "@moq/clock"; import { type Dispose, type Getter, Once, Signal } from "@moq/signals"; import { type Auth, @@ -298,6 +298,7 @@ export class AuthSession implements Handle { #replied = false; #tokens = new Set(); #setupPending = new Signal(0); + #pending = new Signal(0); #acceptor: "undecided" | "default" | RequestQueue = "undecided"; #closed = false; @@ -374,6 +375,63 @@ export class AuthSession implements Handle { while (this.#setupPending.peek() > 0) await this.#setupPending.changed(); } + /** + * Wait for presented tokens that might admit this path, without delaying a covered path. + * Resolves false once `until`, the request's own lifetime, settles first, so an abandoned + * request never waits on a token that may never be answered. `until` is called only once + * the request has to wait. + */ + async ready( + path: Path.Valid, + direction: "publish" | "subscribe", + { signal, until }: { signal?: AbortSignal; until?: () => PromiseLike } = {}, + ): Promise { + let started: Instant | undefined; + let ended = false; + let wake: (() => void) | undefined; + const end = () => { + ended = true; + wake?.(); + }; + try { + while (this.#pending.peek() > 0 && !this.#union.peek()?.[direction].matches(path)) { + if (ended) return false; + if (started === undefined) { + started = now(); + until?.().then(end, end); + } + signal?.throwIfAborted(); + await new Promise((resolve, reject) => { + const cleanup = () => { + pending(); + grant(); + wake = undefined; + signal?.removeEventListener("abort", abort); + }; + const done = () => { + cleanup(); + resolve(); + }; + const abort = () => { + cleanup(); + reject(signal?.reason); + }; + const pending = this.#pending.changed(done); + const grant = this.#union.changed(done); + wake = done; + signal?.addEventListener("abort", abort, { once: true }); + }); + } + return true; + } finally { + if (started !== undefined) { + console.debug( + `request waited for authorization: path=${path} waited=${Nano.toMilli(Instant.durationSince(now(), started))}ms`, + ); + } + } + } + /** Answer one of the peer's token streams, for the life of its token. */ async serve(stream: Stream): Promise { const wire = this.#wire; @@ -417,6 +475,7 @@ export class AuthSession implements Handle { close() { if (this.#closed) return; this.#closed = true; + this.#pending.set(0); for (const token of this.#tokens) token.end(new Error("session closed")); this.#tokens.clear(); this.#recompute(); @@ -427,6 +486,7 @@ export class AuthSession implements Handle { const presented = new Presented(token, setup); this.#track(presented); if (setup) this.#setupPending.update((n) => n + 1); + if (token.byteLength > 0) this.#pending.update((n) => n + 1); void this.#run(wire, presented); return presented; } @@ -507,6 +567,7 @@ export class AuthSession implements Handle { const unanswered = !token.isAnswered; token.end(result); if (unanswered && token.setup) this.#setupPending.update((n) => n - 1); + if (unanswered && token.token.byteLength > 0 && !this.#closed) this.#pending.update((n) => n - 1); this.#tokens.delete(token); // Closed before the union shrinks, so the scope never reads it as a revocation. if (closed) this.close(); @@ -517,6 +578,7 @@ export class AuthSession implements Handle { if (token.isAnswered) return; token.answer(); if (token.setup) this.#setupPending.update((n) => n - 1); + if (token.token.byteLength > 0 && !this.#closed) this.#pending.update((n) => n - 1); } #recompute() { @@ -550,12 +612,14 @@ export async function enforceGrant({ advertised, grant, setupAnswered, + auth, }: { quic: WebTransport; /** The broadcasts this side publishes, undefined once the origin ends. */ advertised: Getter | undefined>; grant: Getter; setupAnswered: Promise; + auth?: AuthSession; }): Promise { const closed = quic.closed.then( () => "closed" as const, @@ -589,6 +653,13 @@ export async function enforceGrant({ for (const path of table.keys()) { if (live.has(path)) continue; if (!current.publish.matches(path)) { + await auth?.ready(path, "publish"); + // A broadcast withdrawn during the wait is no longer ours to judge. + if (!advertised.peek()?.has(path)) continue; + if (grant.peek()?.publish.matches(path)) { + live.add(path); + continue; + } console.error(`publishing outside our grant; closing the session: broadcast=${path}`); quic.close({ closeCode: SessionCode.Unauthorized, diff --git a/js/net/src/ietf/connection.ts b/js/net/src/ietf/connection.ts index 36004008787..aa594f0643a 100644 --- a/js/net/src/ietf/connection.ts +++ b/js/net/src/ietf/connection.ts @@ -240,6 +240,7 @@ export class Connection implements Established { solicit, goaway: this.#goaway, grant: this.#auth.scope, + auth: this.#auth, }); registerWire(this, { consume: (path) => this.#subscriber.consume(path), goaway: this.#goaway }); @@ -307,7 +308,7 @@ export class Connection implements Established { ]; // Fail loud on a publication our grant never covers, once the peer has answered the // credential we presented at setup. - if (this.#client) tasks.push(this.#publisher.runEnforce(this.#auth.setupAnswered())); + if (this.#client) tasks.push(this.#publisher.runEnforce(this.#auth.setupAnswered(), this.#auth)); await Promise.all(tasks); } catch (err) { if (!this.#closed) { diff --git a/js/net/src/ietf/publisher.ts b/js/net/src/ietf/publisher.ts index a22767d9649..297b6e1e8a0 100644 --- a/js/net/src/ietf/publisher.ts +++ b/js/net/src/ietf/publisher.ts @@ -2,6 +2,7 @@ import { Instant, Milli, Nano, now, sleep } from "@moq/clock"; import { type Dispose, type Getter, race, Signal } from "@moq/signals"; import type { Auth, Grant } from "../auth.ts"; import { Gate } from "../auth_gate.ts"; +import type { AuthSession } from "../auth_session.ts"; import { enforceGrant } from "../auth_session.ts"; import type * as broadcast from "../broadcast.ts"; import { Withdrawal } from "../connection/withdrawal.ts"; @@ -1195,8 +1196,8 @@ export class Publisher { * * @internal */ - async runEnforce(setupAnswered: Promise): Promise { - await enforceGrant({ quic: this.#quic, advertised: this.#advertised, grant: this.#grant, setupAnswered }); + async runEnforce(setupAnswered: Promise, auth?: AuthSession): Promise { + await enforceGrant({ quic: this.#quic, advertised: this.#advertised, grant: this.#grant, setupAnswered, auth }); } /** diff --git a/js/net/src/ietf/subscriber.ts b/js/net/src/ietf/subscriber.ts index 506828064cf..4fe804522d8 100644 --- a/js/net/src/ietf/subscriber.ts +++ b/js/net/src/ietf/subscriber.ts @@ -2,6 +2,7 @@ import { Milli } from "@moq/clock"; import { type Dispose, type GetPromise, type Getter, race, Signal } from "@moq/signals"; import * as announce from "../announced.ts"; import type { Grant } from "../auth.ts"; +import type { AuthSession } from "../auth_session.ts"; import * as broadcast from "../broadcast.ts"; import type { Drain } from "../connection/goaway.ts"; import { BroadcastCache } from "../consume.ts"; @@ -35,7 +36,7 @@ import type { Timescale, Timestamp } from "../time.ts"; import type * as track from "../track.ts"; import { untilAborted } from "../util/abort.ts"; import { Deadline, requestOptions } from "../util/deadline.ts"; -import { unwanted } from "../util/demand.ts"; +import { abandoned as demandAbandoned, isAbandoned, unwanted } from "../util/demand.ts"; import { LIMITS, type Limits, SlotMap, Slots } from "../util/limits.ts"; import { overrideBroadcastWire, wireOf } from "../wire.ts"; import type { Session } from "./adapter.ts"; @@ -204,6 +205,7 @@ export class Subscriber { // Our grant (MoQ Auth): a subscription it stops covering is cancelled. Undefined until // the peer answers, which allows everything. #grant?: Getter; + #auth?: AuthSession; /** Marks this subscriber's deliberate local session close. @internal */ close() { @@ -225,6 +227,7 @@ export class Subscriber { solicit, goaway, grant, + auth, limits = LIMITS, }: { /** The session abstraction for bidi streams and request IDs. */ @@ -245,6 +248,7 @@ export class Subscriber { goaway?: GetPromise; /** The union of our tokens' grants (MoQ Auth), which bounds what we subscribe to. */ grant?: Getter; + auth?: AuthSession; /** How many live requests the peer may hold. */ limits?: Limits; }) { @@ -258,6 +262,7 @@ export class Subscriber { this.#solicit = solicit; this.#goaway = goaway; this.#grant = grant; + this.#auth = auth; // A draining peer usually stops publishing namespaces, so reprice from the signal // itself. Waiting for another message would leave the route primary until close. void goaway?.then(() => this.#drainAnnounced()); @@ -750,6 +755,22 @@ export class Subscriber { async #runSubscribe(broadcast: Path.Valid, request: track.Request, valid?: AbortSignal, epoch?: Epoch.Valid) { const refused = unauthorized(broadcast); + try { + for (;;) { + const ready = await this.#auth?.ready(broadcast, "subscribe", { + signal: valid, + until: () => demandAbandoned(request.demand()), + }); + if (ready !== false) break; + // Demand returning before we commit is not abandonment, matching the serving loop. + if (!isAbandoned(request.demand())) continue; + request.reject(new Error("subscribe abandoned before it was authorized")); + return; + } + } catch (err) { + request.reject(error(err)); + return; + } if (this.#denied(broadcast)) { request.reject(refused); return; @@ -790,15 +811,8 @@ export class Subscriber { // Demand returning before we commit is not abandonment, matching the serving loop. const demand = producer.demand(); const waitAbandoned = async (): Promise => { - // An info-only lookup attaches no subscriber yet still waits on SUBSCRIBE_OK for - // the track info, so only demand that arrived and then left is abandonment. - while (demand.wanted.peek() === undefined && demand.closed.peek() === undefined) { - await Signal.race(demand.wanted, demand.closed); - } - for (;;) { - await unwanted(demand); - if (demand.closed.peek() !== undefined || demand.wanted.peek() === undefined) return null; - } + await demandAbandoned(demand); + return null; }; let stream: Stream; diff --git a/js/net/src/lite/connection.ts b/js/net/src/lite/connection.ts index 6b593da3267..57b46f540e2 100644 --- a/js/net/src/lite/connection.ts +++ b/js/net/src/lite/connection.ts @@ -174,6 +174,8 @@ export class Connection implements Established { this.#peerSetup, this.#goaway, this.#auth.scope, + undefined, + this.#auth, ); this.#client = client; registerWire(this, { consume: (path) => this.#subscriber.consume(path), goaway: this.#goaway }); @@ -232,7 +234,7 @@ export class Connection implements Established { // Fail loud on a publication our grant never covers, once the peer has answered the // credential we presented at setup. if (this.#client && hasAuth(this.#version)) { - tasks.push(this.#publisher.runEnforce(this.#auth.setupAnswered())); + tasks.push(this.#publisher.runEnforce(this.#auth.setupAnswered(), this.#auth)); } // Route incoming QUIC datagrams into their subscriptions (lite-05+; runDatagrams diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index f4ec22d7f59..461f7e7091e 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -2,6 +2,7 @@ import { Instant, Milli, Nano, now, sleep } from "@moq/clock"; import { type Dispose, type Getter, Once, race, Signal } from "@moq/signals"; import type { Grant } from "../auth.ts"; import { Gate } from "../auth_gate.ts"; +import type { AuthSession } from "../auth_session.ts"; import { enforceGrant } from "../auth_session.ts"; import type * as broadcast from "../broadcast.ts"; import { Withdrawal } from "../connection/withdrawal.ts"; @@ -1675,9 +1676,9 @@ export class Publisher { * * @internal */ - async runEnforce(setupAnswered: Promise): Promise { + async runEnforce(setupAnswered: Promise, auth?: AuthSession): Promise { if (!this.#grant) return; - await enforceGrant({ quic: this.#quic, advertised: this.#advertised, grant: this.#grant, setupAnswered }); + await enforceGrant({ quic: this.#quic, advertised: this.#advertised, grant: this.#grant, setupAnswered, auth }); } /** diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 08732d797f2..e3e3e4018bd 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -2,6 +2,7 @@ import { Milli } from "@moq/clock"; import { type Dispose, Effect, type GetPromise, type Getter, race, Signal } from "@moq/signals"; import * as announce from "../announced.ts"; import type { Grant } from "../auth.ts"; +import type { AuthSession } from "../auth_session.ts"; import * as broadcast from "../broadcast.ts"; import type { Drain } from "../connection/goaway.ts"; import type { Probe as ProbeStats } from "../connection/stats.ts"; @@ -21,7 +22,7 @@ import * as Time from "../time.ts"; import type * as track from "../track.ts"; import { untilAborted } from "../util/abort.ts"; import { Deadline, REQUEST_TIMEOUT_MS, requestOptions } from "../util/deadline.ts"; -import { unwanted } from "../util/demand.ts"; +import { abandoned, isAbandoned, unwanted } from "../util/demand.ts"; import { LIMITS, type Limits, SlotMap, Slots } from "../util/limits.ts"; import { overrideBroadcastWire, wireOf } from "../wire.ts"; import { @@ -180,6 +181,7 @@ export class Subscriber { // Our grant: a subscription it stops covering is cancelled. Undefined until the peer // answers, and forever on a version without AUTH, which allows everything. #grant?: Getter; + #auth?: AuthSession; /** * Creates a new Subscriber instance. @@ -203,6 +205,7 @@ export class Subscriber { goaway?: GetPromise, grant?: Getter, limits: Limits = LIMITS, + auth?: AuthSession, ) { this.#quic = quic; this.#announces = new Slots(limits.announces, { quic, reason: "too many announcements" }); @@ -212,6 +215,7 @@ export class Subscriber { this.#peerSetup = peerSetup; this.#goaway = goaway; this.#grant = grant; + this.#auth = auth; } // Whether the peer has sent GOAWAY. Requests keep opening here until a replacement @@ -651,6 +655,22 @@ export class Subscriber { return; } const refused = unauthorized(broadcast); + try { + for (;;) { + const ready = await this.#auth?.ready(broadcast, "subscribe", { + signal: consumed.valid, + until: () => abandoned(request.demand()), + }); + if (ready !== false) break; + // Demand returning before we commit is not abandonment, matching the serving loop. + if (!isAbandoned(request.demand())) continue; + request.reject(new Error("subscribe abandoned before it was authorized")); + return; + } + } catch (err) { + request.reject(error(err)); + return; + } // Armed before the first check and held to the end, so a shrink while the // subscription sets up is never missed: it resets whatever reached the wire. const state: SubscribeSetup = { cancel: new AbortController() }; @@ -1048,13 +1068,24 @@ export class Subscriber { maxDelay?: Milli, ): Promise { const refused = unauthorized(broadcast); - // Arm before checking so revocation during setup or response cannot be missed. - const disposeGrant = this.#grant?.subscribe(() => { - if (this.#denied(broadcast)) group.close(refused); - }); - void group.closed.then(() => disposeGrant?.()); - try { + // Every caller leaving ends the wait like it ends the setup below. A caller that + // coalesced back onto the group before we commit keeps it waiting. + while ( + (await this.#auth?.ready(broadcast, "subscribe", { + signal: consumed.valid, + until: () => group.demand().unused(), + })) === false + ) { + if (group.closed.peek() === undefined && group.demand().used.peek()) continue; + throw new StreamError(StreamCode.Cancel, { message: "cancel" }); + } + // Arm after admission, before checking: another unanswered token may still + // widen the union while we wait, which is not a revocation of this fetch. + const disposeGrant = this.#grant?.subscribe(() => { + if (this.#denied(broadcast)) group.close(refused); + }); + void group.closed.then(() => disposeGrant?.()); if (this.#denied(broadcast)) throw refused; if (!supportsTrackStream(this.version)) { throw new Error("fetch group requires moq-lite-05 or newer"); diff --git a/js/net/src/util/demand.ts b/js/net/src/util/demand.ts index 4157274408f..7f0789573a8 100644 --- a/js/net/src/util/demand.ts +++ b/js/net/src/util/demand.ts @@ -7,3 +7,22 @@ export async function unwanted(demand: Demand): Promise { await Signal.race(demand.wanted, demand.closed); } } + +/** + * Resolve once demand that arrived for the track leaves, or it closed. An info-only lookup + * attaches no subscriber, so demand that never arrived is not abandonment. + */ +export async function abandoned(demand: Demand): Promise { + while (demand.wanted.peek() === undefined && demand.closed.peek() === undefined) { + await Signal.race(demand.wanted, demand.closed); + } + for (;;) { + await unwanted(demand); + if (isAbandoned(demand)) return; + } +} + +/** Whether no subscriber wants the track now, or it closed. */ +export function isAbandoned(demand: Demand): boolean { + return demand.closed.peek() !== undefined || demand.wanted.peek() === undefined; +} diff --git a/rs/moq-net/benches/auth.rs b/rs/moq-net/benches/auth.rs index 359227297fc..c001c7da834 100644 --- a/rs/moq-net/benches/auth.rs +++ b/rs/moq-net/benches/auth.rs @@ -26,32 +26,40 @@ struct Pair { _initial: auth::Issued, } -async fn pair(version: Version) -> Pair { +struct Origins<'a> { + publish: &'a origin::Producer, + subscribe: origin::Producer, +} + +async fn pair(version: Version, origins: Option>) -> Pair { let mut mock = moq_net::transport::memory::Config::default(); mock.protocol = Some(version.alpn().to_string()); let (client, server) = create_mock_session_pair(mock); let client = async { - let (session, driver) = moq_net::Client::new() - .with_versions(version.into()) - .connect(harness::now(), client) - .await - .unwrap(); + let mut config = moq_net::Client::new().with_versions(version.into()); + if let Some(origins) = &origins { + config = config.with_publisher(origins.publish); + } + let (session, driver) = config.connect(harness::now(), client).await.unwrap(); harness::spawn(driver); session }; let server = async { - let request = moq_net::Server::new() - .with_versions(version.into()) - .accept_request(harness::now(), server) - .await - .unwrap(); + let mut config = moq_net::Server::new().with_versions(version.into()); + if let Some(origins) = &origins { + config = config.with_subscriber(origins.subscribe.clone()); + } + let request = config.accept_request(harness::now(), server).await.unwrap(); let requests = request.auth().requests().unwrap(); let (session, driver) = request.ok().await.unwrap(); harness::spawn(driver); (session, requests) }; let (client, (server, mut requests)) = futures::join!(client, server); - let initial = requests.next().await.unwrap().accept(auth::Grant::all()); + let initial = requests.next().await.unwrap().accept(match origins { + Some(_) => auth::Grant::default(), + None => auth::Grant::all(), + }); Pair { client, _server: server, @@ -96,7 +104,7 @@ fn admission(c: &mut Criterion) { &(sessions, tokens), |b, &(sessions, tokens)| { let mut pairs = - runtime.block_on(join_all((0..sessions).map(|_| pair(version.parse().unwrap())))); + runtime.block_on(join_all((0..sessions).map(|_| pair(version.parse().unwrap(), None)))); b.iter(|| runtime.block_on(join_all(pairs.iter_mut().map(|pair| pair.round(tokens))))); }, ); @@ -206,5 +214,96 @@ fn renewal(c: &mut Criterion) { group.finish(); } -criterion_group!(benches, admission, renewal); +/// Publications held behind unanswered tokens, sweeping both the token table and paths. +fn waiting(c: &mut Criterion) { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + for version in ["moq-lite-07-wip", "moq-transport-22"] { + let mut group = c.benchmark_group(format!("auth/waiting/{version}")); + group.sample_size(10); + group.warm_up_time(Duration::from_millis(100)); + group.measurement_time(Duration::from_millis(300)); + for tokens in [1, 16, 128] { + for paths in [1, 16, 128] { + group.throughput(Throughput::Elements(paths as u64)); + group.bench_with_input( + BenchmarkId::new(tokens.to_string(), paths), + &(tokens, paths), + |b, &(tokens, paths)| { + let _guard = runtime.enter(); + let mut config = origin::Config::new(Hop::new(2).unwrap()); + config.update_hold = Duration::ZERO; + let (publish, driver) = origin::Producer::new(config); + harness::spawn(driver); + let (receive, driver) = origin::Producer::new(origin::Config::new(Hop::new(1).unwrap())); + harness::spawn(driver); + let mut pair = runtime.block_on(pair( + version.parse().unwrap(), + Some(Origins { + publish: &publish, + subscribe: receive.clone(), + }), + )); + let mut generation = 0; + b.iter(|| { + runtime.block_on(async { + let handle = pair.client.auth(); + let mut presenting = + std::pin::pin!(join_all((0..tokens).map(|_| handle.add("pending")))); + assert!(futures::poll!(presenting.as_mut()).is_pending()); + let mut pending = Vec::with_capacity(tokens); + for _ in 0..tokens { + pending.push(pair.requests.next().await.unwrap()); + } + let prefix = format!("wait/{generation}/"); + generation += 1; + let broadcasts: Vec<_> = (0..paths) + .map(|path| { + let broadcast = + publish.create_broadcast(format!("{prefix}{path}").as_str()).unwrap(); + broadcast.announce(Default::default()).unwrap(); + broadcast + }) + .collect(); + // Let the session reach its admission wait before the final reply widens it. + tokio::task::yield_now().await; + let mut issued = Vec::with_capacity(tokens); + for (index, request) in pending.into_iter().enumerate() { + issued.push(request.accept(if index + 1 == tokens { + auth::Grant::all() + } else { + auth::Grant::default() + })); + } + let tokens: Vec<_> = presenting.await.into_iter().map(Result::unwrap).collect(); + let mut announced = receive.consume().announced(); + let mut seen = std::collections::HashSet::new(); + while seen.len() < paths { + match announced.next().await.unwrap() { + moq_net::announce::Event::Start(update) + | moq_net::announce::Event::Update(update) + | moq_net::announce::Event::Restart(update) + if update.prefix.as_str().starts_with(&prefix) => + { + seen.insert(update.prefix); + } + _ => {} + } + } + drop(broadcasts); + drop(tokens); + join_all(issued.iter().map(auth::Issued::closed)).await; + }) + }); + }, + ); + } + } + group.finish(); + } +} + +criterion_group!(benches, admission, renewal, waiting); criterion_main!(benches); diff --git a/rs/moq-net/src/auth.rs b/rs/moq-net/src/auth.rs index 1235203cd6b..0bdeccb2320 100644 --- a/rs/moq-net/src/auth.rs +++ b/rs/moq-net/src/auth.rs @@ -601,6 +601,25 @@ impl Handle { Poll::Ready(()) } + /// Wait only when a token already presented might widen the grant for this path. + #[cfg_attr(not(feature = "async"), allow(dead_code))] + fn poll_presented(&self, direction: Direction, path: &str, waiter: &kio::Waiter) -> Poll<()> { + let _ = ready_or!(self.state.poll(waiter, |state| { + let (granted, _) = state.parts(direction); + if granted.is_some_and(|grant| grant.matches(path)) + || !state + .tokens + .values() + .any(|slot| !slot.token.is_empty() && slot.answered.is_none() && !slot.withdrawn) + { + Poll::Ready(()) + } else { + Poll::Pending + } + })); + Poll::Ready(()) + } + /// Authorize the peer's connection credential for `grant` on this session, in the /// session's own paths. Calling it again re-authorizes, narrower or wider, replacing the /// last grant. The session's origin handles still bound it, so it never grants more than @@ -777,6 +796,14 @@ pub struct Auth { presented: std::sync::Arc, } +#[cfg(all(test, feature = "async"))] +impl Auth { + /// The id the session presents this credential under. + pub(crate) fn id(&self) -> u64 { + self.presented.id + } +} + /// The slot every clone of one [`Auth`] shares, withdrawn when the last drops. struct Presented { state: kio::Shared, @@ -1619,6 +1646,20 @@ pub(crate) struct Gate { part: Part, /// Whether the scope allowed the path at the last permit. within_limit: bool, + admitted: bool, + waiting: Option, +} + +/// Log the whole wait once, including when its request is cancelled. +struct Waiting { + path: crate::PathOwned, + started: Instant, +} + +impl Drop for Waiting { + fn drop(&mut self) { + tracing::debug!(path = %self.path, waited = ?self.started.elapsed(), "request waited for authorization"); + } } /// Which part of a [`Permit`] a [`Gate`] watches. @@ -1643,6 +1684,8 @@ impl Gate { epoch: 0, part: Part::Both, within_limit, + admitted: false, + waiting: None, } } @@ -1704,11 +1747,38 @@ impl Gate { self.handle.poll_setup_answered(waiter) } + /// Settle tokens that might admit this request before its first wire operation. + #[cfg_attr(not(feature = "async"), allow(dead_code))] + pub(crate) fn poll_ready(&mut self, waiter: &kio::Waiter) -> Poll<()> { + if self.admitted { + return Poll::Ready(()); + } + // A withdrawn route ends the wait too: the request is then judged against the union + // as it stands, so routing can recover instead of parking on a token. + let withdrawn = matches!(&self.part, Part::Announced(live) if live.poll_closed(waiter).is_ready()); + if !withdrawn + && self + .handle + .poll_presented(self.direction, self.path.as_str(), waiter) + .is_pending() + { + self.waiting.get_or_insert_with(|| Waiting { + path: self.path.clone(), + started: Instant::now(), + }); + return Poll::Pending; + } + drop(self.waiting.take()); + self.admitted = true; + Poll::Ready(()) + } + /// Resolve one operation under this gate without dropping the handler's refusal path. #[cfg(feature = "async")] pub(crate) async fn guard(&mut self, operation: impl std::future::Future>) -> Result { let mut operation = std::pin::pin!(operation); kio::wait(|waiter| { + std::task::ready!(self.poll_ready(waiter)); if self.poll_denied(waiter).is_ready() { return Poll::Ready(Err(Error::Unauthorized)); } @@ -1807,6 +1877,9 @@ pub(crate) struct Enforce { /// Every broadcast admitted so far, still announced. live: std::collections::HashSet, setup: bool, + /// Broadcasts outside the union, still announced, waiting on tokens that might cover + /// them, in announcement order. + pending: Vec, } impl Enforce { @@ -1827,29 +1900,53 @@ impl Enforce { self.permit = union.map(|grant| grant.publish); } // No grant yet (the peer never answered with one): nothing to check against. - let Some(permit) = &self.permit else { + if self.permit.is_none() { return Poll::Pending; - }; + } - loop { - let Some(event) = ready_or!(announced.poll_next(waiter)) else { + // Withdrawals queued behind a wait apply before it is judged, so a broadcast that is + // no longer published never closes the session. + while let Poll::Ready(event) = announced.poll_next(waiter) { + let Some(event) = event else { return Poll::Ready(None); }; match event { crate::announce::Event::Start(update) | crate::announce::Event::Update(update) if !self.live.contains(&update.prefix) => { - if !permit.matches(update.prefix.as_str()) { - return Poll::Ready(Some(update.prefix)); + if self.pending.iter().any(|gate| gate.path == update.prefix) { + continue; + } + let mut gate = Gate::new(handle.clone(), update.prefix, Direction::Publish); + if gate.poll_ready(waiter).is_pending() { + self.pending.push(gate); + continue; + } + if !handle.within_grant(Direction::Publish, gate.path.as_str()) { + return Poll::Ready(Some(gate.path)); } - self.live.insert(update.prefix); + self.live.insert(gate.path); } crate::announce::Event::End(update) => { self.live.remove(&update.prefix); + self.pending.retain(|gate| gate.path != update.prefix); } _ => {} } } + + while let Some(index) = self + .pending + .iter_mut() + .position(|gate| gate.poll_ready(waiter).is_ready()) + { + let gate = self.pending.remove(index); + if !handle.within_grant(Direction::Publish, gate.path.as_str()) { + return Poll::Ready(Some(gate.path)); + } + self.live.insert(gate.path); + } + Poll::Pending } } @@ -1857,6 +1954,79 @@ impl Enforce { mod tests { use super::*; + #[moq_clock::test] + async fn a_request_waits_for_an_unanswered_token() { + let handle = Handle::new(true); + let setup = handle.present(Bytes::new(), true).unwrap(); + handle.granted(setup.presented.id, Grant::default()); + let token = handle.present(Bytes::from_static(b"new"), false).unwrap(); + let mut gate = Gate::new( + handle.clone(), + crate::Path::new("room").to_owned(), + Direction::Subscribe, + ); + let waiter = kio::Waiter::noop(); + assert!( + gate.poll_ready(&waiter).is_pending(), + "request refused before token answered" + ); + handle.granted(token.presented.id, Grant::all()); + assert!(gate.poll_ready(&waiter).is_ready()); + assert!(gate.poll_denied(&waiter).is_pending()); + } + + /// A route withdrawn while its request waits on a token ends the wait, and the request + /// is refused against the union as it stands. + #[test] + fn a_withdrawn_route_ends_the_wait() { + let handle = Handle::new(true); + let setup = handle.present(Bytes::new(), true).unwrap(); + handle.granted(setup.presented.id, Grant::default()); + let _unanswered = handle.present(Bytes::from_static(b"new"), false).unwrap(); + let route = kio::Producer::<()>::default(); + let mut gate = Gate::announced( + handle, + crate::Path::new("room").to_owned(), + Direction::Subscribe, + route.consume(), + ); + let waiter = kio::Waiter::noop(); + assert!(gate.poll_ready(&waiter).is_pending(), "waits while the route stands"); + drop(route); + assert!(gate.poll_ready(&waiter).is_ready(), "a withdrawn route kept the wait"); + assert!(gate.poll_denied(&waiter).is_ready()); + } + + #[test] + fn an_empty_token_never_holds_a_request() { + let handle = Handle::new(true); + let _setup = handle.present(Bytes::new(), true).unwrap(); + let mut gate = Gate::new(handle, crate::Path::new("room").to_owned(), Direction::Subscribe); + assert!(gate.poll_ready(&kio::Waiter::noop()).is_ready()); + } + + #[test] + fn a_covered_request_never_waits_for_an_unanswered_token() { + let handle = Handle::new(true); + let first = handle.present(Bytes::from_static(b"first"), false).unwrap(); + handle.granted(first.presented.id, Grant::all()); + let _unanswered = handle.present(Bytes::from_static(b"renewal"), false).unwrap(); + let mut gate = Gate::new(handle, crate::Path::new("room").to_owned(), Direction::Subscribe); + assert!(gate.poll_ready(&kio::Waiter::noop()).is_ready()); + } + + #[test] + fn an_auth_ok_is_queued_after_its_scope_widens() { + let handle = Handle::new(true); + handle.authorize(&Grant::default()); + let serving = Serving::new(handle.clone()); + let request = Request::new(Bytes::from_static(b"token"), serving.issue.clone()); + let _issued = request.accept(Grant::all()); + let issue = serving.issue.read(); + assert!(matches!(issue.outbox.front(), Some(Reply::Grant(_)))); + assert!(handle.within_limit(Direction::Subscribe, "room")); + } + /// A queued request token keeps its request's slot, so the peer's request credit bounds /// how many wait; answering it gives the slot back. #[test] diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index 28c85607b6a..d6d624d15a1 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -2644,9 +2644,25 @@ where Some(live) => crate::auth::Gate::announced(auth, path, direction, live), None => crate::auth::Gate::new(auth, path, direction), }; - if !gate.allows() { - request.reject(Error::Unauthorized); - return; + // Nobody wanting the track any more ends a wait on a token that may never be answered. + loop { + let admitted = kio::wait(|waiter| { + if gate.poll_ready(waiter).is_ready() { + return Poll::Ready(Some(gate.poll_denied(waiter).is_pending())); + } + crate::util::poll_unwanted(&mut request.demand(), waiter).map(|_| None) + }) + .await; + match admitted { + Some(true) => break, + Some(false) => { + request.reject(Error::Unauthorized); + return; + } + None if request.reject_unused(Error::Cancel) => return, + // Demand returned in the gap. + None => continue, + } } let track_name = request.name().to_owned(); @@ -6744,6 +6760,120 @@ mod tests { ); } + /// A subscriber whose session presented a token still unanswered, over a peer that + /// accepts streams and says nothing. + fn waiting_on_auth( + version: Version, + ) -> ( + Subscriber, + crate::lite::test_transport::Log, + crate::auth::Handle, + Vec, + crate::util::TaskSet, + ) { + let session = crate::lite::test_transport::ScriptedSession::new(Vec::new()); + let log = session.log.clone(); + let auth = crate::auth::Handle::new(true); + let setup = auth.present(bytes::Bytes::new(), true).unwrap(); + auth.granted(setup.id(), crate::auth::Grant::default()); + let unanswered = auth.present(bytes::Bytes::from_static(b"new"), false).unwrap(); + let (tasks, task_set) = crate::util::TaskSet::new(); + let subscriber = Subscriber::new( + crate::time::sim(), + session, + crate::origin::Config::new(crate::Hop::new(1).unwrap()).produce(), + Control::new(None, false), + None, + peer::PeerSetup::default(), + crate::Hop::new(1).unwrap(), + None, + version, + tasks, + Default::default(), + ) + .with_auth(auth.clone()); + (subscriber, log, auth, vec![setup, unanswered], task_set) + } + + /// A subscribe whose readers leave while it waits on a token ends, rather than holding + /// its task and track until a token that may never be answered. + #[moq_clock::test] + async fn an_abandoned_subscribe_stops_waiting_for_auth() { + const VERSION: Version = Version::Draft16; + let (mut subscriber, log, _auth, _tokens, _task_set) = waiting_on_auth(VERSION); + + let producer = crate::broadcast::Info::default().produce(); + let mut dynamic = producer.dynamic(); + let consumer = producer.consume(); + let track = consumer.track("video").unwrap(); + let subscription = track.subscribe(None); + let request = dynamic.requested_track().await.expect("no track requested"); + + let mut serving = moq_clock::sim::spawn(async move { + subscriber + .run_subscribe(Path::new("broadcast"), dynamic, None, request) + .await; + }); + assert!( + moq_clock::timeout(std::time::Duration::from_secs(1), &mut serving) + .await + .is_err(), + "finished while the track was wanted" + ); + + drop(subscription); + drop(track); + drop(consumer); + moq_clock::timeout(std::time::Duration::from_secs(1), serving) + .await + .expect("an abandoned subscribe kept waiting on its token") + .unwrap(); + assert!( + !control_message_types(&log, VERSION).contains(&ietf::Subscribe::ID), + "an abandoned subscribe reached the wire", + ); + } + + /// A subscribe under an announced route that is withdrawn while it waits on a token + /// ends, judged against the union as it stands, so routing can recover. + #[moq_clock::test] + async fn a_withdrawn_route_ends_a_subscribe_waiting_for_auth() { + const VERSION: Version = Version::Draft16; + let (mut subscriber, log, _auth, _tokens, _task_set) = waiting_on_auth(VERSION); + + let producer = crate::broadcast::Info::default().produce(); + let mut dynamic = producer.dynamic(); + let consumer = producer.consume(); + let track = consumer.track("video").unwrap(); + let subscription = track.subscribe(None); + let request = dynamic.requested_track().await.expect("no track requested"); + + let route = kio::Producer::<()>::default(); + let live = route.consume(); + let mut serving = moq_clock::sim::spawn(async move { + subscriber + .run_subscribe(Path::new("broadcast"), dynamic, Some(live), request) + .await; + }); + assert!( + moq_clock::timeout(std::time::Duration::from_secs(1), &mut serving) + .await + .is_err(), + "finished while the route stood" + ); + + drop(route); + moq_clock::timeout(std::time::Duration::from_secs(1), serving) + .await + .expect("a withdrawn route kept the subscribe waiting on its token") + .unwrap(); + assert!( + !control_message_types(&log, VERSION).contains(&ietf::Subscribe::ID), + "a subscribe outside the union reached the wire", + ); + drop(subscription); + } + /// A publisher can be serving before its SUBSCRIBE_OK arrives, since data streams are /// independent of the request stream. If the last consumer leaves in that window, the /// subscriber still owes it a cancellation: walking away silently is what leaves it diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index b7b55d2ed42..d781124a96d 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -2809,6 +2809,56 @@ mod tests { } } + /// A track waiting on an unanswered token ends once nobody wants it, rather than + /// holding its task and track until a token that may never be answered. + #[moq_clock::test] + async fn an_unused_track_stops_waiting_for_auth() { + let auth = crate::auth::Handle::new(true); + let _token = auth.present(bytes::Bytes::from_static(b"new"), false).unwrap(); + let subscriber = Subscriber::new(SubscriberConfig { + runtime: crate::time::sim(), + session: SinkSession::default(), + origin: origin::Config::new(crate::Hop::new(1).unwrap()).produce(), + recv_bandwidth: None, + version: VERSION, + peer_setup: crate::lite::PeerSetup::new(false, crate::transport::Binding::InBand), + peer_hop: None, + cost: None, + going_away: Default::default(), + auth, + }); + + let broadcast = crate::broadcast::Info::new().produce(); + let mut dynamic = broadcast.dynamic(); + let consumer = broadcast.consume(); + let mut waiting = Box::pin(consumer.track("video").unwrap().subscribe(None)); + assert!(futures::poll!(waiting.as_mut()).is_pending()); + let request = dynamic.requested_track().now_or_never().unwrap().expect("request"); + + let mut run = TrackServeRun::new( + TrackServe { + validity: Default::default(), + route: None, + subscriber, + path: Path::new("room").to_owned(), + epoch: None, + name: "video".to_string(), + }, + request, + ); + let waiter = kio::Waiter::noop(); + assert!( + kio::Task::poll(&mut run, &waiter).is_pending(), + "waits on the token while the track is wanted" + ); + + drop(waiting); + assert!( + kio::Task::poll(&mut run, &waiter).is_ready(), + "an unused track kept waiting on its token" + ); + } + /// `establish` puts exactly one SUBSCRIBE on the wire, and the id is registered /// before any of it reaches the transport. /// @@ -5488,10 +5538,37 @@ impl TrackServeRun { } } +impl TrackServeRun { + /// Nobody wants the track anymore (the origin failed over, the reader left): stop + /// waiting on a token or a peer that may never answer, which would otherwise hold + /// this task, its TRACK stream, and the track for good. Dropping the fetch resets the + /// stream. Ready once the request is released. + fn poll_unused(&mut self, waiter: &kio::Waiter) -> Poll<()> { + loop { + let TrackRunState::Info { request, .. } = &self.state else { + return Poll::Pending; + }; + let pending = request.as_ref().expect("request pending"); + ready!(crate::util::poll_unwanted(&mut pending.demand(), waiter)); + if pending.reject_unused(Error::Cancel) { + self.state = TrackRunState::Done; + return Poll::Ready(()); + } + // Demand returned in the gap and won inside `reject_unused`: poll again so the + // unused wait is armed for when it leaves. + } + } +} + impl kio::Task for TrackServeRun { type Output = (); fn poll(&mut self, waiter: &kio::Waiter) -> Poll<()> { + // Checked before the auth wait too, so an abandoned request never parks on a token. + if self.poll_unused(waiter).is_ready() { + return Poll::Ready(()); + } + ready!(self.gate.poll_ready(waiter)); // A finished track only waits for its FIN to be acknowledged: nothing is left to revoke. if !matches!(self.state, TrackRunState::Finish(_)) && self.gate.poll_denied(waiter).is_ready() { self.revoke(); @@ -5500,20 +5577,7 @@ impl kio::Task for TrackServeRun { loop { match &mut self.state { TrackRunState::Info { request, info } => { - // Nobody wants the track anymore (the origin failed over, the reader - // left): stop waiting on a peer that may never answer, which would - // otherwise hold this task, its TRACK stream, and the track for good. - // Dropping the fetch resets the stream. - let pending = request.as_ref().expect("request pending"); - if crate::util::poll_unwanted(&mut pending.demand(), waiter).is_ready() { - if pending.reject_unused(Error::Cancel) { - self.state = TrackRunState::Done; - return Poll::Ready(()); - } - // Demand returned in the gap and won inside `reject_unused`: poll - // again so the unused wait is armed for when it leaves. - continue; - } + // Abandonment was checked at the top of this poll, arming its wait. let res = ready!(info.poll_fetch(&self.serve, waiter)); let request = request.take().expect("request pending"); match res { @@ -6231,6 +6295,7 @@ impl ServeLoop { /// is implicit from the request. struct FetchServeRun { validity: crate::model::broadcast::RequestValidity, + gate: crate::auth::Gate, serve: TrackServe, session: S, timescale: Option, @@ -6281,6 +6346,7 @@ impl FetchServeRun { let group = request.sequence(); Self { validity: serve.validity.start(), + gate: serve.gate(), serve, session, timescale, @@ -6364,6 +6430,7 @@ impl kio::Task for FetchServeRun { match &mut self.state { FetchRunState::Open { request } => { + ready!(self.gate.poll_ready(waiter)); let mut stream = match ready!(Stream::poll_open( &mut self.session, self.serve.subscriber.version, diff --git a/rs/moq-net/tests/integration/auth.rs b/rs/moq-net/tests/integration/auth.rs index b18c09d8ec9..56012543ff3 100644 --- a/rs/moq-net/tests/integration/auth.rs +++ b/rs/moq-net/tests/integration/auth.rs @@ -55,7 +55,8 @@ cases!( a_publish_only_session_grants_no_subscribe, an_out_of_scope_announce_aborts_with_the_path, a_broadcast_published_before_the_grant_is_checked, - an_unanswered_token_does_not_suspend_the_check, + an_unanswered_token_holds_the_check, + a_broadcast_withdrawn_while_its_token_waits_is_not_refused, tokens_union_and_withdrawing_one_shrinks_it, an_update_replaces_one_tokens_grant, a_revoked_grant_withdraws_and_can_be_restored, @@ -66,6 +67,7 @@ cases!( an_issued_grant_ends_with_its_session, a_reset_auth_stream_reports_unsupported, a_revoked_grant_cancels_its_subscriptions, + a_live_subscription_survives_union_renewal, a_grant_update_too_large_for_one_message_is_unsupported, nothing_outside_the_grant_reaches_the_peer, ); @@ -604,9 +606,8 @@ async fn a_broadcast_published_before_the_grant_is_checked(version: &'static str .expect("timed out"); } -/// Enforcement waits only for the tokens the session presented itself: a peer that -/// never answers an app-added token cannot suspend it. -async fn an_unanswered_token_does_not_suspend_the_check(version: &'static str) { +/// A publication waits on a presented token, then is judged against the settled union. +async fn an_unanswered_token_holds_the_check(version: &'static str) { within(async { let publisher = produce_origin(2); let mut pair = connect(Options { @@ -617,22 +618,71 @@ async fn an_unanswered_token_does_not_suspend_the_check(version: &'static str) { ..Default::default() }) .await; - let _issued = serve(pair.requests.take().unwrap(), |token| { - token.is_empty().then(|| grant(&["baz"], &[])) - }); - - let auth = pair.client.auth(); - let pending = moq_clock::sim::spawn(async move { auth.add("never answered").await.map(|_| ()) }); + let mut requests = pair.requests.take().unwrap(); + let _setup = requests.next().await.unwrap().accept(grant(&["baz"], &[])); assert_eq!(granted(&pair.client).await, grant(&["baz"], &[])); + let auth = pair.client.auth(); + let pending = moq_clock::sim::spawn(async move { auth.add("new").await }); + let unanswered = requests.next().await.unwrap(); + let broadcast = publisher.create_broadcast("foo/bar").unwrap(); + broadcast.announce(Default::default()).unwrap(); + moq_clock::sleep(Duration::from_secs(1)).await; + use futures::FutureExt; + assert!( + pair.server.closed().now_or_never().is_none(), + "publication refused before token answered" + ); + let _issued = unanswered.accept(grant(&["foo"], &[])); + let _token = pending.await.unwrap().unwrap(); + assert_eq!(granted(&pair.client).await, grant(&["baz", "foo"], &[])); + let bad = publisher.create_broadcast("still-bad").unwrap(); + bad.announce(Default::default()).unwrap(); + assert!(matches!( + pair.server.closed().await, + Error::Session(SessionError::Unauthorized) + )); + let (_, reason) = pair.client_transport.close_reason().expect("closed"); + assert_eq!(reason, "unauthorized: still-bad"); + }) + .await + .expect("timed out"); +} - let bad = publisher.create_broadcast("foo/bar").unwrap(); +/// A broadcast withdrawn while its token is unanswered is not judged once the token settles. +async fn a_broadcast_withdrawn_while_its_token_waits_is_not_refused(version: &'static str) { + within(async { + let publisher = produce_origin(2); + let mut pair = connect(Options { + version: Some(version), + client_publish: Some(publisher.clone()), + server_subscribe: Some(produce_origin(1)), + server_requests: true, + ..Default::default() + }) + .await; + let mut requests = pair.requests.take().unwrap(); + let _setup = requests.next().await.unwrap().accept(grant(&["baz"], &[])); + assert_eq!(granted(&pair.client).await, grant(&["baz"], &[])); + let auth = pair.client.auth(); + let pending = moq_clock::sim::spawn(async move { auth.add("new").await }); + let unanswered = requests.next().await.unwrap(); + let withdrawn = publisher.create_broadcast("foo/bar").unwrap(); + withdrawn.announce(Default::default()).unwrap(); + moq_clock::sleep(Duration::from_secs(1)).await; + withdrawn.unannounce(); + moq_clock::sleep(Duration::from_secs(1)).await; + let _issued = unanswered.accept(grant(&["other"], &[])); + let _token = pending.await.unwrap().unwrap(); + assert_eq!(granted(&pair.client).await, grant(&["baz", "other"], &[])); + // Still enforcing: the next uncovered broadcast is the one named. + let bad = publisher.create_broadcast("still-bad").unwrap(); bad.announce(Default::default()).unwrap(); assert!(matches!( pair.server.closed().await, Error::Session(SessionError::Unauthorized) )); - // The session's close fails the token still waiting on its answer. - assert!(pending.await.unwrap().is_err()); + let (_, reason) = pair.client_transport.close_reason().expect("closed"); + assert_eq!(reason, "unauthorized: still-bad"); }) .await .expect("timed out"); @@ -950,6 +1000,61 @@ async fn connect_announced(version: &'static str) -> (Pair, moq_net::broadcast:: (pair, broadcast) } +async fn a_live_subscription_survives_union_renewal(version: &'static str) { + within(async { + let publisher = produce_origin(1); + let broadcast = publisher.create_broadcast("room/x").unwrap(); + let published = broadcast.create_track("video", timed()).unwrap(); + broadcast.announce(Default::default()).unwrap(); + let received = produce_origin(2); + let mut pair = connect(Options { + version: Some(version), + client_subscribe: Some(received.clone()), + server_publish: Some(publisher), + server_requests: true, + ..Default::default() + }) + .await; + pair.server.auth().authorize(&Grant::default()); + let mut requests = pair.requests.take().unwrap(); + let _setup = requests.next().await.unwrap().accept(Grant::default()); + let auth = pair.client.auth(); + let (first, first_grant) = futures::join!(auth.add("first"), async { + requests.next().await.unwrap().accept(grant(&[], &["room"])) + }); + let first = first.unwrap(); + let remote = received.consume().routed_broadcast("room/x").await.unwrap(); + let mut group = published.append_group().unwrap(); + group + .write_frame(moq_net::Timestamp::from_millis(0).unwrap(), b"first".as_ref()) + .unwrap(); + let mut subscription = remote + .track("video") + .unwrap() + .subscribe(moq_net::track::Subscription::default()) + .await + .unwrap(); + subscription.recv_group().await.unwrap().unwrap(); + let (renewed, _renewed_grant) = futures::join!(auth.add("renewal"), async { + requests.next().await.unwrap().accept(grant(&[], &["room"])) + }); + let _renewed = renewed.unwrap(); + drop(first); + first_grant.closed().await; + let mut group = published.append_group().unwrap(); + group + .write_frame(moq_net::Timestamp::from_millis(1).unwrap(), b"renewed".as_ref()) + .unwrap(); + subscription + .recv_group() + .await + .expect("renewal ended the subscription") + .unwrap(); + }) + .await + .expect("timed out"); +} + /// Losing a grant cancels the subscriptions it covered, in both directions, and /// leaves the session up. async fn a_revoked_grant_cancels_its_subscriptions(version: &'static str) { diff --git a/rs/moq-tokio/src/connection.rs b/rs/moq-tokio/src/connection.rs index 3f4b9521ad6..4c73db1addb 100644 --- a/rs/moq-tokio/src/connection.rs +++ b/rs/moq-tokio/src/connection.rs @@ -468,6 +468,8 @@ impl Resolved { /// distinguishes a permanent give-up from a graceful close. #[derive(Default)] struct State { + /// Tokens to queue before publishing a replacement session to its consumers. + tokens: std::sync::Mutex>>, /// Current connection status, or `None` before the first connect. status: Option, /// How many sessions have been live on this handle, counting the current one. @@ -519,6 +521,7 @@ impl Shared { return; } if let Ok(mut state) = self.state.write() { + auth::replay(&state, session, state.connects + 1); // A migration replaces a live session without a disconnect in between, so // its end is counted here, keeping `started - ended` at 1 while connected. if state.session.is_some() { diff --git a/rs/moq-tokio/src/connection/auth.rs b/rs/moq-tokio/src/connection/auth.rs index aeeb92b77af..f6e5a194b24 100644 --- a/rs/moq-tokio/src/connection/auth.rs +++ b/rs/moq-tokio/src/connection/auth.rs @@ -1,6 +1,9 @@ //! Tokens and grant watches that follow a reconnecting connection. -use std::task::{Poll, ready}; +use std::{ + sync::{Arc, Mutex}, + task::{Poll, ready}, +}; use bytes::Bytes; use futures::FutureExt; @@ -32,10 +35,23 @@ impl Handle { let connection = self.connection.clone(); let pacing = self.pacing; let token = token.into(); + let replay = Arc::new(Replay { + bytes: token, + queued: Mutex::new(Vec::new()), + }); + { + let state = connection.read(); + let mut tokens = state.tokens.lock().unwrap(); + tokens.retain(|token| token.strong_count() > 0); + if let Some(session) = &state.session { + replay.queue(session, state.connects); + } + tokens.push(Arc::downgrade(&replay)); + } let (answered, answer) = tokio::sync::oneshot::channel(); let task = AbortOnDrop::new(tokio::spawn(async move { let mut answered = Some(answered); - if let Err(err) = present(connection, pacing, token, &producer, &mut answered).await { + if let Err(err) = present(connection, pacing, replay, &producer, &mut answered).await { if let Some(answered) = answered.take() { let _ = answered.send(Err(err.clone())); } @@ -50,6 +66,83 @@ impl Handle { } } +type Opening = futures::future::BoxFuture<'static, moq_net::Result>; + +/// Registration shared with session publication, while the task owns its token's life. +pub(super) struct Replay { + bytes: Bytes, + /// Registrations queued as sessions are published, until the task takes them. + queued: Mutex>, +} + +struct Queued { + connects: u64, + session: moq_net::Session, + opening: Opening, +} + +impl Replay { + fn queue(&self, session: &moq_net::Session, connects: u64) { + let handle = session.auth(); + let bytes = self.bytes.clone(); + let mut opening = async move { handle.add(bytes).await }.boxed(); + // The first poll registers the token synchronously, before the session can be + // observed. The replay task replaces this noop waker when it takes the future. + if let Some(result) = opening.as_mut().now_or_never() { + opening = futures::future::ready(result).boxed(); + } + self.queued.lock().unwrap().push(Queued { + connects, + session: session.clone(), + opening, + }); + } + + /// Take the registration queued for session `connects`, moving older ones to `draining`. + fn take(&self, connects: u64, draining: &mut Draining) -> Option { + let mut queued = self.queued.lock().unwrap(); + let mut current = None; + for entry in std::mem::take(&mut *queued) { + match entry.connects.cmp(&connects) { + // Requests on a draining session may still be waiting on this answer. + std::cmp::Ordering::Less => draining.push(entry.session, entry.opening), + std::cmp::Ordering::Equal => current = Some(entry.opening), + // A replacement published since the task's snapshot already holds this token. + // Dropping its registration would withdraw the token from a session consumers + // can see, so leave it for the task's next pass. + std::cmp::Ordering::Greater => queued.push(entry), + } + } + current + } +} + +/// Tokens kept on sessions draining after a migration. Each is held by a task that +/// releases it, and its session, once that session ends; dropping this withdraws them all. +#[derive(Default)] +struct Draining(Vec); + +impl Draining { + /// Keep `held`, a token answered or not, registered on `session` until it ends. + fn push(&mut self, session: moq_net::Session, held: impl Send + 'static) { + self.0.retain(|task| !task.is_finished()); + self.0.push(AbortOnDrop::new(tokio::spawn(async move { + session.closed().await; + drop(held); + }))); + } +} + +pub(super) fn replay(state: &State, session: &moq_net::Session, connects: u64) { + state.tokens.lock().unwrap().retain(|token| { + let Some(token) = token.upgrade() else { + return false; + }; + token.queue(session, connects); + true + }); +} + /// A presented token; dropping it withdraws it from live and draining sessions. pub struct Auth { _task: AbortOnDrop, @@ -193,12 +286,12 @@ fn transient(err: &moq_net::Error) -> bool { async fn present( connection: kio::Consumer, pacing: Pacing, - bytes: Bytes, + replay: Arc, producer: &kio::Producer, answered: &mut Option>>, ) -> crate::Result<()> { let mut snapshot = next(&connection, None).await?; - let mut draining = Vec::new(); + let mut draining = Draining::default(); 'session: loop { let (connects, session) = snapshot; let Some(session) = session else { @@ -211,20 +304,37 @@ async fn present( if let Ok(mut state) = producer.write() { state.grant = None; } - let handle = session.auth(); let mut delay = None; + let mut opening = replay.take(connects, &mut draining); let token = loop { let attempt = async { - if let Some(delay) = delay { - tokio::time::sleep(delay).await; + if opening.is_none() { + if let Some(delay) = delay { + tokio::time::sleep(delay).await; + } + let handle = session.auth(); + let bytes = replay.bytes.clone(); + opening = Some(async move { handle.add(bytes).await }.boxed()); } - handle.add(bytes.clone()).await + opening.as_mut().expect("opening").await }; let result = tokio::select! { biased; - changed = next(&connection, Some((connects, true))) => { snapshot = changed?; continue 'session; } - result = attempt => result, + changed = next(&connection, Some((connects, true))) => Err(changed), + result = attempt => Ok(result), }; + let result = match result { + Ok(result) => result, + Err(changed) => { + // Requests on a draining session may still be waiting on this answer. + if let Some(opening) = opening.take() { + draining.push(session, opening); + } + snapshot = changed?; + continue 'session; + } + }; + opening = None; match result { Ok(token) => break token, // The session ending is no verdict on the token: replay it on the next one. @@ -274,8 +384,7 @@ async fn present( } // A migrated session may still be serving its streams during GOAWAY drain. // Keep its grant until that session ends or the application drops this token. - draining.retain(|(session, _): &(moq_net::Session, auth::Auth)| session.closed().now_or_never().is_none()); - draining.push((session, token)); + draining.push(session, token); } } @@ -357,9 +466,134 @@ mod tests { fn attach(connection: &kio::Producer, session: &moq_net::Session) { let mut state = connection.write().ok().expect("connection observed"); state.connects += 1; + replay(&state, session, state.connects); state.session = Some(session.clone()); } + #[tokio::test(start_paused = true)] + async fn replay_is_registered_before_the_session_is_published() { + let connection = kio::Producer::::default(); + let auth = retained(&connection); + let mut first = pair().await; + attach(&connection, &first.client); + let (token, _issued) = tokio::join!(auth.add("replay"), async { + first.requests.recv().await.unwrap().accept(auth::Grant::all()) + }); + let token = token.unwrap(); + let mut second = pair().await; + let connects = { + let state = connection.read(); + let connects = state.connects + 1; + replay(&state, &second.client, connects); + connects + }; + // The task still observes the first session. The second token reaches the peer + // before publication only if queueing it already polled add's registration. + let request = tokio::time::timeout(std::time::Duration::from_secs(1), second.requests.recv()) + .await + .expect("token was not queued before publication") + .unwrap(); + assert_eq!(request.token().as_ref(), b"replay"); + { + let mut state = connection.write().ok().expect("connection observed"); + state.connects = connects; + state.session = Some(second.client.clone()); + } + let _issued = request.accept(auth::Grant::all()); + let mut watch = token.grant(); + while watch.peek().is_none() { + watch.changed().await.unwrap(); + } + } + + #[tokio::test(start_paused = true)] + async fn a_stale_snapshot_keeps_the_newer_sessions_registration() { + let mut first = pair().await; + let mut second = pair().await; + let replay = Replay { + bytes: "replay".into(), + queued: Mutex::new(Vec::new()), + }; + let mut draining = Draining::default(); + // The replacement queues and publishes while the task still holds the first snapshot. + replay.queue(&second.client, 2); + let request = second.requests.recv().await.unwrap(); + assert!(replay.take(1, &mut draining).is_none()); + let opening = replay + .take(2, &mut draining) + .expect("the newer session's registration was dropped"); + let _issued = request.accept(auth::Grant::all()); + let _token = opening.await.unwrap(); + // A registration for a session the task has moved past stays on it while it drains. + replay.queue(&first.client, 1); + let _stale = first.requests.recv().await.unwrap(); + assert!(replay.take(2, &mut draining).is_none()); + assert!(replay.queued.lock().unwrap().is_empty()); + assert_eq!(draining.0.len(), 1, "the draining session lost its registration"); + } + + /// A draining session that closes releases its token and its session handle at once, + /// not only when a later migration adds another. + #[tokio::test(start_paused = true)] + async fn a_closed_draining_session_releases_its_token() { + let pair = pair().await; + let mut draining = Draining::default(); + let held = Arc::new(()); + draining.push(pair.client.clone(), held.clone()); + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + assert_eq!(Arc::strong_count(&held), 2, "released while its session lives"); + pair.client.abort(moq_net::Error::Cancel); + pair.client.closed().await; + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + assert_eq!(Arc::strong_count(&held), 1, "a closed draining session kept its token"); + drop(draining); + } + + /// A migration keeps an unanswered replay on the session it leaves, whether or not the + /// task had taken it, since requests there may still be waiting on its answer. + #[tokio::test(start_paused = true)] + async fn a_draining_session_keeps_an_unanswered_replay() { + for taken in [false, true] { + let connection = kio::Producer::::default(); + let auth = retained(&connection); + let mut first = pair().await; + attach(&connection, &first.client); + let (token, _first_grant) = tokio::join!(auth.add("renew"), async { + first.requests.recv().await.unwrap().accept(auth::Grant::all()) + }); + let token = token.unwrap(); + let mut second = pair().await; + let third = pair().await; + attach(&connection, &second.client); + if taken { + // Let the task take the queued registration and wait on its answer. + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + attach(&connection, &third.client); + let request = tokio::time::timeout(std::time::Duration::from_secs(5), second.requests.recv()) + .await + .unwrap_or_else(|_| panic!("taken={taken}: the draining session withdrew its token unsent")) + .unwrap(); + assert_eq!(request.token().as_ref(), b"renew"); + let _issued = request.accept(auth::Grant::all()); + let mut grant = second.client.auth().grant(); + tokio::time::timeout(std::time::Duration::from_secs(5), async { + while grant.peek() != Some(auth::Grant::all()) { + grant.changed().await.unwrap(); + } + }) + .await + .unwrap_or_else(|_| panic!("taken={taken}: the draining session never applied its token")); + tokio::time::sleep(std::time::Duration::from_secs(1)).await; + assert_eq!( + second.client.auth().grant().peek(), + Some(auth::Grant::all()), + "taken={taken}: the draining session withdrew its token" + ); + drop(token); + } + } + #[tokio::test(start_paused = true)] async fn a_token_survives_replacement_and_withdraws_from_both_sessions() { let connection = kio::Producer::::default(); diff --git a/test/drill/mutations/fetch-never-sent.patch b/test/drill/mutations/fetch-never-sent.patch index 20d7e5b94a7..aa572c6f864 100644 --- a/test/drill/mutations/fetch-never-sent.patch +++ b/test/drill/mutations/fetch-never-sent.patch @@ -10,10 +10,11 @@ diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber. index 55006c5f7..93f2f9677 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs -@@ -4456,6 +4456,11 @@ impl kio::Task for FetchServeRun { +@@ -6368,7 +6368,12 @@ impl kio::Task for FetchServeRun { match &mut self.state { FetchRunState::Open { request } => { + ready!(self.gate.poll_ready(waiter)); + // MUTATION: never send the FETCH, so nothing ever answers it. + if request.is_some() { + return Poll::Pending;